Modeling Complex Annotations

Modeling annotators and their labels is useful for ensuring data quality. However, while many models have been proposed to handle binary or categorical labels, prior methods do not generalize to complex annotation tasks (e.g., open-ended text, multivariate, structured responses) without devising new models for each specific task. To obviate the need for task-specific modeling, we propose to model distances between labels, rather than the labels themselves. Our methods are agnostic as to the distance function; we leave it to the annotation task requester to specify an appropriate distance function for their task. We propose three methods, including a Bayesian hierarchical extension of multidimensional scaling.

Download paper here

BibTeX

@conference{Braylan-hcompdc19, author = {Alexander Braylan and Matthew Lease}, title = {Modeling Complex Annotations}, booktitle = , year = {2019}, confurl = {https://www.humancomputation.com/2019/attend.html#dc}, url = {../papers/braylan-hcompdc19.pdf}, slides = {../papers/alexbraylan_HCOMP_DC.pptx}, source = {https://github.com/Praznat/annotationmodeling} }