Crowdsourcing Location Estimation
ICSI researchers began experimenting with crowdsourcing as part of the Berkeley Multimodal Location Estimation Project. In that effort, we are developing an automatic system to identify where a video recording was made, using analysis of the video content and information from reference materials, when the video metadata does not contain geotags (GPS coordinates). We are evaluating our systems for their ability to correctly guess the coordinates for videos in terms of distance from the actual geotagged location, but we are also evaluating these systems in comparison with how well human beings do at the same task — i.e., how far from the correct coordinates is a human’s guess?
To develop the human baseline for comparison, we crowdsourced the task to workers on Amazon’s Mechanical Turk platform. We developed a tutorial on location estimation and a qualification task to identify workers who could and would perform the job. We then asked qualified workers to determine the location of non-GPS-tagged videos, and collected data on the accuracy of their guesses and the time it took them to arrive at those guesses. We compared the results to the performance of our automated system to identify which types of videos were easier for humans to geo-locate and which were easier for the computer.
Our Location Estimation Tutorial walked crowdsourced workers through an example task.
Our work on Crowdsourcing Location Estimation was a collaboration between ICSI’s Audio & Multimedia group and researchers at Technische Universität Berlin.
Researchers @ ICSI:
Collaborators @ TU-Berlin:
- Pascal Kelm
- Thomas Sikora
Our work on crowdsourcing for multimodal location estimation was funded as part of National Science Foundation EAGER grant IIS-1128599. The opinions, findings, and conclusions described on this website are those of the researchers and do not necessarily reflect the views of the funders.