Amazon Mechanical Turk (MTurk) is made possible by a large and diverse community of Workers around the world. These Workers contribute time and talent every day to help make Requester customers successful. One of the first questions Requesters ask is “how do I specify who does my work?” This tutorial will help explain the tools at your disposal for defining the audience of Workers that can work on your HITs.
For this tutorial we’ll focus on how to use the MTurk Requester website to create HITs with different Qualification Requirements. These concepts also apply to HITs created though an API. You can find API documentation here and here. You can also learn more about Qualification Tests which require Workers to successfully answer test questions before they can accept your HIT.
When you create a new Project from the MTurk Requester website the first page prompts you to provide requirements for the Workers that do your task.
First, select whether you want to require Masters Workers. Masters Workers have consistently demonstrated a high degree of success in performing a wide range of HITs across a large number of Requesters. Note that there is an additional 5% fee when using Masters Workers.
You can add additional criteria to your task by selecting Qualifications. When you select Add another criterion a drop down is added where you can select qualifications for your task.
The first options are System Qualifications for Location, Approval Rate, and Number of HITs approved. After that are the Premium Qualifications which are qualifications specifying demographic and behavioral profiles of different Worker groups. Finally, there are Qualifications you’ve created yourself. We’ll discuss each type below. Please note you are limited to five Qualifications and can only choose two Premium Qualifications.
The final option is the Adult Content Qualification. When checked, your task is only available to Workers who are willing to see potentially offensive content. Many Requesters that use MTurk for image moderation or other similar tasks use this option. More detail on this can be found here.
There are three system Qualifications you can use. Number of HITs Approved, HIT Approval Rate, and Location. These Qualfications are based on Worker registration and activity information.
You can select Workers based on the country or US state that they registered from. For example, if you wanted to request inputs from Workers in the United States you would start by selecting the Location Qualification, setting the Comparator to is, and selecting United States in the dropdown.
If you wanted to select multiple countries and include both the United Kingdom and United States, you would change the Comparator to is one of and shift- or ctrl-click both the United Kingdom and United States in the list.
You can also exclude locations by selecting is not or is not one of to exclude selected items. For example, the following would exclude Workers from the United States.
It’s also possible to stack Qualifications by adding additional criteria. If we wanted to include Workers from every US state but exclude Workers in Minnesota, we would have an is Location Qualification for the United States and an is not Location Qualification for Minnesota.
When you have multiple Qualifications they are treated as AND operations so both must be met for a Worker to accept this task. Be careful when applying multiple Qualifications to ensure that it’s possible for a Worker to meet the requirements. For example, if we set the requirements below, no Worker would be eligible since it’s impossible to have registered in both the United States and Canada. The correct way to include Workers from both countries is to use the is one of Comparator as described above.
The Approval Rate is the rate that Requesters have approved HITs that Workers complete. For example, if a Worker has completed 5000 HITs and had their work rejected 250 times, their approval rate is 95%.
This is the number of HITs that a Worker has successfully completed since they registered for MTurk. Many Requesters use this as a way to select experienced workers.
Requesters frequently tell us that the combination of these two Qualifications has a significant effect on the overall quality of results.
Many Requesters use MTurk Workers as an audience for surveys. To help narrow your audience to Workers who fit a particular demographic or behavioral profile we have added Premium Qualifications that reflect self-reported information about the Workers on MTurk. Qualifications range from age and gender, to exercise frequency and the type of smartphone Workers own. These Qualifications allow you to select if you want to include or exclude Workers that meet a certain profile. For example, the following would restrict the survey audience to Women who voted in the 2016 US Presidential Election.
Note that there are additional fees for using Premium Qualifications. More information can be found at https://requester.mturk.com/pricing.
The final group of Qualifications are those you can create yourself. These are a great way to identify Workers you want to include or exclude from future work.
For your Qualifications you assign Workers a score, typically 0 to 100. Most Requesters assign 1 or 0 depending on whether or not they want to include or exclude Workers. Others assign a value based on a Worker’s accuracy on previous HITs. How you choose to score your Workers will depend on how your project is structured.
When you add your Qualification to your task, there are a wide range of comparators you can use to filter.
To create a new Qualification go to the Manage tab and select Qualification Types and select the Create New Qualification Type button.
We’ll call this Qualification “Good At Animal Identification”. You can choose any name you like, but be aware that Workers can see Qualifications they have been granted and the Qualification name will display as a requirement to work on your task. Type a description and hit “Create”.
Your Qualification will show up in the list of Qualifications.
If you already have completed tasks in MTurk, you can then go to the Workers tab and select Download CSV file of all of your Workers that you can open in Excel or Google Sheets.
This file contains all of the statistics and Qualifications for all of the Workers that have worked for you in the past. You can update this file with new values for any Qualifications and upload it to MTurk to set new values. To make this easy, we’ll delete any Workers we’re not interested in and all of the columns except for “UPDATE-Good At Animal Identification” which corresponds to the new Qualification we created. Any values we provide here will become the new Qualification values for the Workers we include. Here we will give a value of “1” to the Workers that did well in our evaluation task.
Go back to the Worker tab and select the Upload CSV link to upload the new file you’ve created.
After the file uploads, confirm the Qualification changes.
Now that the Qualification has been created you will be able to add it as a criteria:
If you’ve specified any criteria for your HIT you have the option of deciding it’s visibilty. For many Requesters is easiest just to leave the task marked Public but there are other options.
If your HIT is marked Public, any Worker can preview your task, even if they aren’t qualified. If you mark the task as Private, all Workers will see your task in the list of available work but only those Workers that have the required Qualification will be able to Preview the contents of the task. This is useful if you want to limit the number of Workers that can see the information in your tasks. Hidden is used to hide HITs from showing up in the list of available work unless the Worker has the required Qualifications.
There are a variety of ways that Requesters can use Qualifications to select Workers. We’ve described some methods in the Tutorials below but there are a variety of other cases that we’ll capture in future Tutorials.
If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.