A way to use Mechanical Turk from outside the US. I've needed it with every client I ever worked with, and I need it now and know a dozen companies who do.
CrowdFlower has a de facto monopoly on the outside of the US supply and charges an enormous premium for it - enough to turn off most of them. I certainly don't want to pay several thousand a month for the right to submit jobs, although I was ok with the 25% premium in the old days.
If it's an alternative marketplace, it has to have excellent automation via API. I'm not going to use this for questionnaires, I'll be submitting thousands of jobs automatically.
Hey there—I'm CEO and cofounder of Scale API (YC S16, https://scaleapi.com), which is a developer-first API for human tasks. We allow customers outside the US, and are much more focused on quality than MTurk.
Definitely give us a shot, and feel free to email me if you have any questions! [email protected]
Hi Alex, thanks for your email. Replying here for the benefit of other readers.
We can't use your product for the same reason we can't use most of the alternatives on the market: there's no way to push tasks with custom HTML/CSS templates, as is possible with MTurk. This is important at scale.
For my clients, the standard use cases for e-commerce.
For example, you might have 100,000 products plus another 5,000 new ones listed per month. Your human error rate might be 3%. When a product arrives, its buyer (the corporate person responsible for signing on the brand) gives it a category and it shows up on the site. So 3% of the new products are erroneously categorised by their buyer.
You give X human votes per product, say, 10. "Which of these categories applies to this product?"
If the votes agree with each other, you can just input that as the category. If there's a split, you can feed those products to a more open ended question like "what category is this product" and use that as a starting point for either renaming the categories, finding a new category, or tagging: if it's a 50/50 split, you can just tag the product with both categories.
This is all automated and used to cost me around $200 per 5,000 products.
Another example might be training a data set for a machine learning algorithm. You send a data set to be trained by Turk and use it as training/test sets. I'm keeping this one deliberately vague for now as AFAIK we are the only ones in our space doing this and I don't even want to mention the space due to a relatively smart competitor.
As a hypothetical example, you might be trying to predict the category from the description, gender, picture features, buyer and brand, and the above categorisation tasks can be used to train the algorithms you're testing out.
I've used Fancy Hands (https://www.fancyhands.com/) before and had good results with it, though it's probably not really price-competitive with MT; I paid about $15 for a quite large amount of scraping work that would've been annoying to write software to do.
Do you use custom software to interact with MTurk, or is there already some good software for interacting with it already? I have an image dataset I want classified and I've been putting it off for a while...
Sites like CrowdFlower have point-and-click interfaces that allow you to just upload a CSV and drag and drop its columns as fields in your task template.
What I used to do was host the images on a server, publicly, then upload the URL of the images and use it to display the images within the task. It's OK for one-offs. You get a thousand free rows with the CrowdFlower trial [1] and that's probably the most painless way to go for a classification problem.
In our case though, everything will be written in-house. Automation is how we keep headcount low.
So you essentially just want a broker between yourself and MT? If all I have to do is provide you an account, and you pay up front with bitcoin, I'm happy to do this for you with 25% premium. All it involves for me is going out and buying prepaid debit cards of whatever amount and attaching them to the account. I fill them up with 75% of whatever dollar amount you send me.
Thanks for offering, but I doubt this methodology will scale. We really need a stable company with a product we can trust will both be around in 5 years and that can handle large volumes of tasks.
CrowdFlower has a de facto monopoly on the outside of the US supply and charges an enormous premium for it - enough to turn off most of them. I certainly don't want to pay several thousand a month for the right to submit jobs, although I was ok with the 25% premium in the old days.
If it's an alternative marketplace, it has to have excellent automation via API. I'm not going to use this for questionnaires, I'll be submitting thousands of jobs automatically.
If you have built this already, please email me.