Taskopus

Bitcoin Cash-powered crowdsourcing marketplace

Multi-stage projects

(extracting keywords from texts)

Those examples were very simple. What if I need to do something more complicated, like extracting keywords from texts. Most of the people have no idea what keywords are. So first of all I need to give people an explanation of how to do the task. Then I will select the best people and ask them to do the job.

Or a similar problem: I've got too many lazy workers and want to keep only the best of them.

There is a solution. Think of it as a multi-stage job interview with remote education, but for mere dollars instead of thousands.

So, let's set up our first project that would be used to educate and find people who are able to understand what keywords are and extract them. Think of it as a first stage in multi-stage job interview. I'll put "Extract Keywords" into title, and the following into description:

A keyword is something that people would use to find the text. You will be shown a text. Try to think of the keywords for a provided text.

Since our instructions are a bit long, I would use "Information" field to add them:

For example, if the text is "I am a professional web designer of 10+ years, looking to work for Bitcoins", then the keywords would be: 1) web designer 2) bitcoins 3) professional web designer. But, for example "I am" is not a keyword - it is not something that people would use to search for this text. Similarly "10+ years" is also not something people would use to search for this text, so it's not a keyword. The phrase "looking to work" is also not a keyword.

Now I'll add and information field with this:

The text is: I have been working as web developer for 10 years working on both Linux and FreeBSD systems." What are the keywords here?

Basically some text to test the workers ability to follow the instructions.

In case of batching tutorial, where some users missed that "Brunswick" is a city, I'd add that as one of the questions to filter out people who don't bother checking, maybe something like "Is 'Singapore' a city", "Is 'sunset' a city?", "Is 'ice cream' a city?", "Is 'Brunswick' a city?"..

I'll add the fields: "Keyword 1", which is required, and a text entry, copy, Keyword 2, ..., Keyword 10. I would require three keywords, the rest would be optional. The idea is to have to many to fill, but at the same time to try to see if users didn't read instructions and would include "10 years" or "working" into keywords.

I'll also enable the automatic payments to not have to approve each 1 cent.

The total limit I'll put a 100, but I can pause the project any time. And since this is a regular project, not table-based - I need to limit the number of tasks each user can do to 1. Otherwise users would do the same task multiple times. (You'll get a warning if you try to do that)

Let's run it (Click "Save & publish").

Ok, the first results are in.

This guy did a poor job, so I'll just skip him. (I could also be manually approving people and refusing payments, but it's just 1 cent).

Ok, this guy did well! Let's give him a label of "keywords-1". You'll understand in a minute why. Find "Add label" below the answers of this user and click it.

The users can't see those labels. I'll look through the workers and give more of them the same label.

The number 1 at the end of the label is basically "stage 1". It's my personal preference of how to name the labels, it's up to you how to name your labels. I just find it easier to add a number at the end to mark the stage in the selection process.

A next user did well too, so we add "keywords-1" by clicking on the label. Look through all users and add the label to the users that did well on first stage.

Let's create the second project.

This time I have an Excel file with hundreds of texts from which I want to extract keywords.

That's a lot of work!

I'll copy the first task (then click edit) since it's essentially the same. First of all I want to replace the text with a substitution {{ text }} (that's the name of the column in my excel file).

Then I want to upload my Excel file, which I saved in CSV format. Click "Add table-based input". Now the total limit I'll put in 130 since I have 130 texts and I want each text to be done by 1 worker. Again, you can put 10000 there, but if there are only 130 rows in table data - only 130 would be done.

Remove the limit in "Limit per each user", since now that we have selected our users - a single user can do as many rows as he or she wants.

I'll put "1" in "Times each row", since I want each row done by exactly one worker. Since I've hand-picked them, there is probably very little point in requiring duplicate effort (though it's up to you, sometimes second and third workers might notice something first one didn't).

Now, here comes the magic. I want this task to be only available to the users who did well on my first task. Remember I gave "keywords-1" label to good users? Now I can click Add filters and put in "keywords-1" as a requirement.

So, now only users that have this label can take this task. We can actually give them more money, like 5 cents to incentivize those good users to do our job first. The tasks on workers screen are sorted by median hourly rate (think of it as "if you were to do this task for one hour straight - how much would you earn?" So if a task typically takes 2 minutes and a user pay is 2 cents that means that hourly rate is 2 cents times 30 = $1.20) You can see this number in your projects list. The bigger the number - the higher your project is on worker's screen.

I'll start with 5 cents per task, but look at how long it actually takes to write good keywords. It's possible to change the price during project run, but it changes the median hourly rate accordingly. I have 130 rows, so that should be about $6.50 for whole work.

Let's run it ("Save & publish").

So far 3% of the task has been done and hourly rate is about $4.50. If the workers are working on your task - that means that it's ok for them.

And the results are in. Some of them are better, some of them are not good, so we can remove the label "keywords-1" from users whose results we don't like or we can put "keywords-2" label to those who we like and give the next task only to those users.

But, generally for $6.50 that's a great job!

You can also add links to images in your instructions, to videos (on YouTube) on how to do your task. But remember that you need to change your price accordingly. The time it takes for a user to see your instructional video affects calculated hourly rate of your task.