Updated, 6:32 p.m. EST, Thursday, January 10
In its quest to become the go-to source of breaking news for Web users, Twitter has turned to an unlikely tool: Humans.
On Tuesday, the company revealed that Twitter’s real-time search feature, which allows visitors to search the most relevant tweets by a specific keyword, has recently been relying on an army of workers selected to manually determine the answers to those queries.
Specifically, when a user searches by a trending term, e.g. “Happy Birthday Harry”, Twitter needs to quickly identify which “Harry” the user is looking for so it can display the right account information: Harry Potter or Harry Styles of the pop band “One Direction,” for example.
To stay on top of these continuously changing popular searches, Twitter monitors spikes in new search queries. When a query reaches a certain level of recurrence in a particular amount of time, the system automatically employs a human judge to finalize the search process.
As Twitter data scientist Edwin Chen and senior software engineer of revenue Alpa Jain wrote in an official Twitter Engineering blog post:
“Our custom pool of judges work virtually all day. For many of them, this is a full-time job, and they’re geographically distributed, so our tasks complete quickly at all hours. We can easily ask for thousands of judgments before lunch, and have them finished by the time we need, which makes iterating on our experiments much easier.”
The move is in direct contrast to the mainly algorithmic approach to search pursued for over a decade by search king Google and its rival Microsoft.
Stranger still, Twitter relies on yet another tech giant for its army of workers: Amazon. Amazon’s Mechanical Turk website, an online labor marketplace launched by the retail company in 2005, allows “requesters” like Twitter to post discrete, ultra-low paying computer-oriented jobs called “human intelligence tasks,” (HITs) that pay out between a few dollars to a few cents. The site pairs these job opportunities with willing human laborers, self-described “Turkers,” with Amazon pocketing a 10 percent cut per each match.
The system is named after a famed 18th century Hungarian hoax, “The Turk,” a complex puppet that was advertised by its creator as an automaton (proto-robot), capable of beating human players in chess. In reality, a person hid inside controlling the machine.
Amazon’s modern system, by contrast, openly notes that humans are completing the tasks, and since the website launched as an external product in 2005, many companies have turned to it to outsource undesirable, low-skilled computer tasks. Examples of jobs posted on Amazon’s Mechanical Turk recently include: “provide a short descriptive title for a piece of adult content by looking through images,” and “Find company contact information,” given a photo and image online.
The way Twitter uses Amazon’s Mechanical Turk to crowdsource answers to Twitter realtime search queries is unique from how most companies use the service, though. As Chen and Jain described in their post, Twitter handpicks its own small pool of trusted Turkers, eschewing Amazon’s built-in filters and the work-matching services of third party company Crowdflower.
Instead, Twitter finds its workers as follows:
“our judges are culled from the best of Mechanical Turk, they’re experts at the kinds of tasks we send, and can often provide higher quality at a faster rate than what even in-house judges provide. For example, they’ll often use the forums and chatrooms to collaborate amongst themselves to give us the best judgments, and they’re already familiar with the Firefox and Chrome scripts that help them be the most efficient at their work.
Asked by TPM how many Turkers it employs, how specifically it evaluates them, and how much it pays, Twitter declined to specify.
“We recently launched this particular system,” a Twitter spokesperson told TPM. “It’s worth noting, however, that we have used human computation systems for some time, and for a variety of purposes — for example: to measure performance of our search algorithms.”
In its blog post describing the system, Twitter also included the following video assembled by its custom Turker search army, which Twitter said demonstrates “the kind of top quality our workers provide.” The video is a “singing telegram” praising Twitter’s Turkers — a series of still, stock images with lyrics set to a piano and vocal song that includes the line: “And with all teams there’s always one that goes the extra mile / The one that works their ass off, yet does so with a smile.”
The vocals and music were provided by one specific Twitter Turker, “workasaurusrex,” who Chen and Jain called “amazing” in their blog post.
“Workasaurusrex” seemed to appreciate the nod. As the self-identified male Turker wrote in a post on the forum CloudMeBaby on Wednesday morning:
“Jack and co. posted a new entry in their Twitter Engineering blog today that gives some cool insight into the turker’s role in what they do. I’ve been checking out the twitter comments (link at bottom of the blog entry) and am getting a kick out of folks’ thoughts on the human side of the process…not necessarily ‘shock’ that real-life people are part of the efforts behind the twitter curtain, but it definitely perked some ears. Always nice to see positive reviews/thoughts/comments about turking!”
But “workasaurusrex” didn’t always hold such a high opinion of Turking. As he wrote in another forum, TurkerNation, in July 2012:
“I forget sometimes how crazy it sounds to be doing this kind of work. I was telling a friend about how I spent my weekend plugging away at a task, finishing a couple thousands HITs and now I’m waiting, crossing my fingers it will go OK and all will be approved. She looked at me and said ‘you mean you did all that and you might not get paid?’ And sadly I responded by saying, ‘It’s happened before unfortunately. And not only might I not get paid, but worse my rating could drop and I would be severly limited in the work I can do in the future.’ She said nothing more and just looked at me like I was an idiot.”
Salon in 2006 published a first-hand account of a writer who worked as a Turk and communicated with other turkers, as well as critics of the system, who labeled it a “virtual sweatshop.”
TPM reached out to workasaurusrex for more of the Turker’s thoughts on the enterprise and we’ll update if and when we receive a response.
Late update: The Turker responsible for the vocals and music on the video shared by Twitter, workasaurusrex, responded to TPM to explain the context of his forum posts and provided the following further insights on turking and the Twitter video:
“The video was originally as a thank you to a Twitter employee,” workasaurusrex wrote. “The project was completed around the first week of November. The instructions were pretty basic: to take the lyrics created by other turkers and craft a singing telegram. I was given flexibility to be creative (and looking back now at the emails from the beginning, piano was not even part of it, I offered that part as well).”
Indeed, the original project description was merely for workasaurusrex to create the audio for the singing telegram, but he offered “to add the video part because I thought it would be fun.”
Workasaurusrex told TPM that he has been a Turker since 2009, “though not seriously until 2010” and that he “really enjoy[s] it, especially now after a couple of years of really getting to know the system and how to seek out work that is worthwhile and interesting.”
Workasaurusrex also explained that his latter forum post about the uncertainty of turking wasn’t a criticism of the system.
“My comments to my coworker were my funny way of reflecting the potential pitfalls but were in no way reflective of what typically happens to experienced turkers,” Workasaurusrex said. “I am VERY selective in what I do, and I would not do a couple thousand HITs for someone unless I had a very strong idea that things would be OK.”
Workasaurusrex’s full time day job is as a special education teacher. He has four kids and also coaches two soccer teams. He’s worked as part of Twitter’s select group of custom turkers on other tasks for the website.
Workasaurusrex had attempted to contact TPM prior to the initial publication of this story, but this reporter’s spam filter miscategorized workasaurusrex’s email as spam and went undiscovered until now.
