The hidden humans powering the AI economy
CBC
Since January, Tina Lynn Wilson of Hamilton, Ont., has been freelancing for a company called DataAnnotation.
The 45-year-old says she loves the work, which involves checking responses from an AI model for grammar, accuracy and creativity. It calls for analytical skills and an eye for detail — and she also gets some interesting projects, like choosing the better of two samples of poetry.
“Because it is a creative response, there would be no fact-checking involved. You would have to indicate … what the better reply is and why.”
The work Wilson does is part of a huge, yet not well-known, network of gig workers of the emerging AI economy. Companies such as Outlier AI and Handshake AI hire them to be "artificial intelligence trainers, contracting with large AI platforms to help them train their models.
Some data annotation work is poorly paid — even exploitative, in other parts of the world — but there's a broad range of jobs in training, tending to and correcting AI. It's labour the tech giants seem to prefer not to talk about. And as models advance, they will require more specialized training — meaning companies may soon no longer need many of the very humans who helped make them what they are today.
We often hear that today's generative AI is trained on vast amounts of data to teach it how human ideas typically go together. Sometimes called pre-training, that’s only the first step. For these systems to produce responses that are accurate, useful and not offensive, they need to be further refined, especially if they're going to work in narrow fields in the real world.
This is called fine-tuning, and it relies on human expertise. It's basically gig work: done on a per-assignment basis, without guaranteed hours. The Canadian AI trainers we spoke to made about $20 an hour, though some more specialized work can pay around $40. Still, inconsistency can be a problem.
"You cannot rely on this as a main source of income," said Wilson, who described her work as that of a generalist. Many other annotators consider it a side hustle as well, she said.
Reinforcement learning from human feedback is a type of fine-tuning that relies on people evaluating AI outputs.
Wilson’s work involves evaluating how “human” an AI response sounds.
“This is especially true for voice responses,” she said. “‘Is this something a human would like to hear?’”
So, when ChatGPT or Claude sounds uncannily human, that's because humans have trained it to be so.
"It's still the output of a software product," said Brian Merchant, a tech journalist specializing in labour and digital technology. "You need quality assurance of the output of a commercial, for-profit, software product.”
Outlier AI has more than 250,000 active contributors across 50 countries, said Fiorella Riccobono, a spokesperson for Scale AI, its parent company. Eighty-one per cent hold at least a bachelor’s degree, she said. The company was not able to provide Canada-specific numbers.
