Citizen science, which asks the public to help out science projects, has produced some spectacular successes. But finding a way to grab and maintain hold of the public's attention can be a challenge. That has led to a number of projects that turn the science challenge into a game, finding ways of making a "win" into scientific progress.
But scientists have also figured out ways of hijacking existing games, including using pre-existing fan bases that recruit players through in-game rewards. Now, there's a progress report on an effort to turn EVE Online players into cell biology experts. Thanks to some in-game rewards, more than 300,000 players contributed roughly 33 million calls on where in a cell a protein was located. This not only greatly expanded a public database of information on proteins, but it enabled the researchers to better train a neural network to do the same thing.
Call it
While in many cases it has been possible to determine or infer what a protein does, that only gives us a partial idea of its actual function. That's because many proteins are shipped to specific locations in cells. So while two proteins may look similar in terms of the order and identity of their amino acids, one may be shipped to the nucleus, where it interacts with DNA, while its relative gets sent to the cell's surface, where it acts on proteins in the surroundings. So figuring out where a protein normally resides within cells can go a long way toward helping us figure out its normal functions.
So how do you figure out where in a cell a protein ends up? It's relatively simple if you have antibodies that stick to the protein. If you label those antibodies with fluorescent tags and float them through a dead cell, they'll stick to wherever the protein is found and make it glow. Combine that with a sufficiently good microscope, and it's possible to get a good image of where the protein normally resides in the cell. To an extent, it's even possible to automate this process.
The problem is that you end up with a huge collection of images, each of which has to be manually classified by a person who understands the interior of cells.
Faced with just this sort of huge collection, an international team of researchers decided to train gamers to do it for them. Working with an organization called Massively Multiplayer Online Science, they arranged for the developers of EVE Online to build an in-game interface called Project Discovery that could be adapted for the image classification task. Participants would be rewarded by in-game badges and currency.
Flawless victory
The gamers responded in a big way, with more than 300,000 of them taking part in the year that the experiment was running, and about 60,000 of them went through all the training images and contributed to the experimental data. Collectively, they evaluated more than 33 million images of fluorescently labeled cells, with nearly 24 million of those considered high-quality evaluations.
Where did the low-quality evaluations come from? At the start, the researchers rewarded people for getting an answer "right" with in-game currency. But people quickly figured out that "right" simply meant that a lot of people agreed on the same answer, and they started gaming the system. That loophole was eventually shut down. Based on feedback from the initial users, the researchers added a few difficult-to-evaluate images to the training set, as the first training set didn't help people identify rare and unusual patterns of protein distribution.
Given the massive interest, the gamers looked over every single image an average of 78 times, allowing the researchers to get six times the minimum number of calls they needed to consider an image properly evaluated. There were a few cellular structures the gamers weren't good at telling apart, but the overall quality was better than any automated system that had been tried on the data. And the authors estimate that, if they were to pay $0.05 per task on Mechanical Turk, the project would have cost them more than $1.5 million dollars.
If you would like to see some of the results, they're available at the Human Protein Atlas. Thanks to the gamers, the Atlas was able to add five new categories to its existing set of protein locations.
Lessons learned
The paper notes that the whole project was dependent on the civic-mindedness of EVE's developers: "Participation in [the project] on behalf of the gaming company (CCP games) is voluntary based on their desire to promote scientific research and foster good will in their player base." That sort of generosity won't always be available (although Eve has moved on to an exoplanet-focused research project). In addition, this won't be appropriate for every type of project; the researchers behind this one note that, if they hadn't had a massive database of images available, the gamers would have rapidly exhausted all their material. "The players are very fast," they note in an understatement.
Because they couldn't guarantee that gamers would be around to help them out as more images made their way into the database, the Human Protein Atlas team decided to train a deep-learning network on the images used in the project. In essence, they took the information generated by the gamers and used it to improve an AI that could perform the same task. And it did indeed make for an improvement over any previous automated classifier system (and a number of previous versions had been attempted.)
The final conclusion of the researchers was that an expert with experience of what the inside of a cell looks like is still better than either 300,000 gamers or a well-trained computer algorithm. But as that sort of expertise tends to be rare and somewhat expensive, gamers may have a place in cell biology, at least until neural networks improve a bit from where they are now.
Nature Biotechnology, 2018. DOI: 10.1038/nbt.4225 (About DOIs).
[contf] [contfnew] 
Ars Technica
[contfnewc] [contfnewc]
