Airlines, insurance companies, hospitals and many other organizations are trying to figure out how to corral all their data and turn it into something useful.
Kaggle, a start-up, has figured out a way to connect these companies with the mathematicians and scientists who crunch numbers for a living or a hobby. On Thursday, it announced it had raised $11 million from investors including Khosla Ventures, Index Ventures and Hal Varian, Google‘s chief economist.
“We’re really making big data into a sport,” said Anthony Goldbloom, Kaggle’s founder.
Through Kaggle, an organization can run a competition by proposing a riddle it needs answered, offering its data, and granting cash prizes. Numbers crunchers on Kaggle have bested years of academic research by predicting how long people with HIV would live, outperformed commonly used algorithms for mapping dark matter and discovered that people who order vegetarian meals from airlines are more likely to show up for the flight.
Here’s how it works. An organization runs a competition by proposing a riddle it needs answered and offering years of data. Anyone can enter to crunch the data and answer the riddle to win a cash prize.
Currently, a health plan is offering $3 million to someone who can take years of patients’ medical claim data and produce an equation to predict whether patients will be hospitalized in the next year, so doctors can try to intervene beforehand. It has almost 8,000 entries so far. A bank is offering $5,000 to someone who can use credit score data to come up with a better way to predict whether someone who receives a loan will enter financial distress in the next two years.
The 17,000 people who have registered to answer questions include professors, a glaciologist and an English major. About half are computer scientists but the best performers are physicists, Mr. Goldbloom said. They do it because “data science is an area where it’s really difficult to prove your own skills and get access to interesting data sets,” said Jeremy Howard, Kaggle’s chief data scientist.
They know about life as data scientists. Mr. Goldbloom studied econometric modeling and worked for the Australian government before starting Kaggle last year. Mr. Howard was a consultant and entrepreneur who caught Mr. Goldbloom’s eye because he kept winning competitions on Kaggle.
Kaggle charges the organizations that host competitions a fee and eventually hopes to take a cut of the prize money. It will also host private competitions for organizations that don’t want to publicly post their data sets online, Kaggle said Thursday. They choose the most elite number crunchers on Kaggle and ask them to sign non-disclosure agreements to compete.
In private competitions, every entrant gets paid. The idea is to someday make it possible for data scientists to earn a living income on Kaggle, Mr. Goldbloom said.
“The incomes of data scientists begin to look more like tennis or golf players or hedge fund managers, rather than sitting in the bowels of Allstate‘s actuarial team,” he said.