The open data movement has reached a new and significant milestone this month. It arguably began 24 years ago when Cern scientist Sir Tim Berners-Lee first invented the world wide web. Last year in October, just four months after taking part in the opening ceremony of the London Olympics, Berners-Lee co-founded the Open Data Institute with Sir Nigel Shadbolt.
Within just a few months, the ODI has attracted multiple members, support from universities and governments, has helped dozens of startup companies, has set up certification for open data, and is holding its first annual summit next week at the Museum of London. As a promoter for open data practice, can the ODI kickstart a new social revolution as much as the world wide web has done? Will sharing open data become as established in modern life and language as sending texts and using Facebook? It's quite possible.
The ODI is a catalyst for change, but it sits within a long and distinguished history of open or shared data. Over thousands of years, we have always recorded and shared information with a view to improving our lives. The term 'open data' has become firmly established with the formation of maker movements and other organisations over the last decade. Its practice has accelerated, thanks largely to the world wide web. But, by other names, it has always been with us. So what are the roots of our instinct to share and collaborate, and was this always open data?
First, the recent past. On 21 January 2009, on his first full official day in office, Barack Obama issued a memo on the Freedom of Information Act to federal departments. The new president announced: "In the face of doubt, openness prevails." He quoted the phrase "sunlight is said to be the best of disinfectants", talked of "accountability through transparency" and said such "accountability is in the interest of the government and the citizenry alike". In an extraordinary move, he officially made open and machine readable the default setting for government. It was a stark contrast to his predecessor George W Bush. The data.gov website, launched in May 2009, mirrored by many new government sites in other countries that same year, set about the release of hundreds of thousands of datasets from 295 government APIs and 175 agencies, and has made accessible multiple apps and tools.
Where that sits with the NSA files, Prism and Edward Snowden is a matter of fierce debate. Is the issue of spying on private data, whether of world leaders or private citizens, the US government's evil Dr Hyde to its good Dr Jekyll over open data policy? Open data, by its very definition, must play a positive role – it has to because, well, it's open. It can only be a force for good. It defends privacy and is designed to utilise public data to improve people's lives. My intention here is not to set this up as a good-versus-evil story, but in Hollywood terms, if the NSA scandal was eventually turned into a Fifth Estate Wikileaks-type movie, Berners-Lee would definitely be portrayed as one of the good guys.
Now for a potted history. Has the sharing and recording of data always played a benevolent role? Over 20,000 years ago Palaeolithic humans, as well as developing tools, would record numbers of objects by marking notches in stone, wood and bone. Who were they doing that for? Themselves? Their community? They were counting, and counting on data, but it wasn't necessarily 'open'.
Formal written language, however, could be described as the beginning of data sharing. Sumerian writing was established in Mesopotamia around 3500BC, and, alongside Egyptian hieroglyphics in the same period, began to record all kinds of data from government taxes to schooling and religious rites. You couldn't call it open, but it was a vital development. In some ways it was the beginning of a new artform.
Pythagoras (572BC-495BC) is best known for his theorem, but his influence over other philosophers including Plato became a cornerstone in the foundations of western culture. He set out to show how mathematics can be used to systematically understand and compute aspects of nature, music, and the world. He is quoted as saying: "Friends share all things. Number is the ruler of forms and ideas."
So going back further to more than 17,000 years ago to an original piece of art, the Lascaux cave paintings in southwestern France depicted hunting scenes and, as some later analysts suggests, early attempts at star charts. Were they in some ways data for public consumption, or only private advantage, or were they simply a colourful expression of experience?
Art and data can indeed go hand in hand. Zoom into the present for the moment, and information can be beautiful, as data journalist and designer David McCandless proves. In one of its first moves to promote open data, the ODI in London even commissioned some artworks to reflect open data's role in the world. Among such, an earlier piece, Vending Machine by Ellie Harrison (2009) (slide 22) dispenses free packets of crisps every time the BBC News feed, to which it connects through a program, mentions any of a selection of key words associated with the recession. During the last budget it emptied out its entire contents. Talk about credit crunch.
But you can't get very far on crisps. Open data, or its predecessor, has always helped us find our way through the world. History reveals traces of us leaving markers for others to follow. Inukshuk landmarks, artworks in themselves, would help Inuit people find land routes and fishing areas in Alaska and Greenland. Around 1150BC the Turin Papyrus in Egypt became one of the first topographical maps – not just useful to one person, but to many. Turning myth into method, indigenous Australians would use "songlines" to guide them across the routes of their ancestors, "singing out the name of everything that crossed their path - birds, animals, plants, rocks, waterholes – and so singing the world into existence", as Bruce Chatwin put it in his remarkable 1987 book, The Songlines.
One of the first real pioneers of shared open data was 19th-century oceanographer and meteorologist Matthew Fontaine Maury. After extensive analysis of ships' logs, rain, winds and currents, he mapped the information on charts and in 1855 published his book, The Physical Geography of the Sea. Here he made his intention clear, "that each may have before him, at a glance, the experience of all". He sent his findings to others and encouraged their contributions, launching a worldwide project for safe navigation.
So is satnav a form of modern open data equivalent that has continued Maury's vision? Not really - it is purchased. Satellites aren't public property, and though apparently free to use, Google Maps, for example, is made up of large amounts of leased copyright material.
Returning to public documents, here's an extract from a certain tome published in the year 1086: "There are 9 villeins each on 1 virgate and 41 cottars who pay 40 shillings a year for their gardens … meadow for 11 ploughs, and woodland for 100 pigs". Where is this rural setting? Central London. It's an extract from the Domesday Book describing the parish of "The Land of St. Peter of Westminster In Ossulstone". It may have been all about ownership, but here lay a true landmark of national surveying.
History moved on, and so did language, with an urge to make it more open and accessible to all with the opening of a new chapter that described "a table alphabeticall conteyning and teaching the true writing, and understanding of hard vsuall English words". Teacher Robert Cawdrey compiled what was probably the first alphabetical dictionary of the English language in 1604 with some intention of universal education, even though he did not regard all men, nor indeed women as equal, his book being "gathered for the benefit & helpe of ladies, gentlewomen, or any other vnskilfull persons … to vse the same aptly themselues".
What about the natural world? "Nature does not proceed by leaps and bounds," proclaimed another pioneer, Swedish botanist Carl Linnaeus (1707-78), but he made a giant leap by establishing the binomial nomenclature, or naming scheme for all living things, becoming the father of taxonomy and establishing a basis for the work of many who followed, including Alfred Russel Wallace and Charles Darwin. This could surely be seen as another milestone in universal data language in the spirit of openness.
One of the heroes of open data worked in 1854 with a truly brave, hands-on approach. Dr John Snow reported: "On proceeding to the spot, I found that nearly all the deaths had taken place within a short distance of the Broad Street pump." Snow identified a water pump as the source of a cholera outbreak in what is now Broadwick Street in Soho, London. He disproved the common theory that the disease came from a miasma or "bad air". It preceded Louis Pasteur's germ theory. His analysis using maps of how the disease spread is seen as the birth of epidemiology.
Without the computer, modern open data would not be possible, and it could be seen to have been formed by a thread of moments throughout history. The Sumerian abacus (c 2700 BC) became possibly the first computational tool. From 1669 and with publication in 1703, Gottfried Wilhelm Leibniz developed logic in a formal, mathematical sense with his writings on the binary numeral system. In 1837 Charles Babbage first described his Analytical Engine, a machine with expandable memory, an arithmetic unit, and logic processing capabilities. And then in 1936, Alan Turing developed his Turing machine, a hypothetical device that manipulates symbols on a strip of tape according to a table of rules. It became the basis of modern computing.
There are countless gaps in this potted history, admittedly, and I welcome more examples. Many have led up to that key moment, when in 1989, Berners-Lee proposed a system in which: "HyperText is a way to link and access information of various kinds as a web of nodes in which the user can browse at will." What happens now is anyone's guess. But if we are successfully going to fight for social, environmental or economic progress, on everything from climate change to energy bills, improving transport or fighting disease, open data will certainly play a very big part.
