What might the corporate library of the future look like? As the Web becomes increasingly personalized and intelligent, with algorithms that understand our interests and deep learning systems that can make sense of everything from text to movies, how might all of these tools come together to reimagine the corporate reference library of tomorrow?
Half a century after the debut of the modern keyword search engine, we still rely on carefully selected textual keywords and phrases to search our vast archives of human knowledge. Boolean queries and complex search operators can all help, but at the end of the day the nearly unimaginable wealth of human information is still accessed through the lowly keyword.
The rise of deep learning algorithms may finally help us move beyond the limitations of trying to guess the exact wording an author of a document may have used.
Today even the most powerful enterprise search engines are frequently stymied by synonyms and different ways of saying the same thing. In contrast, library patrons are ever more accustomed to “intelligent” search engines that can search across minor and even major linguistic differences.
Search on Google for a specific phrase and documents that mention related terms are likely to show up even without containing the keyword itself. To the end user, it doesn’t matter that this may merely be because some other page on the Web used the keyword in linking to the page, rather than the result of some intelligent algorithm. All that matters is that they ran a search and relevant content was returned.
Moreover, since most searches return more than a single result, the ranking of the returned material is absolutely critical. Users have become accustomed to Web search engines that can bring to bear hundreds upon hundreds of metrics and the structure of the entire Web in real-time in deciding what to surface on the first page of results. In contrast, enterprise search engines often rely primarily on half-century-old simple textual relevance.
Web search engines increasingly personalize our results, using our past history of what kinds of search results we click on to learn the kinds of pages and sources we prefer. Thus, ten people all running the same search at the same moment might get ten very different first pages of results. In contrast, few enterprise search systems support much in the way of individual-level personalization across an entire company.
Global companies must work across languages, yet when it comes to their library, typically only a small number of languages are supported. An engineering patron of an American company's library looking for past designs of a particular component might be assisted by library personnel searching in English and perhaps a few European languages. A Chinese technical report that is a perfect match would likely be entirely missed, if the library subscribed to it at all.
Here again, today’s users are increasingly accustomed to being able to look across languages with seamless machine translation. While Google Translate is far from flawless, its accuracy is “good enough” to understand the basic gist of a typical document and its transparent integration into Google’s Chrome browser means that when browsing across the Web, the language a page is written in is slowly becoming less important over time.
In fact, my own open GDELT Project pioneered the model of mass machine translation at global scale. Absolutely every news article it monitors globally across 65 languages is machine translated in real-time, allowing researchers to search in English to identify relevant coverage globally regardless of what language it was written in. When they click on a link to visit a given page, Chrome’s built-in Google Translate integration translates the page for them instantly.
Together, this mass machine translation for search and processing and real-time in-browser translation offers a first glimpse at a post-linguistic world in which the language(s) we speak are no longer the only languages through which we can listen to the world.
We increasingly have tools that can not only make sense of the written word at higher semantic levels but can also do better at extracting the latent emotional undercurrents that affect how we understand that text.
In many ways, however, the digital era has ushered in a move past the written word. We are quickly becoming a visual-first society. We no longer tweet with words that the weather is nice. We post a photograph of a cloudless blue sky.
Researchers are no longer content merely to keyword search text, they need to be able to penetrate the visual world.
In 2014 I reimagined the concept of the book, collaborating with the Internet Archive to extract the images and their captions from 600 million pages of public domain books dating back 500 years from over 1,000 worldwide libraries. Creatively repurposing the OCR that had already been performed on the books, the project was the first to attempt to extract book images at this scale. In doing so, it demonstrated that the technical tools and workflows existed to make such mass image extraction feasible, rather than being limited to the small experiments of the past.
A year later I applied Google’s then-nascent Cloud Vision API image understanding technology to begin exploring the visual realm of the world’s news, analyzing the imagery in worldwide online news coverage. Today that effort has cataloged more than half a billion worldwide news images, making it possible to explore how we “see” the world through the news.
In the years since, there are now tools to understand video and speech in much the same way.
In fact, there are even streamlined workflows that combine the tools together, allowing enterprises to upload a vast archive of scanned documents and have them not only OCR’d into searchable text but seamlessly annotated with entities, emotions and structure.
Over the years, I’ve demonstrated the power of bringing advanced document understanding to everything from Wikipedia to 21 billion words of academic literature, including, with permission, the holdings of JSTOR, DTIC and the Internet Archive’s 1.6 billion PDFs. My own doctoral dissertation involved the mass processing, with permission, of more than 4.7 million news articles totaling 1.3 billion words from LexisNexis.
Each of these projects demonstrated not only the feasibility of applying a vast array of advanced data mining techniques to massive text archives of the kind corporate libraries work with but also the entirely new interaction and understanding modalities those algorithms could provide, such as mapping concepts and the flow of ideas.
Corporate research libraries today will find they can purchase bulk data mining licenses to many of their large subscriptions, allowing them to mass download that content and process it locally or in a secure cloud account, from machine translation to image extraction and advanced documented understanding. Using the preexisting AI suites from cloud vendors like Google, companies don’t even need their own AI staff. They can simply use an enterprise API to download tens of millions of journal articles into a cloud storage directory, along with their own corporate document archives, then pipe that collection through multi-lingual OCR, machine translation, image analysis, document analysis and more advanced workflows, customized models and even entirely bespoke analyses. Archives of technical videos and talks, engineering voice notes and other content can all be integrated into a single searchable and intelligent archive.
Yet mass analytics aren’t limited to the documents corporate libraries typically manage. The ability to perform hyperscale machine learning and automated deep learning model construction are also part of today's AI-forward cloud, meaning libraries can help their patrons manage more traditional numeric data as well.
Over time it is likely that the cloud will even begin to offer tools for helping to search all of this content after it has been transformed, leveraging the same AI-powered personalization and intelligent results ranking that has become so ubiquitous on the Web.
In fact, corporate libraries do not actually need to learn these cloud AI offerings themselves. A growing number of major enterprise document repositories and search systems offer built-in integrations, making adding these features as simple as flipping a switch.
Putting this all together, the corporate library of the future may actually become one of the primary entrance points for AI into the enterprise, bringing to bear the vast power of cloud AI to understanding a company’s vast knowledge archives that today are either merely residing unused on file servers or at best searched by the half-century-old keyword.
The rise of accurate and scalable AI, available ready-to-go through the cloud, will ultimately transform the modern corporate library, making it a technological centerpiece of the enterprise and one of the landing points for semantic AI. In turn, such tools will free corporate librarians to spend less time searching and managing content and more time working directly with patrons to understand that content and to leverage scalable understanding tools to their maximum.
In the end, like all fields, AI is coming for the library. Not to replace humans but to free them to do the far more interesting tasks they got into the library field for in the first place.
