8 Lessons from 20 Years of Hype Cycles

As a VC at Icon Ventures and a twenty year veteran of productizing and marketing high tech for VMware, Netscape and others, I've always been fascinated by how new technologies emerge and come to market. One of the major artifacts that tries to capture the state of our market and industry each year is the annual Gartner Hype Cycle - which I always read with interest. Just last month, I had an interesting thought: "Has anyone gone back and done a retrospective of Gartner Hype Cycles - because I'd totally read that article". A quick Google search didn't turn up anything useful, so I decided I'd do the work and write it myself. This article is the result.

As most of you know, the Gartner Hype Cycle for Emerging Technologies is practically an institution in high tech. First published in 1995, the Hype Cycle proposed a standard adoption model for new technologies. In this model, technologies all go through a process of :

1. Emergence: "The Technology Trigger"
2. Excessive enthusiasm: "The Peak of Inflated Expectations"
3. Excessive disappointment : "The Trough of Disillusionment"
4. Gradual, practical adoption: "The Slope of Enlightenment" and "The Plateau of Productivity"

By way of illustration, below is the first Hype Cycle - from 1995. And it's truly a fascinating historical document. Some of its technologies that have become so ubiquitous, that they're now background noise (Object-Oriented Programming). Some technologies have simply disappeared from public consciousness (Emergent Computing). Still others are technologies that we thought were almost baked but actually took decades longer to reach full maturity (Speech Recognition).   

The most hyped technology in 1995 was Intelligent Agents. Two years later, Office 97 introduced Clippy, the enthusiastic, but incompetent assistant which was so poorly received that it effectively killed off the idea for a generation. Today, twenty years later, we’re once again trying to build intelligent assistants, although now we call them Chatbots, and the core tech - contextual reasoning in a broad domain - is still a hard problem.

I think of the Gartner Hype Cycle as a Hero's Journey for technologies. And just like the hero's journey, the Hype Cycle is a compelling narrative structure. When we consider many of the technologies in use today, we tend to recall that they were overhyped when they first arrived, but eventually found their way to mainstream usage. But ... is that really how technologies emerge and gain adoption? After analyzing every Gartner Hype Cycle for Emerging Technology from 2000 to 2016 - all seventeen years of the post dotcom era - I’ve come to believe that the median technology doesn’t obey the Hype Cycle. We only think it does because when we recollect how technologies emerge, we're subject to cognitive biases that distort our recollection of the past:  

• Hindsight bias: we unconsciously "improve" our memory of past predictions.
• Survivor bias: it's much easier to remember the technologies that succeed (we're surrounded by them) rather than the technologies that fail.

For example, Emergent Computation - the earliest technology on the 1995 Hype Cycle up above - is a great illustration of survivor bias. (Emergent computing, by the way, is computing based on distributed evolutionary algorithms - a kind of cousin to neural network based machine learning. (I had to look it up)). Today if I asked 20 Silicon Valley technologists to name which technologies succeeded and failed since 1995, I think I can guarantee that no-one would name Emergent Computation. But yet, there it is in the 1995 Hype Cycle, important enough to merit one of just ten slots in that year's listing.

(And incidentally, my intention is not to call out Gartner’s accuracy as a firm specifically. With some notable exceptions, such as the technology terms that Gartner coins itself, I think of the Gartner Hype Cycle as mostly a reflection of industry consensus.) 

But our inability to remember the past in proper context is not the only lesson from taking a deep dive into Gartner's past Hype Cycles. After analyzing every year from 2000 on, I think I can say with confidence that we are simply not very good at predicting the future. I've learned that lesson and seven more from my deep dive into the data. Read on for the details:

Lesson 1. We're terrible at making predictions. Especially about the future.

No surprise to any experienced Silicon Valley hands. In general, we're bad at making predictions. Out of the more than 200 unique technologies that have ever appeared on a Gartner Hype Cycle for Emerging Technology, just a handful of technologies - Cloud Computing, 3D Printing, Natural Language Search, Electronic Ink - have been identified early and traveled even somewhat predictably through a Hype Cycle from start to finish. 

Lesson 2. An alarming number of technology trends are flashes in the pan.

High tech has a pronounced propensity for getting extremely excited about a technology for a very short period of time. Out of the more than 200 technologies ever listed, just over 50 individual technologies appear for just a single year on the Hype Cycle - never to reappear again. Yes, it’s true that many of the Hype Cycle’s one hit wonders survive today, enjoying minor success or mindshare: Crowdsourcing - 2013, HTML5 - 2012, BYOD - 2012, Podcasting -2005. But it’s equally true that past Hype Cycles contain a long list of technologies that seem as poorly considered as parachute pants or perms. Just some of the one-hit wonders: Social TV (2011), Truth Verification (2004), Folksonomies (2006) and Expertise Location (2007). 

Lesson 3. Lots of technologies just die. Period.

Closely related to the last lesson, the last two decades are a graveyard of technologies that died permanent and premature deaths. By my rough count, an additional 20% of all technologies that were tracked for multiple years on the Hype Cycle became obsolete before reaching any kind of mainstream success. Some of the most notable technologies that appeared on multiple hype cycles but ultimately died include:

• Ultrawideband: a short range radio technology, ultrawideband reached peak hype in the 2004 hype cycle but was abandoned by 2008
• RSS Enterprise: after the success of RSS as a consumer news reader format, it was thought that RSS could become a dominant Enterprise format for information dissemination. Named as a rising tech in the 2006 Hype Cycle, it was already consigned to the trough of disillusionment by 2007 (its final appearance).
• 802.16 WiMAX: WiMAX was a competitor to LTE for the 4th generation cellular standard. Although it had minor deployment, WiMAX was essentially dead shortly after arrival. First appearing on the Hype Cycle in 2005 at peak hype, WiMAX was relegated to the trough of disillusionment in 2006 and then disappeared.
• Desktop Linux for Business: For anyone under 30, yes, this was a thing. First appearing in 2003 at peak hype, it appeared again in 2004 in that position, moving toward the trough in 2005 before disappearing completely. Desktop linux vendors just never managed to displace Windows as the mainstream business desktop OS. And VMware enabled people to run Linux as an app on Windows, never allowing it to gather footprint as an OS.   
• Mesh Networks: mesh networking is a networking architecture that builds efficient routable networks from peer to peer interactions among network nodes. It's just in the last two years that the first broad-based consumer mesh networking has reached the market in the form of Eero and other home network meshes. But before this year, Mesh Networking has the distinction of being one of the longest tracked Hype Cycle technologies never to reach wide adoption. Its history? Mesh networking first appeared on the 2003 Hype Cycle as an emerging technology and subsequently appeared in 9 of the next 11 years as a post-hype or trough technology - making its last appearance in 2013's Hype Cycle. Although I'm not a subject matter expert, my recollection is that mesh networking turned out to simply be really hard to build. Here is the history of the mesh networks entry across all the years it has appeared on a Hype Cycle.

And those are just five of the many technologies that went to the graveyard without passing Go or collecting $200. There are more (Broadband Over Power Lines!). See the Appendix below for all the technologies that didn’t make it. Sic transit gloria mundi.

Lesson 4: The technical insight is often correct, but the implementation isn't there

I was often struck by how many times the Hype Cycle had an insight that was essentially correct, but the technology or the market just wasn’t ready yet. Some of the best examples:

• WS-Enabled Business Models: riding on the wave of enthusiasm for WS* based web services in the early 2000's, WS*-enabled business models first appeared on the Hype Cycle in 2003 as a declining tech, and appeared again in 2005 at the Trough of Disillusionment. For those unfamiliar, WS* was a set of technology standards that used XML formats over SOAP as the basis for remote API invocation. The WS* movement essentially collapsed under its own technical bloat. Developers hated writing to the API's and there were many specification problems. But yet, the core insight was essentially correct. In the last few years, the rise of Twilio, Plaid, Checkr and others has validated the essential insight made all the way back in 2003: that there would be successful businesses whose only product offering was a valuable API.
• Public Authentication Services: First appearing in 2002 as an emerging technology, this prediction was presumably based on the release of Microsoft Passport - a Microsoft public authentication service based on WS* APIs. Passport also failed because of its WS* foundation. It took the development of Oauth in 2007 by an informally cooperating group from Google, Twitter and Magnolia to make the promise real. Today, of course, third party public login via Google, Twitter and Facebook is ubiquitous. Again, an insight that was correct but the implementation was faulty.
• Tera-Architecture: listed as an emerging technology in just two Hype Cycles - 2006 and 2007 - I was unfamiliar with the term and had to look it up. Astoundingly, Tera-Architecture (a term that Gartner coined but then seemingly abandoned) described something quite close to a modern data ingestion pipeline for big data. It's true that it took the release of open source projects like Kafka and Samza to make very large scale ingestion pipelines widespread, but the essential insight into its requirements were all described by Gartner in the mid 2000’s in a solid piece of prognostication.

Lesson 5: We've been working on a few core technical problems for decades

There are a number of core technologies that appear again and again in different guises over the years in Hype Cycles, sometimes under multiple aliases. Each reincarnation makes progress and leaves lessons for its successors without really breaking through. These are the technical marathons of the Hype Cycle.

• Speech recognition: as we saw earlier, Speech Recognition appears in the very first Hype Cycle in 1995, where it's climbing the Plateau of Productivity. In reality, speech recognition was far from mature in 1995. It's only - possibly - with deep learning breakthroughs in the last two years that we have reached human equivalent recognition in speech recognition. Two decades later.
• Internet micropayments: alternatively known as ecash, epayments, cryptocurrency and in its latest incarnation - bitcoin - we've been working on enabling small internet payments between untrusted parties since the dawn of the internet. Today micropayments still only work well in walled gardens like the Apple ecosystem. Perhaps it will take another generation beyond bitcoin for this to really work.
• Data analysis: large scale data and content analysis has come and gone from Hype Cycles in three waves over the last two decades: data mining (90's) gave way to analytics (2000's) which in turn gave way to big data (2010's). It seems to always take a new generation of architectures to cope with the inexorable expansion in the scope and size of data we want to analyze, and who we expect to perform that analysis.

Lesson 6: Some technologies keep receding into the future

There are some notable technologies that recur on the Hype Cycle and every time they appear they seem equally scifi. Although at some point, I'm sure they will not. The most notable are:

• Quantum Computing: as early as 2000, quantum computing was considered more than a decade away (and likely still is).
• Brain/Computer Interfaces: (also aliased under Human Augmentation) despite notable progress on neural control of prosthetics, thought controlled computing is still a work in progress with general availability lurking at least a decade away.
• Context delivery: it's been obvious for many years that capturing and brokering context is crucial to everything from content to commerce, but a general approach to this seems to continue to recede. This doesn't stop the term from continuing to pop up on Hype Cycles as an early stage tech.

Lesson 7: Lots of technologies make progress when no-one is looking

Look at enough Hype Cycles and you can see a pattern where many technologies make steady and sometimes breakthrough progress after they're considered played out. Like machine learning in the aughts, many technologies are doggedly moved forward by researchers, startups and large tech companies when their previous generation is widely seen as having failed. A couple of my favorite examples:

• Head Mounted Displays: The first generation of head mounted displays appeared in the late 1990’s and made it onto the Hype Cycle for 2001. But limitations of that era’s screen technology made the first generation dead on arrival. It's only in the last few years, the arrival of far more advanced head mounted displays for VR and AR has spurred a renaissance of hope. Enabled by higher resolution and higher frame-rate displays, cheap motion sensors, and a new generation of machine learning algorithms, these new displays incorporate almost two decades of software and hardware advances. (And they're still not quite there.)
• Speech Generation: speech generation, text to speech and speech to speech translation appear on multiple Hype Cycles (2002, 2005, 2006...) But again, it took deep learning breakthroughs to finally generate near-human performance just in the last year.

For technologists looking for out-of-favor technologies that may be biding their time before their next breakthrough, past Hype Cycles can be a fruitful source of ideas. Some of the most intriguing technologies that in my humble opinion, may be due for their second or third round of visibility:

• Peer-to-peer computing: last seen on the Hype Cycle in 2002, (although you could argue that Ethereum constitutes the second or perhaps third coming of peer-to-peer) there are plenty of interesting advances in peer to peer research over the last decade that could enable a new generation to emerge.
• Business process engines/platforms: listed in the 2005 Hype Cycle, it was expected that general purpose rules engines would power the next generation of business apps. Instead, we got highly functionalized applications like Salesforce and Workday. Perhaps IFTT, Zapier and others herald a renaissance of business process engines?

Lesson 8: Many major technologies flew under the Hype Cycle radar

One final lesson. It's remarkable the number of major technologies from the last 20 years that were either identified late or simply never appeared on a Hype Cycle. In technology, so many things that look trivial or transitory turn out to be the foundation of the next generation of business and consumer platforms. A brief list of the technologies that should clearly have been listed as important emerging technologies since the dot com wave:

• x86 Virtualization: arguably the single most important new data center technology of the last decade, pioneered by VMware
• NoSQL: the massive adoption wave of non-SQL based databases that started in the mid-2000's and gave us MongoDB, Cassandra, Redis, Couch and more
• Map/Reduce/Hadoop: the foundation of this generation of large scale data analysis
• Open Source: the proliferation of open source as a licensing model that resulted in the commoditization of infrastructure software, the rise of community code sharing and the enablement of cloud models

Those are just a few of the major technology trends that never surfaced as part of an Emerging Technology Hype Cycle - although many of them appear on the various functional, vertical and specialist Hype Cycles that Gartner has produced in ever increasing diversity over the last decade.

The Lessons About The Lessons

What I take away from my analysis of these Hype Cycles is not just how difficult it is to make predictions, and how much wasted effort goes into technologies that doesn't tend to work, but also how exciting and wondrous is the progress that we've made in technology. The labor of the last two decades has given rise to an Age of Wonders: of self-driving cars, computers that almost understand us, and unfathomable scales of data analysis. More than ever, I feel personally privileged to work among and invest in the teams creating our technological future.

Update: If you liked this post, you might also enjoy my deep dive into the Magic Quadrant - looking at 16 years of data for the Business Intelligence category.

---------------------------------------------------------------------------------------------------------

Appendix: The Raw Data Dump

For my analysis, I tracked down every Hype Cycle for Emerging Technologies published by Gartner between 1995 and 2016, and created a list of all individual entries by year. I decided to discard 1995 through 1999because the dotcom technology categories seemed mostly irrelevant to our current technology conversations.

Next, I merged multiple terms into a single category if I felt that the difference between the two terms was minor. For example, I decided that "Personal Fuel Cells" and "Micro Fuel Cells" really referred to the same thing. As another example, I decided that Context Delivery Architectures, Context Brokering and Context Services all centered around the idea of Context abstraction, so I treated that as a single term. I'm very aware that I may have merged over-aggressively, but some of the more obscure and older terms are poorly documented on the web, so I'd be happy to receive feedback on how I could improve here.

Next, in order to track movement of each technology over multiple Hype Cycles, I coded each entry by where the term appeared. To code position, I came up with an eight point scale for Hype Cycle position, from stage 1 (earliest) to stage 8 (mature). My general categorization of the stages can be seen in the figure below.

Lastly, I color coded each entry from light red (stage 1), to deep red (stage 3) all the way to green (stage 8). And using Excel's value based color coding, I interpolated those colors across the range. What this gives is what I think as a nicely efficient 10,000 foot view of Hype Cycles from 2000 to 2016.

Below is the resulting database of Hype Cycle entries sorted by year of first appearance. For example, the first entry - Synthetic Characters (analagous to our current chatbots) - first appeared at Stage 2 in the 2000 Hype Cycle, appeared again in the 2001 Hype Cycle at the same position, and then never appeared thereafter.

Google Spreadsheet Link: https://docs.google.com/spreadsheets/d/1NkC0g60q-6w72nksayvdfzCT5oOmBy97XBCGw-tW1p8/edit?usp=sharing

'If you made it all the way down here, thanks for reading - I hope you find the retrospective useful. Comments welcome!

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.