This Insane Map Shows All The Beauty And Horror Of The Dark Web

Want to know what the dark web looks like without even having to go there? A map released Tuesday by research and development outfit Hyperion Gray is bringing to life the often misunderstood and decried corner of the internet.

In what the company believes is an unprecedented effort to illuminate hidden services on the dark web, it allows anyone to zoom in and out of images containing the homepage of each website based on the Tor network. (Tor, run by the Tor Project, is designed to provide anonymity for its users and the hidden services within are considered to make up the largest section of the dark web.)

Be warned, though. The content within the Dark Web Map can be graphic. As Hyperion Gray's disclaimer notes: "These sites include mature and/or offensive content, including pornography, violence and racism."

The map shows 6,608 dark web sites crawled during January 2018. That includes all manner of webpages, from the amusing to the horrific. Amongst the myriad pages on the map, Forbes has already seen evidence of extreme sexual content, credit card cloning products, pictures of poop and penises, a large number of Bitcoin scams, Propublica's Tor website and a range of whistleblower pages based on the SecureDrop service. That includes Forbes' own SecureDrop service (feel free to drop us a tip securely anytime).

What's apparent from a cursory view of the map, zooming in and out of different areas, is that a significant proportion of mapped sites have been set up for illegal means, but there are also plenty of legal services such as whistleblower and personal pages. That's something Hyperion Gray CTO Alejandro Caceres (and co-founder with Amanda Towler) discovered too. "I like Tor's narrative that the technology preserves political freedom, privacy, freedom of speech," he said.

"But the data clearly indicates that this is at the very least a double-edged sword. While we didn't put out this research to take a side on the debate about hidden services, it's not something that can be ignored either - there's a lot of shitty stuff out there.

"I should add that I use Tor daily for many reasons, and hidden services shouldn't be taken as the core of what the Tor project is and does." Caceres also donates to the Tor Project.

From Darpa to dark web cartography

The map was partly based on an experiment carried out with the MEMEX project at Darpa, the research arm of the Pentagon. As Forbes has previously reported, the aim of MEMEX was to create simple search engines for the dark web and had proven useful for cops investigating human trafficking.

But the digital cartography was entirely self-funded, explained Mark Haase, computer scientist at Hyperion Gray and the man behind the map. Haas started with a list of 60,000 Tor addresses, also known as onions because of their .onion extensions. He wrote a script that downloaded the home page for each and took a screenshot. About 6,600 sites responded, whilst the other 53,000 either timed out or returned errors. Haase then manually redacted any screenshots that appeared to contain illegal content, going by U.S. laws.

Users will note the clusters of different sites. They're bunched together thanks to a "similarity algorithm" that assigns each page a score between 0.0 and 1.0. That similarity is based on the structure of the website's code. "The layout algorithm tries to put sites that are similar to each other close together on the map and tries to move dissimilar sites out of the way, which causes groups of similar pages to move towards each other and form into clusters," Haase explained to Forbes over encrypted chat. "Two sites can be 'similar' in terms of HTML structure but contain totally different content."

The map should provide a comprehensive view of the dark web, even if it doesn't give the entire picture. For instance, many .onion domains don't go to a webpage, but to file or chat servers. And some can be hidden, even from computer scientists who've contracted with the U.S. government. "It's representative, but definitely not exhaustive," added Haase. "If somebody really wants to hide an onion, it's very difficult for an outsider to find it."

Haase hopes that there are two audiences for the Dark Web Map. "First, novice users who only have a hazy idea of what 'dark web' means, maybe they can get a better sense for what it is - maybe not radically different from the open web that they already know - and a sense of what content is on it. Second, for expert users, we are planning a series of articles doing a deeper technical analysis and hopefully coming out with some novel observations about the structure of the dark web."

"One thing i think that's interesting about this visualization is that it likely either confirms or challenges people's existing perceptions," Towler added. "But the data itself doesn't have a motive. It just is what it is."

Anyone who believes they’ve seen illegal content on the map, such as child exploitation material, should send an email to contact@hyperiongray.com with the details.

This isn't the first time Hyperion Gray has gone out looking for nasty things on the web. It previously created a search engine for sites with common vulnerabilities, dubbed PunkSpider, hoping to expose poor security across the internet.