.com on top of .xxx in porn links count

If you thought pornography occupied a lot of space on the net, think again. It occupies a vast space: according to MetaCert, which has developed a web-flitering technology, a scan of 645 miilion web pages - out of 3 billion or so online - contains more than 31 billion links to porn.

But it also shows that the creation of the ".xxx" web domain suffix in 2010, which was meant to create a space where pornographers, and pornography, would congregate and leave the rest of the web relatively clean, has had little effect so far.

Instead, according to a new analysis by MetaCert, a web-blocking firm, the .xxx domains contain just 0.56% of the sites that it has to block to create a "clean" web - while the original .com, .net, and .org domains contain 83.32%, 8.13% and 0.9% of the relevant domains respectively, adding up to 92.35% of the total that it blocks. Meanwhile a surprisingly high number of pornography sites - around 26% - are located in the Netherlands, and the countrywide .nl suffix hosts 2.41% of the pornography found by the company.

"Every day we are given a list of .xxx domains that are registered," Paul Walsh, chief executive of MetaCert, explained to the Guardian. "We upload the domains to our system and they are then automatically labeled and crawled. All other sites/TLDs to which they link and also contain porn, are then labeled and crawled for more sites."

And porn sites turn out to be surprisingly interconnected: MetaCert, baased in San Francisco, found that there are approximately 5,000 URLs per pornography website, and that on average each site links to another 10 sites.
MetaCert has been indexing sites around the clock, but so far the findings reinforce the suspicions of web activists who thought the creation of the .xxx suffix would be a bonanza for the company which allocates web domains there - but wouldn't actually make porn any easier to block. The ".com" suffix is available worldwide and houses some of the world's biggest brands.

Major companies have questioned the benefits of .xxx, feeling it was necessary to preserve their brand by defensively buying .xxx domains and not using them. Meanwhile pornography businesses have been less quick to buy them because they would be too easy to filter out. Many search engines automatically filter out searches containing ".xxx" on their default "safe" settings.

MetaCert says that as well as having the world's biggest set of porn URLs, it also offers a reliable filtering service - because it uses its own DNS (domain name server), the service which translates human-readable addresses such as "guardian.co.uk" into machine-readable strings of numbers. "This means, whether or not they are already deploying existing filtering technology, virtually any type of network can hook into our system without any additional hardware or software and be up and running within hours, not days or weeks," Walsh says.

He thinks that companies such as BT and Starbucks which offer customers free or paid-for web browsing in public locations should be keen to adopt his company's technology.

Blocking sites has been controversial because innocent sites - or sites which contain content that is legal in one country but not another - can be trapped. Walsh claims though that MetaCert has an error rate of less than 0.3% and that sites can be removed or added to the blocklist in two hours or less.

John Carr, an independent campaigner for child safety online, said that he endorses the company's product: "it is by far the most effective solution available on the market for ISPs and companies that provide Wi-Fi or other forms of public access to customers or members of the public."

The location of porn sites is also heavily focussed in the US, with 58.9% located in there. The Netherlands is the next largest, with 27%, followed by the UK (7.4%) and Germany (1.5%).

Walsh said the company has been researching and developing its crawling and classification platform for the past six years, and has just spent 12 months tweaking the crawler-based algorithms to construct the world's largest database of pornography web sites.

But Walsh things that the new top-level domains (TLDs) announced earlier this year by Icann, which allocated .xxx and has given permission for more than a hundred new ones - including .sex and .love - to be created - won't be a serious problem. "Given that the number of .xxx domains represents 0.56% of our data set and they have been on sale for a year - it's coming up to the anniversary this week - I would say that new TLDs will not pose any issues for two reasons. Firstly because I don't anticipate an upsurge in domains sales specifically for pornography and secondly, our system is constantly crawling, identifying and labelling new sites."