Welcome to the ninth OnionScan Report. The aim of these reports is to provide an accurate and up-to-date analysis of how anonymity networks are being used in the real world.
Over the last year we have produced reports into the state of onion services. In this final report we will examine the most up to date analysis of the OnionSphere compiled from the largest OnionScan crawl we have ever conducted.
Note: This research could not exist without our supporters, and we hope we can continue to deliver new insights, research and technology in the future. To help us do so, please support us
A Year In OnionScan
Over the last year we have written many reports relating to anonymity, security and the dark web.
Our first ever report covered estimates of crime on the dark web, counting onion services and hidden service security, we then looked at what platforms are used on the dark web.
For our third report we produced the first maps of the dark web analyzing relationships between onion services.
Our forth report examined how https certificates were being used (and misused) by dark web sites.
In September we took a close look at the hosting provider Freedom Hosting II and demonstrated that they were hosting a significant portion of the dark web.
Towards the end of 2016 we demonstrated a new attack to extract co-hosting relationships from hidden services. And at the start of 2017 we demonstrated how much information we could extract from a popular dark marketplace.
Much has changed in the onionsphere in that time, and yet much has remained the same. The issues that we identified in all of those reports haven't gone away. Hosting an anonymous service is still very difficult.
The Death of Freedom Hosting II
In early February the free hosting provider Freedom Hosting II was compromised along with all of the sites it was hosting.
OnionScan data was used as part of the initial analysis to confirm the impact.
While it is still difficult to say how many of the 10,000 sites compromised in the database leak were serving content before the compromise - we can definitely say that over 2000 sites that were part of the OnionScan master list are no longer being hosted - and that this has had a significant impact on the overall size of the onionsphere - an estimate we put at between 15-20% of all dark web site content on the tor network.
An analysis of the database dump revealed that at least 3 of the largest databased in the dump were forums related to sharing and discussing child sexual exploitation - this would appear to align with statements made by the hackers, that it was only after they had gained access to Freedom Hosting II that they decided to destroy the hosting provider and leak the database.
When we examine the smaller databases, we find them to be populated with blogs, small political forums and personal sites - whatever the main motive behind the attack, it is clear that it was not without collateral.
The Shape of the OnionSphere Today
Without venturing into guesswork, we can say that 2017 has not been a good year to be operating a hidden service.
After the downfall of Freedom Hosting II we undertook the largest OnionScan crawl to date and examined over 30,000 onion services extracted from both the freedom hosting database and our existing master list.
Of the 30,000 queried just over 4,400 were online when we scanned (as always the scans were conducted across a number of days in an attempt to make the set as large as possible). We believe that many of the others in our list are now defunct.
These 4,400 hidden services are far fewer than previous scans. We believe that the Freedom Hosting II takedown not only removed many thousands of active sites but also may have affected other hosting providers who were hosting some infrastructure on top of Freedom Hosting II.
The sudden disappearance of Sigaint, an encrypted email provider, may also be associated with the decline of some hidden services.
Without venturing into guesswork, we can say that 2017 has not been a good year to be operating a hidden service.
Overall we present the following numbers regarding the makeup of the onion sphere in March 2017.
- HTTP Detected - ~4000
- TLS Detected - ~250 (In line with previous counts - seemingly unaffected by FHII)
- SSH Detected - ~270 (much lower, mostly due to the FHII hack)
- FTP Detected - < 10 (much lower, again expected to be related to FHII)
- SMTP Detected - < 100
- VNC Detected - < 10
- Bitcoin Nodes Detected - ~220 (much higher, likely because of better bitcoin capability in OnionScan)
Since our initial explorations back in April we can see that web sites still dominate the landscape. SSH endpoints are down substantially mostly linked to the downfall of Freedom Hosting II.
Otherwise not much as changed. Even in terms of security the state remains much the same as it was a year ago.
Apache mod_status Exposures still site between 7-10% of sites (we detected just over 400 in our recent scans) - this might even represent a slight increase in the number of misconfigured Apache installs.
Other kinds of vulnerabilities (open directories, EXIF metdata, host header co-hosting leaks) are also within the levels that we described over the last year.
We were able to extract nearly a thousand unique IP addresses from our data set belonging to both services and clearnet clients accessing misconfigured hidden services.
A New Map
When we examine larger scale correlations we still see that the vast majority of dark web sites can be connected to one or more clusters based on outbound/inbound links, bitcoin addresses and/or other types of identifiers.
Truth be told, it is simply far too easy to correlate hidden services to each other. This is because it is far too easy to misconfigure software or otherwise leak information.
Operational patterns appear across our new map, as can been seen in the images below there are several clusters of hosted nodes that are connected through Apache mod_status leaks - only a few of these sites are ever linked to outside of the cluster.
The fact that we can find multiple examples of this pattern likely indicates that, as we discussed back in our first report, that most hidden services are likely operated by a small collection of organizations and people rather than by individuals.
This is possibly driven by a few factors - the difficulty of setting up and maintaining a website, combined with the skills required to run a Tor hidden service make offloading that work to a 3rd party tempting.
However as seen with Freedom Hosting, and the other leaks we have demonstrated, these relationships creates additional security risks - and may in the end completely compromise any anonymity or privacy.
In order to truly solve the problem of usable, anonymous publishing we need a new approach.
The Future of OnionScan
We have always states that the end goal for OnionScan was to "make OnionScan not work" - by that we meant that we want to build a world where anonymity applications do not/cannot leak the kinds of information that OnionScan uses to draw correlations.
To achieve this goal we must move away from a pure OnionScan development and start investing in new technologies.
Over the course of the next few months we will be experimenting with and releasing prototypes, and papers wit the aim of coalescing on a new direction for anonymous publishing.
The end goal: a tool that can be used by anyone that will host a hidden service securely, privately and anonymously. A tool that will empower and aid the user in making decisions that protect themselves and others.
This doesn't mean that OnionScan is going away.
OnionScan will still be under active development - we plan to release 0.3 later on this year which will make scanning more efficient, and introduce more in-depth scanning of various protocols.
OnionScan will also feature in many of the new projects we build - we believe that active scanning on anonymity applications is vital to maintain their security - to that end we believe OnionScan, or a derivative, can be used to continually monitor for potential deanonymization vectors.
So, while this may be the last in-depth OnionScan Report we produce for a while - this is not the end of OnionScan - and will certainly not be our last report.
This research could not exist without our supporters, and we hope we can continue to deliver new insights, research and technology in the future. To help us do so, please support us
