A while ago, I decided to test a new analogue of Shodan — the Netlas.io platform only to find out that it is quite useful, and deserves a dedicated article. So, let’s go!
Usage
The search interface of Netlas is available after registration and authorization only. As I understand, the search system won’t be free in the future, but now it is in the alpha-testing stage, so one can use Netlas almost without strict restrictions.
It is worth noting that for each returned result you spend one NetlasCoin (you get 1000 for the registration). You can press the “Renew coins” button on your profile page to refill your coins to the maximum amount again.
Comparison with others
I think there is no need to explain why any IPv4 scan datasets are useful. Like in the case of DB leaks from some service — you can find something that crawlers of another search engine lost due to timeout or other network issues, or maybe missed intentionally.
That’s why I began by making a comparison of results for Netlas and other similar platforms. My goal was to understand how they differ in coverage, functionality, and usefulness for OSINT. Additionally, of course, to check how useful Netlas can be in general!
Total results number
Firstly, let’s check the official numbers for scan results. It’s obvious that often such information is only marketing bait, but it also gives us clues to understanding the profile of a search engine.
Unfortunately, there is no general standard for measuring the quality of such search engines. But if you think about it, you realize that there are no general-purpose platforms — some focus on IoT devices, some mostly check websites and their vulnerabilities, and some perform scans for specific common ports.
Also, there is a need to make constant scans round-the-clock because many hosts die (port closed, IP change), and all the gathered data becomes unactual within a day! Do you imagine the complexity of making a universal port scan system for such conditions?
So, how can we measure the quality of port scan results? Well, let’s dive into publicly available data:
I have found numbers useful for comparison only for Censys (rounded) and for ZoomEye. As you see, Shodan is not represented in the table: yeah, it is too good to publish its statistics, as well as some other platforms, such as BinaryEdge and Fofa.
Also, I should note that I didn’t try to make a comparison with archives of scan data, such as Opendata (Sonar). It would be excellent to compare the completeness of any scanner results with a certain known bulk of port scan data, but it is too enterprising and academic compared to my goal.
So, what are the conclusions from the gathered information?
- Netlas is focused on domains, SSL/TLS certificates, and HTTP responses (as I suppose, from any service capable of answering with a text banner).
- (Netlas domains) = 1/2 * (ZoomEye web pages). We see numbers of one order, that’s okay, looks plausible.
- (Netlas responses) = 2.5 * (Censys hosts) = 1/3 * (Censys services). Netlas has more responses than hosts Censys has hosts, but also it has a lesser number of scanned devices. Maybe, Netlas scans a smaller set of ports.
- (Netlas certificates) = 1.5 * (Netlas domains). It is really interesting, why are there so many certificates? Let’s explore this further.
Filters usage
Of course, general statistics are not enough, so let’s evaluate quality by more understandable criteria. For example, by several search results with certain filters.
Netlas developers understand that filters are important and suggest query examples with filters for Mongo, Confluence, Lync/Skype for Bussiness, and vBulletin forum engines straight on the search page.
Well, let’s use them. Firstly, we’ll search MongoDB in several search engines.
Are you surprised that Shodan has the smallest number of results? It is really easy to explain: the oldest and best-known network search engine is also known for removing all the inconvenient results (of course, it is also inconvenient to publish general statistics then).
Censys has a little more than Shodan and Netlas. I think it is a sign of quality that the examined by Netlas database is comparable to a database of the platform with the most transparent results.
But why do we have 3-5 times more results in Chinese platforms like Fofa and ZoomEye? Let’s check.
If you look attentively into their search interface, you’ll find the answer. There are filters by years and percents for each of them. As you see, their databases contain a lot of old and dead hosts/ports. On the one hand, it may be useful for research purposes. On the other hand, it raises doubts about the quality of the latest results.
Netlas also has a scan menu in the right upper corner of the interface. Well, not bad, at least we understand what was scanned and when. But I honestly tried a lot to find it! :)
Certainly, it will not be correct to compare the quality of databases with one MongoDB filter. That’s why I continued searching with other filters. But we have already spent a lot of time on comparisons, so find the results in the article appendix while we continue to analyze Netlas’ specific features.
And, finally, take a look at the full search web interface of Netlas:
Netlas features useful for OSINT
Certificates
As I mentioned before, Netlas can gather not only text banners from ports but also TLS certificates.
Some time ago I released a document about email intelligence. I took crt.sh as an example of a platform collecting information about new certificates from certificate transparency logs. As we see in Netlas documentation (login required) they do the same thing, but also save certificates from scanned hosts.
How it may be useful? If you search the real IP of a site behind Cloudflare, you often can find a typical server misconfiguration: the site respond with a certificate not only for a request to a web page by domain name (https://example.com) but also for a request to a web page by IP (https://1.2.3.4/).
And of course, a complete network scan of 80/443 ports will find such IP and will save information about a site certificate. Thus, it will be enough to make a Netlas search just by domain name to find its original IP.
Unlike Censys, which I used before for such purposes, Netlas also collect domain from the CZDS registry, so it is guaranteed that you will see both answers from a domain name and IP on the same results page, that’s very handy.
Contacts
Netlas have separate crawlers for site contacts and filters for search by them. Looks like this information is being parsed from pages’ footers: phone numbers, physical addresses, and geo positions.
I’ve made a screenshot with an example of a search by contact and statistics screen view. You can sort data by various parameters and explore it in a table view, unlike the default view with raw answers.
Domains and IP whois
As the documentation says, Netlas gather information about domains naturally from anywhere: starting with CZDS and ending with trivial HTTP redirects. It would not have been able to do a high-quality search without it.
The situation with WHOIS is simpler — Netlas has a “WHOIS” info block for each IP (not for the site). Unfortunately, contacts from standard fields are removed, but you can find mentions of the owner’s email/name in other fields.
Surely it would be cool to see full personal data here as in platforms like bgp.he.net, but this information is greatly cherished in our time. And that’s right. :)
Probably, you’ve thought that it is a very good idea to automate the search by contacts in Netlas. Yeah, I agree, and this is not a hard job, cause we can use API documentation and Python SDK.
I’ve tried a little and made a simple tool for such search by email — atlas-email-search. I’ve used the following API requests:
- Certificates search
- Contacts search
- IP whois information search
- Gathered text banners search
# ./netlas-email-search.py root@localhost
Whois search did't return results
SSL search can return 49710 results
Downloading 20 results...
Downloaded first 20 results only. You can get all the results manually with a query <certificate.subject.email_address:root@localhost>
Contacts search can return 6 results
Downloading 6 results...
Webpage search can return 288451 results
Downloading 20 results...
Downloaded first 20 results only. You can get all the results manually with a query <http.body:root@localhost>
Downloaded data was saved to following files:
SSL_root@localhost_results.json
Contacts_root@localhost_results.json
Webpage_root@localhost_results.jsonThe tool downloads the first 20 results and makes tips about search continuation — I believe it’s a good help for a quick start into Netlas search.
Other features
As you saw the “CVE” info block for each site on the screenshot above. They are being mapped by software versions at the time of scan and require additional verification. However, Netlas also propose links to exploits right on the results page. :)
Well, but which interesting features does Netlas have?
- Favicon search. Not an exclusive feature, but it allows you to search by a hash and through a downloaded image.
- Tags. Currently, about 900 tags by devices, services, and software versions are supported.
- DNS-records search. You can find all the servers that use certain mail providers. Or, for example, all the TXT records with incorrect SPF strings.
- Redirects. Netlas follows all the redirects, saving their results. It is very useful for fingerprinting web software that responds with an almost empty page for requests to
/, but then leads to a “human” login page with valuable information.
Conclusion
The list of network scanners has been updated, and it’s good. There are interesting and promising features in Netlas, and in terms of quality of results, it can compete with Shodan and almost catch up with Censys.
It is already possible to write scripts with Netlas API usage, and you can also use practically infinite limits for the time of alpha testing, not only for OSINT purposes, as we discussed above.
Do you know other interesting features? Do you disagree with my findings? Do not hesitate to write to me, I will be glad to receive feedback! You can also check out my Github profile.
Appendix. Search results statistics for different scanners
---------------------------------------------------------------------
MongoDB
---------------------------------------------------------------------
Netlas | monogdb:* | 75,502
Shodan | product:"MongoDB" | 65,484
Fofa | app="MongoDB-数据库" | 297,561
ZoomEye | app:"MongoDB" | 631,220
Censys | services.mongodb.build_info.version:* | 115,033
BinaryEdge | type:mongodb | 119,291
---------------------------------------------------------------------
Elasticsearch
---------------------------------------------------------------------
Netlas | elasticsearch.elastic_search_main:* | 35,418
Shodan | product:"Elastic" | 21,855
Fofa | app="elastic-Elasticsearch" | 84,586
ZoomEye | service:"elasticsearch" | 48,496
Censys | services.elasticsearch.system_info.name:* | 35,601
BinaryEdge | type:elasticsearch | 34,595
---------------------------------------------------------------------
PostgreSQL
---------------------------------------------------------------------
Netlas | postgres.is_ssl:* | 576,105
Shodan | product:"PostgreSQL" | 655,169
Fofa | app="PostgreSQL" | 1,194,013
ZoomEye | app:"PostgreSQL DB" | 2,543,351
Censys | services.postgres.protocol_error.code:* | 842,725
BinaryEdge | product:"PostgreSQL DB" | 903,669
---------------------------------------------------------------------
MySQL
---------------------------------------------------------------------
Netlas | mysql:* | 3,214,687
Shodan | product:"MySQL" | 2,992,729
Fofa | protocol="mysql" | 7,727,585
ZoomEye | app:"MySQL" | 89,734,244
Censys | services.mysql.server_version:* | 2,489,613
BinaryEdge | product:mysql | 4,853,960
---------------------------------------------------------------------
Redis
---------------------------------------------------------------------
Netlas | redis:* | 99,698
Shodan | product:"Redis key-value store" | 26,011
Fofa | protocol="redis" | 328,167
ZoomEye | app:"Redis key-value store" | 1,121,815
Censys | services.redis.os:* | 39,310
BinaryEdge | product:redis | 196,838