A while ago, I decided to test a new analog of Shodan — the Netlas.io platform only to find out that it is quite useful, and deserves a dedicated article. So, let’s go!
Usage
The search interface of Netlas is available after registration and authorization only. As I understand, the search system won’t be free in the future, but now it is on an alpha-testing stage, so one can use Netlas almost without strict restrictions.
It is worth noting that for each returned result you spend one NetlasCoin (you get 1000 for the registration). You can press the “Renew coins” button on your profile page to refill your coins to the maximum amount again.
Comparison with others
I think there is no need to explain why any IPv4 scan datasets are useful. Like in the case with DB leaks from some service — you can find something that crawlers of another search engine lost due to timeout or other network issues, or maybe missed intentionally.
That’s why I began by making a comparison of results for Netlas and other similar platforms. My goal was to understand how they differ in coverage, functionality, and usefulness for OSINT. Additionally, of course, to check how useful Netlas can be in general!
Total results number
Firstly, let’s check the official numbers for scan results. It’s obvious that often such information is only marketing bait, but it also gives us clues to understanding the profile of a search engine.
Unfortunately, there is no general standard for measuring the quality of such search engines. But if you think about it, you realize that there are no general-purpose platforms — some have the focus on IoT devices, some mostly check websites and their vulnerabilities, some perform scans for specific common ports.
And also there is a need to make constant scans round-the-clock because many hosts die (port closed, IP change), and all the gathered data become unactual within a day! Do you imagine the complexity of making a universal port scan system for such conditions?
So, how can we measure the quality of port scan results? Well, let’s dive into publicly available data:
I have found numbers useful for comparison only for Censys (rounded) and for ZoomEye. As you see, Shodan is not represented in the table: yeah, it is too good to publish its statistics, as well as some other platforms, such as BinaryEdge and Fofa.
Also, I should note that I didn’t try to make a comparison with archives of scan data, such as Opendata (Sonar). It would be excellent to compare the completeness of any scanner results with a certain known bulk of port scan data, but it is too enterprising and academic compared to my goal.
So, what are the conclusions from the gathered information?
- Netlas is focused on domains, SSL/TLS certificates, and HTTP responses (as I suppose, from any service capable to answer with a text banner).
- (Netlas domains) = 1/2 * (ZoomEye web pages). We see numbers of one order, that’s okay, looks plausible.
- (Netlas responses) = 2.5 * (Censys hosts) = 1/3 * (Censys services). Netlas has more responses than hosts Censys has hosts, but also it has a lesser number of scanned devices. Maybe, Netlas scans a smaller set of ports.
- (Netlas certificates) = 1.5 * (Netlas domains). It is really interesting, why are there so many certificates? Let’s explore this further.
Filters usage
Of course, general statistics are not enough, so let’s evaluate quality by more understandable criteria. For example, by a number of search results with certain filters.
Netlas developers understand that filters are important and suggest query examples with filters for Mongo, Confluence, Lync/Skype for Bussiness, vBulletin forum engines straight on the search page.
Well, let’s use them. Firstly, we’ll search MongoDB in several search engines.
Are you surprised that Shodan has the smallest number of results? It is really easy to explain: the oldest and best-known network search engine is also known for removing all the inconvenient results (of course, it is also inconvenient to publish general statistics then).
Censys has a little more than Shodan and Netlas. I think it is a sign of quality that examined by Netlas database is comparable to a database of the platform with the most transparent results.
But why do we have 3-5 times more results in Chinese platforms like Fofa and ZoomEye? Let’s check.
If you look attentively into their search interface, you’ll find the answer. There are filters by years and percents for each of them. As you see, their databases contain a lot of old and dead hosts/ports. On the one hand, it may be useful for research purposes. On the other hand, it raises doubts about the quality of the latest results.
Netlas also has a scans menu in the right upper corner of the interface. Well, not bad, at least we understand what was scanned and when. But I honestly tried a lot to find it! :)
Certainly, it will not be correct to compare the quality of databases with one MongoDB filter. That’s why I continued searching with other filters. But we have already spent a lot of time on comparisons, so find the results in the article appendix while we continue to analyze Netlas’ specific features.
And, finally, take a look at the full search web interface of Netlas:
Netlas features useful for OSINT
Certificates
As I mentioned before, Netlas can gather not only text banners from ports but also TLS certificates.
Some time ago I released a document about email intelligence. I took crt.sh as an example of a platform collecting information about new certificates from certificate transparency logs. As we see in Netlas documentation (login required) they do the same thing, but also save certificates from scanned hosts.
How it may be useful? If you search the real IP of a site behind Cloudflare, you often can find a typical server misconfiguration: site respond with certificate not only for a request to a web page by domain name (https://example.com) but also for a request to a web page by IP (https://1.2.3.4/).
And of course, a complete network scan of 80/443 ports will find such IP and will save information about a site certificate. Thus, it will be enough to make a Netlas search just by domain name to find its original IP.
Unlike Censys, which I used before for such purposes, Netlas also collect domain from the CZDS registry, so it is guaranteed that you will see both answers from a domain name and IP on the same results page, that’s very handy.
Contacts
Netlas have separate crawlers for site contacts, and filters for search by them. Looks like this information is being parsed from pages’ footers: phone numbers, physical addresses, geo positions.
I’ve made a screenshot with an example of search by contact and statistics screen view also. You can sort data by various params and explore it in a table view, unlike the default view with raw answers.
Domains and IP whois
As the documentation says, Netlas gather information about domains naturally from anywhere: starting with CZDS and ending with trivial HTTP redirects. It would not have been able to do a high-quality search without it.
The situation with whois is simpler — Netlas has a “WHOIS” info block for each IP (not for the site). Unfortunately, contacts from standard fields are removed, but you can find mentions of owner email/name in other fields.
Surely it would be cool to see full personal data here as in platforms like bgp.he.net, but this information is greatly cherished in our time. And that’s right. :)
Probably, you’ve got thoughts that it is a very good idea to automate the search by contacts in Netlas. Yeah, I agree, and this is not a hard job, cause we can use API documentation and Python SDK.
I’ve tried a little and made a simple tool for such search by email — netlas-email-search. I’ve used the following API requests:
- Certificates search
- Contacts search
- IP whois information search
- Gathered text banners search
# ./netlas-email-search.py root@localhost Whois search did't return results SSL search can return 49710 results Downloading 20 results... Downloaded first 20 results only. You can get all the results manually with a query <certificate.subject.email_address:root@localhost> Contacts search can return 6 results Downloading 6 results... Webpage search can return 288451 results Downloading 20 results... Downloaded first 20 results only. You can get all the results manually with a query <http.body:root@localhost> Downloaded data was saved to following files: SSL_root@localhost_results.json Contacts_root@localhost_results.json Webpage_root@localhost_results.json
The tool downloads the first 20 results and makes tips about search continuation — I believe it’s a good help for a quick start into Netlas search.
Other features
As you saw the “CVE” info block for each site on the screenshot above. They are being mapped by software versions in time of scan and require additional verification. However, Netlas also propose links to exploits right on the results page. :)
Well, but which interesting features also does Netlas have?
- Favicon search. Not an exclusive feature, but it allows to search by a hash and through a downloaded image.
- Tags. Currently, about 900 tags by devices, services, and software versions are supported.
- DNS-records search. You can find all the servers that use certain mail providers. Or, for example, all the TXT records with incorrect SPF string.
- Redirects. Netlas follows all the redirects, saving their results. It is very useful for fingerprinting of web software that respond with an almost empty page for request to
/, but then leads to a “human” login page with valuable information.
Conclusion
The list of network scanners has been updated, and it’s good. There are interesting and promising features in Netlas, and in terms of quality of results, it can compete with Shodan and almost catches up with Censys.
It is already possible to write scripts with Netlas API usage, and you can also use practically infinite limits for the time of alpha testing, not only for OSINT purposes, as we discussed above.
Do you know other interesting features? Do you disagree with my findings? Do not hesitate to write me, I will be glad to receive feedback! You can also check out my Github profile.
Appendix. Search results statistics for different scanners
--------------------------------------------------------------------- MongoDB --------------------------------------------------------------------- Netlas | monogdb:* | 75,502 Shodan | product:"MongoDB" | 65,484 Fofa | app="MongoDB-数据库" | 297,561 ZoomEye | app:"MongoDB" | 631,220 Censys | services.mongodb.build_info.version:* | 115,033 BinaryEdge | type:mongodb | 119,291 --------------------------------------------------------------------- Elasticsearch --------------------------------------------------------------------- Netlas | elasticsearch.elastic_search_main:* | 35,418 Shodan | product:"Elastic" | 21,855 Fofa | app="elastic-Elasticsearch" | 84,586 ZoomEye | service:"elasticsearch" | 48,496 Censys | services.elasticsearch.system_info.name:* | 35,601 BinaryEdge | type:elasticsearch | 34,595 --------------------------------------------------------------------- PostgreSQL --------------------------------------------------------------------- Netlas | postgres.is_ssl:* | 576,105 Shodan | product:"PostgreSQL" | 655,169 Fofa | app="PostgreSQL" | 1,194,013 ZoomEye | app:"PostgreSQL DB" | 2,543,351 Censys | services.postgres.protocol_error.code:* | 842,725 BinaryEdge | product:"PostgreSQL DB" | 903,669 --------------------------------------------------------------------- MySQL --------------------------------------------------------------------- Netlas | mysql:* | 3,214,687 Shodan | product:"MySQL" | 2,992,729 Fofa | protocol="mysql" | 7,727,585 ZoomEye | app:"MySQL" | 89,734,244 Censys | services.mysql.server_version:* | 2,489,613 BinaryEdge | product:mysql | 4,853,960 --------------------------------------------------------------------- Redis --------------------------------------------------------------------- Netlas | redis:* | 99,698 Shodan | product:"Redis key-value store" | 26,011 Fofa | protocol="redis" | 328,167 ZoomEye | app:"Redis key-value store" | 1,121,815 Censys | services.redis.os:* | 39,310 BinaryEdge | product:redis | 196,838
