I've seen some posts about differences between the different OSINT search engines that scan the internet (eg. Shodan, Zoomeye, Censys, etc).
Basically, most of them talk about the differences in port coverage or detected technologies, which is definitely an important element to think about.
But nobody talks about the differences in the quality of the results or the ease of use in these systems.
There are a lot of services and types of services to cover when scanning the entire internet, but in my opinion, there are really not that much that can be abused to gain profit from.
Most apps that are valuable and relevant, use the same ports:
Web apps mostly use 80, 443, 8080, 8443, 3443, 8000
Mail services use 110, 143, 993, 995, 25, 587, 465
VPN services use 1194, 443, 500, 1723
FTP uses 20 and 21
SSH uses 22
DNS use 53
SQL Servers ports 1433, 3306, 5432, 27017, 27018, 1521, 6379
Windows services (that are sometimes open to the internet) mostly are 445, 137, 139, 135, 3389
There are also lots of other programs with default ports of their own (like cPanel, some AV servers, and SCADA devices).
But these are the general ports that are relevant (imo) in the context of cyber security and are not just random SIP or IOT devices.
ALL OF THESE LISTED PORTS are being scanned by Shodan, Zoomeye, Fofa, Criminal IP, and Censys.
So what's the difference between them???
The frequencies
Some of them scan a specific port once a day, some scan it once a week, and some may scan it somewhere in between.
It really differs depending on the specified port.
I can't put my finger myself on the frequencies regarding which service scans which port at what frequencies - and I'm sure it changes a lot over time.
You can refer to "A Survey on Cyberspace Search Engines" - https://link.springer.com/chapter/10.1007/978-981-33-4922-3_15 for more details about the frequencies.
The query language
So FOFA, Zoomeye & Censys have their own query language where you can use AND, OR and NOT (in some cases).
Shodan and Criminal IP do not. It really affects the quality of life when using these kinds of systems.
If for example, I want to perform a scan of a technology based on a couple of parameters: favicon and Title.
To find all services that use the favicon or the title - In Shodan\Criminal IP I'll have to use 2 queries (that most likely will cost me more quota than I need because of duplicates), And in Zoomeye\Fofa\Censys I can use 1 query to find the services and without spending double the quota on duplicates.
It is also way more comfortable to use 1 query instead of multiple.
The API and general use
Censys recently updated their license program. In most search engines (Zoomeye, Fofa, Shodan, Criminal IP) you have a quota that you can use however you like.
But in Censys - you can't. You can get a maximum of 5,000 results per search (for the 500$ monthly plan).
Kind of strange but it's a problem we need to address regarding this search engine.
Criminal IP is shit
It's shit.
I hate it.
It has some weird bugs here and there and I hate to use it.
The UNKNOWN factor (Firewall DROPs)
So far we discussed the technical details of the scaners\search engines.
There is one last element to be addressed and it's regarding the amount of results we get for a specific service\port.
Let's take HTTP\80 in the United States as an example:
Fofa has 105M unique IPs
Zoomeye has 60M unique IPs
Censys has 53M unique IPs
Shodan has 42M unique IPs
Criminal IP has 20M unique IPs
The reason for the dramatic difference is because of FW DROPS. When you scan the entire internet, it's easy to get blocked by lots of FWs and ISPs.
It can be a really difficult thing to maintain a network of scan devices that won't get blocked all the time.
Some scanners\search engines are performing better than others, but the most important thing to note is that it's always beneficial to use multiple scanners\search engines to get the most results.
Some targets I found on Shodan that haven't been found on any of the other scanners.
And I can say the same about them all, each of them can find devices\services that the others didn't manage to access because of this issue (Even Criminal IP which is a piece of shit).
Tools and utils
https://transl8.watchdawg.io/ - is a very useful site to help you write queries to the different search engines according to their syntax.
https://github.com/cipher387/awesome-ip-search-engines - contains a lot of useful info in it (articles, cheatsheets, Tools, etc)
Hope I helped you understand this world a little better.
If you have better ways to scan the internet, or some general thoughts regarding this topic, feel free to discuss them in the comments or contact me in pm.
I'm looking for a business partner to scan the internet with me, if you have more results than the search engines, contact me in pm.
Basically, most of them talk about the differences in port coverage or detected technologies, which is definitely an important element to think about.
But nobody talks about the differences in the quality of the results or the ease of use in these systems.
There are a lot of services and types of services to cover when scanning the entire internet, but in my opinion, there are really not that much that can be abused to gain profit from.
Most apps that are valuable and relevant, use the same ports:
Web apps mostly use 80, 443, 8080, 8443, 3443, 8000
Mail services use 110, 143, 993, 995, 25, 587, 465
VPN services use 1194, 443, 500, 1723
FTP uses 20 and 21
SSH uses 22
DNS use 53
SQL Servers ports 1433, 3306, 5432, 27017, 27018, 1521, 6379
Windows services (that are sometimes open to the internet) mostly are 445, 137, 139, 135, 3389
There are also lots of other programs with default ports of their own (like cPanel, some AV servers, and SCADA devices).
But these are the general ports that are relevant (imo) in the context of cyber security and are not just random SIP or IOT devices.
ALL OF THESE LISTED PORTS are being scanned by Shodan, Zoomeye, Fofa, Criminal IP, and Censys.
So what's the difference between them???
The frequencies
Some of them scan a specific port once a day, some scan it once a week, and some may scan it somewhere in between.
It really differs depending on the specified port.
I can't put my finger myself on the frequencies regarding which service scans which port at what frequencies - and I'm sure it changes a lot over time.
You can refer to "A Survey on Cyberspace Search Engines" - https://link.springer.com/chapter/10.1007/978-981-33-4922-3_15 for more details about the frequencies.
The query language
So FOFA, Zoomeye & Censys have their own query language where you can use AND, OR and NOT (in some cases).
Shodan and Criminal IP do not. It really affects the quality of life when using these kinds of systems.
If for example, I want to perform a scan of a technology based on a couple of parameters: favicon and Title.
To find all services that use the favicon or the title - In Shodan\Criminal IP I'll have to use 2 queries (that most likely will cost me more quota than I need because of duplicates), And in Zoomeye\Fofa\Censys I can use 1 query to find the services and without spending double the quota on duplicates.
It is also way more comfortable to use 1 query instead of multiple.
The API and general use
Censys recently updated their license program. In most search engines (Zoomeye, Fofa, Shodan, Criminal IP) you have a quota that you can use however you like.
But in Censys - you can't. You can get a maximum of 5,000 results per search (for the 500$ monthly plan).
Kind of strange but it's a problem we need to address regarding this search engine.
Criminal IP is shit
It's shit.
I hate it.
It has some weird bugs here and there and I hate to use it.
The UNKNOWN factor (Firewall DROPs)
So far we discussed the technical details of the scaners\search engines.
There is one last element to be addressed and it's regarding the amount of results we get for a specific service\port.
Let's take HTTP\80 in the United States as an example:
Fofa has 105M unique IPs
Zoomeye has 60M unique IPs
Censys has 53M unique IPs
Shodan has 42M unique IPs
Criminal IP has 20M unique IPs
The reason for the dramatic difference is because of FW DROPS. When you scan the entire internet, it's easy to get blocked by lots of FWs and ISPs.
It can be a really difficult thing to maintain a network of scan devices that won't get blocked all the time.
Some scanners\search engines are performing better than others, but the most important thing to note is that it's always beneficial to use multiple scanners\search engines to get the most results.
Some targets I found on Shodan that haven't been found on any of the other scanners.
And I can say the same about them all, each of them can find devices\services that the others didn't manage to access because of this issue (Even Criminal IP which is a piece of shit).
Tools and utils
https://transl8.watchdawg.io/ - is a very useful site to help you write queries to the different search engines according to their syntax.
https://github.com/cipher387/awesome-ip-search-engines - contains a lot of useful info in it (articles, cheatsheets, Tools, etc)
Hope I helped you understand this world a little better.
If you have better ways to scan the internet, or some general thoughts regarding this topic, feel free to discuss them in the comments or contact me in pm.
I'm looking for a business partner to scan the internet with me, if you have more results than the search engines, contact me in pm.
