In case anyone needs a all around good IP api with excellent threat intelligence that is not that expensive as IPinfo is, I suggest to use https://ipapi.is/
https://ipapi.is/ is not as good in regards to geolocation, but our hosting detection is more advanced and https://ipapi.is/ is much cheaper...
Hey, I am happy to see you here! What do you think of the event? I think you might find it interesting how these IPs were submitted. I would love to hear your thoughts.
In our previous interaction, I mentioned that I learned about CGNAT-based web scraping from your article on your personal blog. In this event, people took it to a whole other level. People were churning carrier IPs by the hundreds to win a pair of socks.
I am not sure if I would say that your hosting detection is more advanced than ours because we have already productized every aspect discussed in your steps.
Step 1: "Download a List of Top 1 Million Domain Names" → 10 million domain names available for free download here: https://host.io/rankings
Step 6: Crawl the Website and Classify the Website Text → https://host.io
This is just scratching the surface; a lot is happening internally. We have a team of talented engineers and a decade of experience. I am not trying to be disrespectful; I just wanted to share the information, that's all.
Our pricing approach is that we have an uncompromising approach to IP data, and organizations pay for the quality. However, for developers, small businesses, and students, we provide a generous yet highly accurate database and API for geolocation, which is free to access. We offer 50k reqs/month, tokenless API access, and a free IP to Country ASN database.
In general ipinfo.io has the best data and the best product out there in the IP API niche. It is what it is. Your data is the most accurate and most updated.
Hosting detection is a finite process, meaning that there is a finite amount of hosting providers out there to detect after all. Challenge lies in staying up to date.
So maybe we can make a compromise and state that ipinfo.io is likely a bit better than ipapi.is in hosting detection.
But ipapi.is for sure detects some hosting provider's that you don't. Examples:
Thank you. I hope you don't think I am pushing back or being sarcastic in any way because I personally like your work and really appreciate your comment. This is a typical conversation in a very "HN" way. Sometimes, it's hard to determine tone in written communication.
I appreciate that you make a wide variety of data accessible, and that has a positive impact. Our goal is to be the most accurate IP data provider, with geolocation being our foremost priority. It requires a "significant" amount of effort to simply be better than the rest of the industry. If customers want that bump in quality they will pay for our services. The ProbeNet has about 700 servers now, and nobody is investing in that level of infrastructure for IP data
accuracy.
The whole game for us is accuracy. Consider our free IP to Country database. It is so accurate that it goes down to individual IP address (`/32`) levels for country-level locations. This high level of accuracy, however, results in a bigger file size. Some developers compromise on accuracy over file size. While rounding up the ranges to `/24` and updating them twice a week would reduce cost, it does not align with our philosophy.
> ipapi.is for sure detects some hosting provider's that you don't
For the last one, it is not associated with an ASN, so the company field can be used as a proxy. As that company's IP data is a bit vague labeling it as hosting or business is difficult.
We have four types of "types": business, education, hosting, and ISP. This applies to both ASN and companies/organizations. For us, it is not simply a binary classification of hosting and non-hosting. Instead, we categorize IP ranges based on a statistical model with assigned weights. These ranges are then aggregated to determine AS types. When our customers say why is that this or that, you can show them the underlying reasoning.
But still, I grok and understand you. I really appreciate your feedback.
But I picked just one faulty classification of ipinfo.io, that's not fair, I know. I only wanted to point out that what you are doing is exactly the same as https://ipapi.is/ is doing and that we both make mistakes
----
You are using the 700 measuring servers to interpolate geolocations of IP addresses, right?
That works sometimes, but more than often it does not. It does not scale either.
Active latency triangulation of every IPv4 address (let's not even speak about IPv6) is simply not possible. The reasons are manyfold:
- Most hosts don't reply to ICMP
- Many routers block ICMP traffic, or they throttle / downgrade it, thus skewing measurements
- Traffic from your probing servers is probably not handled in the same way as is normal residential ISP traffic
- You have to constantly measure all IPv4, since IPs are constantly reassigned, which is simply not possible with only 700 servers
Latency triangulation works in theory, but in practice it is just not applicable to the full IP space.
Having said that, active geolocation with probing servers is still better than not doing it :D
Latency triangulation works much better in a passive way, meaning that a client is visiting a server that is under your control and you triangulate the client with JS for example (web sockets).
But I doubt that ipinfo.io has a significant share of the Internet's traffic...
You're not missing anything - those are all real problems! We've done a lot of work to overcome many of them, and others are active areas of research and development for us.
We do scan and traceroute all IPv4 and (known) IPv6 space, once a week, so our measurement data can be out of date by at most a week. We have other signals that an IP might have moved within that timeframe though.
We definitely don't have a significant share of total internet traffic - but we do get 6BN API requests a day.
https://ipapi.is/ is not as good in regards to geolocation, but our hosting detection is more advanced and https://ipapi.is/ is much cheaper...