Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discussion: New Format #743

Open
CameronMunroe opened this issue Dec 12, 2024 · 15 comments
Open

Discussion: New Format #743

CameronMunroe opened this issue Dec 12, 2024 · 15 comments
Assignees
@Munzy
Labels
enhancement New feature or request

Comments

@CameronMunroe
Copy link
Collaborator

Just an open discussion...

Initially, this project was built to identify likely sources of "hosting" traffic to help with blocking and detecting possibly SPAM.

Over the past few weeks and months, I've wanted to make this more of an open source identification of AS Networks in general.

I think we will leave this repo intact, and maybe create a new one with the new ideals in mind.

Any thoughts on this?
@CameronMunroe CameronMunroe added the ip data IP Data issues, or improvements. label Dec 12, 2024
@CameronMunroe CameronMunroe added enhancement New feature or request and removed ip data IP Data issues, or improvements. labels Dec 12, 2024
@finnbear
Copy link
Contributor

finnbear commented Dec 12, 2024 •
edited
Loading

For better precision & accuracy, perhaps it should have more granularity than this repository:

  • If AS0 contains AS1, the format should support identifying AS0 as one type (applies to IP's in AS0 except those in AS1) and AS1 as another type (applies to all IP's in AS1)
    • Would need a whitelist to do this in the current format
  • If AS0 contains many IP ranges, such as some hosting and some consumer, the format should support identifying them appropriately.

Related: #742

@CameronMunroe
Copy link
Collaborator Author

For better precision & accuracy, perhaps it should have more granularity than this repository:

* If AS0 contains AS1, the format should support identifying AS0 as one type (applies to IP's in AS0 except those in AS1) and AS1 as another type (applies to all IP's in AS1)
  
  * Would need a whitelist to do this in the current format

* If AS0 contains many IP ranges, such as some hosting and some consumer, the format should support identifying them appropriately.

Related: #742

Already do this regarding blackbox the service.

We based it on the most specific announcement. For example, 8.8.8.8 is announced by Level 3 and Google. Since Google has the most specific announcement we base it upon them.

@finnbear

This comment was marked as off-topic.

Sign in to view
@CameronMunroe
Copy link
Collaborator Author

Yes that service.

In the backend we use https://ip2asn.ipinfo.app/api/v1/8.8.8.8 and select the 0 record. This selects the most specific announcement, which is the default used for routing.

For the actual blackbox service, I don't have any published or downloadable feeds.

@finnbear

This comment was marked as off-topic.

Sign in to view
@abdullahdevrel
Copy link

I work for IPinfo. Would using our free IP to Country ASN database work for this purpose? I understand that for ASNs like Cogent, which provide connectivity to many companies but do not actually operate an ASN database, it would not be very effective in identifying them.

However, generally speaking, if you add another metadata layer which is the country level data per ASN, you can make the hosting range declaration more granular. Let me know what you think.

@CameronMunroe
Copy link
Collaborator Author

I work for IPinfo. Would using our free IP to Country ASN database work for this purpose? I understand that for ASNs like Cogent, which provide connectivity to many companies but do not actually operate an ASN database, it would not be very effective in identifying them.

However, generally speaking, if you add another metadata layer which is the country level data per ASN, you can make the hosting range declaration more granular. Let me know what you think.

I assume this is for personal use only, and not for paid products. Correct?

@abdullahdevrel
Copy link

@CameronMunroe No, the free database is licensed under CC-BY-SA 4.0, which is a commercially permissive license that only requires public attribution, that's all.

@finnbear
Copy link
Contributor

finnbear commented Dec 13, 2024 •
edited
Loading

identification of AS Networks

Depending on what is meant by "identification," these GitHub repositories might also be useful. Their license is more permissive.

(I use these for mapping this ASN list to an IP list)

if you add another metadata layer which is the country level data per ASN, you can make the hosting range declaration more granular.

@abdullahdevrel Are you referring to the fact that there are multiple entries in that database per ASN?

All entries that I could find have the same ASN name and domain name so, while more granular, any blocking decision would have to be based on country, which doesn't seem as useful as attributing IP ranges to a Cogent customer name.

Some entries for AS174 from the IP Info IP to Country and ASN database CSV 
149.14.124.106,149.14.124.106,SA,Saudi Arabia,AS,Asia,AS174,Cogent Communications,cogentco.com
205.160.110.103,205.160.110.103,SA,Saudi Arabia,AS,Asia,AS174,Cogent Communications,cogentco.com
154.54.75.80,154.54.75.87,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
154.54.61.238,154.54.61.238,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
2001:978:3::1a0,2001:978:3::1a7,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
130.117.2.93,130.117.2.94,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
154.45.64.0,154.45.64.255,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
154.54.37.37,154.54.37.37,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
149.11.82.120,149.11.82.255,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
149.11.82.0,149.11.82.111,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
2001:550:0:1000::9a36:385a,2001:550:0:1000::9a36:385a,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
154.54.21.189,154.54.21.189,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
130.117.51.107,130.117.51.136,SE,Sweden,EU,Europe,AS174,Cogent Communications,cogentco.com
120.138.35.0,120.138.35.255,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.222.38.9,203.222.38.31,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2402:4480:30e::,2402:4480:30e:ffff:ffff:ffff:ffff:ffff,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2001:550:0:1000::9a36:57d1,2001:550:0:1000::9a36:57d1,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
154.54.192.0,154.54.192.255,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.222.63.85,203.222.63.85,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.222.160.24,203.222.160.25,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
154.24.30.44,154.24.30.47,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2001:550:0:1000::9a18:4b81,2001:550:0:1000::9a18:4b82,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2001:550:0:1000::9a1a:4016,2001:550:0:1000::9a1a:4018,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2600:0:1:1239:203:222:40:9,2600:0:1:1239:203:222:40:9,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.98.195.0,203.98.195.255,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
154.18.187.0,154.18.187.255,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
154.54.140.15,154.54.140.19,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2402:4480:2:2::,2402:4480:2:2::e:ffff,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.222.38.99,203.222.38.101,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
154.18.2.120,154.18.2.191,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
2402:4480:2:2::d6:0,2402:4480:2:2::e0:ffff,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com
203.222.160.223,203.222.160.223,SG,Singapore,AS,Asia,AS174,Cogent Communications,cogentco.com

However, your website does offer the customer names of Cogent IP ranges:

Some entries for ASN174 from IPinfo website 
Netblock	Company	Num of IPs
[100.43.22.0/23](https://ipinfo.io/AS174/100.43.22.0/23)	APOG-HOWARD	512
[102.129.145.0/24](https://ipinfo.io/AS174/102.129.145.0/24)	Internet Utilities Africa (PTY) LTD	256
[102.129.152.0/24](https://ipinfo.io/AS174/102.129.152.0/24)	Internet Utilities Africa (PTY) LTD	256
[102.129.153.0/24](https://ipinfo.io/AS174/102.129.153.0/24)	Internet Utilities Africa (PTY) LTD	256
[102.129.232.0/24](https://ipinfo.io/AS174/102.129.232.0/24)	Digital Energy Technologies Limited	256
[102.129.234.0/24](https://ipinfo.io/AS174/102.129.234.0/24)	Digital Energy Technologies Limited	256
[102.129.235.0/24](https://ipinfo.io/AS174/102.129.235.0/24)	Digital Energy Technologies Limited	256
[102.129.252.0/24](https://ipinfo.io/AS174/102.129.252.0/24)	Digital Energy Technologies Limited	256
[102.165.16.0/24](https://ipinfo.io/AS174/102.165.16.0/24)	Digital-Energy-Technologies-LTD	256
[102.165.48.0/24](https://ipinfo.io/AS174/102.165.48.0/24)	Internet Utilities Africa (PTY) LTD	256

@abdullahdevrel
Copy link

@finnbear I did not want to pitch a paid service to an OSS project, but yes, we have a paid product called IP to Company API that provides exactly what you want.

The company data is publicly available through our website, so I would say to use the free ASN + Country database we have for the first layer of data addition. Then, through community contribution, you can use our site as a reference to get granular range information like here: #742

In terms of the quality of the free database we have, it is seriously unmatched by anything out there. Our geolocation database is based on active measurement and, on the other hand, the ASN database is produced through a combination of WHOIS and BGP peering data. It is also updated daily and provides full accuracy.

@finnbear
Copy link
Contributor

finnbear commented Dec 13, 2024 •
edited
Loading

@abdullahdevrel thanks!

I have a question: In your opinion, if an open source project like the one proposed here used your CC-BY-SA 4.0 licensed data, would you expect all users of that project be to acknowledge IPinfo? Or just acknowledge that project?

I ask because your website says "Attribution Required" and prominently lists adding a link to IP info on a project website or social media as a means of attribution.

However, creative commons lists GPLv3 as a compatible license to BY-SA.

AFAIK, only AGPLv3 (but not regular GPLv3) would prevent unattributed use in a web-server (that only makes decisions based on IP data, and doesn't redistribute it).

@abdullahdevrel
Copy link

Just project itself. Just adding a link to the README.md is enough.

Example:

image

image

I really do not know much about comparable licensing. Sorry. With our license, just pasting a link in the README.md is enough.

@finnbear
Copy link
Contributor

Okay, cool. I appreciate that, as a user, I would only have to worry about one license, i.e. the proposed project's, and not IPinfo too :)

@abdullahdevrel
Copy link

My friend, just writing and maintaining code is difficult enough, and then adding licensing on top of it is a nightmare. We thought we would not let any OSS maintainers worry about licensing when it comes to using our free database.

We looked around to see what the most popular open-source license is, which apparently is CC-BY-SA as it is used by Stack Overflow and Wikipedia. So, we went with that.

@CameronMunroe
Copy link
Collaborator Author

Thinking i might do something like

Inputs > Hosting > 1.md (Evidence inside of file)
Inputs > ISP  > 3.md (Evidence inside of file)

and then have an output directory where they combined into a nice list.

Current thoughts are we would have:

  • ISP (AT&T)
  • Mobile (T-Mobile, Verizon)
  • Hosting (ColoCrossing)
  • Transit (Cogent)
  • Cloud (AWS)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Munzy Munzy

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Munzy @finnbear @CameronMunroe @abdullahdevrel