[Feature Request] Multiple IP list sources, list merging, and deduplication logic

[KD-MM2]

Description:
I would like to suggest a feature to improve the way geoip-shell fetches and manages IP lists by adding multi-source support and deduplication. Currently, geoip-shell defaults to RIPE and treats imported local lists and fetched lists independently.

Proposed Features:

1. Multiple Options for IP List Sources

• Instead of defaulting to RIPE, provide an interactive checklist or command-line flag during setup to select from various available IP list sources (e.g., RIPE, IPDeny, IPinfo, MaxMind).
• For sources that require authentication (like IPinfo or MaxMind), prompt the user to run a credential setup step before fetching.

2. Source Merging and Deduplication

• Allow fetching IP lists from multiple selected sources simultaneously.
• Implement a merging and deduplication process to combine the fetched lists into a single, optimized final source before passing it to the firewall backend.
  • Example: Fetch VN_ipv4 from RIPE and VN_ipv4 from IPDeny → merge and deduplicate overlapping CIDRs → output a unified VN_ipv4 list.
• Use this single optimized merged list for configuring the nftables or iptables ruleset, reducing the size of the IP sets in memory.

3. Deduplication of Online Lists with Local (Imported) Lists

• Extend the deduplication logic to cover user-imported local IP lists.
• When a user imports a local_allow_ipv4 or local_block_ipv4 file, the script should compare it against the fetched online lists (e.g., country lists) and deduplicate overlapping subnets or specific IPs to ensure the most efficient firewall rules possible, while respecting the precedence of blocklists over allowlists.

Use Case / Motivation:
This would allow users to rely on multiple upstream providers for better accuracy and redundancy without bloating nftables sets or ipset with duplicate entries. Deduplicating local lists against online lists will further optimize memory usage and firewall performance.

[friendly-bits]

Allow fetching IP lists from multiple selected sources simultaneously.

Are you talking about one setting which would configure all IP lists (e.g. you chose to geoblock DE and FR and fetch from RIPE and IPDENY, so IP lists for both countries are fetched from both sources and later combined), or are you talking about per-country source selection?

If it's the former, this should not be too difficult to implement, although frankly I'm not sure how useful this would be (at least RIPE and IPDENY IP lists are essentially the same, except IPDENY aggregates the IP ranges which makes the download smaller).

If it's the latter then this would require a lot of work and probably I wouldn't go there.

• Implement a merging and deduplication process to combine the fetched lists into a single, optimized final source before passing it to the firewall backend.

Basic deduplication is not very hard to implement. However basic deduplication wouldn't catch all duplicate IP addresses. For example, if one list includes the range 8.8.8.1/16 and another list includes the range 8.8.8.8/32 (which is of course contained in the first range) then string-based deduplication would not catch and eliminate the duplicate. This would require more clever processing, i.e. IP ranges aggregation. I'm not sure how to do this efficiently. I've seen a Python script by @dewdmadbro which appears to be doing such aggregation (among other things) but I haven't tested it, so I don't know how efficient it is. Perhaps the author could comment (not sure whether the above mention pings them or not). Regardless, ideally I would like to avoid adding Python as a dependency, so if this is possible to implement without it, that would be better. I wonder whether there is some clever way to achieve this with grepcidr.

Deduplication of Online Lists with Local (Imported) Lists

This is essentially same as above: string-based deduplication is easy to implement but it would miss IP ranges which are partially or completely contained in other ranges.

[KD-MM2]

Hi, thanks for the reply! To clarify:

1. Multiple IP list sources simultaneously

I am referring to the former (a global selection). The idea is to configure a global list of sources (e.g., picking both RIPE and MaxMind), and then the script fetches the IP lists for the selected countries from all chosen sources and combines them.

Why this is useful: While RIPE and IPDeny might be very similar, commercial or third-party databases like MaxMind or IPinfo often have different classifications or quicker updates for reassigned IPs. Combining a primary registry (RIPE) with a commercial provider (MaxMind) increases coverage and catches edge cases where one provider might miscategorize an IP.

2. Merging and CIDR Aggregation (Without Python)

You make a great point about string-based deduplication (sort -u) failing to catch overlapping subnets (e.g., 8.8.8.1/16 vs 8.8.8.8/32). You definitely don't need a heavy Python dependency to solve this! Here are two highly efficient ways to handle CIDR aggregation in shell environments:

Option A: Native Firewall Features (Zero Dependencies)
If the user is on nftables, you can offload the aggregation entirely to the kernel. nftables has an auto-merge feature specifically for this. If you define the sets with flags interval; auto-merge;, nftables will automatically merge overlapping ranges and eliminate subsets when loading the IPs into memory.

Option B: Lightweight C-based CLI tools (File-level)
If you need to optimize the files themselves before feeding them to the firewall (or to support iptables/ipset), there are tiny, lightning-fast C utilities built exactly for this:

• iprange: Used heavily by projects like FireHOL. It can instantly merge and optimize multiple lists. Usage is as simple as: iprange list1.txt list2.txt > merged_optimized.txt.
• aggregate: Another classic lightweight tool (by ISC) that takes a list of prefixes and outputs the shortest possible list of non-overlapping prefixes.

Using either iprange or relying on nftables auto-merge would solve the subset/overlap problem elegantly without touching Python or complex grepcidr loops.

[dewdmadbro]

Basic deduplication is not very hard to implement. However basic deduplication wouldn't catch all duplicate IP addresses. For example, if one list includes the range 8.8.8.1/16 and another list includes the range 8.8.8.8/32 (which is of course contained in the first range) then string-based deduplication would not catch and eliminate the duplicate. This would require more clever processing, i.e. IP ranges aggregation. I'm not sure how to do this efficiently. I've seen a Python script by @dewdmadbro which appears to be doing such aggregation (among other things) but I haven't tested it, so I don't know how efficient it is. Perhaps the author could comment (not sure whether the above mention pings them or not). Regardless, ideally I would like to avoid adding Python as a dependency, so if this is possible to implement without it, that would be better. I wonder whether there is some clever way to achieve this with grepcidr.

Well it did ping me, but that repo is templated from another user as I don't know much about python. I used it as an opportunity to try and understand running github actions.(still mostly magic to me) I am still trying to get my other stuff to play nice with nftables and bash scripting. So I am afraid that I won't be much help here, but this has given me other things to consider and learn about, Thanks

[friendly-bits]

iprange: Used heavily by projects like FireHOL. It can instantly merge and optimize multiple lists. Usage is as simple as: iprange list1.txt list2.txt > merged_optimized.txt.

I was not aware of this utility. Looks like it fits the bill, and is very efficient.

aggregate: Another classic lightweight tool (by ISC) that takes a list of prefixes and outputs the shortest possible list of non-overlapping prefixes.

This one fits the bill in terms of functionality but performance seems not great for large lists (judging by benchmarks I've seen online).

So technically this sort of smart decuplication is possible, albeit it requires adding a dependency on iprange. I'm still not persuaded that many people would use this feature. I'll hold off on implementing this for now, until more users chime in and request this feature.