Lookup Alternative - TXT Map-File

[iwikus]

Hi, I was searching if there is any lua geoip plugin for HAproxy and found this your project. I see you have implemented it by doing lookups to location service or (untested) lua mmdb library. For the the first option, this is possible do it other and I think more easy way, since there you need to do that caching for performance reasons.

Using geoip database or ipinfo it is posible to generate geoip.txt with all ip ranges and countries - I am doing it here (big file) http://iwik.org/ipcountry/geoip.txt
Just yesterday have switched source of data and details are here on my blog https://blog.erben.sk/2024/09/14/new-country-ip-ranges-generator/ including link to source.
And you can use it for ACL like this

# GeoIP ACL - allow only from SK and CZ
acl acl_geoloc_sk_cz src,map_ip(/etc/haproxy/geoip.txt) -m reg -i (SK|CZ)
acl acl_internal src  10.0.0.0/8 192.168.0.0/16
http-request deny if !acl_geoloc_sk_cz !acl_internal

Or in frontend part, set variable
http-response set-header geoip-countrycode %[src,map_ip(/etc/haproxy/geoip.txt)]

Or just in custom logformat (end of this huge line)

log-format vhost:%[capture.req.hdr(0)]|client:%ci|time:%Ts|method:%HM|protocol:%HV|uri:"%[capture.req.uri]"|status:%ST|outbytes:%B|inbytes:%U|referer:"%[capture.req.hdr(4)]"|ua:"%[capture.req.hdr(5)]"|cr
eate:%Tt|sslprot:%sslv|sslciph:%sslc|ssl_sni:%[ssl_fc_sni]|ssl_fc_has_early:%[ssl_fc_has_early]|backend_name:%b|frontend_name:%f|country:%[src,map_ip(/etc/haproxy/geoip.txt)]

Anyway, when I find some time, I will try doing some test with that lua mmdb library.

[superstes]

Good idea 👍
Will test it.
I initially thought the mmdb lookup is (much) more efficient than others. But with the overhead of the backend request and the HAProxy map_ip being very fast, this can be a good option.

[superstes]

We should compare the memory consumption though.
Could be relevant for small VPS setups. 🤔

[abdullahdevrel]

Please reach out if you have any questions about IPinfo. I mentioned in @iwikus's blog that the size of the database will be smaller because the issue of smaller ranges resulting in a bigger file size is a known issue for us. However, it will not be significantly smaller compared to other databases. I believe an MMDB-based solution is the optimal choice because it is both fast and compact. However, I am not very familiar with the project so I am not sure which option would be the best fit.

[superstes]

From what I see from a first view:

• IPv4 Ranges to Country-Code is doable
• IPv6 Ranges to Country-Code is very large
• If you want to query more than the Country-Code, the text file sizes grows rapidly (ASN/ISP, City, ...)

As described in haproxy/haproxy#223 they will use loads of RAM.

As for the performance - I will try to find some time for benchmarks. Not sure how performant it is with (global-) large-scale DBs.
Source references:
https://github.com/haproxy/haproxy/blob/master/src/map.c#L1221
https://github.com/haproxy/haproxy/blob/master/src/pattern.c#L1016
https://github.com/haproxy/haproxy/blob/master/src/pattern.c#L956
https://github.com/haproxy/haproxy/blob/master/src/pattern.c#L988

Regarding the pure-LUA implementation - this would be nice to have. But I have not found time to test it, as https://github.com/anjia0532/lua-resty-maxminddb seems to have some nginx-specific (resty) code in it. Changes to the script might be needed.

[iwikus]

I still did not get into testing that lua code...
But for memory consumption I did not see any problem - with 20MB TXT map file for countries (ipv4 & ipv6 together), HAproxy is using less then 512MB ram.

[WRMSRwasTaken]

Hi!

In case this might be interesting, someone else did a pure Lua implementation without any external modules here: https://github.com/toonvd/haproxy-lua-geoip2

I'm running this one in production since ~4 Months with both GeoLite2-Country and GeoLite2-ASN (but only up to ~500 req/s so I can't really tell high load scenarios) but besides the breakage on Lua 5.5 (see my PR on that repo) it's been working flawlessly so far!

One thing I've had to adjust was to disable memory trimming (no-memory-trimming) because otherwise the HAProxy internal watchdog was sometimes firing on the old process on reload.

There's also an Implementation in Rust which gets compiled down to bytecode to run in the Lua VM (https://github.com/khvzak/haproxy-geoip2) which I originally found and the Lua one being mentioned in the issues for apparently being faster.