getProxy and geodb

[for7raid]

What is easiest way to integrate https://www.maxmind.com/ geo database into proxy?
The idea is to split traffic and choose upstream depending on destination web site, my country sites should outgone directly, but internation traffic thought another upstream.

Now I can do it with this way:

1. Download geo db
2. Convert it to json array of objects tuples ip:country
3. Load json in getProxy
4. Make a choice depending on destination ip and relevant county

But I afraid, json will be prohibitively large.

Thanx in advance.

[Snawoot]

Answering your question directly, yes - nearly 3.7 billion lines of JSON for just IPv4 addresses aren't really practical: dumbproxy will be loading that too long on startup and memory consumption will be insane.

Instead you can come up with aggregated CIDR prefixes mapped to countries. E.g.:

{
  "192.0.2.0/24": "ZZ",
  ...
}

the resulting list won't be too large and dp scripting engine will chew it easily. Many years ago I converted geoip db like that for nginx with that C program: https://gist.github.com/Snawoot/7c6f0b2122b3ea7fcb35

With some adjustments it will do the trick for your case too.

Now, matching part: you'll have resolved destination address inside proxy function, but map contains address prefixes. In order to check address against map you'll have to check it for presence of any prefix which could contain that IP address. E.g. for IP address 192.0.2.1 you'll have to check 192.0.2.1/32, 192.0.2.0/31, 192.0.2.0/30, ..., 0.0.0.0/0. That's basically how net:hash ipset matcher works in Linux kernel.

[for7raid]

What do you think about idea to create another one dialer called geodb ?

1. geodb://path/to/file.mmdb
2. if dialer was called, next call of getProxy will contain property geo in dst parameter, filled with country_name, country_iso_code and etc fields from db.

Need to add package https://github.com/oschwald/geoip2-golang.

I'm not a Go developer, but I'd like to try implement this solution, if you agree and do not have any other suggestions.

[for7raid]

Or another solution - implement geodb function in js execution engine, like readFile. It is simply and more natively to use in getProxy and bwLimit too.

[Snawoot]

@for7raid There is native MMDB support coming for scripting engine: #222 That way you'll be able to query about whatever IP address, source or destination. It will be merged and released soon.

[for7raid]

Is it beautiful, thank you very much!

[Snawoot]

I've released v1.48.0 and posted an example to wiki: https://github.com/SenseUnit/dumbproxy/wiki/GeoIP-support

[for7raid]

Amazing!