Improve performance on Linux with many network interfaces

[wjordan]

The existing implementation of New() has a performance issue on Linux for the case when there are a large number (~ thousands) of network interfaces on the system.

Calling Addrs() on a net.Interface triggers a netlink request (RTM_GETADDR) that dumps addresses for all network interfaces, then filters the response messages to the ones matching the specified interface. When iface.Addrs() is separately called on every item returned by net.Interfaces(), this results in O(n²) data being sent through netlink, and can cause significant performance issues on systems managing thousands of network interfaces (virtual TAP devices, in my case).

Ideally a fix will eventually be implemented upstream (see discussion), but in the meantime, a workaround is to collect all network addresses through a single netlink call and assign the addresses to each network interface by index. I've created a simple drop-in module at github.com/wjordan/netinterfaces to help apply the fix with minimal diff to the existing codebase (see the implementation here).

I've included a simple benchmark test against the New() function, and compared the results before/after this PR on a single production host (with 1329 network interfaces):

before (~8255 ms/op)

goos: linux
goarch: amd64
pkg: github.com/libp2p/go-netroute
cpu: AMD EPYC 7502P 32-Core Processor               
BenchmarkNew-64    	       1	8142832032 ns/op
BenchmarkNew-64    	       1	7926576416 ns/op
BenchmarkNew-64    	       1	8625423001 ns/op
BenchmarkNew-64    	       1	7759228885 ns/op
BenchmarkNew-64    	       1	8347337935 ns/op
BenchmarkNew-64    	       1	8783752409 ns/op
BenchmarkNew-64    	       1	8592556706 ns/op
BenchmarkNew-64    	       1	8162635159 ns/op
BenchmarkNew-64    	       1	7887760609 ns/op
BenchmarkNew-64    	       1	8525044047 ns/op
PASS

after (~141 ms/op)

goos: linux
goarch: amd64
pkg: github.com/libp2p/go-netroute
cpu: AMD EPYC 7502P 32-Core Processor               
BenchmarkNew-64    	       8	 131353497 ns/op
BenchmarkNew-64    	       8	 146446498 ns/op
BenchmarkNew-64    	       8	 161298688 ns/op
BenchmarkNew-64    	       8	 135098898 ns/op
BenchmarkNew-64    	       8	 133536285 ns/op
BenchmarkNew-64    	       7	 169874107 ns/op
BenchmarkNew-64    	       8	 145653096 ns/op
BenchmarkNew-64    	       8	 137948240 ns/op
BenchmarkNew-64    	       8	 132781627 ns/op
BenchmarkNew-64    	       8	 184450081 ns/op
PASS

-98.28%

$ benchstat go-netroute-master.txt go-netroute-perf_fix.txt
goos: linux
goarch: amd64
pkg: github.com/libp2p/go-netroute
cpu: AMD EPYC 7502P 32-Core Processor               
       │ go-netroute-master.txt │       go-netroute-perf_fix.txt       │
       │         sec/op         │    sec/op     vs base                │
New-64             8255.0m ± 4%   141.8m ± 20%  -98.28% (p=0.000 n=10)

[willscott]

It would be great if we could figure out 3 years later how to upstream this into the standard library, rather than needing this utility layer for each client of calls to net.Interfaces. I wonder if it's worth another round of pinging maintainers on a resolution path here?