thousands of ips on one machine

Maybe this is common knowledge, it took me a couple hours to figure out.. I was having a problem across several machines where opening new connections would incur a significant delay (500-750ms).. The problem would only happen after I configured over 10k ips.

There's an obscure sysctl for linux that will speed up your software's bind() call that is made while creating a new connection.

net.ipv4.ip_nonlocal_bind

This tells the kernel to allow ip addresses that do not exist on the interface to be used. In the kernel's net code, setting this sysctl to 1 tells the netcode to not check the interfaces to see if the ip address exists. Bypassing this check when you have 10's of thousands of ips on a server makes a significant difference in opening new connections. The only downside is you will not receive an error while attempting to bind a non-configured address, your connection will timeout..

Code:

err = -EADDRNOTAVAIL;
	if (!sysctl_ip_nonlocal_bind &&
	    !(inet->freebind || inet->transparent) &&
	    addr->sin_addr.s_addr != htonl(INADDR_ANY) &&
	    chk_addr_ret != RTN_LOCAL &&
	    chk_addr_ret != RTN_MULTICAST &&
	    chk_addr_ret != RTN_BROADCAST)
		goto out;