I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.
If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.
One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.
(this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example
Agree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible.
Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.
Actually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.
You may not have read the entire article, but he clearly explains that their tech stack is FreeBSD + Erlang not Linux + Erlang.
There is no /proc/net/tcp on FreeBSD. Hell, there is no /proc unless it is specifically mounted by the administrator, but tools certainly don't use it to get data.
You may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.
netstat -s |grep "connections established" is pretty fast and does the job if you're not filtering for certain ports or IPs. But running plain old netstat on a system with so many connections is a really bad idea. I almost fired myself once over a crufty cron job I left running...
If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.
One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.
I cannot find a link to my story though...