Topic 'Re: maximum number of open sockets, files, threads, etc..?' in forum 'All Things Unix' - dslreports.com

Re: maximum number of open sockets, files, threads, etc..?

Mon, 14 Jul 2008 14:55:04 EDT

xargs posted :

said by justin:

you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle.

I'm working on a server that has just those requirements. It does appear that there's a 64K per-process open file limit, though I can't find any documentation to that effect. It's not a system-wide limit -- clients can open 64K connections into the server process from the same machine, giving 128K total open files:

$ cat /proc/sys/fs/file-nr135360  0       296295

A call to getrlimit(RLIMIT_NOFILE) tells me the per-process limit is 250,000 (that's how I've configured it in /etc/security/limits.conf) but I can't seem to actually achieve that many open connections.

Have you found any information that this limit can be increased?

Re: maximum number of open sockets, files, threads, etc..?

Thu, 24 Apr 2008 02:24:30 EDT

justin posted : dammit .. now I read some more about nginx, I want to change our front end. Which means converting a bunch of ReWrite rules, testing and so on and so forth.

But since it offers proper limits per simultaneous IP and such huge scalability and such immense capacity for KeepAlive and open but quiet connections, I can't see why to stick with apache2

Re: maximum number of open sockets, files, threads, etc..?

Thu, 24 Apr 2008 02:14:14 EDT

justin posted : I like this part:

quote:
Because web connections aren’t as long lived as IMAP connections, we stayed with Apache for our frontends for a while longer. However we’ve now switched over to using nginx for our frontend web proxy as well, which has also allowed us to increase the keep-alive timeout for HTTP connections to 5 minutes, which should result in a small perceptible improvement when moving between pages.

The net result of all this is that each frontend proxy server currently maintains over 10,000 simultaneous IMAP, POP, Web & SMTP connections (including many SSL ones) using only about 10% of the available CPU on 3.20GHz Netburst Xeon based CPUs.

nginx seems to have done it right: a small amount of processes each using epoll. If the kernel doesn't waste a whole heap of memory with each open connection, this would scale really well.

Re: maximum number of open sockets, files, threads, etc..?

Thu, 24 Apr 2008 01:49:49 EDT

shdesigns posted :

said by Steve:

Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling.

If they can't be, then this smells like a great optimization to add to the kernel.

Steve

/proc/sys/net/ipv4:

tcp_rmem - 4096 87380 16777216
tcp_wmem - 4096 87380 16777216
tcp_mem - 98304 131072 196608

I think those are min/default/max

edit: I bet Apache opens sockets with its own settings.
--
Scott Henion

Embedded Systems Consultant, shenion on #ATU @irc.freenode.net
SHDesigns home

Re: maximum number of open sockets, files, threads, etc..?

Thu, 24 Apr 2008 01:40:43 EDT

BeesTea posted :

said by cvig:

nginx is another interesting HTTP server

Unrelated to the thread topic, but of interest.

This happens to have been the HTTPd of choice of the authors of stormworm, the much discussed malware. It's in fact so unique, many vendors chose to trigger IDS on its banner.

For example:Cisco Intellishield

Edit: Changed tense
--
Overpower, overcome.

Re: maximum number of open sockets, files, threads, etc..?

Thu, 24 Apr 2008 00:57:36 EDT

Steve posted :

said by shdesigns:

Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also.

Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling.

If they can't be, then this smells like a great optimization to add to the kernel.

Steve
--
Stephen J. Friedl | Unix Wizard | Microsoft Security MVP | Tustin, California USA | my web site

Re: maximum number of open sockets, files, threads, etc..?

Wed, 23 Apr 2008 23:16:52 EDT

justin posted : It is a good point.

65k open sockets with 128k rmem and wmem buffers take 8.3gb

But things might not be that bad inside the kernel. Perhaps the space is only taken if it is needed:

A paper was published
»www.stdlib.net/~colmmacc/Apachec···dout.pdf

that showed off an ftp server whose record was 27,000 simultaneous downloads (under apache). And there rmem_default/rmem_max was 262k. The machine has 4gb of ram.

Unless my math is faulty, that SHOULD take 7gb of ram for read socket buffers alone (rmem_default). But clearly it didn't.

Re: maximum number of open sockets, files, threads, etc..?

Wed, 23 Apr 2008 23:04:14 EDT

shdesigns posted : As memory is allocated per socket the limit of the # of sockets is probably moot.

Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also. Thousands of sockets and threads would consume more memory than you have and probably thash so hard the limit of the # of sockets would probably be the least of your worries.

Given the 1024*1024 you mentioned and the probable 64k minimum buffer on a TCP sliding window socket that is 64gigs or RAM.

Gobs of RAM and 64-bit may reach the limits. I bet few are getting close to this in practice so few developers consider them.
--
Scott Henion

Embedded Systems Consultant, shenion on #ATU @irc.freenode.net
SHDesigns home

Re: maximum number of open sockets, files, threads, etc..?

Wed, 23 Apr 2008 23:00:36 EDT

cvig posted : nginx is another interesting HTTP server -- on their website they claim 10,000 inactive HTTP keep-alive connections consume only 2.5M. From the LKML thread though it sounds like the real limitation is 32-bit with 8K stack. If you go 64-bit a lot of the limitations will be eased (at least in terms of max threads).

»nginx.net/

This sounds like the ideal time to bring up ideas of making a test program :).

Re: maximum number of open sockets, files, threads, etc..?

Wed, 23 Apr 2008 22:36:48 EDT

justin posted : no, and you are right. "it depends".

I think the kernel gods like to say that as you approach large numbers you "aren't doing things right", so they get less interested in the answers.

So if the current practical maximum number of threads is, say, 32000 .. it is of no interest to them that this is hard to achieve because no properly written application should require that many threads.

The issue of maximums comes up more often than it should because future web servers will need to handle 100,000+ concurrent connections. Apache still has one foot in the past and needs one process or thread per open connection. Lighttpd is a single process single thread and uses epoll() so appears to be limited by the maximum number of open file descriptors per process (which to me says it is till capped at 65k?).

"Comet" which is the next generation of "Ajax", requires a quantum leap on number of concurrent connections. Instead of Ajax scripts on each browser polling for answers, they want to hold open a connection, and get events. This is very useful for lots of things: dynamic displays of data, chat .. anything that changes can now change immediately yet take no resources in polling when there are no changes to be made. Such a more logical way of doing things. Google is already using it all over: google chat, finance.google.com for example.

But of course "comet" needs a web infrastructure (front end servers, load balancers, routers, firewalls, web servers, application servers) that is much better at far more simultaneously open connections.

And funnily enough it becomes a lot easier to handle denial of service attacks as well: something that can handle 100k+ open connections can easily limit to a few per IP address, and easily identify and kill ones that misbehave. Half of a good denial of service attack is to try to open too many active connections and hold them (the other half is flooding).

Re: maximum number of open sockets, files, threads, etc..?

Wed, 23 Apr 2008 16:03:24 EDT

cvig posted : Have you been following this:

»kerneltrap.org/Linux/Defaulting_To_4K_Stacks

If so, it's clear that a lot of these limits are "it depends". 64 bit or 32 bit? Etc... There is some good information there within the lkml thread on the page if you read between the main points.

It is surprising that the theoretical limits aren't discussed more -- it would be interesting to see some tests with the smallest possible thread size one could maintain to see the max number of threads in practice.

Have you found any concrete answers yet to any of your questions?

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 16:17:32 EDT

justin posted : ok, so you can set it to whatever you want for the box by echoing into file-max. So the theory limit is 1024x1024 (memory willing?) fds?

#define NR_OPEN (1024*1024) /* Absolute upper limit on fd num */
#define INR_OPEN 1024 /* Initial setting for nfile rlimits */

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 16:05:54 EDT

dave posted : Yeah, but fds are per-process indexes into a per-process table.

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 16:00:50 EDT

justin posted : that is ancillary memory per socket. The maximum number that can be open on a machine is capped by maximum FDs (an fd is a 16 bit number, right?), not total memory taken by all sockets.

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:54:22 EDT

evilghost posted :

said by justin:

said by evilghost:

Max num of connections is handled by /proc/sys/net/core/optmem_max

Nope:
»www.linuxinsight.com/proc_sys_ne···max.html

Yep:
»linux.die.net/man/3/cmsg

Insufficient optmem results in failure to bind socket. tcp_rmem, net.core.rmem_max, net.core.wmem_max, and tcp_wmem also control socket allocation.

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:13:33 EDT

justin posted :

said by evilghost:

Max num of connections is handled by /proc/sys/net/core/optmem_max

Nope:
»www.linuxinsight.com/proc_sys_ne···max.html

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:11:58 EDT

justin posted : those are current max numbers, and they can be changed.

I'm after what the real upper limits actually are.

eg, from my lkml topic it seems that max number of processes (or threads) is 65k, and potentially 128k but then /proc blows up.

So presumably 65k open files (and therefore sockets) is a hard limit everywhere.

There are interesting implications. No you would not want 40k threads, or 40k processes, but you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle.

Yet it appears to be an impossible task. Even on a 64bit machine with a whole pile of memory.

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:08:25 EDT

evilghost posted : Max num of connections is handled by /proc/sys/net/core/optmem_max

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:05:00 EDT

justin posted : I've been googling for a while and the answers are unclear otherwise I wouldn't bother posting here.

In part because (for threads) it depends on the stack size you limit things to. For fds and sockets wanting more than 65k (especially per process, but per maybe per machine) is a very specialized thing.

Even this topic (on max threads) came to an uncertain resolution:

»lkml.org/lkml/2006/8/15/222

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 15:03:07 EDT

evilghost posted :

said by justin:

I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ...

Ok, use this approach:

#Maximum number of open files:cat /proc/sys/fs/file-max   #Allocated, free allocated, and max FDs:cat /proc/sys/fs/file-nr   #Max threads:cat /proc/sys/kernel/threads-max #Max half-open TCP:cat /proc/sys/net/core/somaxconn

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 14:51:33 EDT

sporkme posted : UTSL!

Additionally, I'm sure one of the distros with good docs (Gentoo??) must have stuff like this spelled out somewhere.

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 14:49:36 EDT

justin posted : I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ...

Re: maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 14:28:05 EDT

evilghost posted : Limits can be enforced using /etc/security/limits.conf but I'm not sure that's what you're asking.

Ulimit is probably what you're looking for...

ulimit -a

Or optionally, a manual approach:

Maximum number of open files: cat /proc/sys/fs/file-maxAllocated, free allocated, and max FDs:  cat /proc/sys/fs/file-nr

maximum number of open sockets, files, threads, etc..?

Tue, 22 Apr 2008 14:17:38 EDT

justin posted : It is surprisingly hard to get accurate information on this.

My question is pretty simple. What are the global (machine wide) upper limits assuming all tweaking, for number of open FDs, number of open tcp (or udp) sockets, number of active threads.

I'm aware of some limits. For example, a small stack is needed to create lots of threads and then the machine won't let me create more than 25k threads (over 3 processes).

I don't think you can have more than 65k open FDs (globally?), at the least because it is a 16 bit quantity? and if you can't have more than 65k open FDs you can't have more than 65k open sockets either.. or (FDs+Sockets must total 65k) or something..

64 bit linux, machines with 65gb of memory: are these limits still the same?