justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
justin
Mod
2008-Apr-22 2:17 pm
maximum number of open sockets, files, threads, etc..?It is surprisingly hard to get accurate information on this.
My question is pretty simple. What are the global (machine wide) upper limits assuming all tweaking, for number of open FDs, number of open tcp (or udp) sockets, number of active threads.
I'm aware of some limits. For example, a small stack is needed to create lots of threads and then the machine won't let me create more than 25k threads (over 3 processes).
I don't think you can have more than 65k open FDs (globally?), at the least because it is a 16 bit quantity? and if you can't have more than 65k open FDs you can't have more than 65k open sockets either.. or (FDs+Sockets must total 65k) or something..
64 bit linux, machines with 65gb of memory: are these limits still the same? |
|
evilghost Premium Member join:2003-11-22 Springville, AL |
Limits can be enforced using /etc/security/limits.conf but I'm not sure that's what you're asking. Ulimit is probably what you're looking for... ulimit -a
Or optionally, a manual approach: Maximum number of open files: cat /proc/sys/fs/file-max
Allocated, free allocated, and max FDs: cat /proc/sys/fs/file-nr
|
|
justin..needs sleep Mod join:1999-05-28 2031 |
justin
Mod
2008-Apr-22 2:49 pm
I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ... |
|
sporkmedrop the crantini and move it, sister MVM join:2000-07-01 Morristown, NJ |
UTSL!
Additionally, I'm sure one of the distros with good docs (Gentoo??) must have stuff like this spelled out somewhere. |
|
evilghost Premium Member join:2003-11-22 Springville, AL |
to justin
said by justin:I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ... Ok, use this approach: #Maximum number of open files:
cat /proc/sys/fs/file-max
#Allocated, free allocated, and max FDs:
cat /proc/sys/fs/file-nr
#Max threads:
cat /proc/sys/kernel/threads-max
#Max half-open TCP:
cat /proc/sys/net/core/somaxconn
|
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
to sporkme
I've been googling for a while and the answers are unclear otherwise I wouldn't bother posting here. In part because (for threads) it depends on the stack size you limit things to. For fds and sockets wanting more than 65k (especially per process, but per maybe per machine) is a very specialized thing. Even this topic (on max threads) came to an uncertain resolution: » lkml.org/lkml/2006/8/15/222 |
|
evilghost Premium Member join:2003-11-22 Springville, AL |
to justin
Max num of connections is handled by /proc/sys/net/core/optmem_max |
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
to evilghost
those are current max numbers, and they can be changed.
I'm after what the real upper limits actually are.
eg, from my lkml topic it seems that max number of processes (or threads) is 65k, and potentially 128k but then /proc blows up.
So presumably 65k open files (and therefore sockets) is a hard limit everywhere.
There are interesting implications. No you would not want 40k threads, or 40k processes, but you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle.
Yet it appears to be an impossible task. Even on a 64bit machine with a whole pile of memory. |
|
justin |
to evilghost
said by evilghost:Max num of connections is handled by /proc/sys/net/core/optmem_max Nope: » www.linuxinsight.com/pro ··· max.html |
|
|
evilghost Premium Member join:2003-11-22 Springville, AL 1 edit |
Yep: » linux.die.net/man/3/cmsgInsufficient optmem results in failure to bind socket. tcp_rmem, net.core.rmem_max, net.core.wmem_max, and tcp_wmem also control socket allocation. |
|
justin..needs sleep Mod join:1999-05-28 2031 |
justin
Mod
2008-Apr-22 4:00 pm
that is ancillary memory per socket. The maximum number that can be open on a machine is capped by maximum FDs (an fd is a 16 bit number, right?), not total memory taken by all sockets. |
|
dave Premium Member join:2000-05-04 not in ohio |
dave
Premium Member
2008-Apr-22 4:05 pm
Yeah, but fds are per-process indexes into a per-process table. |
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
justin
Mod
2008-Apr-22 4:17 pm
ok, so you can set it to whatever you want for the box by echoing into file-max. So the theory limit is 1024x1024 (memory willing?) fds?
#define NR_OPEN (1024*1024) /* Absolute upper limit on fd num */ #define INR_OPEN 1024 /* Initial setting for nfile rlimits */ |
|
cvig join:2003-12-31 Marquette, MI 1 edit |
to justin
Have you been following this: » kerneltrap.org/Linux/Def ··· K_StacksIf so, it's clear that a lot of these limits are "it depends". 64 bit or 32 bit? Etc... There is some good information there within the lkml thread on the page if you read between the main points. It is surprising that the theoretical limits aren't discussed more -- it would be interesting to see some tests with the smallest possible thread size one could maintain to see the max number of threads in practice. Have you found any concrete answers yet to any of your questions? |
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
justin
Mod
2008-Apr-23 10:36 pm
no, and you are right. "it depends".
I think the kernel gods like to say that as you approach large numbers you "aren't doing things right", so they get less interested in the answers.
So if the current practical maximum number of threads is, say, 32000 .. it is of no interest to them that this is hard to achieve because no properly written application should require that many threads.
The issue of maximums comes up more often than it should because future web servers will need to handle 100,000+ concurrent connections. Apache still has one foot in the past and needs one process or thread per open connection. Lighttpd is a single process single thread and uses epoll() so appears to be limited by the maximum number of open file descriptors per process (which to me says it is till capped at 65k?).
"Comet" which is the next generation of "Ajax", requires a quantum leap on number of concurrent connections. Instead of Ajax scripts on each browser polling for answers, they want to hold open a connection, and get events. This is very useful for lots of things: dynamic displays of data, chat .. anything that changes can now change immediately yet take no resources in polling when there are no changes to be made. Such a more logical way of doing things. Google is already using it all over: google chat, finance.google.com for example.
But of course "comet" needs a web infrastructure (front end servers, load balancers, routers, firewalls, web servers, application servers) that is much better at far more simultaneously open connections.
And funnily enough it becomes a lot easier to handle denial of service attacks as well: something that can handle 100k+ open connections can easily limit to a few per IP address, and easily identify and kill ones that misbehave. Half of a good denial of service attack is to try to open too many active connections and hold them (the other half is flooding). |
|
cvig join:2003-12-31 Marquette, MI 1 edit |
cvig
Member
2008-Apr-23 11:00 pm
nginx is another interesting HTTP server -- on their website they claim 10,000 inactive HTTP keep-alive connections consume only 2.5M. From the LKML thread though it sounds like the real limitation is 32-bit with 8K stack. If you go 64-bit a lot of the limitations will be eased (at least in terms of max threads). » nginx.net/This sounds like the ideal time to bring up ideas of making a test program . |
|
shdesignsPowered By Infinite Improbabilty Drive Premium Member join:2000-12-01 Stone Mountain, GA (Software) pfSense ARRIS SB6121
|
to justin
As memory is allocated per socket the limit of the # of sockets is probably moot.
Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also. Thousands of sockets and threads would consume more memory than you have and probably thash so hard the limit of the # of sockets would probably be the least of your worries.
Given the 1024*1024 you mentioned and the probable 64k minimum buffer on a TCP sliding window socket that is 64gigs or RAM.
Gobs of RAM and 64-bit may reach the limits. I bet few are getting close to this in practice so few developers consider them. |
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
justin
Mod
2008-Apr-23 11:16 pm
It is a good point. 65k open sockets with 128k rmem and wmem buffers take 8.3gb But things might not be that bad inside the kernel. Perhaps the space is only taken if it is needed: A paper was published » www.stdlib.net/~colmmacc ··· dout.pdfthat showed off an ftp server whose record was 27,000 simultaneous downloads (under apache). And there rmem_default/rmem_max was 262k. The machine has 4gb of ram. Unless my math is faulty, that SHOULD take 7gb of ram for read socket buffers alone (rmem_default). But clearly it didn't. |
|
SteveI know your IP address
join:2001-03-10 Tustin, CA |
to shdesigns
said by shdesigns: Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also. Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling. If they can't be, then this smells like a great optimization to add to the kernel. Steve |
|
BeesTeaInternet Janitor Premium Member join:2003-03-08 00000 1 edit |
to cvig
said by cvig:nginx is another interesting HTTP server Unrelated to the thread topic, but of interest. This happens to have been the HTTPd of choice of the authors of stormworm, the much discussed malware. It's in fact so unique, many vendors chose to trigger IDS on its banner. For example: Cisco IntellishieldEdit: Changed tense |
|
shdesignsPowered By Infinite Improbabilty Drive Premium Member join:2000-12-01 Stone Mountain, GA (Software) pfSense ARRIS SB6121
1 edit |
to Steve
said by Steve:Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling. If they can't be, then this smells like a great optimization to add to the kernel. Steve /proc/sys/net/ipv4: tcp_rmem - 4096 87380 16777216 tcp_wmem - 4096 87380 16777216 tcp_mem - 98304 131072 196608 I think those are min/default/max edit: I bet Apache opens sockets with its own settings. |
|
justin..needs sleep Mod join:1999-05-28 2031 Billion BiPAC 7800N Apple AirPort Extreme (2011)
|
to BeesTea
I like this part: quote: Because web connections arent as long lived as IMAP connections, we stayed with Apache for our frontends for a while longer. However weve now switched over to using nginx for our frontend web proxy as well, which has also allowed us to increase the keep-alive timeout for HTTP connections to 5 minutes, which should result in a small perceptible improvement when moving between pages.
The net result of all this is that each frontend proxy server currently maintains over 10,000 simultaneous IMAP, POP, Web & SMTP connections (including many SSL ones) using only about 10% of the available CPU on 3.20GHz Netburst Xeon based CPUs.
nginx seems to have done it right: a small amount of processes each using epoll. If the kernel doesn't waste a whole heap of memory with each open connection, this would scale really well. |
|
justin |
to BeesTea
dammit .. now I read some more about nginx, I want to change our front end. Which means converting a bunch of ReWrite rules, testing and so on and so forth.
But since it offers proper limits per simultaneous IP and such huge scalability and such immense capacity for KeepAlive and open but quiet connections, I can't see why to stick with apache2 |
|
xargs join:2000-08-25 Camarillo, CA |
to justin
said by justin:you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle. I'm working on a server that has just those requirements. It does appear that there's a 64K per-process open file limit, though I can't find any documentation to that effect. It's not a system-wide limit -- clients can open 64K connections into the server process from the same machine, giving 128K total open files: $ cat /proc/sys/fs/file-nr
135360 0 296295
A call to getrlimit(RLIMIT_NOFILE) tells me the per-process limit is 250,000 (that's how I've configured it in /etc/security/limits.conf) but I can't seem to actually achieve that many open connections. Have you found any information that this limit can be increased? |
|