dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
54755

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin

Mod

maximum number of open sockets, files, threads, etc..?

It is surprisingly hard to get accurate information on this.

My question is pretty simple. What are the global (machine wide) upper limits assuming all tweaking, for number of open FDs, number of open tcp (or udp) sockets, number of active threads.

I'm aware of some limits. For example, a small stack is needed to create lots of threads and then the machine won't let me create more than 25k threads (over 3 processes).

I don't think you can have more than 65k open FDs (globally?), at the least because it is a 16 bit quantity? and if you can't have more than 65k open FDs you can't have more than 65k open sockets either.. or (FDs+Sockets must total 65k) or something..

64 bit linux, machines with 65gb of memory: are these limits still the same?

evilghost
Premium Member
join:2003-11-22
Springville, AL

evilghost

Premium Member

Limits can be enforced using /etc/security/limits.conf but I'm not sure that's what you're asking.

Ulimit is probably what you're looking for...
ulimit -a
 

Or optionally, a manual approach:
Maximum number of open files: cat /proc/sys/fs/file-max
Allocated, free allocated, and max FDs:  cat /proc/sys/fs/file-nr
 

justin
..needs sleep
Mod
join:1999-05-28
2031

justin

Mod

I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ...

sporkme
drop the crantini and move it, sister
MVM
join:2000-07-01
Morristown, NJ

sporkme

MVM

UTSL!

Additionally, I'm sure one of the distros with good docs (Gentoo??) must have stuff like this spelled out somewhere.

evilghost
Premium Member
join:2003-11-22
Springville, AL

evilghost to justin

Premium Member

to justin
said by justin:

I know that, but 'unlimited' actually isn't. So I'm after the real upper limits determined by kernel design ...
Ok, use this approach:

#Maximum number of open files:
cat /proc/sys/fs/file-max  
 
#Allocated, free allocated, and max FDs:
cat /proc/sys/fs/file-nr  
 
#Max threads:
cat /proc/sys/kernel/threads-max
 
#Max half-open TCP:
cat /proc/sys/net/core/somaxconn
 

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin to sporkme

Mod

to sporkme
I've been googling for a while and the answers are unclear otherwise I wouldn't bother posting here.

In part because (for threads) it depends on the stack size you limit things to. For fds and sockets wanting more than 65k (especially per process, but per maybe per machine) is a very specialized thing.

Even this topic (on max threads) came to an uncertain resolution:

»lkml.org/lkml/2006/8/15/222

evilghost
Premium Member
join:2003-11-22
Springville, AL

evilghost to justin

Premium Member

to justin
Max num of connections is handled by /proc/sys/net/core/optmem_max

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin to evilghost

Mod

to evilghost
those are current max numbers, and they can be changed.

I'm after what the real upper limits actually are.

eg, from my lkml topic it seems that max number of processes (or threads) is 65k, and potentially 128k but then /proc blows up.

So presumably 65k open files (and therefore sockets) is a hard limit everywhere.

There are interesting implications. No you would not want 40k threads, or 40k processes, but you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle.

Yet it appears to be an impossible task. Even on a 64bit machine with a whole pile of memory.
justin

justin to evilghost

Mod

to evilghost
said by evilghost:

Max num of connections is handled by /proc/sys/net/core/optmem_max
Nope:
»www.linuxinsight.com/pro ··· max.html

evilghost
Premium Member
join:2003-11-22
Springville, AL

1 edit

evilghost

Premium Member

said by justin:

said by evilghost:

Max num of connections is handled by /proc/sys/net/core/optmem_max
Nope:
»www.linuxinsight.com/pro ··· max.html
Yep:
»linux.die.net/man/3/cmsg

Insufficient optmem results in failure to bind socket. tcp_rmem, net.core.rmem_max, net.core.wmem_max, and tcp_wmem also control socket allocation.

justin
..needs sleep
Mod
join:1999-05-28
2031

justin

Mod

that is ancillary memory per socket. The maximum number that can be open on a machine is capped by maximum FDs (an fd is a 16 bit number, right?), not total memory taken by all sockets.
dave
Premium Member
join:2000-05-04
not in ohio

dave

Premium Member

Yeah, but fds are per-process indexes into a per-process table.

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin

Mod

ok, so you can set it to whatever you want for the box by echoing into file-max. So the theory limit is 1024x1024 (memory willing?) fds?

#define NR_OPEN (1024*1024) /* Absolute upper limit on fd num */
#define INR_OPEN 1024 /* Initial setting for nfile rlimits */
cvig
join:2003-12-31
Marquette, MI

1 edit

cvig to justin

Member

to justin
Have you been following this:

»kerneltrap.org/Linux/Def ··· K_Stacks

If so, it's clear that a lot of these limits are "it depends". 64 bit or 32 bit? Etc... There is some good information there within the lkml thread on the page if you read between the main points.

It is surprising that the theoretical limits aren't discussed more -- it would be interesting to see some tests with the smallest possible thread size one could maintain to see the max number of threads in practice.

Have you found any concrete answers yet to any of your questions?

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin

Mod

no, and you are right. "it depends".

I think the kernel gods like to say that as you approach large numbers you "aren't doing things right", so they get less interested in the answers.

So if the current practical maximum number of threads is, say, 32000 .. it is of no interest to them that this is hard to achieve because no properly written application should require that many threads.

The issue of maximums comes up more often than it should because future web servers will need to handle 100,000+ concurrent connections. Apache still has one foot in the past and needs one process or thread per open connection. Lighttpd is a single process single thread and uses epoll() so appears to be limited by the maximum number of open file descriptors per process (which to me says it is till capped at 65k?).

"Comet" which is the next generation of "Ajax", requires a quantum leap on number of concurrent connections. Instead of Ajax scripts on each browser polling for answers, they want to hold open a connection, and get events. This is very useful for lots of things: dynamic displays of data, chat .. anything that changes can now change immediately yet take no resources in polling when there are no changes to be made. Such a more logical way of doing things. Google is already using it all over: google chat, finance.google.com for example.

But of course "comet" needs a web infrastructure (front end servers, load balancers, routers, firewalls, web servers, application servers) that is much better at far more simultaneously open connections.

And funnily enough it becomes a lot easier to handle denial of service attacks as well: something that can handle 100k+ open connections can easily limit to a few per IP address, and easily identify and kill ones that misbehave. Half of a good denial of service attack is to try to open too many active connections and hold them (the other half is flooding).
cvig
join:2003-12-31
Marquette, MI

1 edit

cvig

Member

nginx is another interesting HTTP server -- on their website they claim 10,000 inactive HTTP keep-alive connections consume only 2.5M. From the LKML thread though it sounds like the real limitation is 32-bit with 8K stack. If you go 64-bit a lot of the limitations will be eased (at least in terms of max threads).

»nginx.net/

This sounds like the ideal time to bring up ideas of making a test program .

shdesigns
Powered By Infinite Improbabilty Drive
Premium Member
join:2000-12-01
Stone Mountain, GA
(Software) pfSense
ARRIS SB6121

shdesigns to justin

Premium Member

to justin
As memory is allocated per socket the limit of the # of sockets is probably moot.

Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also. Thousands of sockets and threads would consume more memory than you have and probably thash so hard the limit of the # of sockets would probably be the least of your worries.

Given the 1024*1024 you mentioned and the probable 64k minimum buffer on a TCP sliding window socket that is 64gigs or RAM.

Gobs of RAM and 64-bit may reach the limits. I bet few are getting close to this in practice so few developers consider them.

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin

Mod

It is a good point.

65k open sockets with 128k rmem and wmem buffers take 8.3gb

But things might not be that bad inside the kernel. Perhaps the space is only taken if it is needed:

A paper was published
»www.stdlib.net/~colmmacc ··· dout.pdf

that showed off an ftp server whose record was 27,000 simultaneous downloads (under apache). And there rmem_default/rmem_max was 262k. The machine has 4gb of ram.

Unless my math is faulty, that SHOULD take 7gb of ram for read socket buffers alone (rmem_default). But clearly it didn't.

Steve
I know your IP address

join:2001-03-10
Tustin, CA

Steve to shdesigns

to shdesigns
said by shdesigns:

Each socket may gave 256k TX and RX buffers (seems to be default minimum) with file descriptors allocated also.
Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling.

If they can't be, then this smells like a great optimization to add to the kernel.

Steve

BeesTea
Internet Janitor
Premium Member
join:2003-03-08
00000

1 edit

BeesTea to cvig

Premium Member

to cvig
said by cvig:

nginx is another interesting HTTP server
Unrelated to the thread topic, but of interest.

This happens to have been the HTTPd of choice of the authors of stormworm, the much discussed malware. It's in fact so unique, many vendors chose to trigger IDS on its banner.

For example:Cisco Intellishield

Edit: Changed tense

shdesigns
Powered By Infinite Improbabilty Drive
Premium Member
join:2000-12-01
Stone Mountain, GA
(Software) pfSense
ARRIS SB6121

1 edit

shdesigns to Steve

Premium Member

to Steve
said by Steve:

Can TX and RX buffer sizes be tuned separately? I would think that for web-type service, maxing out the TX buffer and minimizing the RX buffer would make a big diff in scaling.

If they can't be, then this smells like a great optimization to add to the kernel.

Steve
/proc/sys/net/ipv4:

tcp_rmem - 4096 87380 16777216
tcp_wmem - 4096 87380 16777216
tcp_mem - 98304 131072 196608

I think those are min/default/max

edit: I bet Apache opens sockets with its own settings.

justin
..needs sleep
Mod
join:1999-05-28
2031
Billion BiPAC 7800N
Apple AirPort Extreme (2011)

justin to BeesTea

Mod

to BeesTea
I like this part:
quote:
Because web connections aren’t as long lived as IMAP connections, we stayed with Apache for our frontends for a while longer. However we’ve now switched over to using nginx for our frontend web proxy as well, which has also allowed us to increase the keep-alive timeout for HTTP connections to 5 minutes, which should result in a small perceptible improvement when moving between pages.

The net result of all this is that each frontend proxy server currently maintains over 10,000 simultaneous IMAP, POP, Web & SMTP connections (including many SSL ones) using only about 10% of the available CPU on 3.20GHz Netburst Xeon based CPUs.
nginx seems to have done it right: a small amount of processes each using epoll. If the kernel doesn't waste a whole heap of memory with each open connection, this would scale really well.
justin

justin to BeesTea

Mod

to BeesTea
dammit .. now I read some more about nginx, I want to change our front end. Which means converting a bunch of ReWrite rules, testing and so on and so forth.

But since it offers proper limits per simultaneous IP and such huge scalability and such immense capacity for KeepAlive and open but quiet connections, I can't see why to stick with apache2
xargs
join:2000-08-25
Camarillo, CA

xargs to justin

Member

to justin
said by justin:

you might want 100k simultaneous open tcp conversations. It isn't clear to me why this should be so difficult if the majority of the open conversations were idle.
I'm working on a server that has just those requirements. It does appear that there's a 64K per-process open file limit, though I can't find any documentation to that effect. It's not a system-wide limit -- clients can open 64K connections into the server process from the same machine, giving 128K total open files:
$ cat /proc/sys/fs/file-nr
135360  0       296295
 
A call to getrlimit(RLIMIT_NOFILE) tells me the per-process limit is 250,000 (that's how I've configured it in /etc/security/limits.conf) but I can't seem to actually achieve that many open connections.

Have you found any information that this limit can be increased?