C500K from Urban Airship
Urban Airship is generous enough to publish how they tune the Linux Kernel to handle over 500K concurrent users. This article is just my note to fill up some background info to facilitate better understanding of what and why they did that.
To start with, they want to squeeze the system and would like it to handle as many connections as possible. The tradeoff is they should have this box doing less. That means less code, cpu-usage, and ram-usage. So, its main job is to deal with client connections and submit the task to the queue.
Check memory usage
[root@f3 ~]# sysctl -a | grep mem
net.ipv4.udp_wmem_min = 4096
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_mem = 1549632 2066176 3099264
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_mem = 196608 262144 393216
net.ipv4.igmp_max_memberships = 20
net.core.optmem_max = 20480
net.core.rmem_default = 129024
net.core.wmem_default = 129024
net.core.rmem_max = 131071
net.core.wmem_max = 131071
vm.lowmem_reserve_ratio = 256 256 32
vm.overcommit_memory = 0
Check system max in Internet connection
[root@f3 ~]# sysctl -a | grep file-max
fs.file-max = 1587124
//how many open file descriptors are currently being used.
[root@f3 ~]# more /proc/sys/fs/file-nr
1020 0 1587124
- 1020: total allocated file descriptor
- 0: total free allocated file descriptor
- 1587124: max number of file descriptor allowed on the system
//how many files are open.
[root@f3 ~]# lsof | wc -l
2497
[root@f3 ~]# lsof -u trffcapp | wc -l
[root@f3 ~]# vi /etc/security/limits.conf
Check max connection per user
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 512
virtual memory (kbytes, -v) unlimited
$ launchctl limit
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 1024 2048
maxfiles 2048 4096
$ sysctl -a | grep files
kern.maxfiles = 32768
kern.maxfilesperproc = 16384
kern.maxfiles: 32768
kern.maxfilesperproc: 16384
kern.num_files: 2049
temporary changes:
sudo launchctl limit maxfiles 16384 32768
sudo ulimit -n 32768
sudo sysctl -w kern.maxfilesperproc=16384
sudo sysctl -w kern.maxfiles=32768
permanent changes, you need to go to the actual file:
/etc/sysctl.conf (kern.maxfilesperproc=65536)
/etc/launchd.conf
The ulimit level is set low to prevent one poor shell script from flooding the kernel with open files.
The kern.maxfilesperproc is there to leave a little room in the max files count so that one process can use most but not all of the open file handler space from the kernel.