nginx 1.7.7 のベンチマーク

検証作業とかちゃんとできてないなぁという反省から nginx のベンチマークをとってみることに．
バージョンは現時点の Mainline version である 1.7.7 ．
担当サービス的に同時接続数を重視した．目標は同時接続数 10,000 ．
検証に使った仮想サーバの環境は以下のとおり．

OS	CPU	Mem
CentOS 6.4 64bit	4 Core	4 GB

ベンチマーク用nginxのコンパイルオプション

./configure \
--prefix=/usr/local/nginx-1.7.7 \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--user=nginx \
--group=nginx

なるべくプレーンなものになるようにした．

チューニング

同時接続数を増やしながら設定を煮詰めていく．

同時接続数 600 でエラー発生

[root@client ~]# ab -c 600 -n 6000 'http://10.100.47.58/10k.dat'

<<< snip >>>

Complete requests:      6000
Failed requests:        47
   (Connect: 0, Receive: 0, Length: 47, Exceptions: 0)

<<< snip >>>

nginx のエラーログ

2014/11/11 18:19:50 [crit] 6448#0: *46675 open() "/usr/local/nginx-1.7.7/html/10k.dat" failed (24: Too many open files), client: 10.100.37.239, server: localhost, request: "GET /10k.dat HTTP/1.0", host: "10.100.47.58"

プロセスがオープンできるファイルの数が上限に達した模様．

制限値を確認してみる

[root@benchnginx ~]# ps -aefww | grep 'worker'
nginx     6448  6447  0 18:03 ?        00:00:05 nginx: worker process
root      6504  1928  0 18:21 pts/0    00:00:00 grep worker

[root@benchnginx ~]# cat /proc/6448/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             30507                30507                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       30507                30507                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Max open files 1024 なので足りてなさそう．

設定変更

@@ -8,6 +8,7 @@

 #pid        logs/nginx.pid;

+worker_rlimit_nofile  102400;

 events {
     worker_connections  1024;

nginx.confのworker_rlimit_nofileを設定．

反映

[root@benchnginx logs]# /usr/local/nginx-1.7.7/sbin/nginx -s reload

[root@benchnginx logs]# ps -aefww | grep worker
nginx     8856  6447  2 19:18 ?        00:00:14 nginx: worker process
root      9929  1928  0 19:30 pts/0    00:00:00 grep worker

[root@benchnginx logs]# cat /proc/8856/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             30507                30507                processes
Max open files            102400               102400               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       30507                30507                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

反映されたことを確認できた．

同時接続数 1200 でエラー発生

[root@client ~]# ab -c 1200 -n 12000 'http://10.100.47.58/10k.dat'

<<< snip >>>

Complete requests:      12000
Failed requests:        12000
   (Connect: 0, Receive: 0, Length: 11840, Exceptions: 160)

<<< snip >>>

nginx のエラーログ

[root@benchnginx logs]# grep -v ' 200 ' access.log | tail

※出力なし

messages

[root@benchnginx logs]# tail /var/log/messages

※出力なし

ListenOverflows

[root@benchnginx logs]# cat /proc/net/netstat |
> awk '
>     {
>         if(NR == 1) label = $21
>         if(NR == 2) value = $21
>     }
>     END {
>         printf "%s: %d\n", label, value
>     }
> '
ListenOverflows: 1742

ListenOverflowsが出てる → 接続リクエストを受けきれなくなった模様．

設定変更

@@ -11,7 +11,7 @@
 worker_rlimit_nofile  4096;

 events {
-    worker_connections  1024;
+    worker_connections  10240;
 }

worker_connectionsの値を大きくした．
- このディレクティブは worker あたりの接続数だと思っていたけど，どうも nginx 全体の接続数っぽいので要検証．

同時接続数 10000 に到達

[root@client ~]# ab -c 10000 -n 100000 'http://10.100.47.58/10k.dat'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.100.47.58 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/1.7.7
Server Hostname:        10.100.47.58
Server Port:            80

Document Path:          /10k.dat
Document Length:        10240 bytes

Concurrency Level:      10000
Time taken for tests:   14.741 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      1078278186 bytes
HTML transferred:       1052579436 bytes
Requests per second:    6783.58 [#/sec] (mean)
Time per request:       1474.148 [ms] (mean)
Time per request:       0.147 [ms] (mean, across all concurrent requests)
Transfer rate:          71431.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  900 1691.1    370    9572
Processing:    15  458 202.6    492    1500
Waiting:        5  245 148.9    243    1465
Total:         18 1358 1719.1    876   10424

Percentage of the requests served within a certain time (ms)
  50%    876
  66%    904
  75%   1176
  80%   1212
  90%   3758
  95%   4126
  98%   9617
  99%   9695
 100%  10424 (longest request)

この結果がチューニングした時の基準になりそう．

少しだけ最適化

@@ -1,6 +1,8 @@

 #user  nobody;
-worker_processes  1;
+worker_processes  4;
+
+worker_cpu_affinity 1000 0100 0010 0001;

 #error_log  logs/error.log;
 #error_log  logs/error.log  notice;

CPU コア数に合わせた設定にしてみた．

最適化後の結果

[root@client ~]# ab -c 10000 -n 100000 'http://10.100.47.58/10k.dat'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.100.47.58 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/1.7.7
Server Hostname:        10.100.47.58
Server Port:            80

Document Path:          /10k.dat
Document Length:        10240 bytes

Concurrency Level:      10000
Time taken for tests:   12.525 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      1077834884 bytes
HTML transferred:       1052139634 bytes
Requests per second:    7983.89 [#/sec] (mean)
Time per request:       1252.523 [ms] (mean)
Time per request:       0.125 [ms] (mean, across all concurrent requests)
Transfer rate:          84036.23 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       78  517 678.3    382    3559
Processing:   125  696 158.7    699    1718
Waiting:      108  359 113.3    358    1502
Total:        279 1213 695.2   1116    4340

Percentage of the requests served within a certain time (ms)
  50%   1116
  66%   1162
  75%   1180
  80%   1185
  90%   1210
  95%   3757
  98%   4075
  99%   4156
 100%   4340 (longest request)

最適化前後の比較

最適化前10回の平均値

Requests per second:    6706.79 [#/sec] (mean)

Connection Times (ms)
               mean[+/-sd] median max
Connect:        804 1574.1 354     9406
Processing:     510  231.3 561     1916
Waiting:        272  146.0 284     1617
Total:         1313 1617.7 958    10715

最適化後10回の平均値

Requests per second:    8700.57 [#/sec] (mean)

Connection Times (ms)
               mean[+/-sd] median max
Connect:        413 319.8   375   3322
Processing:     695 179.3   688   2732
Waiting:        350 130.7   343   2281
Total:         1108 377.4  1093   4489

結果を見比べてみると
- 一秒当たりに処理できるリクエスト数が約30%向上した．
- 接続にかかる時間が大幅に短縮された．
  - 複数 worker で処理することにより待ち時間が減ったためと思われる．
  - これにより一番時間が掛かったリクエストも 10715ms から 4489ms になっている．
- Processing と Waiting では逆に若干の性能劣化が見られる．
  - 複数 worker へ処理を振り分ける分のオーバーヘッドかな？
  - 標準偏差は若干良くなっているので処理の安定性は高そう．

noob

底辺オペレーターのメモ

nginx 1.7.7 のベンチマーク

ベンチマーク用nginxのコンパイルオプション

チューニング

同時接続数 600 でエラー発生

nginx のエラーログ

制限値を確認してみる

設定変更

反映

同時接続数 1200 でエラー発生

nginx のエラーログ

messages

ListenOverflows

設定変更

同時接続数 10000 に到達

少しだけ最適化

最適化後の結果

最適化前後の比較