2016ǯ06��03��

pixiv private isucon 2016 ��ά (5/5)

¤Ï¤Æ¤Ê¥Ö¥Ã¥¯¥Þ¡¼¥¯¤ËÅÐÏ¿

����������:

�ܥȥ�ͥå�õ��

����ϰ��ֽŤ��ä��ȥåץڡ����˥ڡ�������å����Ƴ�����Ʒ�Ū�˹�®��������ΤΡ�CPU�ʳ����ܥȥ�ͥå��ˤʤä��礷����®���ϤǤ��ޤ���Ǥ����� ������������ dstat �����Ω�Ĥ��Ȥ�¿���Ǥ���

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  2   0  98   0   0   0| 596k  271k|   0     0 |   0     0 | 606   755
  1   0 100   0   0   0|   0     0 | 192B 1190B|   0     0 | 119   177
  0   0 100   0   0   0|   0     0 |  66B  350B|   0     0 | 103   158
  1   0  98   1   0   0| 512k    0 | 537B  783B|   0     0 | 962  1035
 48   7  38   3   0   5|  16M  728k|4590k   60M|   0     0 |  24k   20k
 44   7  36   5   0   8|  11M  376k|6568k   71M|   0     0 |  34k   26k
 44  11  35   4   0   7|  18M  312k|6190k   71M|   0     0 |  33k   27k
 48   8  33   3   0   8|  14M  272k|7784k   71M|   0     0 |  31k   24k
 41   7  42   4   0   6|  17M  352k|8097k   72M|   0     0 |  31k   23k
 44   9  38   4   1   5|  14M  256k|8019k   71M|   0     0 |  33k   25k
 44  10  35   3   0   8|  10M  448k|7705k   71M|   0     0 |  32k   25k
 36   9  46   3   0   6|  14M  224k|7196k   71M|   0     0 |  32k   22k
 38  10  39   6   0   7|  16M   18M|7812k   71M|   0     0 |  31k   22k
 47   9  28  11   0   4|  20M   47M|6683k   72M|   0     0 |  29k   19k
 45   7  39   3   0   6|  10M  696k|6113k   71M|   0     0 |  33k   24k
 42   7  39   4   0   8|  12M  672k|3825k   71M|   0     0 |  33k   24k
 51   8  31   3   0   6|  13M  512k|6750k   71M|   0     0 |  31k   22k
 44   9  39   2   0   5|5496k  648k|8045k   71M|   0     0 |  32k   24k
 53   9  28   3   0   6|  20M  544k|5832k   71M|   0     0 |  29k   20k
 41   7  42   3   0   7|  14M  616k|7459k   71M|   0     0 |  35k   24k
 48   9  31   4   0   8|  17M  608k|5869k   71M|   0     0 |  30k   23k
 38  11  40   3   0   8|  14M  752k|8956k   71M|   0     0 |  32k   24k
 44  11  36   4   0   5|  13M  576k|7181k   71M|   0     0 |  33k   25k
 47   7  35   7   0   5|  14M   62M|5753k   65M|   0     0 |  30k   22k
...

net �� send �� 71M ��ĥ���դ��Ƥ�Τ��ܤˤĤ��ޤ��͡����ץꥵ���С��� c4.large �ʤΤǤ���ʤ��Ǥ��礦��

�Ǥ����� pixiv ����� Blog ���� �Υ����꡼�󥷥�åȤ򸫤�ȡ�¿���Υ����ब 40k ����Ķ���Ƥ��ơ��ȥåץ������180k������ã���Ƥ��ޤ������ä���40k���μ������Ӱ���ɤˤ֤Ĥ��äƤ�Τˡ��Ѥ����� (Twitter �� pixiv �� @catatsuy ����ˡ��ݡ�����Υ������ȥ��ޥ�ɥ饤��ˤ��٥���ޡ������Υ�������Ʊ��Ǥ��뤳�Ȥ��ǧ���ޤ�����)

���ơ�2���ܤε����ǡ����Τ褦�˸��äƤ��ޤ����͡�

�Ӱ�� /image/ ������Ū���Ӱ�ͥå��ˤʤ�褦���ä��顢�����Ϻư��̤��ˤ����Τǡ��إå�����Ŭ�ڤ����ꤹ��Х��饤����Ȥ� "304 Not Modified" ���֤���褦�ˤʤä��礭�ʥ֥졼�����롼�����뤫�⤷��ʤ���

��˥��饤����ȤΥ���å�������˱ƶ���Ϳ�������ʥإå������ɲä��Ƥߤޤ��礦�� "nginx ��Ū�ե����� ����å���" �ǥ����äƤ���äݤ��Τ�õ���ޤ�

  upstream app {
    server 127.0.0.1:8080;
    keepalive 64;
  }

server {
  listen 80;

  client_max_body_size 10m;
  root /home/isucon/private_isu/webapp/public/;

  location / {
    try_files $uri @app;
  }
  location ~* \.(?:ico|css|js|gif|jpe?g|png)$ {
    try_files $uri @app;
    expires max;
    add_header Pragma public;
    add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    etag off;
  }
  location @app {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://app;
  }
}

��¬���Ƥߤޤ���

������:

{"pass":true,"score":182435,"success":174747,"fail":0,"messages":[]}

pixiv ���� isucon �Υȥåפ�Ʊ�����餤�Υ��������Ǥޤ�����(����ϥץ��ե�������ä��ꤤ���������ʤ���Υ������ʤΤǡ��ž夲������⤦�������Ӥ�Ϥ��Ǥ�)

top, dstat:

Tasks:  81 total,   3 running,  78 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.7 us, 11.5 sy,  0.0 ni,  3.0 id,  0.5 wa,  0.0 hi,  4.2 si,  0.0 st
KiB Mem:   1022972 total,   954144 used,    68828 free,    25432 buffers
KiB Swap:        0 total,        0 used,        0 free.   557948 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1397 isucon    20   0  377096  97572   8880 S 100.5  9.5   0:27.35 app.dc4a3eaa
  731 mysql     20   0 1059292 183956  10552 S  65.9 18.0   2:26.08 mysqld
 1396 www-data  20   0   92088   5304   3144 R  18.6  0.5   0:04.99 nginx
 1069 isucon    20   0   33140  10476   2636 S   5.7  1.0   0:12.10 tmux
  308 memcache  20   0  334624   9744   2228 S   0.7  1.0   0:01.38 memcached
  110 root      20   0       0      0      0 S   0.3  0.0   0:00.08 jbd2/xvda2-8
 1034 root      20   0       0      0      0 S   0.3  0.0   0:02.89 kworker/1:0
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  45    76 
 21   5  71   2   0   1|  11M    0 |1987k   12M|   0     0 |8731  9184 
 73  15   5   2   0   5|  17M  904k|4574k   25M|   0     0 |  19k   20k
 78  13   3   1   0   6|  20M  272k|3815k   18M|   0     0 |  15k   17k
 80   9   6   2   0   4|  25M  272k|7221k   20M|   0     0 |  16k   17k
 81  11   4   1   0   4|  18M  272k|5953k   20M|   0     0 |  15k   16k
 81  10   4   0   0   5|  24M  256k|5363k   24M|   0     0 |  15k   17k
 76  13   6   2   0   4|  20M  272k|8293k   18M|   0     0 |  16k   17k
 82  11   2   1   0   5|  15M  552k|2665k   18M|   0     0 |  16k   16k
 80  14   3   1   0   4|  14M  408k|2870k   15M|   0     0 |  15k   16k
 79  12   5   1   0   4|  16M  360k|5475k   19M|   0     0 |  17k   18k
 79  11   5   2   0   3|  16M   65M|5456k   21M|   0     0 |  15k   16k
 77  12   6   1   0   4|  19M  760k|6763k   23M|   0     0 |  19k   20k
 79  12   4   2   0   4|  23M  592k|6824k   19M|   0     0 |  16k   17k
 79  12   5   2   0   3|  11M  632k|5994k   20M|   0     0 |  16k   18k
 82  11   3   1   0   4|  11M  680k|3447k   16M|   0     0 |  15k   17k
 83  11   2   0   0   4|  13M  792k|5076k   18M|   0     0 |  16k   18k
...

CPU��Ȥ��ڤ�褦�ˤʤä��Τ�Ƚ��ޤ���

access.log ����:

Request by count
117444 GET /image/*
7987 GET /
6394 GET /js/jquery-2.2.0.js
6394 GET /css/style.css
6394 GET /js/jquery.timeago.ja.js
6394 GET /favicon.ico
6394 GET /js/jquery.timeago.js
6394 GET /js/main.js
4028 POST /login
1459 GET /login
1176 GET /posts/*
959 POST /register
958 GET /admin/banned
873 GET /@user
731 GET /logout
609 POST /
330 GET /posts?max_created_at=
268 POST /comment
1 GET /initialize

Request by total time
98.996 0.299987878788 GET /posts?max_created_at=
84.66 0.0105997245524 GET /
84.077 0.0963081328751 GET /@user
52.172 0.0129523336643 POST /login
39.666 0.0337295918367 GET /posts/*
33.621 0.0552068965517 POST /
19.801 0.0738843283582 POST /comment
17.144 0.0117505140507 GET /login
...

Request by out bytes
894159053 7613 GET /image/*
185613643 23239 GET /
16456443 18850 GET /@user
11202870 33948 GET /posts?max_created_at=
4098331 3484 GET /posts/*
...

image ��ž���̤������˸��ä����Ȥ���ǧ�Ǥ��ޤ����ޤ������פ����� access.log �򸫤�ȡ������̤� 304 ���֤��Ƥ��뤳�Ȥ�Ƚ��ޤ���

(;��) �ɤ�Ķ����

2���ܤε����� 304 Not Modified ���֤��뤳�Ȥ�ͽ�ۤ��Ƥ����Τϡ������ѡ��Ǥ�ʤ�Ǥ⤢��ޤ��󡣰��ǯ (2014ǯ) �η辡�Ǥ��ä���Ǥ������λ��Ϥ����ɤ�Ķ����줺�������פ��򤷤ޤ�����

���λ����ɤ�Ķ���������ब2�����ढ��ޤ�������ͥ�����������Ū�ե�������֤������ nginx ������Υƥ�ץ�ʤ�����������Υ���סˤ�Ȥä��餤���ʤꥹ�������夬�ä����ɲ��������ä��Τ����ꤷ���ܿͤˤ�ʬ����ʤ��ä���������ͥ��������Ͻ�ͥ��������Υ������򸫤ơ�ɬ���ɤ�Ķ������ˡ������Ϥ����פȻ�Ժ�����������ɤ�Ķ�����ȵ������Ƥ��ޤ����ʶ�̣���������������� blog ������õ���ƤߤƤ���������

�������Υ���פ������ϥޤ롢�Ȥ����ɤ�Ķ�����Ͻ����λ�ʪ�Ǥ��� �Ŀ�Ū�ˤϥ��塼�˥󥰤���Ȥ��ϥѥ�᡼������1��1�Ĥ����äƸ��̳�ǧ�������Ȥ����Ǥ�����������Ȼ��֤���ä����ʤ������٥���ޡ������ε�ư�ˤĤ��Ƥ�ͽ�ۤ����Ƥ��ʤ����ɤ�Ķ�����ʤ��Τǡ���Ϥ�������Υ���פ϶��Ϥ����ˤʤ�ޤ���

�������٥���ޡ�������ư���ͽ�ۤ���Ժ������������ϡ��������¦�ο������ɤ� ISUCON �ϡ���ǽ���Τ����Ժ����򤹤뤿��ΰ����Ф�(����Ǥ����� HTTP �إå����ȥ���å���˴ؤ����μ�)�������Ϥʤɤ������ޤ��� ���˷辡��ͥ������������Ϥɤ�⤳��ǽ�Ϥ��ä�ͥ��Ƥ�����������Ȼפ��ޤ����񤷤��Ǥ���ͥ�����ܻؤ����ٶ�����ϣ�򤷤ޤ��礦��

���ҹʤ�

CPU�ͥå��ˤʤä����ȤǤ�����CPU�ץ��ե�����򸫤Ƥߤޤ��礦��

(pprof) top30 -cum
7.63s of 29.87s total (25.54%)
Dropped 601 nodes (cum <= 0.15s)
Showing top 30 nodes out of 296 (cum >= 4.36s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%     28.68s 96.02%  runtime.goexit
         0     0%     0%     24.58s 82.29%  net/http.(*conn).serve
         0     0%     0%     23.35s 78.17%  net/http.(*ServeMux).ServeHTTP
         0     0%     0%     23.35s 78.17%  net/http.serverHandler.ServeHTTP
         0     0%     0%     23.32s 78.07%  github.com/zenazn/goji/web.(*Mux).ServeHTTP
         0     0%     0%     23.30s 78.00%  github.com/zenazn/goji/web.(*cStack).ServeHTTP
         0     0%     0%     23.30s 78.00%  github.com/zenazn/goji/web/middleware.RequestID.func1
         0     0%     0%     23.30s 78.00%  net/http.HandlerFunc.ServeHTTP
         0     0%     0%     23.25s 77.84%  github.com/zenazn/goji/web/middleware.Logger.func1
         0     0%     0%     22.71s 76.03%  github.com/zenazn/goji/web.(*mStack).newStack.func1
     0.01s 0.033% 0.033%     22.71s 76.03%  github.com/zenazn/goji/web.(*router).route
         0     0% 0.033%     22.71s 76.03%  github.com/zenazn/goji/web/middleware.AutomaticOptions.func1
         0     0% 0.033%     22.71s 76.03%  github.com/zenazn/goji/web/middleware.Recoverer.func1
     0.01s 0.033% 0.067%     18.11s 60.63%  github.com/zenazn/goji/web.netHTTPHandlerFuncWrap.ServeHTTPC
         0     0% 0.067%     12.85s 43.02%  main.getPosts
         0     0% 0.067%     12.46s 41.71%  github.com/jmoiron/sqlx.(*DB).Select
         0     0% 0.067%     12.46s 41.71%  github.com/jmoiron/sqlx.Select
     0.07s  0.23%   0.3%     11.69s 39.14%  github.com/jmoiron/sqlx.scanAll
     1.72s  5.76%  6.06%      9.78s 32.74%  runtime.mallocgc
     0.16s  0.54%  6.60%      8.62s 28.86%  runtime.systemstack
     5.02s 16.81% 23.40%      7.29s 24.41%  runtime.scanobject
     0.15s   0.5% 23.90%      5.91s 19.79%  runtime.newobject
         0     0% 23.90%      5.33s 17.84%  runtime.gcAssistAlloc
         0     0% 23.90%      5.33s 17.84%  runtime.gcAssistAlloc.func1
     0.04s  0.13% 24.04%      5.33s 17.84%  runtime.gcDrainN
     0.04s  0.13% 24.17%      5.22s 17.48%  database/sql.(*Rows).Next
     0.03s   0.1% 24.27%      5.04s 16.87%  github.com/go-sql-driver/mysql.(*textRows).Next
     0.34s  1.14% 25.41%      5.01s 16.77%  github.com/go-sql-driver/mysql.(*textRows).readRow
         0     0% 25.41%      4.55s 15.23%  github.com/zenazn/goji/web.handlerFuncWrap.ServeHTTPC
     0.04s  0.13% 25.54%      4.36s 14.60%  main.makePosts


(pprof) list getPosts
Total: 29.87s
ROUTINE ======================== main.getPosts in /home/isucon/private_isu/webapp/golang/app.go
         0     12.85s (flat, cum) 43.02% of Total
         .          .    593:           fmt.Println(terr)
         .          .    594:           return
         .          .    595:   }
         .          .    596:
         .          .    597:   results := []Post{}
         .     11.19s    598:   rerr := db.Select(&results, "SELECT `id`, `user_id`, `body`, `mime`, `created_at` FROM `posts` WHERE `created_at` <= ? ORDER BY `created_at` DESC", t.Format(ISO8601_FORMAT
))
         .          .    599:   if rerr != nil {
         .          .    600:           fmt.Println(rerr)
         .          .    601:           return
         .          .    602:   }
         .          .    603:
         .      1.03s    604:   posts, merr := makePosts(results, getCSRFToken(r), false)
         .          .    605:   if merr != nil {
         .          .    606:           fmt.Println(merr)
         .          .    607:           return
         .          .    608:   }
         .          .    609:
         .          .    610:   if len(posts) == 0 {
         .          .    611:           w.WriteHeader(http.StatusNotFound)
         .          .    612:           return
         .          .    613:   }
         .          .    614:
         .          .    615:   fmap := template.FuncMap{
         .          .    616:           "imageURL": imageURL,
         .          .    617:   }
         .          .    618:
         .          .    619:   template.Must(template.New("posts.html").Funcs(fmap).ParseFiles(
         .       10ms    620:           getTemplPath("posts.html"),
         .          .    621:           getTemplPath("post.html"),
         .      620ms    622:   )).Execute(w, posts)
         .          .    623:}
         .          .    624:
         .          .    625:func getPostsID(c web.C, w http.ResponseWriter, r *http.Request) {
         .          .    626:   pid, err := strconv.Atoi(c.URLParams["id"])
         .          .    627:   if err != nil {

(pprof) list makePosts
Total: 29.87s
ROUTINE ======================== main.makePosts in /home/isucon/private_isu/webapp/golang/app.go
      40ms      4.36s (flat, cum) 14.60% of Total
         .          .    188:}
         .          .    189:
         .          .    190:func makePosts(results []Post, CSRFToken string, allComments bool) ([]Post, error) {
         .          .    191:   var posts []Post
         .          .    192:
         .       20ms    193:   for _, p := range results {
         .      720ms    194:           err := db.Get(&p.CommentCount, "SELECT COUNT(*) AS `count` FROM `comments` WHERE `post_id` = ?", p.ID)
         .          .    195:           if err != nil {
         .          .    196:                   return nil, err
         .          .    197:           }
         .          .    198:
         .          .    199:           query := "SELECT * FROM `comments` WHERE `post_id` = ? ORDER BY `created_at` DESC"
         .          .    200:           if !allComments {
         .          .    201:                   query += " LIMIT 3"
         .          .    202:           }
         .          .    203:           var comments []Comment
         .      920ms    204:           cerr := db.Select(&comments, query, p.ID)
         .          .    205:           if cerr != nil {
         .          .    206:                   return nil, cerr
         .          .    207:           }
         .          .    208:
         .          .    209:           for i := 0; i < len(comments); i++ {
      20ms      1.89s    210:                   uerr := db.Get(&comments[i].User, "SELECT * FROM `users` WHERE `id` = ?", comments[i].UserID)
         .          .    211:                   if uerr != nil {
         .          .    212:                           return nil, uerr
         .          .    213:                   }
         .          .    214:           }
         .          .    215:
         .          .    216:           // reverse
         .          .    217:           for i, j := 0, len(comments)-1; i < j; i, j = i+1, j-1 {
         .          .    218:                   comments[i], comments[j] = comments[j], comments[i]
         .          .    219:           }
         .          .    220:
         .          .    221:           p.Comments = comments
         .          .    222:
      20ms      780ms    223:           perr := db.Get(&p.User, "SELECT * FROM `users` WHERE `id` = ?", p.UserID)
         .          .    224:           if perr != nil {
         .          .    225:                   return nil, perr
         .          .    226:           }
         .          .    227:
         .          .    228:           p.CSRFToken = CSRFToken
         .          .    229:
         .          .    230:           if p.User.DelFlg == 0 {
         .       30ms    231:                   posts = append(posts, p)
         .          .    232:           }
         .          .    233:           if len(posts) >= postsPerPage {
         .          .    234:                   break
         .          .    235:           }
         .          .    236:   }

getPosts ���Ť��Τϡ����� getIndex ���Ť��Τ�ľ��������Ʊ������ LIMIT ������ɤ������Ǥ��͡�

makePosts �⥯������ꤲ�����Ƥ���Τǡ� Post �μ����Ȥ��� User �μ����� INNER JOIN �ǰ쵤�ˤ��롢�����Ȱ����ϥ����˥���å��夷�Ƥ��ޤ����Ȥˤ��ޤ��� ���ȥƥ�ץ졼�ȤΥ���ѥ��������ˤ���褦�ˤ��Ƥ����ޤ��� Go ���ץ�¦�ǽ��Ϥ��Ƥ�������� tmux �� ssh �� CPU �򿩤äƤ�ΤǤ��������ߤ�Ƥ��ޤ��ޤ��礦�� (nginx �Υ�����ߤ��Τϰ��ֺǸ�Ǥ�)

diff --git a/app.go b/app.go
index f246ad6..eca8413 100644
--- a/app.go
+++ b/app.go
@@ -187,54 +184,78 @@ func getFlash(w http.ResponseWriter, r *http.Request, key string) string {
 	}
 }
 
-func makePosts(results []Post, CSRFToken string, allComments bool) ([]Post, error) {
-	var posts []Post
+var (
+	commentM     sync.Mutex
+	commentStore map[int][]Comment = make(map[int][]Comment)
+)
 
-	for _, p := range results {
-		err := db.Get(&p.CommentCount, "SELECT COUNT(*) AS `count` FROM `comments` WHERE `post_id` = ?", p.ID)
+func getCommentsLocked(postID int) []Comment {
+	if cs, ok := commentStore[postID]; ok {
+		return cs
+	}
+
+	var cs []Comment
+	query := ("SELECT comments.id, comments.comment, comments.created_at, users.id, users.account_name " +
+		" FROM `comments` INNER JOIN users ON comments.user_id = users.id " +
+		" WHERE `post_id` = ? ORDER BY comments.`created_at`")
+
+	rows, err := db.Query(query, postID)
+	if err != nil {
+		log.Println(err)
+		return cs
+	}
+	for rows.Next() {
+		var c Comment
+		err := rows.Scan(&c.ID, &c.Comment, &c.CreatedAt, &c.User.ID, &c.User.AccountName)
 		if err != nil {
-			return nil, err
+			log.Println(err)
+			continue
 		}
+		cs = append(cs, c)
+	}
+	rows.Close()
 
-		query := "SELECT * FROM `comments` WHERE `post_id` = ? ORDER BY `created_at` DESC"
-		if !allComments {
-			query += " LIMIT 3"
-		}
-		var comments []Comment
-		cerr := db.Select(&comments, query, p.ID)
-		if cerr != nil {
-			return nil, cerr
-		}
+	commentStore[postID] = cs
+	return cs
+}
 
-		for i := 0; i < len(comments); i++ {
-			uerr := db.Get(&comments[i].User, "SELECT * FROM `users` WHERE `id` = ?", comments[i].UserID)
-			if uerr != nil {
-				return nil, uerr
-			}
-		}
+func getComments(postID int) []Comment {
+	commentM.Lock()
+	defer commentM.Unlock()
+	return getCommentsLocked(postID)
+}
 
-		// reverse
-		for i, j := 0, len(comments)-1; i < j; i, j = i+1, j-1 {
-			comments[i], comments[j] = comments[j], comments[i]
-		}
+func appendComent(c Comment) {
+	commentM.Lock()
+	cs := getCommentsLocked(c.PostID)
+	commentStore[c.PostID] = append(cs, c)
+	commentM.Unlock()
+}
+
+func makePosts(results []Post, CSRFToken string, allComments bool) ([]Post, error) {
+	var posts []Post
 
+	for _, p := range results {
+		comments := getComments(p.ID)
+		if !allComments && len(comments) > 3 {
+			comments = comments[len(comments)-3:]
+		}
 		p.Comments = comments
+		p.CSRFToken = CSRFToken
 
 		perr := db.Get(&p.User, "SELECT * FROM `users` WHERE `id` = ?", p.UserID)
 		if perr != nil {
 			return nil, perr
 		}
 
-		p.CSRFToken = CSRFToken
-
-		if p.User.DelFlg == 0 {
-			posts = append(posts, p)
+		if p.User.DelFlg != 0 {
+			continue
 		}
+		posts = append(posts, p)
 		if len(posts) >= postsPerPage {
 			break
 		}
 	}
-
 	return posts, nil
 }
 
@@ -411,8 +432,9 @@ func getLogout(w http.ResponseWriter, r *http.Request) {
 }
 
 var (
-	indexTemplate *template.Template
-	postsTemplate *template.Template
+	indexTemplate       *template.Template
+	postsTemplate       *template.Template
+	accountNameTempalte *template.Template
 
 	indexPostsM         sync.Mutex
 	indexPostsT         time.Time
@@ -434,6 +456,13 @@ func init() {
 		getTemplPath("posts.html"),
 		getTemplPath("post.html"),
 	))
+
+	accountNameTempalte = template.Must(template.New("layout.html").Funcs(fmap).ParseFiles(
+		getTemplPath("layout.html"),
+		getTemplPath("user.html"),
+		getTemplPath("posts.html"),
+		getTemplPath("post.html"),
+	))
 }
 
 func renderIndexPosts() {
@@ -506,7 +535,6 @@ func getAccountName(c web.C, w http.ResponseWriter, r *http.Request) {
 	}
 
 	results := []Post{}
-
 	rerr := db.Select(&results, "SELECT `id`, `user_id`, `body`, `mime`, `created_at` FROM `posts` WHERE `user_id` = ? ORDER BY `created_at` DESC", user.ID)
 	if rerr != nil {
 		fmt.Println(rerr)
@@ -527,12 +555,10 @@ func getAccountName(c web.C, w http.ResponseWriter, r *http.Request) {
 	}
 
 	postIDs := []int{}
-	perr := db.Select(&postIDs, "SELECT `id` FROM `posts` WHERE `user_id` = ?", user.ID)
-	if perr != nil {
-		fmt.Println(perr)
-		return
+	for _, r := range results {
+		postIDs = append(postIDs, r.ID)
 	}
-	postCount := len(postIDs)
+	postCount := len(results)
 
 	commentedCount := 0
 	if postCount > 0 {
@@ -557,16 +583,7 @@ func getAccountName(c web.C, w http.ResponseWriter, r *http.Request) {
 
 	me := getSessionUser(r)
 
-	fmap := template.FuncMap{
-		"imageURL": imageURL,
-	}
-
-	template.Must(template.New("layout.html").Funcs(fmap).ParseFiles(
-		getTemplPath("layout.html"),
-		getTemplPath("user.html"),
-		getTemplPath("posts.html"),
-		getTemplPath("post.html"),
-	)).Execute(w, struct {
+	accountNameTempalte.Execute(w, struct {
 		Posts          []Post
 		User           User
 		PostCount      int
@@ -595,7 +612,7 @@ func getPosts(w http.ResponseWriter, r *http.Request) {
 	}
 
 	results := []Post{}
-	rerr := db.Select(&results, "SELECT `id`, `user_id`, `body`, `mime`, `created_at` FROM `posts` WHERE `created_at` <= ? ORDER BY `created_at` DESC", t.Format(ISO8601_FORMAT))
+	rerr := db.Select(&results, "SELECT posts.`id`, `user_id`, `body`, `mime`, posts.`created_at` FROM `posts` INNER JOIN `users` ON posts.user_id=users.id WHERE users.del_flg = 0 AND posts.`created_at` <= ? ORDER BY `created_at` DESC LIMIT 20", t.Format(ISO8601_FORMAT))
 	if rerr != nil {
 		fmt.Println(rerr)
 		return
@@ -806,9 +823,24 @@ func postComment(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	query := "INSERT INTO `comments` (`post_id`, `user_id`, `comment`) VALUES (?,?,?)"
-	db.Exec(query, postID, me.ID, r.FormValue("comment"))
-
+	now := time.Now()
+	commentStr := r.FormValue("comment")
+	query := "INSERT INTO `comments` (`post_id`, `user_id`, `comment`, `created_at`) VALUES (?,?,?,?)"
+	res, err := db.Exec(query, postID, me.ID, commentStr, now)
+	if err != nil {
+		log.Println(err)
+		return
+	}
+	lid, _ := res.LastInsertId()
+	c := Comment{
+		ID:        int(lid),
+		PostID:    postID,
+		UserID:    me.ID,
+		Comment:   commentStr,
+		CreatedAt: now,
+		User:      me,
+	}
+	appendComent(c)
 	renderIndexPosts()
 	http.Redirect(w, r, fmt.Sprintf("/posts/%d", postID), http.StatusFound)
 }
@@ -921,6 +953,7 @@ func main() {
 
 	go http.ListenAndServe(":3000", nil)
 
+	goji.DefaultMux = web.New()
 	goji.Get("/initialize", getInitialize)
 	goji.Get("/login", getLogin)
 	goji.Post("/login", postLogin)

��¬

������:

{"pass":true,"score":271542,"success":256436,"fail":0,"messages":[]}

pixiv ���� isucon ��ͥ��������Υ�������Ķ���뤳�Ȥ��Ǥ����Ȼפ��ޤ���

top:

ks:  82 total,   4 running,  78 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57.1 us, 13.8 sy,  0.0 ni, 16.5 id,  5.1 wa,  0.0 hi,  7.3 si,  0.2 st
KiB Mem:   1022972 total,   926504 used,    96468 free,    27152 buffers
KiB Swap:        0 total,        0 used,        0 free.   432620 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                 1494 isucon    20   0  544796 191292   8940 S  81.5 18.7   0:29.06 app.7f063c1
  731 mysql     20   0 1125024 182444  10552 S  44.9 17.8   3:04.05 mysqld
 1524 www-data  20   0   92056   5276   3140 R  27.6  0.5   0:09.65 nginx
  308 memcache  20   0  336672  11756   2228 S   1.7  1.1   0:02.25 memcached
   31 root      20   0       0      0      0 S   0.7  0.0   0:00.40 kswapd0
   89 root       0 -20       0      0      0 S   0.3  0.0   0:00.73 kworker/0:1H
    1 root      20   0   28636   4736   3100 S   0.0  0.5   0:01.31 systemd

����ä� idle ���ФƤ������ʡ�

dstat:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  82   283 
 11   3  82   2   0   2|  12M    0 |1684k   15M|   0     0 |7607  5952 
 55  13  17   9   0   7|  49M  864k|  11M   44M|   0     0 |  27k   20k
 58  15  14   8   0   6|  48M  416k|  13M   43M|   0     0 |  28k   20k
 48  14  21  12   0   5|  43M   63M|  12M   40M|   0     0 |  25k   19k
 55  17  14   7   0   7|  48M  528k|  14M   42M|   0     0 |  25k   18k
 55  12  19   8   0   6|  45M  488k|  12M   42M|   0     0 |  27k   20k
 52  17  16   6   0   8|  35M  592k|  14M   38M|   0     0 |  27k   19k
 50  12  19  14   0   6|  34M   59M|  11M   38M|   0     0 |  25k   17k
 56  12  18   6   0   7|  42M  720k|  11M   45M|   0     0 |  28k   20k
 51  15  18   9   0   8|  45M  704k|  14M   39M|   0     0 |  26k   19k
 52  17  17   7   0   7|  39M 3936k|  14M   45M|   0     0 |  27k   19k
 47  17  18  13   0   5|  41M   56M|  12M   39M|   0     0 |  24k   17k
 53  13  19   9   0   6|  45M  848k|  13M   41M|   0     0 |  27k   20k
 57  15  16   6   0   6|  38M  776k|  13M   42M|   0     0 |  27k   19k
 53  14  20   9   0   4|  28M   58M|  11M   29M|   0     0 |  24k   16k
 53  13  19   6   0   8|  44M  672k|  13M   38M|   0     0 |  26k   18k
 57  13  16   9   0   6|  46M  872k|  10M   38M|   0     0 |  25k   18k
 53  12  22   7   0   6|  44M  776k|  12M   41M|   0     0 |  28k   20k
 54  16  16   8   0   6|  33M   57M|9049k   33M|   0     0 |  24k   18k
 51  16  21   5   0   7|  32M  888k|  12M   38M|   0     0 |  27k   19k
 54  13  19   7   0   7|  49M 2264k|  11M   46M|   0     0 |  26k   18k
 54  14  19   6   0   7|  36M  680k|  12M   42M|   0     0 |  28k   19k
 48  16  19  10   0   7|  33M   52M|  14M   31M|   0     0 |  23k   16k
 55  15  15   7   0   7|  42M  720k|  12M   34M|   0     0 |  26k   20k
 59  14  17   5   0   6|  33M  744k|  12M   35M|   0     0 |  25k   18k
 47  15  25   7   0   6|  34M  736k|9604k   43M|   0     0 |  28k   19k
 47  11  27   8   0   6|  31M   51M|  12M   39M|   0     0 |  25k   17k
 54  18  17   5   0   7|  34M  816k|  12M   33M|   0     0 |  27k   20k
 55  16  16   6   0   6|  35M  888k|  14M   34M|   0     0 |  27k   19k
 57  13  18   6   0   6|  34M  720k|  11M   37M|   0     0 |  27k   19k
 52  12  18  12   0   6|  36M   53M|  11M   34M|   0     0 |  23k   17k
 62  12  15   4   0   7|  26M  720k|9309k   29M|   0     0 |  25k   19k
 46  20  19   5   0   9|  30M  560k|  16M   35M|   0     0 |  28k   20k
 59  12  20   5   0   5|  23M  544k|  10M   36M|   0     0 |  25k   17k
 53  13  21   8   0   5|  25M   47M|  11M   31M|   0     0 |  23k   16k
 54  14  18   6   0   9|  38M  584k|  14M   35M|   0     0 |  27k   21k
...

�Ӱ��Ⱦʬ���ȤäƤ��ޤ��͡��礭�� js ������Τǡ� `nginx �� gzip_static �⥸�塼���Ȥ��ʤɤǤ⤦�����Ӱ�򳫤�����Ȼפ��ޤ���

pprof:

(pprof) top30 -cum
6.95s of 22.25s total (31.24%)
Dropped 621 nodes (cum <= 0.11s)
Showing top 30 nodes out of 321 (cum >= 2.44s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%     20.30s 91.24%  runtime.goexit
     0.02s  0.09%  0.09%     17.03s 76.54%  net/http.(*conn).serve
         0     0%  0.09%     14.60s 65.62%  net/http.(*ServeMux).ServeHTTP
         0     0%  0.09%     14.60s 65.62%  net/http.serverHandler.ServeHTTP
         0     0%  0.09%     14.58s 65.53%  github.com/zenazn/goji/web.(*Mux).ServeHTTP
         0     0%  0.09%     14.57s 65.48%  github.com/zenazn/goji/web.(*cStack).ServeHTTP
         0     0%  0.09%     14.57s 65.48%  github.com/zenazn/goji/web.(*mStack).newStack.func1
         0     0%  0.09%     14.57s 65.48%  github.com/zenazn/goji/web.(*router).route
         0     0%  0.09%     14.57s 65.48%  net/http.HandlerFunc.ServeHTTP
         0     0%  0.09%     10.03s 45.08%  github.com/zenazn/goji/web.netHTTPHandlerFuncWrap.ServeHTTPC
     0.01s 0.045%  0.13%      6.12s 27.51%  html/template.(*Template).Execute
         0     0%  0.13%      5.58s 25.08%  text/template.(*Template).Execute
     0.12s  0.54%  0.67%      5.57s 25.03%  text/template.(*state).walk
     0.01s 0.045%  0.72%      5.20s 23.37%  text/template.(*state).walkTemplate
     0.04s  0.18%   0.9%      4.68s 21.03%  text/template.(*state).walkRange
     0.01s 0.045%  0.94%      4.65s 20.90%  text/template.(*state).walkRange.func1
     0.97s  4.36%  5.30%      4.60s 20.67%  runtime.mallocgc
         0     0%  5.30%      4.46s 20.04%  github.com/zenazn/goji/web.handlerFuncWrap.ServeHTTPC
     0.23s  1.03%  6.34%      4.38s 19.69%  runtime.systemstack
     0.04s  0.18%  6.52%      4.07s 18.29%  text/template.(*state).evalPipeline
     0.08s  0.36%  6.88%      4.03s 18.11%  text/template.(*state).evalCommand
     2.54s 11.42% 18.29%      3.71s 16.67%  runtime.scanobject
     0.10s  0.45% 18.74%      3.09s 13.89%  text/template.(*state).evalCall
     0.03s  0.13% 18.88%      3.07s 13.80%  text/template.(*state).evalFunction
         0     0% 18.88%      2.99s 13.44%  main.getAccountName
     2.48s 11.15% 30.02%      2.70s 12.13%  syscall.Syscall
         0     0% 30.02%      2.62s 11.78%  main.getPosts
     0.01s 0.045% 30.07%      2.56s 11.51%  reflect.Value.Call
     0.26s  1.17% 31.24%      2.52s 11.33%  reflect.Value.call
         0     0% 31.24%      2.44s 10.97%  github.com/jmoiron/sqlx.(*DB).Get

�ƥ�ץ졼�Ȥμ¹ԤȤ��Ť��Τȡ�����GC���Ť��Ǥ��͡�

�ƥ�ץ졼�Ȥ��Ť��Τϡ��ڡ�������å�����ȥȥåץڡ����ʳ��Ϥɤ줯�餤����å���ҥå�Ψ��������뤫�狼��ޤ��� valyala/quicktemplate �ʤɤΡ���ե쥯������¿�Ѥ��ʤ���®�ʥƥ�ץ졼�ȥ��󥸥�˾�괹����ۤ����ᤤ���⤷��ޤ���

GC�����ϡ�������¹Բ���򤵤�˺�äƤ����С�������֥������Ȥ���︺����Τ�ͭ���������ɤ��ǥ��������Ȥ�¿ȯ���Ƥ���Τ����ǧ���������� pprof -alloc_objects ��ȤäƤߤƤ���������

�ޤȤ�

������: 4745 (�������) -> 39743 (����) -> 182435 (Not Modified) -> 271542 (�ǽ���)

�ɤ�Ķ�������̤ǡ����ޤޤǿ���Ǻ��Ǥ������������쵤�˥����פ��ޤ�����

��ά���Ϥ����ޤǤˤ��ޤ������Ǥ���Ф���˥��塼�˥󥰤�³���ƤߤƤ��������������Ǽºݤ˼��ư�����Ƥ��ʤ����Ȥϡ����֤ǤϤۤܳμ¤ˤǤ��ޤ��� ��ξ��ϡ����Τ褦�ʥ��塼�˥󥰤�500k����Ķ���ޤ�����������Ǥ�ޤ��Ӱ��Ȥ��ڤäƤ��ʤ��� nginx ���� app �� MySQL �����Ѥ���CPU�������礭���ΤǤ���˾��������Ϥ��Ǥ���

  • gzip_static ��Ȥ� (�������Ͼ夬��ޤ���Ǥ��������Ӱ��;͵����ޤ���)
  • �ƥ�ץ졼�Ȥ��Ť� posts ����ʬ�� valyala/quicktemplate ��ȤäƤߤ�
  • �桼�����ò¥ª¥ï¿½ï¿½ï¿½ê¥­ï¿½ï¿½Ã¥ï¿½ï¿½å¤·ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½JOIN�����
  • ���å���󥹥ȥ��� memcached ���饪�������ڤ��ؤ���
  • nginx �� app �δ֤� unix domain socket�����ڤ��ؤ���
  • nginx ���Ť��ʤäƤ��Ƥ����Τ� worker_processes �� 2 �ˤ���
  • �����ϤΥꥯ�����ȤǤ����� time.Sleep() ���Ԥ����ơ��������ؤαƶ����礭�����ʥꥪ�ν����˥꥽�����������
  • ���������������ڤ� (�ǽ����Ū�ե�����������Ǹ�������Υ�������������ä�)

�����ޤǤΥ��ߥåȥ����� methane/pixiv-private-isucon-2016 ���֤��Ƥ���ޤ��� �Ǥ������Ǥ���������Ϥ� 500k ��Ķ�����ܻؤ��ƤߤƤ���������

�Ǹ�ˡ�ISUCON �������ˤȤƤ��ɤ������󶡤���ĺ���� pixiv ����˴��դ��ޤ���


@methane

songofacandy at 18:53��Comments(0)��TrackBack(0)��ISUCON | golang

�ȥ�å��Хå�URL

���ε����˥����Ȥ���

̾��:
URL:
  ����òµ­²ï¿½: ɾ��: ��    ��
 
 
 
Blog�⸡��
�ǿ�����
Archives
���Υ֥����ˤĤ���
DSAS�Ȥϡ�KLab �����ۤ����Ѥ��Ƥ��륳��ƥ�ĥ����ӥ��Ѥ�Linux�١����Υ���ե�Ǥ�������5����Υǡ������󥿤ˤƹ��ۤ������Ѥ��Ƥ��ޤ������桹��DSAS����Ȥ��䤹�����������ˡ������Ƥ����ϤDZ��ѤǤ��뤳�Ȥ��ܻؤ��ơ��������ɤ˶Ф���Ǥ��ޤ���
���Υ֥����Ǥϡ������ DSAS �ǻȤäƤ��뵻�ѤξҲ�䡢�¸����Ƥߤ���̤���𡢥ȥ�֥�˴������ޤ줿���ηи��̤ʤɡ���������������������򿥤�ޤ��ƾҲ𤷤Ƥ��������Ȼפ��ޤ���
�ǿ�������
<%==comments[n].author%>
<% } %>