pixiv thumbnails
15 likes5,538 views
- Pixiv is an illustration communication site with over 5 million users and 3.3 billion monthly page views. - The presenter works on infrastructure and software engineering at Pixiv, where he is responsible for image upload, thumbnail generation, data storage, caching strategies and more. - Pixiv generates 12 or more thumbnails for each image uploaded to optimize loading and browsing on different devices. With over 30 million images, this amounts to over 30 terabytes of thumbnails.
More Related Content
Similar to pixiv thumbnails (12)
More from Tatsuhiko Kubo (12)
pixiv thumbnails
- 1. pixiv thumbnails Tatsuhiko Kubo [email protected]
- 2. Introduction to pixiv ● Illust Communication Site ○ http://www.pixiv.net ● Users ○ 5 million ● Monthly page view ○ 3.3 billion ● Network Traffic ○ Over 6Gbps
- 3. About me ● Tatsuhiko Kubo(H.N：bokko) ● @cubicdaiya ● Infrastructure & Software Engineer@pixiv. Inc
- 4. My Works@pixiv. Inc Responsible for ● Middle-Ware Development ● Technical Operation & Administration ● Datastore & Caching Strategy ● Image Upload & Transformation ● Illust & User Recommendation ● Notification( called Popboard at pixiv) etc...
- 5. My works@private ■Software ● neoagent ○ A Yet Another Memcached Protocol Proxy Server ● dtl ○ diff template library with C++ ● ngx_small_light ○ Dynamic Transformation Module for Nginx ■Writing ● Software Design 2009 Sep ○ どのようにして差分を導き出すのか～diffの動作原理を知る～ ○ http://gihyo.jp/dev/column/01/prog/2011/diff_sd200906 And More -> http://cccis.jp
- 6. Agenda ● Following mechanism at pixiv ○ Image Upload ○ Thumbnail Generation
- 7. Scale of pixiv according to thumbnails ● pixiv has about 30,000,000 illusts and comics ● Each illust has about 12 ～ too many thumbnails ● 20,000 illusts and comics are uploaded every day ● Total volume is about 30TB
- 8. Image Upload
- 9. Image Upload Detail 1 ● Generate too many thumbnails ○ 12 ～ too many thumbnails ● Save a original image and thumbnails to storage ○ not NFS ○ with in-house WebDAV Client(ImageClient)
- 14. ImageClient ● Enable WebDAV operations to multiple servers transparently ○ put image ○ delete image ○ move image ○ make directory PUT ImageClient ○ move directory DELETE MOVE PUT DELETE ○ delete directory MKCOL MOVE MKCOL Image Image Storage ・・・・・ Storage
- 15. Image Upload Detail 2 ● A image is uploaded semi-asynchronously ○ Generating thumbnails takes a long time
- 16. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
- 17. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
- 18. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
- 19. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
- 20. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted
- 21. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action
- 22. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork server
- 23. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker upload worker ・ ・ ・ ・ ・ upload worker
- 24. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails upload worker Genrate Thumbnails ・ ・ ・ ・ ・ upload worker Genrate Thumbnails
- 25. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails delete lock file upload worker Genrate Thumbnails delete lock file ・ ・ ・ ・ ・ upload worker Genrate Thumbnails delete lock file
- 26. Inside Image Upload ● User Side Action ○ Apache ○ PHP ■ ImageClient ● Server Side Action ○ daemontools ○ Python ■ python-prefork, python-q4m, python-worker ● python-q4m and python-worker are in-house libraries ○ Q4M ■ Used As Upload Job Queue
- 28. Two types of thumbnails at pixiv ● Static Thumbnail ● Dynamic Thumbnail
- 29. Static Thumbnail
- 30. Static Thumbnail Generated on upload
- 31. Detail of generating thumbnails ● pixiv uses ImageMagick ○ ImageMagick is not fast ○ But quality of generated image is good ○ Quality is more important than speed for us ○ Of course, optimization is important, too
- 33. libjpeg-turbo
- 34. libjpeg-turbo ● Optimized libjpeg for x86 and x86_64 ● Replace libjpeg simply by LD_LIBRARY_PATH ● ImageMagick uses libjpeg
- 35. benchmark of libjpeg and libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo
- 36. benchmark of libjpeg and libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo libjpeg-turbo is faster than libjpeg by 10% on x86_64
- 37. Disable OpenMP
- 38. Disable OpenMP ● Latest ImageMagick is OpenMP enabled at default ○ This is very slow in multi-process environment ● How to disable OpenMP ○ Re-compile with '--disable-openmp' ○ OMP_NUM_THREADS=1 ● pixiv takes the latter ○ Re-compiling and Re-packaging are complicated
- 39. OMP_NUM_THREADS=1 CPU Usage of Application Server
- 40. OMP_NUM_THREADS=1 CPU usage in peak dropped by 150～200% CPU Usage of Application Server
- 41. OMP_NUM_THREADS=1 Load Average of Application Server
- 42. OMP_NUM_THREADS=1 Load Average became dust Load Average of Application Server
- 44. Dynamic Thumbnail Generated on demand
- 45. Why is dynamic thumbnail needed? ● Static thumbnail consumes disk space ○ Dynamic thumbnail does not consume disk space ● Preparing new static thumbnails takes a long time ○ Dynamic thumbnail is ready in a second！
- 46. Other Companies' Cases ● Cookpad ○ mod_tofu ○ http://www.slideshare.net/mirakui/ss-8150494 ● livedoor(Now, NHN Japan) ○ mod_small_light ○ http://www.slideshare.net/livedoor/smalllight2
- 47. pixiv's Case
- 49. mod_small_light-1.1.1 pixiv Edition5
- 50. mod_small_light-1.1.1 pixiv Edition5 heavily customized...
- 53. Resize image with mod_small_light GET /tank.jpg GET /small_light(dw=100,dh=100)/tank.jpg
- 54. Many Options q image quality(0～100) of output format(jpg,gif,png,tiff) e processing engine(imlib2,imagemagick) cc canvas color p pattern name with SmallLightPatternDefine etc Too many options
- 55. Pattern Name
- 56. Resize image with mod_small_light GET /small_light(p=small)/tank.jpg GET /tank.jpg or GET /small_light(dw=100,dh=100,e=imagemagick,jpeghint=y)/tank.jpg
- 57. Why not original mod_small_light? ● Some thumbnails are special ○ comic cover ○ various cropping algorithms depending on aspect ratio ● Default output format is JPEG ○ It is good for us that input and output formats are the same
- 58. Why not original mod_small_light？ ● No support for CMYK ○ pixiv must support CMYK ● Needed support for strange aspect ratio ○ da=l -> long-edge ○ da=s -> short-edge ○ da=p -> pixiv-edge (only pixiv Edition)
- 59. Special Thumbnail 1 ● comic cover
- 60. Special Thumbnail 2 ● special cropping algorithm
- 61. Special Thumbnail 2 ● special cropping algorithm
- 62. Special Thumbnail 2 ● special cropping algorithm crop center of image if image is landscape
- 63. Special Thumbnail 3 ● special cropping algorithm
- 64. Special Thumbnail 3 ● special cropping algorithm
- 65. Special Thumbnail 3 ● special cropping algorithm crop top of image if image is portrait
- 66. Many Extend Options rmprof remove image profiles crop_square crop image squarly cover add manga cover samec conform dw & dh & cw & ch extendl extend long-edge etc twenty new options in all
- 67. Summary ● Optimization of image processing is very important ○ Generating thumbnail takes a long time ○ Let's tune and desynchronize ● pixiv has two types of thumbnails ○ Static Thumbnail ○ Dynamic Thumbnail ● Dynamic Thumbnail is diskspace-saving and flexible ○ Big image storage is expensive ○ Easy to correspond to shift of application ■ New thumbnail types for new designs
- 68. Thanks！