Scala Use Cases at Hatena

Transcript

1. Scala Use Cases at Hatena גࣜձࣾ͸ͯͳ Takaya Tsujikawa https://www.flickr.com/photos/30081184@N02/3769383676

2. Takaya Tsujikawa (@mechairoi) blog.chairoi.me mechairoi ! - 2009 Hatena Intern

   (Perl) - 2011 Joined Hatena (Perl) - 2013~ Mackerel team lead engineer (Scala)

3. Table of Contents • Web development with Perl & problems

   • Why Scala? • Web development with Scala • Pros & Cons
4. None

5. • Internet Company 2001~ • Hatena Bookmark • Hatena Blog

   • Flipnote Hatena etc. • Almost products by Perl

6. Why Perl?

7. – http://www.paulgraham.com/hp.html “We need a language that lets us scribble

   and smudge and smear, not a language where you have to sit with a teacup of types balanced on your knee and make polite conversation with a strict old aunt of a compiler”

8. Why Perl? (200x) • CPAN • Rapid prototyping • Code

   density ⛳️ • Duck typing • Perl Community (YAPC etc)

9. a decade later … https://www.flickr.com/photos/josh/11029669/in/photolist-YwJH-dZrHB3-dZro3Y-dZkJSF-dZrn6b-dZkTkH-dZm2se-dZkN5R-dZrFnG-dZm1Lc-aEwYeQ-8cKHdX-aGrYEa-593eo9-eUa2aU- eZXyHe-f1cTP9-dZkQzP-dZrJps-dZkYtg-dZm3gp-dZkAwV-dZkyLc-dZrhAo-dZrjiC-dZrgmN-dZrifq-dZri1Y-dZkS2a-dZkVoM-eUar2C-dZrhLu-a3hhH4-dZruES-eTXXbc-dZkUhB-dZkCJF- dZrmTY-dZrCXS-dZkEYz-dZkZsz-dZkZJZ-dZrrTb-dZkGH2-dZrkZ5-dZrp1N-dZrpNG-dZrHYQ-dZrq5C-dZrsA1/

10. Web applications become more complex https://www.flickr.com/photos/adrian_s/8271860/in/photolist-JoWh-2J1ULn-5MBjL1-7Jb7po-PBm6n-4YxhWQ-ftWYrC-56q8A8-byH7Vo-g6K6Gv-aiH2Kq-oqCruv-wJ9B-6qtmTj- bzvmcN-8SLbUW-4EUk2k-7WsSCV-6WR5CX-bQkpxT-EN7qe-5i3Tcf-aUrm1H-8ZRy8r-aBUFd2-6jBeaS-8N6ZoM-ig1HkK-4Ut5PG-8t8Q6D-5KjrEv-4cTcqP-3feujT-7Bpbqh- atG3JH-22M7W-8cwtgh-5jQDxs-EN7nt-cc87U3-6Zmv7D-4q13tN-9MqM4o-dK4pWN-8agRiA-GK1af-crBgSU-7NVEST-9Lg7jB-6KRFX Business Library

    Update Legacy Code New features Mobile

11. Resistance to collapse • Coding rule • Code review •

    Test / Continuous integration • Testing is require but not sufficient • Static code analysis IUUQTXXXqJDLSDPNQIPUPT!/

12. ʊਓਓਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹSudden Runtime Errorɹʻ ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^Yʉ

13. We need static typing https://www.flickr.com/photos/hetgacom/10420400556/in/photostream/

14. Why xxScala?

15. New Project!

16. A New Kind of Application Performance Management

17. Push architecture host1 host2 host3 Mackerel Agent

18. Graphs by roles cpu usage of proxies loadavg5 of apps

19. Monitoring • Mail • Webhook

20. Background • 2013/11 Start • 1 engineer + Outsource •

    B2B • Low dependency to existing system.(ex. Account system) • Owned service • Less restriction Chance of introduction of new technology

21. Go Graphite PostgreSQL

22. Go Graphite PostgreSQL

23. Requirements • Modern type system • Lots of libraries •

    Case studies of web development • Member’s Skill

24. Why Scala (1/4) • Modern type system • Static typing

    • Algebraic data type (case class, sealed trait) • Type inference • Ad hoc polymorphism (implicit conversion) • B2B Safe refactoring and improvement

25. Why Scala (2/4) • Lots of libraries

26. Why Scala (3/4) • Case studies of web development

27. Why Scala (4/4) • Member’s skills • some members had

    been studied/was interested in Scala • Product must not depend on individual skills

28. Other languages • Java8 • Not available yet • Haskell

    • Hard to assemble team • Go/Ruby • No modern type system

29. Development with Scala

30. • Fullstack WAF • Routing • Type-safe template engine •

    play.api.{data.Form, libs.Json} • Pretty error messages • play-auth/stackable-controller

31. • Type safe template engine (Twirl) • Our designer writes

    Scala expression • Hard for designer • Rewriting Angularjs template • 2.2.1 → 2.2.2 → 2.3.1

32. Design policy • MVC • Classes • object models.Users #create

    • Side effects • case class models.User (email: String, … ) • Immutable data container • No side effects, no variable.

33. In Perl case ! Side effects are used everywhere

34. Design policy • Less variable, less mutable collection, less partial

    function ( Option#get ) • User defined types (Tagged Type) • UserId / HostId instead of Int • UserName instead of String Increase guarantee by type

35. In Perl case ! No static typing Just try hard

36. Design policy • Mapping primitive type and domain types by

    type classes )551CPEZ ! 63* ! %BUBTUPSF "QQMJDBUJPO .PEFM .BQQJOHCZ UZQFDMBTT 4USJOH +40/ KBWBMBOH42- DBTFDMBTTFT 6TFS )PTU

37. Routing GET /hosts/:id controllers.Hosts.retrieve(id: HostId) implicit def hostIdFormat: Formatter[HostId] =

    new Formatter[HostId] { def bind(key: String, data: Map[String, String]) : Either[Seq[FormError], HostId] = ??? ! def unbind(key: String, value: HostId) = ??? } <a href=“@(routes.Hosts.retrieve(hostId)”>…</a>

38. Modules • Modularize for compile speed ( In progress..) $PSF5ZQFT

    $PSF .PEFMT $PSF$POUSPMMFST $PSF 7JFXT "MFSU5ZQFT "MFSU .PEFMT "MFSU$POUSPMMFST "MFSU 7JFXT 3PPU 3FWFSTF 3PVUFS

39. • Schema code generation • Type-safe queries and raw queries

    • All Queries in our product are type-safe • Extends for “FOR UPDATE OF t1” • 1.0.1 → 2.0.0 → 2.0.3 ( → 2.1.0)

40. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap

41. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for {! (log, logLater) <- Tables.alertLogs(orgId) leftJoin! Tables.alertLogs(orgId) on { case (log, logLater) =>! log.createdAt < logLater.createdAt &&! log.alertId === logLater.alertId! }! if log.alertId inSetBind entities.map(_.id)! if logLater.createdAt.?.isNull! } yield (log.alertId, log.status)).list.toMap

42. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap

43. Not human readable select x2.x3, x2.x4 from (select x5."updated_at" as

    x6, x5."status" as x4, x5."alert_id" as x3, x5."created_at" as x7, x5."operator_ id" as x8, x5."id" as x9, x5."trigger" as x10 from "alert_logs" x5, "alerts" x11 where (x11."id" = x5."alert_id") and (x11."org_id" = ?)) x2 left outer join (select x12."updated_at" as x13, x12."status" as x14, x12."alert_id" as x15, x12."created_at" as x16, x12."operator_id" as x17, x12."id" as x18, x12."trigger" as x19 from "alert_logs" x12, "alerts" x20 where (x20."id" = x12."alert_id") and (x20."org_id" = ?)) x21 on (x2.x7 < x21.x16) and (x2.x3 = x21.x15) where (x2.x3 in (?)) and (x21.x16 is null)

44. Not human readable select x2.x3, x2.x4 from (select x5.x6 as

    x7, x5.x8 as x9, x5.x10 as x11, x5.x12 as x13, x5.x14 as x15, x5.x16 as x17, x5.x18 as x3, x5.x19 as x20, x5.x21 as x22, x5.x23 as x24, x5.x25 as x26, x5.x27 as x28, x5.x29 as x30, x5.x31 as x32, x5.x33 as x34, x5.x35 as x36, x37.x38 as x39, x37.x40 as x4, x37.x41 as x42, x37.x43 as x44, x37.x45 as x46, x37.x47 as x48, x37.x49 as x50, x37.x51 as x52 from (select x53."updated_at" as x6, x53."image_url" as x8, x53."hosts_count_limit" as x10, x53."webhook_url" as x12, x53."enabled_experimental_features" as x14, x53."created_at" as x16, x53."id" as x18, x53."owner_team_id" as x19, x53."name" as x21, x53."gravatar_email" as x23, x53."is_staff" as x25, x53."notification_activates_at" as x27, x53."plan_id" as x29, x53."plan_expired_at" as x31, x53."author_id" as x33, x53."token" as x35 from "orgs" x53) x5 left outer join (select x54."updated_at" as x38, x54."agv_count" as x40, x54."max_count" as x41, x54."total" as x43, x54."created_at" as x45, x54."base_datetime" as x47, x54."id" as x49, x54."org_id" as x51 from "host_metrics_counts" x54) x37 on x5.x18 = x37.x51) x2 left outer join (select x55."updated_at" as x56, x55."agv_count" as x57, x55."max_count" as x58, x55."total" as x59, x55."created_at" as x60, x55."base_datetime" as x61, x55."id" as x62, x55."org_id" as x63 from "host_metrics_counts" x55) x64 on (x2.x3 = x64.x63) and (x2.x48 < x64.x61) where ((x2.x3 in (?)) and (x64.x61 is null)) and (x2.x48 > {ts '2014-09-02 17:27:18.039'}) • Where does code generate this query?

45. In Perl case • No type-safe • Publisher class in

    SQL Comment SELECT * FROM user WHERE user_id = ?; -- Hatena::User

46. Development flow

47. IDE AND / OR

48. Git Branches operation NBTUFS EFWFMPQ GFBUVSF GFBUVSF 3FMFBTF

49. Git Branches operation NBTUFS EFWFMPQ GFBUVSF GFBUVSF 3FMFBTF No accident

    on merge

50. In Perl case ＿人人人人人人人人人＿ ＞�Merge without conflicts�＜ ‾Y^Y^Y^Y^Y^Y^Y^Y‾ Can't locate

    object method "retrieve_by_id" via package "Entry" at error.pl line 21. Entry->retrieve_by_id(10); Entry->retrieve(id => 10); Entry->retrieve_by_id(20); 3FGBDUPSJOH "EEFEDBMMT

51. Task management • Github issues/pull requests • Tag • Milestone

52. Scrum • Sprint = Milestone • Burn down chart •

    radekstepan/github-burndown-chart

53. Jenkins/CI • Test & Packaging parallel • 1h → 20min

    • Git integration

54. git-pr-release • Auto generate pull-request • master ← develop •

    Check list / issues • Everyone can deploy • motemen/git-pr-release

55. Deploy • Restarting JVM is slow → 502 • Capistrano

    • Custom deploy strategy • scp jar files from Jenkins • Rolling deploy • daemontools https://www.flickr.com/photos/gsfc/9807812154/in/photolist-fWFARy-dBeVvk-96iJmd-atSZgv-hFZzw8-aoA2hH-j2aiC2-mrQV5-kwW6fT-9MP2Ju-edaJM7-M3WWJ-kmUfnB-dxxRza- Cjb4u-4YSS4-8JpmiE-4yjdpB-bbibXF-btu7MA-bGoWje-kTaXqi-jJpjHn-5RV54-4rMctW-bGoWnD-ecFBPp-9Yf9KD-9XsGPA-LVDt-aRcVjM-5xgexk-9S6d4F-4y4qNv-eZwRu-osQq9g-ebMkug- ed9XJo-qyM6W-aUyKZH-4cxiEU-bGoWsk-btu7F1-776kcJ-9zuC1x-j2bH1R-bypWd7-e6BdUN-bGoWeX-btu7A7

56. In Perl case • Capistrano • git clone • Server::Starter

    • Graceful restart

57. Scala Pros & Cons

58. Pros https://www.flickr.com/photos/quinndombrowski/5200218267/in/photolist-6k8go4-a3y271-8VwtHD-skiqN-nN34SF-bveV1T-cjNRK9-oddpGc-6SXagE-fea96C-6MLPeT-a7pJhj-wDorR- a55adA-8uYbhg-4HP2Qm-dhuorr-aYXd4F-eLGkd-e4eFKc-9ugzpc-6MZ9UU-bWXfCV-9GH9mD-59UEA7-aG95zM-3fJmjC-cHPX65-8aBfdr-j7GYKE-e6ViAm-bAhx1p-feQRPa-92JZzp-kLtFHa- XWd9L-9zbeGs-9DdEqR-8maR73-nBgwNS-8WFsTq-7wSNi8-MvvjP-5DMyrq-8Kowmj-54rxHi-BqUvC-fLbCYs-7F9di2-nR8Kt4

59. Pros • Peace of Mind • Refactoring/Upgrade is easy •

    Can focus to architecture/design and logic in review • Functional programming • Collection libraries

60. Cons • Slow compilation • modularize • high spec machine

61. Cons • Require Java/JVM literacy • Rich language functions •

    Hard to learn • De-facto standard Job queue? • Mocking is difficult in tests

62. Conclusion ! Cannot go back to Perl

63. Any Question?

64. We are hiring