Hadoop filesystem at Twitter
Twitter runs multiple large Hadoop clusters that are among the biggest in the world. Hadoop is at the core of our data platform and provides vast storage for analytics of user actions on Twitter. In this post, we will highlight our contributions to ViewFs, the client-side Hadoop filesystem view, and its versatile usage here. ViewFs makes the interaction with our HDFS infrastructure as simple as a
Scala.io
Finagle is Twitter's open source RPC library that allows composing asynchronous RPC requests like functions. It provides three key abstractions: Futures for asynchronous computations, Services for RPC functions, and ServiceFactories for creating Services. Finagle handles load balancing, connection pooling, failure detection, and other distributed systems concerns through composable layers above a
How Twitter Uses Redis to Scale - 105TB RAM, 39MM QPS, 10,000+ Instances - High Scalability -
Yao Yue has worked on Twitter’s Cache team since 2010. She recently gave a really great talk: Scaling Redis at Twitter. It’s about Redis of course, but it's not just about Redis. Yao has worked at Twitter for a few years. She's seen some things. She’s watched the growth of the cache service at Twitter explode from it being used by just one project to nearly a hundred projects using it. That's many
Twitterã®ã‚ャッシュを支ãˆã‚‹Redis - ワザノム| wazanova
https://www.youtube.com/watch?v=rP9EKvWt0zo 1 comment | 0 points | by WazanovaNews â– comment by Jshiike | ç´„1æ™‚é–“å‰ Twitterã®Yao YuãŒã€å¤§è¦æ¨¡ã‚µãƒ¼ãƒ“スã®ã‚ャッシュã«ãŠã„ã¦Redisを活用ã™ã‚‹å–組ã¿ã«ã¤ã„ã¦ç´¹ä»‹ã—ã¦ã„ã¾ã™ã€‚ 1) Redisを採用ã—ã¦ã„ã‚‹ç†ç”± ã‚ャッシュã ã‘ã§ã€ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã¨ã—ã¦ã¯åˆ©ç”¨ã—ã¦ã„ãªã„。 主ãªã¨ã“ã‚ã§ã¯ã€Twitterã®ã‚¿ã‚¤ãƒ ラインã§åˆ©ç”¨ã—ã¦ã„る。ホーム画é¢ã§ã‚ã‚Œã€ãƒ¦ãƒ¼ã‚¶ç”»é¢ã§ã‚ã‚Œã€ã‚¿ã‚¤ãƒ ラインã¯Tweetã®ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ãªã®ã§ã€key/valueストア型ã®Redisを利用ã™ã‚‹ã‚±ãƒ¼ã‚¹ã¨ã—ã¦æœ€é©ã€‚ 以å‰ã¯memcachedを使ã£ã¦ã„ãŸãŒã€å•é¡Œã«ãªã£ãŸã®ã¯ã€ã‚¿ã‚¤ãƒ ラインã§ãŠãã‚‹read/writeã¯ã€ï¼ˆãƒ¦ãƒ¼ã‚¶ãŒé–²è¦§ã—ã¦ã„る範囲ã«è¿½åŠ åæ˜ ã™ã‚‹ã¨ã„ã†ã“ã¨ãªã®
ツイッターã€èµ¤å—大幅拡大も売上高éŽåŽ»æœ€é«˜
米ツイッターãŒç¾åœ°æ™‚é–“2014å¹´7月29æ—¥ã«ç™ºè¡¨ã—ãŸ4～6月期決算ã§ã¯ã€æœ€çµ‚æ益ãŒ1å„„4464万ドル（約147億円）ã®èµ¤å—ã¨ãªã‚Šã€å‰å¹´åŒæœŸã®4222万ドル（約43億円）ã‹ã‚‰å¤§å¹…ã«æ‹¡å¤§ã—ãŸã€‚ç ”ç©¶é–‹ç™ºè²»ãŒè†¨ã‚‰ã‚“ã ã“ã¨ãŒåŽŸå› ã ã¨ã„ã†ã€‚ 一方ã€å£²ä¸Šé«˜ã¯3å„„1216万ドル（約318億円）ã§å‰å¹´åŒæœŸæ¯”ã®ç´„2.2å€å¢—ã§ã€éŽåŽ»æœ€é«˜ã‚’更新。ã¾ãŸ6月末時点ã®åˆ©ç”¨è€…æ•°ã¯2å„„7100万人ã§ã€1å¹´å‰ã¨æ¯”ã¹ã¦24％増ã¨ãªã£ãŸã€‚
All-pairs similarity via DIMSUM
We are often interested in finding users, hashtags and ads that are very similar to one another, so they may be recommended and shown to users and advertisers. To do this, we must consider many pairs of items, and evaluate how “similar†they are to one another. We call this the “all-pairs similarity†problem, sometimes known as a “similarity join.†We have developed a new efficient algorithm to so
BigQueryã«Twitterã®ã‚¿ã‚¤ãƒ ラインを入れる [Ruby]
Twitter ã®ã‚¿ã‚¤ãƒ ラインをä¿å˜ã—ã¦ãŠãã¨ãªã«ã‹ã¨ä¾¿åˆ©ãªã®ã§ã€è‰²ã€…ã¨ä¿å˜å½¢å¼ã‚’変ãˆãªãŒã‚‰ 4 å¹´ãらã„記録ã—続ã‘ã¦ã„る。ツイートã®ä¿å˜ãŒä¾¿åˆ©ã™ãŽã‚‹ã®ã§ã€ãƒ„イセーブã¨ã„ã†ã‚µãƒ¼ãƒ“ス化ã¾ã§ã—ãŸã€‚ã‹ã¤ã¦ã¯ãƒ†ã‚ストã§ã€MongoDB ã‚„ MySQL ã¨ã‹ Groonga ã¨ã‹ã„ã‚ã„ã‚ã‚„ã£ã¦ããŸã€‚ã©ã‚Œã‚‚å•é¡Œãªã„ã‚“ã ã‘ã©ã€å¢—ãˆç¶šã‘ã‚‹ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’ä¿å˜ã™ã‚‹å ´æ‰€ã¨ã—ã¦è€ƒãˆã‚‹ã¨ BigQuery ãŒç¾ä»£ã«ãƒžãƒƒãƒã—ã¦ã‚‹ã‚ˆã†ãªã®ã§ãã¡ã‚‰ã«ç§»è¡Œã—ãŸã€‚ BigQuery ã« TL ã‚’ä¿å˜ã™ã‚‹ã¨ã§ãã‚‹ã“㨠TL ã®å…¨ã¦ã®ãƒ‡ãƒ¼ã‚¿ã‚’フルスã‚ャンã§ãる。ã“ã‚Œã¯ã‹ãªã‚Šä¾¿åˆ©ã§ã€ä»Šå›žã“ã®ãƒ–ãƒã‚°è¨˜äº‹ã‚’書ãã«ã‚ãŸã£ã¦ã‚‚ ‘BigQuery’ ã‚’ TL ã‹ã‚‰æ¤œç´¢ã™ã‚Œã°ã€ä¿¡é ¼ã§ãるフォãƒãƒ¼ã‚¤ãƒ³ã‚°ã®äººã€…ã®å£°ã‚’見るã“ã¨ãŒã§ããŸã€‚ã“ã‚Œã«ã‚ˆã‚Šã€ŒæŸ CA 社ã§ã¯ 5000 å°ã® MongoDB クラスタ㧠BigQuery ã«å¯¾æŠ—ã—ã¦ã„ã‚‹ã€ã¨ã„ã†
![BigQueryã«Twitterã®ã‚¿ã‚¤ãƒ ラインを入れる [Ruby]](https://cdn-ak-scissors.b.st-hatena.com/image/square/eb99bf7e3074ef0fe5df0ca25a644e696c62ad78/height=288;version=1;width=512/https%3A%2F%2Fs3-ap-northeast-1.amazonaws.com%2Fkksg%2Fblog%2F53b4ac19c38aa5513f47ea75.png)
社内輪èªä¼šã§ã€ŒTopical Keyphrase Extraction from Twitterã€ã‚’紹介ã—㟠- skozawa's blog
会社ã§æœ€è¿‘始ã¾ã£ãŸè«–æ–‡ã®è¼ªè¬›ã§æ‹…当ã ã£ãŸã®ã§ã€ç™ºè¡¨ã‚’ã—ãŸã€‚ è«–æ–‡ã¯ã§ãã‚‹ã ã‘自分ã®åˆ†é‡Žã®ãƒˆãƒƒãƒ—カンファレンスã‹ã‚‰é¸ã¶ã¨ã„ã†ã“ã¨ã ã£ãŸã®ã§ã€è‡ªç„¶è¨€èªžå‡¦ç†ã®å›½éš›ä¼šè°ã®ACLã‹ã‚‰è«–文をé¸ã‚“ã§ç´¹ä»‹ã—ãŸã€‚ 今回紹介ã—ãŸã®ã¯ã€å°‘ã—å¤ã„ã‘ã©ã€ACL2011ã‹ã‚‰Topical Keyphrase Extraction from Twitter å‚åŠ ã—ã¦ã‚‹äººã¯ãã‚Œãžã‚Œåˆ†é‡ŽãŒé•ã†ã®ã§ã€ã©ã®éƒ¨åˆ†ã‚’ã©ã®ç¨‹åº¦èª¬æ˜Žã™ã‚Œã°ã„ã„ã‹ãŒé›£ã—ã„。 内容 内容ã¯Twitterã‹ã‚‰ãƒˆãƒ”ックã®ã‚ーフレーズを抽出ã™ã‚‹æ‰‹æ³•ã®æ案ã§ã€æ案ã¯ä¸»ã«ä»¥ä¸‹ã®2点 Context-sensitive Topical PageRank ã«ã‚ˆã‚‹ã‚ーワードスコアリング relevanceã¨interestingnessを用ã„ãŸã‚ーフレーズスコアリング 紹介ã—ãŸã‚‚ã®ã‚’å°‘ã—ä¿®æ£ã—ãŸã‚‚ã®ã‚’アップã—ãŸã€‚ 感想 1ã¤ç›®ã¯ã‚¹ã‚³ã‚¢ãƒªãƒ³ã‚°ã®éš›ã«ãƒˆãƒ”ックを考慮ã™ã‚‹ã“ã¨ã«ã‚ˆ
ã€è¿‘å‚ツイート検索】特定ツイートå‰å¾Œã®ã‚¿ã‚¤ãƒ ラインを表示ã™ã‚‹ãƒ¦ãƒ¼ã‚¶ãƒ¼ã‚¹ã‚¯ãƒªãƒ—ト試作 - 風柳メモ
残念ãªãŠçŸ¥ã‚‰ã› 諸般ã®äº‹æƒ…ã«ã‚ˆã‚Š - è¿‘å‚ツイート検索 - Twitter原寸ã³ã‚…ー - Twitterメディアダウンãƒãƒ¼ãƒ€ ã‚’ã¯ã˜ã‚ã¨ã™ã‚‹æ‹™ä½œãƒ–ラウザ拡張機能ã®é–‹ç™ºãŠã‚ˆã³å…¬é–‹ã‚’ä¸æ¢è‡´ã—ã¾ã™ å„ストアやGitHubリãƒã‚¸ãƒˆãƒªç‰ã¯9月16日を目処ã«é †æ¬¡éžå…¬é–‹(é…ä¿¡åœæ¢)ã¨ã™ã‚‹äºˆå®šã§ã™ ã”愛顧ã‚ã‚ŠãŒã¨ã†ã”ã–ã„ã¾ã—㟠https://t.co/bd9S572YL1— 風柳 (@furyutei) 2023å¹´8月31æ—¥ Unfortunate Announcement Due to various circumstances, I will be discontinuing the development and release of my browser extensions, including: - twDisplayVicinity - Twitter View Original Imag
Twitter details its Manhattan real-time database – Old GigaOm
Twitter’s service is nothing if not fast-moving, and on Tuesday night the company published a blog post detailing the database that helps it keep up. Called Manhattan, it’s a distributed, real-time database built to serve multiple teams and applications within the company. It’s also something of an indictment against existing open source database technologies, at least when it comes to handling th
Hadoop/Storm ã®çµ±åˆã‚’実ç¾ã™ã‚‹ Twitter ã® SummingBird - Okapies' Archive
Twitter ㌠SummingBird ã‚’æ£å¼ãƒªãƒªãƒ¼ã‚¹ã—ã¦æ—©äºŒãƒ¶æœˆã€‚「日本語ã®ç´¹ä»‹è¨˜äº‹ãŒã»ã¨ã‚“ã©å‡ºã¦ãªã„ãªã€ã¨æ°—付ã„ãŸã®ã§ã€èª¿æŸ»ãŒã¦ã‚‰ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ SummingBird ã¨ã¯ï¼Ÿ MapReduce ãªãƒ—ãƒã‚°ãƒ©ãƒ を書ããŸã‚ã® Scala/Java ライブラリ。最大ã®ç‰¹å¾´ã¯ã€ã²ã¨ãŸã³ SummingBird ã§æ›¸ã„ãŸã‚¸ãƒ§ãƒ–㯠Hadoop ã§ã‚‚ Storm ã§ã‚‚åŒã˜ã‚ˆã†ã«å®Ÿè¡Œã§ãã‚‹ã“ã¨ã€‚ SummingBird ã§ã¯ã€Hadoop を使ã†ã€Œãƒãƒƒãƒãƒ¢ãƒ¼ãƒ‰ã€ã¨ã€Storm を使ã†ã€Œãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ モードã€ã«åŠ ãˆã¦ã€äºŒã¤ã‚’åŒæ™‚ã«å®Ÿè¡Œã™ã‚‹ã€Œãƒã‚¤ãƒ–リッドモードã€ãŒã‚る。ãƒã‚¤ãƒ–リッドモードã§ã¯ã€ã‚¸ãƒ§ãƒ–ã®ä½œè€…ãŒç‰¹ã«é…æ…®ã—ãªãã¦ã‚‚ã€ãƒãƒƒãƒã¨ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã®å‡¦ç†çµæžœã‚’自動的ã«ãƒžãƒ¼ã‚¸ã§ãる。 ãƒã‚¤ãƒ–リッドモードã§ã¯ã€åŒã˜ã‚¸ãƒ§ãƒ–ã‚’ Hadoop 㨠Storm ã§åŒæ™‚ã«å®Ÿè¡Œã§ãã‚‹ã®ã§ã€Hadoop ã®è€éšœå®³æ€§
Twitter: データ分æžåŸºç›¤æ”¹å–„å–り組㿠- ワザノム| wazanova.jp
https://www.facebook.com/photo.php?v=10151697364230687&set=vb.9445547199&type=2&theater Twitterã®Analyticsインフラãƒãƒ¼ãƒ ãŒã€ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®æ”¹å–„ã«å–り組んã§ããŸäº‹ä¾‹ã‚’紹介ã—ã¦ã„ã¾ã™ã€‚ 1) 背景 ４億tweet/日を発信 & 消費ã—ã¦ã„るユーザã®ã‚¢ã‚¯ãƒ†ã‚£ãƒ“ティをã€Twitter社内ã®å¤šãã®ãƒãƒ¼ãƒ ãŒãã‚Œãžã‚Œã®è¦³ç‚¹ & 様々ãªåˆ©ç”¨å½¢æ…‹ã§åˆ†æžãƒ‡ãƒ¼ã‚¿ã‚’å¿…è¦ã¨ã™ã‚‹ãŸã‚ã€é‡ãŠã‚ˆã³ãƒ‡ãƒ¼ã‚¿ã®ä¾å˜é–¢ä¿‚ãŒã€ç›¸å½“大ãã複雑ãªã‚‚ã®ã«ãªã£ã¦ã„る。Analyticsインフラã¯ã€1000ノードã‚ã‚‹Hadoopã®ã‚¯ãƒ©ã‚¹ã‚¿ã‚’ã„ãã¤ã‹ã‚‚ã¤è¦æ¨¡ã€‚ ストレージフットプリント & I/Oを減らã™ã ã‘ã§ãªãã€ä»–ã®æ–¹æ³•ã§ãƒ—ãƒã‚»ã‚¹ã‚¹ãƒ”ードをã‚ã’ã‚‹ã“ã¨ã«å–り組んã§ã„る。 2) Parquet （「Hadoop用ã®ã‚«ãƒ©ãƒ ナストレージフォー
Netty 4 at Twitter: Reduced GC Overhead
At Twitter, Netty (@netty_project) is used in core places requiring networking functionality. For example: Finagle is our protocol agnostic RPC system whose transport layer is built on top of Netty, and it is used to implement most services internally like Search TFE (Twitter Front End) is our proprietary spoon-feeding reverse proxy which serves most of public-facing HTTP and SPDY traffic using Ne
Hadoop Hardware @Twitter: Size does matter!
At Twitter we started out with a large monolithic cluster that served most of the use-cases. As the usage expanded and the cluster grew accordingly, we realized we needed to split the cluster by access pattern. This allows us to tune the access policy, SLA, and configuration for each cluster. We will explain our various use-cases, their performance requirements, and operational considerations and
ã¯ã˜ã‚ã®è¨€èªžã®è³žå‘³æœŸé™ - Kato Kazuyoshi
ライブドアブãƒã‚°ã® PSGI 化ã®è©± ã¯è‰¯ã„ã¯ãªã—ã ã¨æ€ã†ã€‚一方ã§ã€ç§ã¯ã‚ã‚“ã¾ã‚Š Perl ãŒå¥½ãã˜ã‚ƒãªã„ã®ã§ã€10å¹´ã«ã‚ãŸã£ã¦ç”Ÿã続ã‘㟠Perl アプリケーションãŒã€æ¬¡ã®10å¹´ã«ã‚€ã‘ã¦ã‚¢ãƒƒãƒ—ã‚’ã¯ã˜ã‚ã¦ã„ã‚‹ã®ã¯ã€ã¡ã‚‡ã£ã¨ã—ãŸãƒ›ãƒ©ãƒ¼ã§ã‚‚ã‚る。 Twitter 㨠Ruby 㨠JVM ライブドアブãƒã‚°ãŒã€å°†æ¥ã«å‘ã‘㦠mod_perl ã‹ã‚‰ PSGI + Starlet ã«ã‹ãˆãŸã‚ˆã†ã«ã€å°†æ¥ã«å‘ã‘ã¦ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°è¨€èªžã‚’ã‹ãˆã‚‹äººé”ã‚‚å˜åœ¨ã™ã‚‹ã€‚最近ã®äº‹ä¾‹ã§æœ‰åãªã®ã¯ã€Twitter ã® Ruby ã‹ã‚‰ JVM 言語群ã¸ã®ç§»è¡Œã ã‚ã†ã€‚ OSCON Java 2011 ã® Twitter: From Ruby on Rails to the JVM ã§ã¯ã€JVM ã¸ã®ç§»è¡Œã«è‡³ã£ãŸç†ç”±ã¨ã—㦠Ability to handle server workloads A real concurrency
BigData Counts
During work and play, many times one has to make ‘back of the envelope’ calculations. Sometimes it is hard to get a perspective on scale and various aspects of big data – numbers like millions and billions or even GB/TB/PB et al. So I have started collecting a few representative numbers that can add perspective to any calculation. Please suggest more … PetaByte vs ExaBytes vs ZettaByte vs YoottaBy
大è¦æ¨¡ã€è¤‡é›‘ã€åˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ ã«å¿…è¦ãªæ–°ãŸãªãƒ‘ラダイムã¨è€ƒãˆæ–¹ã€€ï¼ã€€Twitter Storm：ヨãƒã‚ºITå–„å•ç”：オルタナティブ・ブãƒã‚°
NoSQL Now 2013㮠コンファレンスã§ã¯ã€ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ«ã®ä»–ï¼’ã¤ã®åŸºèª¿è¬›æ¼”ã«å‚åŠ ã—ãŸã€‚ï¼’ã¤ã¨ã‚‚似通ã£ãŸãƒ†ãƒ¼ãƒžã ãŒã€å°‘ã—ç•°ãªã£ãŸç‚¹ã‚’強調ã—ãŸã€‚電力系統ã¯é›»åŠ›ã ã‘ã§ã¯ãª ãã€é›»åŠ›æ¬é€ã«é–¢ã™ã‚‹èŽ«å¤§ãªé‡ã®ãƒ‡ãƒ¼ã‚¿ã‚„æƒ…å ±ã‚‚ä¼é”ã™ã‚‹ã€‚ã“ã‚Œã¯ã€é›»è©±ã®ã‚·ã‚¹ãƒ†ãƒ ã«ä¼¼ã¦ã„る。電話システムã¯ã‚³ãƒ³ãƒˆãƒãƒ¼ãƒ«ãƒ»ã‚·ã‚¹ãƒ†ãƒ (シグナリング)ã¨éŸ³ 声(データ)ã 。電話システムも電力系統も国家ã®æ ¹å¹¹ã‚’ãªã™é‡è¦ãªã‚¤ãƒ³ãƒ•ãƒ©ã§ã‚ã‚Šã€äºˆæœŸã§ãる故障や予期ã§ããªã„æ•…éšœã®éš›ã«ã‚‚サービスをåœæ¢ã™ã‚‹ã“ã¨ã¯ã§ ããªã„。電力ã®æ供を絶やã•ãªã„ãŸã‚ã«ã€ã‚¤ãƒ³ãƒ•ãƒ©ã‚„ãã‚Œã«ä»˜éšã™ã‚‹ã‚·ã‚¹ãƒ†ãƒ ã¯å†—長性ã€å›žå¾©æ€§ã€è‡ªå·±å›žå¾©æ€§ã‚„,フォールトトレラント 性を備ãˆã¦è¨è¨ˆã•ã‚Œæ§‹ç¯‰ã•ã‚Œãªã‘ã‚Œã°ãªã‚‰ãªã„。 コンピュータã®ã‚·ã‚¹ãƒ†ãƒ ã‚‚åŒæ§˜ã 。大è¦æ¨¡ãªåˆ†æ•£ã‚³ãƒ³ãƒ”ュータ・システムをモニターã—コント ãƒãƒ¼ãƒ«ã™ã‚‹ãŸã‚ã«ã¯ã€çªç„¶ã§ã‚ã£ã¦ã‚‚ä¸å¯é¿ãªå•é¡Œã«å¯¾å¿œã§ãるよã†ã«è¨è¨ˆãƒ»æ§‹ç¯‰ã•
Dremel made simple with Parquet
Columnar storage is a popular technique to optimize analytical workloads in parallel RDBMs. The performance and compression benefits for storing and processing large amounts of data are well documented in academic literature as well as several commercial analytical databases. The goal is to keep I/O to a minimum by reading from a disk only the data required for the query. Using Parquet at Twitter,
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - twitter/sslconfig: Twitter's OpenSSL Configuration
twitter/hpack · GitHub - Header Compression for HTTP/2
{{#tags}}- {{label}}
{{/tags}}