From Clouds to Roots: Performance Analysis at Netflix
Recent posts: 29 Oct 2024 » AI Flame Graphs 22 Jul 2024 » No More Blue Fridays 24 Mar 2024 » Linux Crisis Tools 17 Mar 2024 » The Return of the Frame Pointers 10 Mar 2024 » eBPF Documentary 28 Apr 2023 » eBPF Observability Tools Are Not Security Tools 01 Mar 2023 » USENIX SREcon APAC 2022: Computing Performance: What's on the Horizon 17 Feb 2023 » USENIX SREcon APAC 2023: CFP 02 May 2022 » Brendan
The Netflix Tech Blog: Hadoop Platform as a Service in the Cloud
by Sriram Krishnan and Eva Tse, Cloud Big Data Platform Hadoop has become the de facto standard for managing and processing hundreds of terabytes to petabytes of data. At Netflix, our Hadoop-based data warehouse is petabyte-scale, and growing rapidly. However, with the big data explosion in recent times, even this is not very novel anymore. Our architecture, however, is unique as it enables us to
Genie - Hadoop Platform as a Service at Netflix
In a prior tech blog (http://nflx.it/XoySYR), we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazon’s Simple Storage Service (S3) as our "source of truth", leveraging the elasticity of the cloud to run multiple dynamically resizable Hadoop clusters to support various workloads, and our horizontally scalab
Using Presto in our Big Data Platform on AWS
by Eva Tse, Zhenxiao Luo, Nezih Yigitbasi @ Big Data Platform team At Netflix, the Big Data Platform team is responsible for building a reliable data analytics platform shared across the whole company. In general, Netflix product decisions are very data driven. So we play a big role in helping different teams to gain product and consumer insights from a multi-petabyte scale data warehouse (DW). Th
Scryer: Netflix’s Predictive Auto Scaling Engine — Part 2
In part 1 of this series, we introduced Scryer, Netflix’s predictive autoscaling engine, and discussed its use cases and how it runs in Netflix. In this second installment, we will discuss the design of Scryer ranging from the technical implementation to the algorithms that drive its predictions. Design of ScryerScryer has a simple data flow architecture. On a very high level, historical data flow
Netflix running Presto in the AWS Cloud
Netflix runs Presto in its AWS cloud environment to enable low-latency ad-hoc queries on petabyte-scale data stored in S3. Some key things Netflix did include optimizing Presto to read from and write directly to S3, fixing bugs, integrating Presto with its EMR and Ganglia monitoring, and deploying a 100+ node Presto cluster that handles over 1000 queries per day. Performance testing showed Presto
Elasticsearch in Netflix
The document discusses Netflix's use of Elasticsearch for querying log events. It describes how Netflix evolved from storing logs in files to using Elasticsearch to enable interactive exploration of billions of log events. It also summarizes some of Netflix's best practices for running Elasticsearch at scale, such as automatic sharding and replication, flexible schemas, and extensive monitoring.Re
Netflixã¯ã©ã®ã‚ˆã†ã«æ˜ 画をジャンル分ã‘ã—ã¦ã„ã‚‹ã‹ - ä¸å¯è¦–点
æ˜ åƒã‚³ãƒ³ãƒ†ãƒ³ãƒ„ã®ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ã¨ã„ãˆã°Netflixã€ç¾åœ¨4400万人ã®ãƒ¦ãƒ¼ã‚¶ãƒ¼ï¼ˆæœ‰æ–™ä¼šå“¡ï¼‰ãŒã„ã‚‹æˆç†Ÿã—ãŸã‚µãƒ¼ãƒ“スã§ã™ãŒã€ç¾åœ¨ã‚‚ã™ã”ã„ペースã§æˆé•·ã—ã¦ã„ã¾ã™ã€‚ Netflixã€ç¬¬4å››åŠæœŸæ±ºç®—ã§å¤§å¹…増益--åŠ å…¥è€…æ•°ã¯400万人増 - CNET Japan 利用ã§ãる地域ã¯é™ã‚‰ã‚Œã¾ã™ãŒã€æ—¥æœ¬ã§ã‚‚レコメンデーションã®ã‚³ãƒ³ãƒ†ã‚¹ãƒˆNetflix prizeã®é–‹å‚¬ã‚„ã€AWSã‚’ã„ã¡æ—©ã活用ã—ãŸä¼æ¥ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã¦ã„ã¾ã™ã€‚ Netflixã¯å…ˆã«ç´¹ä»‹ã—ãŸNetfix Prizeã§ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã®æ€§èƒ½å‘上ã«æ‡¸è³žé‡‘ã‚’ã‹ã‘ãŸã»ã©ã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ãŒã‚µãƒ¼ãƒ“スã®é‡è¦ãªä½ç½®ã‚’å ã‚ã¦ã„ã¾ã™ã€‚ 視è´ã•ã‚ŒãŸæ˜ ç”»ã®2/3ã¯ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³çµŒç”±ã‚‰ã—ã„ã§ã™ã€‚ Todd Yellin(Vice President of Product Innovation at Netflix)ã¯ã€ã€Œæ˜ 画をピッタリã®äººã«ãƒ”ッタリã®ã‚¿ã‚¤ãƒŸãƒ³ã‚°ã§
Scryer: Netflix’s Predictive Auto Scaling Engine
To deliver the best possible experience to Netflix customers around the world, it is critical for us to maintain a robust, scalable, and resilient system. That is why we have built (and open sourced) applications ranging from Hystrix to Chaos Monkey. All of these tools better enable us to prevent or minimize outages, respond effectively to outages, and/or anticipate the kinds of operational gaps t
猿ãŒæš´ã‚Œã¦ã‚¯ãƒ©ã‚¦ãƒ‰ã®éšœå®³ã«å¼·ããªã‚‹Netflixã®ã‚·ã‚¹ãƒ†ãƒ - ワザノム| wazanova.jp
http://techblog.netflix.com/2013/10/introducing-chaos-to-c.html 「æ´å²ä¸Šã®æœ‰åãªé–‹ç™ºãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‹ã‚‰ã¾ãªã¶ã¹ãã“ã¨ã€ã‚’ã¾ã¨ã‚ã¦ã„ãŸã¨ãã«ã€Videoã®ä¸ã§ã€ã‚る大型ãƒã‚±ãƒƒãƒˆã‚¨ãƒ³ã‚¸ãƒ³ã®é–‹ç™ºã«ãŠã„ã¦ã€ä¿¡é ¼æ€§ãƒ†ã‚¹ãƒˆã®ãŸã‚ã«å°åž‹çˆ†å¼¾ã‚’エンジンã®å™´å°„å£è¾ºã‚Šã§çˆ†ç™ºã•ã›ã¦è€æ€§ã‚’調ã¹ãŸäº‘々ã®ã‚¨ãƒ”ソードãŒç´¹ä»‹ã•ã‚Œã¦ã„ã¦ã€æ›´ã«ç¶šã„ã¦ã€Œãƒãƒƒãƒˆæ¥ç•Œã§åŒã˜ã‚ˆã†ãªã“ã¨ã‚’ã‚„ã£ã¦ã‚‹ã®ã¯Netflixãらã„ã 。ã€ã¨ã„ã†èª¬æ˜ŽãŒã‚ã‚Šã¾ã—ãŸã€‚ãã®æ™‚ã¯ä½•ã®ã“ã¨ã ã‹ã‚ˆãã‚ã‹ã‚‰ãªã‹ã£ãŸã®ã§ãƒ–ãƒã‚°ã§ã¯å–り上ã’ãªã‹ã£ãŸã®ã§ã™ãŒã€ä»Šå›žè¦‹ã¤ã‘ã¾ã—ãŸã€‚ 以å‰ç´¹ä»‹ã—ãŸã‚ˆã†ã«ã€åŒ—ç±³ã®ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆãƒˆãƒ©ãƒ•ã‚£ãƒƒã‚¯ã®30%以上をå ã‚ã‚‹Neflixã¯ã‚¤ãƒ³ãƒ•ãƒ©ã‚’Amazonã«å…¨é¢çš„ã«ç§»è¡Œã—ã¦ã„ã¾ã™ã€‚クラウドã«ç§»è¡Œã—ãŸå¾Œã®å¦ã³ã¨ã—ã¦ã€ 自社データセンターã®æ™‚ã¯ã€å€‹åˆ¥ã®ãƒãƒ¼ãƒ‰ã‚¦ã‚§ã‚¢ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ãŒéšœå®³
1
ランã‚ング
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - Netflix/SimianArmy: Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.
{{#tags}}- {{label}}
{{/tags}}