[B! data-engineering] yanbeã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yanbe id:yanbe

data-engineeringã«é–¢ã™ã‚‹yanbeã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (17)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

dbtã§ç®¡ç†ã—ã¦ã„ã‚‹ãƒ‡ãƒ¼ã‚¿ã®å¯ç”¨æ€§ã‚’elementaryã§åˆ†æžã™ã‚‹ - yasuhisa's blog
ãƒ‡ãƒ¼ã‚¿ã®å¯ç”¨æ€§ã‚’å¯è¦–åŒ–ã—ãŸã„ ãƒ‡ãƒ¼ã‚¿ã®å¯ç”¨æ€§ã®è§£åƒåº¦ã‚’ä¸Šã’ãŸã„: elementary-data elementaryã«ã‚ˆã‚‹ç´°ã‹ã„å¯è¦–åŒ– å¤§é›‘æŠŠã«ãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆå˜ä½ã§å¯ç”¨æ€§ã‚’å¯è¦–åŒ–ã—ãŸã„ ã¾ã¨ã‚ ãƒ‡ãƒ¼ã‚¿ã®å¯ç”¨æ€§ã‚’å¯è¦–åŒ–ã—ãŸã„ ãƒ‡ãƒ¼ã‚¿å“è³ªã¯æ£ç¢ºæ€§ã‚„æœ€æ–°æ€§ãªã©æ§˜ã€…ãªé …ç›®ã«åˆ†è§£ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ãŒã€å¯ç”¨æ€§(Availability)ã¯ãã®ä¸ã§ã‚‚åŸºç¤Žçš„ãªé …ç›®ã§ã™ã€‚ä½¿ã„ãŸã„ã¨ãã«ãƒ‡ãƒ¼ã‚¿ãŒä½¿ãˆãªã„ã¨å›°ã‚‹ã®ã§ã€‚ è‡ªåˆ†ãŒæ‰€å±žã—ã¦ã„ã‚‹ãƒãƒ¼ãƒ ã¯dbt(cli)ãŠã‚ˆã³dbt cloudã‚’ä½¿ã£ã¦ã„ã¾ã™ãŒã€å¯ç”¨æ€§ã‚’è€ƒãˆã‚‹ã®ã§ã‚ã‚Œã° cli: dbt runã®å®Ÿè¡Œçµæžœ dbt cloud: Jobsã®å®Ÿè¡Œçµæžœ ã‚’ãã‚Œãžã‚Œç¢ºèªã—ãŸã‚Šã€ã“ã‘ã¦ã„ã‚‹ã‚ˆã†ã§ã‚ã‚Œã°ã‚¢ãƒ©ãƒ¼ãƒˆã‚’é£›ã°ã™ã¨ã„ã†é‹ç”¨ãŒå¤šã„ã¨æ€ã„ã¾ã™ã€‚ã“ã‚Œã ã‘ã ã¨ã€Œã„ã¤ã“ã‘ãŸã€ã—ã‹åˆ†ã‹ã‚‰ãªã„ã®ã§ã€Datadogã‚’ä½¿ã£ã¦ã€Œã„ã¤ã“ã‘ãŸã€ã€Œã„ã¤å¾©æ—§ã—ãŸã€ã€Œã“ã‘ã¦è½ã¡ã¦ã„ãŸæ™‚é–“ã¯ã©
yanbe 2023/09/16
dbt

data-engineering
ãƒªãƒ³ã‚¯
Data Developer Platform
Home Data Developer PlatformÂ¶ A Data Platform Specification, open for adoption by any data platform developer. A modern way to run data engineering teamsÂ¶ Data teams are drained from continuously plumbing integrations and fragile pipelines, which leaves little to no time to focus on the real deal - data and data applications. Businesses that have a good grasp on data realise that today data makes
yanbe 2023/09/12
data-engineering
ãƒªãƒ³ã‚¯
Ascend.io: The Data Automation Platform
Deliver your data with market-leading automation on Ascendâ€™s Unified Data Engineering Platform.
yanbe 2023/06/30
data-engineering
ãƒªãƒ³ã‚¯
Incident management for data teams
Data teams are no strangers to incidents. But thereâ€™s often no standard way of acting on data issues and many teams only sporadically declare data incidents. In this post Iâ€™ve teamed up with Jack Cook, analytics manager, from incident.io. incident.io have helped hundreds of engineering teams manage tens of thousands of incidents so thereâ€™s a thing or two we can learn from them. The five steps of i
yanbe 2023/06/04
data-engineering
ãƒªãƒ³ã‚¯
Rill | 5 Founders define the Modern Data Stack
We recently hosted an in-person event in San Francisco to discuss the Modern Data Stackâ€”ETL pipelines, quality monitoring, operational analytics, fast OLAP stores, and headless BI. The panel included: Census - Founder, CEO Boris Jabes. Census is a data automation platform that provides an easy way to sync your data warehouse with the apps you use. Bigeye - Co-founder, CTO Egor Gryaznov. Bigeye is
yanbe 2023/04/26
data-engineering
ãƒªãƒ³ã‚¯
ã‚¹ãƒ—ãƒ¬ãƒƒãƒ‰ã‚·ãƒ¼ãƒˆã‚’æ´»ç”¨ã—ã¦çµ„ç¹”æ¨ªæ–çš„ã«ãƒ†ãƒ¼ãƒ–ãƒ«ãƒ»ã‚«ãƒ©ãƒ ã®èª¬æ˜Žæ–‡ã‚’å…¥åŠ›ã—ãŸè©± - High Link ãƒ†ãƒƒã‚¯ãƒ–ãƒã‚°
ã“ã‚“ã«ã¡ã¯ã€‚æ ªå¼ä¼šç¤¾High Link ã§æ¥å‹™å§”è¨—ã¨ã—ã¦åƒã„ã¦ã„ã‚‹ã€ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®ikkiï¼ˆ@ikki_mzï¼‰ã§ã™ã€‚ ç§ãŸã¡ãƒ‡ãƒ¼ã‚¿ãƒãƒ¼ãƒ ã§ã¯ã€ã€Œãƒ‡ãƒ¼ã‚¿ã®æ°‘ä¸»åŒ–ã€ã‚’æŽ¨é€²ã—ã¦ãŠã‚Šã€å…¨ç¤¾å“¡ãŒãƒ‡ãƒ¼ã‚¿åˆ©æ´»ç”¨ã‚’è¡Œãˆã‚‹ã‚ˆã†ã«ã€dbtã‚’ç”¨ã„ãŸåˆ†æžåŸºç›¤ã®æ•´å‚™ã«å–ã‚Šçµ„ã‚“ã§ã„ã¾ã™ã€‚ tech.high-link.co.jp ãƒ‡ãƒ¼ã‚¿ã®æ°‘ä¸»åŒ–ã‚’æŽ¨é€²ã—ã¦ã„ãã«ã‚ãŸã‚Šã€ãƒ†ãƒ¼ãƒ–ãƒ«ãƒ»ã‚«ãƒ©ãƒ ã®èª¬æ˜Žæ–‡ã¯éžå¸¸ã«é‡è¦ãªå½¹å‰²ã‚’å ã‚ã¾ã™ã€‚ãƒ†ãƒ¼ãƒ–ãƒ«ã‚„ã‚«ãƒ©ãƒ ãŒä½•ã‚’æ„å‘³ã—ã¦ã„ã‚‹ã‹ã®èª¬æ˜Žã¯ã€åˆ†æžã‚’ã™ã‚‹ä¸Šã§ã¯ã¨ã¦ã‚‚é‡è¦ã§ã™ã€‚ ã—ã‹ã—ã€ã“ã®ãƒ†ãƒ¼ãƒ–ãƒ«ã‚„ã‚«ãƒ©ãƒ ã®èª¬æ˜Žã¯ãªã‹ãªã‹åŽ„ä»‹ã§ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‚’é–‹ç™ºã—ãŸé–‹ç™ºã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã¨ã‚³ãƒŸãƒ¥ãƒ‹ã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ã¨ã‚‰ãªã„ã¨ã€èª¬æ˜Žæ–‡ã‚’æ£ç¢ºã«æ›¸ãã“ã¨ãŒã§ãã¾ã›ã‚“ã€‚ ãã“ã§ç§ãŸã¡ã¯ã€dbtãƒ»ã‚¹ãƒ—ãƒ¬ãƒƒãƒ‰ã‚·ãƒ¼ãƒˆã‚’ä½¿ã£ã¦ã€ãƒ†ãƒ¼ãƒ–ãƒ«ã‚„ã‚«ãƒ©ãƒ ã®èª¬æ˜Žæ–‡ã®å…¥åŠ›ã‚’ã™ã‚‹ã¨ã„ã†ã€çµ„ç¹”æ¨ªæ–çš„ãªãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚’å®Ÿæ–½ã—ã¾ã—ãŸã€‚ èƒŒæ™¯ã¨èª²é¡Œ dbt de
yanbe 2023/04/24
data-engineering
ãƒªãƒ³ã‚¯
Enterprise Data Team
The GitLab Enterprise Data Team is responsible for empowering every GitLab team member to contribute to the data program and generate business value from our data assets. Welcome to the Enterprise Data Team Handbook Our Vision is to Contribute to GitLabâ€™s journey of becoming the leading AllOps platform by responsibly harnessing the power of data. In pursuit of our vision, we will focus on 4 outcom
yanbe 2023/04/20
data_management

data-engineering
ãƒªãƒ³ã‚¯
The next step for analytics engineering innovation | dbt Labs
If you use dbt today, youâ€™ve bought into our way of looking at the world. Originally authored in 2016, our manifesto has changed the way that analytics is done inside the modern data stack. From the post: Analytics teams have a workflow probl em today. Too often, analysts operate in isolation, and this creates suboptimal outcomes. Knowledge is siloed. We too often rewrite analyses that a colleague
yanbe 2023/04/20
data-engineering
ãƒªãƒ³ã‚¯
Big Data is Dead
For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size. â€œYour data is too big for your puny systems,â€ was the diagnosis, and the cure was to buy some new fancy techno logy that can handle massive scale. Of course, after the Big Data task force purchased all new tooling and migrated from Legacy systems, people found t
yanbe 2023/03/06
data-engineering
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ãŒäº‹æ¥è²¢çŒ®ã—ã¦ã„ã‚‹ã“ã¨ã‚’ç¤ºã™ãŸã‚ã®å–ã‚Šçµ„ã¿
2023å¹´2æœˆ16æ—¥é–‹å‚¬ã€ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ ãƒ¡ãƒ‡ã‚£ã‚¢äº‹æ¥éƒ¨ä¸»å‚¬ã®ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ã«é–¢ã™ã‚‹å‹‰å¼·ä¼šã€Œãƒ¡ãƒ‡ã‚£ã‚¢ã‚µãƒ¼ãƒ“ã‚¹ã«ãŠã‘ã‚‹ãƒ‡ãƒ¼ã‚¿ãƒ»AIã®æ´»ç”¨äº‹ä¾‹ #2ã€ç™»å£‡è³‡æ–™ã§ã™ã€‚ https://cyberagent.connpass.com/event/270224/
yanbe 2023/02/19
data-engineering
ãƒªãƒ³ã‚¯
ãƒžãƒ¼ã‚±ã‚¿ãƒ¼è‡ªèº«ãŒãƒ‡ãƒ¼ã‚¿ã‚’ç®¡ç†ã™ã‚‹ãŸã‚ã«ã€åºƒå‘Šé‹ç”¨ã«dbtã‚’å°Žå…¥ã—ãŸçµŒç·¯ã¨åŠ¹æžœ - MonotaRO Tech Blog
ã“ã‚“ã«ã¡ã¯ã€ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°éƒ¨é–€åºƒå‘Šã‚°ãƒ«ãƒ¼ãƒ—ã®å°æž—ã§ã™ã€‚ã“ã®è¨˜äº‹ã§ã¯ã‚ªãƒ³ãƒ©ã‚¤ãƒ³åºƒå‘Šé‹ç”¨ã«ä½¿ã£ã¦ã„ã‚‹ãƒ‡ãƒ¼ã‚¿å¤‰æ›å‡¦ç†ã‚’dbtã«ç§»è¡Œã—ãŸéŽç¨‹ã¨å¾—ã‚‰ã‚ŒãŸåŠ¹æžœã«ã¤ã„ã¦ã”ç´¹ä»‹ã—ã¾ã™ã€‚ ãƒ¢ãƒŽã‚¿ãƒã‚¦ã§ã¯ã€å…¨ç¤¾çš„ãªãƒ‡ãƒ¼ã‚¿æ´»ç”¨ç ”ä¿®ãªã©ã«ã‚ˆã‚Šã€ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°ã®ã‚ˆã†ãªãƒ“ã‚¸ãƒã‚¹ç³»ã®éƒ¨ç½²ã§ã‚‚ã€SQLã‚’è‡ªèº«ã§æ›¸ã„ã¦ãƒ‡ãƒ¼ã‚¿æŠ½å‡ºã‚’è¡Œã„ã€æ•°å—ã«åŸºã¥ã„ãŸæ„æ€æ±ºå®šã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ãã®ä¸€æ–¹ã§ã€é›†è¨ˆå¾Œã®æ•°å€¤ã®ã‚ºãƒ¬ã‚„ãƒ‰ãƒ¡ã‚¤ãƒ³å›ºæœ‰ã®ãƒ‡ãƒ¼ã‚¿ã®å“è³ªç®¡ç†ãªã©ã€æ´»ç”¨ãŒé€²ã‚“ã ä¼æ¥ãªã‚‰ã§ã¯ã®èª²é¡Œã¨ã„ã†ã®ã‚‚è¡¨é¢åŒ–ã—ã¦ãã‚‹ã‚ˆã†ã«ãªã£ã¦ãã¾ã—ãŸã€‚ ã‚ªãƒ³ãƒ©ã‚¤ãƒ³åºƒå‘Šé‹ç”¨ã«ãŠã„ã¦ã¯ã€æŠ•ä¸‹ã—ãŸè²»ç”¨ãªã©é…ä¿¡å®Ÿç¸¾ã®ãƒ¬ãƒãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã€åºƒå‘Šåª’ä½“ã¸ã®ãƒ‡ãƒ¼ã‚¿é€ä¿¡ãªã©ã®ã„ã‚ã‚†ã‚‹ELTã‚’å®‰å®šçš„ã«å›žã™ä»•çµ„ã¿ãŒå¿…è¦ã«ãªã‚Šã¾ã™ãŒã€å‡¦ç†ã®è‡ªå‹•åŒ–ã‚„ãƒ‡ãƒ¼ã‚¿ã®å“è³ªã¾ã§æ±‚ã‚ã‚‰ã‚Œã‚‹ã‚ˆã†ã«ãªã‚‹ã¨ã€ã€Œãƒ‡ãƒ¼ã‚¿ãŒæŠ½å‡ºã§ãã‚‹ã€ã ã‘ã§ã¯é™ç•ŒãŒè¦‹ãˆã¦ãã¦ã„ã¾ã—ãŸã€‚ãã“ã§ä»Šå›žã€ãƒžãƒ¼ã‚±ã‚¿ãƒ¼è‡ªèº«ãŒãƒ‡ãƒ¼ã‚¿ã‚’ç®¡ç†ã™ã‚‹ç«‹å ´ã«
yanbe 2023/02/15
data-engineering
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆå‘ã‘æ€§èƒ½å•é¡Œå¯¾å¿œã®åŸºç¤Ž
ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆã®çš†ã•ã‚“ã€æ¬¡ã®ã‚ˆã†ãªæ€§èƒ½å•é¡Œã«ã§ã‚ã£ãŸã“ã¨ãªã„ã§ã—ã‚‡ã†ã‹ã€‚ã€Œãƒ‡ãƒ¼ã‚¿ã®åŠ å·¥å‡¦ç†ãŒé…ã„ã‹ã‚‰ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã‚¿ã‚¤ãƒ—ã‚’ä¸Šã’ãŸãŒé€Ÿããªã‚‰ãªã‹ã£ãŸã€ã€Œæ©Ÿæ¢°å¦ç¿’ã®å¦ç¿’ãŒé…ã„ã‹ã‚‰ã€GPUã‚’å¢—ã‚„ã—ãŸãŒã€é€Ÿããªã‚‰ãªã‹ã£ãŸã€ã“ã†ã„ã£ãŸã¨ãã«ã©ã†ã™ã‚Œã°ã‚ˆã„ã‹èª¬æ˜Žã—ã¾ã™ã€‚
yanbe 2023/01/23
data-engineering
ãƒªãƒ³ã‚¯
20230118 kazaneya TeckTalk3 Data Standards and Open Data Initiatives by the Digital Agency of Japan
ãƒ‡ã‚¸ã‚¿ãƒ«åºãŒæ‰‹ãŒã‘ã‚‹ãƒ‡ãƒ¼ã‚¿æ¨™æº–ã€Œæ”¿åºœç›¸äº’é‹ç”¨æ€§ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ï¼ˆGovernment Interoperability Framework; GIFï¼‰ã€ãŠã‚ˆã³ã‚ªãƒ¼ãƒ—ãƒ³ãƒ‡ãƒ¼ã‚¿æ–½ç–ã«ã¤ã„ã¦ã®ç´¹ä»‹è³‡æ–™ã§ã™ã€‚ é¢¨éŸ³å±‹TeckTalk #3ã€Œãƒ‡ã‚¸ã‚¿ãƒ«åºã®ãƒ‡ãƒ¼ã‚¿æ¨™æº–ã‚„ã‚ªãƒ¼ãƒ—ãƒ³ãƒ‡ãƒ¼ã‚¿ã®æ°‘é–“æ´»ç”¨ã‚’è€ƒãˆã‚ˆã†ã€ã«ã¦ç™ºè¡¨â€¦
yanbe 2023/01/20
data-engineering
ãƒªãƒ³ã‚¯
Vertex AI ã® Streaming Ingestion ã§ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ AI ã‚’å®Ÿç¾ | Google Cloud å…¬å¼ãƒ–ãƒã‚°
â€»ã“ã®æŠ•ç¨¿ã¯ç±³å›½æ™‚é–“ 2022 å¹´ 9 æœˆ 21 æ—¥ã«ã€Google Cloud blog ã«æŠ•ç¨¿ã•ã‚ŒãŸã‚‚ã®ã®æŠ„è¨³ã§ã™ã€‚ ä¸æ£è¡Œç‚ºã®æ¤œå‡ºã€åºƒå‘Šã®ã‚¿ãƒ¼ã‚²ãƒƒãƒˆè¨å®šã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ ã‚¨ãƒ³ã‚¸ãƒ³ã‚’ã¯ã˜ã‚ã€æ•°å¤šãã®æ©Ÿæ¢°å¦ç¿’ï¼ˆMLï¼‰ã®ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«ã¯ã€ã»ã¼ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã®äºˆæ¸¬ãŒå¿…è¦ã§ã™ã€‚ã“ã†ã—ãŸäºˆæ¸¬ã®ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã¯ã€æœ€æ–°ãƒ‡ãƒ¼ã‚¿ã¸ã®ã‚¢ã‚¯ã‚»ã‚¹ã«å¤§ããä¾å˜ã—ã¦ãŠã‚Šã€æ•°ç§’ã®é…ã‚Œã§ã‚‚å°‘ãªã‹ã‚‰ã¬å½±éŸ¿ã«ã¤ãªãŒã‚Šã¾ã™ã€‚ã¨ã¯ã„ãˆã€é«˜ã‚¹ãƒ«ãƒ¼ãƒ—ãƒƒãƒˆã®æ›´æ–°ã¨ä½Žãƒ¬ã‚¤ãƒ†ãƒ³ã‚·ã®ãƒ‡ãƒ¼ã‚¿å–å¾—ã«å¿…è¦ãªã‚¤ãƒ³ãƒ•ãƒ©ã‚¹ãƒˆãƒ©ã‚¯ãƒãƒ£ã‚’æº–å‚™ã™ã‚‹ã®ã¯å®¹æ˜“ã§ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚ ä»Šæœˆã‹ã‚‰ã€Vertex AI Matching Engine ã¨ Feature Store ã¯ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã®ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°å–ã‚Šè¾¼ã¿æ©Ÿèƒ½ã€Streaming Ingestion ã® Preview ç‰ˆã‚’æä¾›é–‹å§‹ã—ã¾ã™ã€‚Matching Engine ã¯ãƒ™ã‚¯ãƒˆãƒ«æ¤œç´¢ç”¨ã®ãƒ•ãƒ«ãƒžãƒãƒ¼ã‚¸ãƒ‰
yanbe 2023/01/09
data-engineering
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢é“ã®ä¿ºã®ãƒã‚¤ãƒ–ãƒ«
å…ˆäººã®çŸ¥æµã«å¦ã¶ ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢é“ã§ã€æœ¬å½“ã«è‰¯ã‹ã£ãŸï¼èªã¿ç‰©ã‚’ã€ä¸å®šæœŸã«è¿½è¨˜ã—ã¦ã„ãã€‚ A Beginnerâ€™s Guide to Data Engineering â€” Part I ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã‚’ã“ã‚Œã‹ã‚‰å§‹ã‚ã‚‹äººã«ã€å¿…ãšè–¦ã‚ã‚‹è¨˜äº‹ã€‚ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®åŸºæœ¬ã‚’å¦ã¹ã‚‹ã‹ã¤ã€ã©ã†ã„ã†ä¸–ç•Œã«åºƒãŒã£ã¦ã„ãã®ã‹ã¾ã§ã€ä¸€æ°—ã«å¦ã¹ã‚‹ã®ã§ã¨ã¦ã‚‚è‰¯ã„ã€‚ Functional Data Engineering â€” a modern paradigm for batch data processing é–¢æ•°åž‹ãƒ‘ãƒ©ãƒ€ã‚¤ãƒ ã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã®æ§‹ç¯‰æ–¹æ³•ã€‚ã“ã‚Œã‚’åˆã‚ã¦èªã‚“ã æ™‚ã®è¡æ’ƒã¯ä»Šã§ã‚‚å¿˜ã‚Œãªã„ã—ã€ãƒ•ãƒ«ã‚¹ã‚¯ãƒ©ãƒƒãƒã‹ã‚‰dbtã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã«ãªã£ã¦ã‚‚å¥åœ¨ãªè¨è¨ˆæ‰‹æ³•ã€‚ Engineers Shouldnâ€™t Write ETL: A Guide to Building a High Function
yanbe 2023/01/03
data-engineering
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿ãƒžãƒ¼ãƒˆé–‹ç™ºãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚’é€šã—ã¦æ°—ã¥ã„ãŸãƒ¡ãƒ³ãƒ†ãƒŠãƒ³ã‚¹ã—ã‚„ã™ã„ã‚¯ã‚¨ãƒªã®æ›¸ãæ–¹ï½œrossokuno
ãƒžãƒãƒ¼ãƒ•ã‚©ãƒ¯ãƒ¼ãƒ‰ã€€åˆ†æžæŽ¨é€²å®¤ã€€ã‚¢ãƒŠãƒªãƒ†ã‚£ã‚¯ã‚¹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®å¥¥é‡Žã§ã™ï¼ ç§ã¯ã€ç¾åœ¨ãƒ“ã‚¸ãƒã‚¹ãƒ¦ãƒ¼ã‚¶ãƒ¼ãŒä½¿ã„ã‚„ã™ã„ãƒ‡ãƒ¼ã‚¿ãƒžãƒ¼ãƒˆé–‹ç™ºãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«æºã‚ã£ã¦ãŠã‚Šã€ãã®ä¸ã§dbt Coreã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã®æ§‹ç¯‰ã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ãƒ‡ãƒ¼ã‚¿ã‚¢ãƒŠãƒªã‚¹ãƒˆ1åã€ã‚¢ãƒŠãƒªãƒ†ã‚£ã‚¯ã‚¹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢2åã®åˆè¨ˆ3åã§é–‹ç™ºã‚’é€²ã‚ã¦ã„ã¾ã™ã€‚ ã“ã®ãƒ–ãƒã‚°ã§ã¯ã€ãã®ãƒ‡ãƒ¼ã‚¿ãƒžãƒ¼ãƒˆé–‹ç™ºãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã«ãŠã„ã¦ã€ãƒ¡ãƒ³ãƒ†ãƒŠãƒ³ã‚¹ã—ã‚„ã™ã„ã‚¯ã‚¨ãƒªã‚’ä½œã‚‹ãŸã‚ã®ã‚¢ã‚¤ãƒ‡ã‚¢ãŒå‡ºã¦ããŸã®ã§å…±æœ‰ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ ä¼ãˆãŸã„ã“ã¨ã¯å¤§ããä»¥ä¸‹ã®2ã¤ã§ã™ã€‚ SQLãƒªãƒ³ã‚¿ãƒ¼SQLfluffã‚’å…¥ã‚Œã‚‹ã¨ãã‚Œãªã‚Šã«åˆ†ã‹ã‚Šã‚„ã™ã„ã‚¯ã‚¨ãƒªã«ãªã‚‹ ãƒªãƒ³ã‚¿ãƒ¼ã§ã¯ã‚«ãƒãƒ¼ã§ããªã„åˆ†ã‹ã‚Šã‚„ã™ã„æ›¸ãæ–¹ã®ã‚¢ã‚¤ãƒ‡ã‚¢ ã§ã¯æ—©é€Ÿç´¹ä»‹ã—ã¦ã„ãã¾ã™ï¼ SQLãƒªãƒ³ã‚¿ãƒ¼SQLfluffã‚’å…¥ã‚Œã‚‹ã¨ãã‚Œãªã‚Šã«åˆ†ã‹ã‚Šã‚„ã™ã„ã‚¯ã‚¨ãƒªã«ãªã‚‹SQLfluffã¨ã„ã†SQLãƒªãƒ³ã‚¿ãƒ¼ãŒã‚ã‚Šã¾ã™ã€‚ã™ã§ã«æ—¥æœ¬èªžã§ã‚‚ç´¹ä»‹ã•ã‚Œã¦ã„ã‚‹è¨˜
yanbe 2022/12/21
data-engineering
ãƒªãƒ³ã‚¯
Data Reliability Engineering ã‚’ã¯ã˜ã‚ãŸè©± | ãƒ¡ãƒ«ã‚«ãƒªã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°
ã“ã‚“ã«ã¡ã¯ã€ãƒ¡ãƒ«ã‚«ãƒªUS ã§Data Engineerã‚’ã—ã¦ã„ã‚‹ @hatone ã§ã™ã€‚ ãƒ¡ãƒ«ã‚«ãƒªUSã®Data Engineering Team(DEãƒãƒ¼ãƒ ) ã§ã¯ã€MLã‚„BIã€ä¼šè¨ˆã€ãƒžãƒ¼ã‚±ãƒ†ã‚£ãƒ³ã‚°ãªã©ã€å¤šå²ã«ã‚ãŸã‚‹ãƒãƒ¼ãƒ ã¨ãƒ—ãƒãƒ€ã‚¯ãƒˆã«ãƒ‡ãƒ¼ã‚¿ã‚’æä¾›ã™ã‚‹å¿…è¦ä¸å¯æ¬ ãªãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã‚’æ‹…å½“ã—ã¦ã„ã¾ã™ã€‚ ä»Šå¹´ã®7æœˆã« Data Reliability Engineering Team (DRE: ãƒ‡ãƒ¼ã‚¿ä¿¡é ¼æ€§ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ãƒãƒ¼ãƒ ) ã¨ã„ã†ãƒãƒ¼ãƒ ã‚’ç«‹ã¡ä¸Šã’ã¾ã—ãŸã€‚æ—¥ã€…å¢—ãˆç¶šã‘ã‚‹DBä¸Šã®ãƒ¬ã‚³ãƒ¼ãƒ‰ã¨ãã®ãƒ‡ãƒ¼ã‚¿ã®æ´»ç”¨ã‚’æ»žã‚Šãªãå®Ÿç¾ã™ã‚‹ãŸã‚ã€Data Engineeringã«å¯¾ã—DevOpsã¨SREã®åŽŸå‰‡ã‚’é©ç”¨ã™ã‚‹ã“ã¨ã§ã‚¹ã‚±ãƒ¼ãƒ«ã•ã›ã¦ã„ã“ã†ã¨ã—ã¦ã„ã¾ã™ã€‚ Mercari Advent Calendar 2022 ã®19æ—¥ç›®ã§ã¯ã€ãƒãƒ¼ãƒ å„ã€…ãŒå®‰å¿ƒã—ã¦ãƒ‡ãƒ¼ã‚¿ã‚’åˆ©ç”¨ã§ãã‚‹ã‚ˆã†ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã®ç›£è¦–ãƒ»è‡ªå‹•
yanbe 2022/12/19
data-engineering
ãƒªãƒ³ã‚¯
1