DuckDB を使ã£ãŸãƒ‡ãƒ¼ã‚¿å“質ä¿è¨¼ã®å®Ÿè·µ - Timee Product Team Blog
ã“ã®è¨˜äº‹ã¯ Timee Advent Calendar 2024 シリーズ 1 ã®5日目ã®è¨˜äº‹ã§ã™ã€‚ ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€‚タイミー㮠DRE ãƒãƒ¼ãƒ ã® chanyou ã§ã™ã€‚2024å¹´ã®3月㫠DRE ãƒãƒ¼ãƒ ã«ã‚¸ãƒ§ã‚¤ãƒ³ã—ã¦ã€ç¤¾å†…ã®ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã‚’作ã£ã¦é‹ç”¨ã—ã¦ã„ã¾ã™ã€‚ DuckDB を使ã£ã¦ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã§æ‰±ã†ãƒ‡ãƒ¼ã‚¿ã®å“質をä¿è¨¼ã—始ã‚ãŸã®ã§ã€ãã®å†…容をã”紹介ã—ã¾ã™ã€‚ データå“質ã¨å®Œå…¨æ€§ タイミーã®ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã§é‡è¦–ã—ã¦ã„るデータå“質 タイミーã§ã¯ã€DMBOK ã‚’å‚考ã«ä»¥ä¸‹ã®ãƒ‡ãƒ¼ã‚¿å“質をé‡è¦–ã—ã¦è¨è¨ˆã‚„日々ã®é‹ç”¨ã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ 特性 æ„味 完全性 データãŒæ¬ æã—ã¦ã„ãªã„ã‹ é©æ™‚性 å¿…è¦ãªã¨ãã«ã™ãã«ãƒ‡ãƒ¼ã‚¿ã‚’å‚ç…§ã§ãる㋠一æ„性 データãŒé‡è¤‡ã—ã¦ã„ãªã„㋠一貫性 型・タイムゾーン・表記æºã‚Œãªã©ã€å€¤ã®æ›¸å¼ã‚„æ„味ãŒçµ±ä¸€ã•ã‚Œã¦ã„る㋠今回ã¯å®Œå…¨æ€§ã«ãƒ•ã‚©ãƒ¼ã‚«ã‚¹ã—ã¾ã™ã€‚ 完全性ãŒæãªã‚れるタイミング 上記ã®é€šã‚Š
DataFrame ã‚’ Validation ã™ã‚‹ pandera 入門
ã¯ã˜ã‚ã« Python を用ã„ã¦ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚’è¡Œã†ã«ã‚ãŸã‚Šã‚ˆã使ã‚れるライブラリã¨ã—㦠pandas ãŒã‚ã‚Šã¾ã™ã€‚ pandas ã¯å¤§å¤‰ä½¿ã„å‹æ‰‹ã®è‰¯ã„ライブラリã§ã™ãŒã€å¤šãã®å ´åˆãƒ‡ãƒ¼ã‚¿ã‚’丸ã”㨠pd.DataFrame åž‹ã§ä¿æŒã™ã‚‹ãŸã‚「ã©ã®ã‚ˆã†ãªåˆ—ã‚’æŒã£ã¦ã„ã‚‹ã®ã‹ã€ã€ã€Œå„列ãŒã©ã®ã‚ˆã†ãªåž‹ã‹ã€ã€ã€Œå„列ã®å€¤ã«ã©ã®ã‚ˆã†ãªå€¤ãŒå…¥ã‚Šã†ã‚‹ã®ã‹ã€ç‰ãŒã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ã‚’一見ã—ãŸã ã‘ã§ã¯åˆ†ã‹ã‚‰ãªã„ã“ã¨ãŒå¤šã„ã§ã™ã€‚ çµæžœã¨ã—ã¦å‡¦ç†ãŒãƒ–ラックボックス化ã—ã¦ã—ã¾ã„ã€ãƒ‡ãƒãƒƒã‚°ã‚³ã‚¹ãƒˆã®å¢—åŠ ã‚„ã‚³ãƒ¼ãƒ‰ã®å¯èªæ€§ä½Žä¸‹ã¨ã„ã£ãŸå•é¡Œã‚’生ã˜ã•ã›ã‚‹ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ ã“ã®å•é¡Œã¸ã®è§£æ±ºç–ã®ä¸€ã¤ã¨ã—ã¦ã€æœ¬è¨˜äº‹ã§ã¯ãƒ‡ãƒ¼ã‚¿ãƒ•ãƒ¬ãƒ¼ãƒ ã®ãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³æ©Ÿèƒ½ã‚’æä¾›ã™ã‚‹ãƒ©ã‚¤ãƒ–ラリã§ã‚ã‚‹ pandera を紹介ã—ã¾ã™ã€‚ pandera ã¨ã¯ データ処ç†ãƒ‘イプラインã®å¯èªæ€§ã¨ãƒãƒã‚¹ãƒˆã•ã‚’高ã‚ã‚‹ãŸã‚ã« dataframe ã«å¯¾ã—ã¦ãƒ‡ãƒ¼ã‚¿æ¤œè¨¼ã‚’è¡Œã†æ©Ÿèƒ½ã‚’æä¾›ã™ã‚‹ãƒ©
Pydanticã«6行書ãåŠ ãˆã‚‹ã ã‘ã§ã€ArgumentParserã¨åŒã˜ä½¿ã„æ–¹ãŒã§ãã‚‹ã‹ã‚‰ä¾¿åˆ©ã‚ˆã€ã¨ã„ã†å°ãƒã‚¿ - Qiita
ã“ã®è¨˜äº‹ã«ã¤ã„㦠Pythonã®ãƒãƒªãƒ‡ãƒ¼ã‚¿ãƒ¼ï¼ˆPydantic）をã€Pythonã®å¼•æ•°ã®ãƒ‘ーサーã¨ã—ã¦ä½¿ã†æ–¹æ³•ã‚’紹介ã—ã¾ã™ 何ãŒå¬‰ã—ã„ã®ï¼Ÿ Pydantic以外ã®OSSライブラリã¯ä¸è¦ã§ã™ Pythonファイルã«æ¸¡ã•ã‚ŒãŸå¼•æ•°ã‚’検証ã€åž‹å¤‰æ›ã•ã›ã‚‹ã“ã¨ãŒã§ãã¾ã™ Pydanticã§å®šç¾©ã‚’書ãã ã‘ã§ã‚ˆã„ãŸã‚ã€ArgumentParserよりも楽ã§ã™ IDEã®è£œå®ŒãŒåŠ¹ãよã†ã«ãªã‚Šã¾ã™ 方法 BaseModelを継承ã—ãŸã‚¯ãƒ©ã‚¹ã«ã€ä»¥ä¸‹ã®é–¢æ•°ã‚’書ãåŠ ãˆã¾ã™ @classmethod def parse_args(cls): parser = ArgumentParser() for k in cls.schema()["properties"].keys(): parser.add_argument(f"-{k[0:1]}", f"--{k}") return cls.parse_obj(parser
Introduction to Data Validation: MLOps ã«ãŠã‘ã‚‹é‡è¦æ€§ã¨ãã®åˆ†é¡žã€å®Ÿç”¨ä¸Šã®æ³¨æ„点ã«ã¤ã„㦠- Qiita
スã‚ーマã«åŸºã¥ã Validation ã®æ©Ÿæ¢°å¦ç¿’ã«ç‰¹æœ‰ãªèª²é¡Œ スã‚ーマã«åŸºã¥ã Validation ã®ã†ã¡ã€æ©Ÿæ¢°å¦ç¿’ã«ç‰¹æœ‰ã®å•é¡Œã‚’ã“ã“ã§ã¯3ã¤å–り上ã’ã¾ã™ã€‚ 機械å¦ç¿’モデルã®ç‰¹å¾´é‡ã¯èŽ«å¤§ã«ãªã‚‹ã“ã¨ãŒã‚ã‚Šã€æ‰‹å‹•ã§ã™ã¹ã¦ã‚’書ã出ã™ã®ã¯ç¾å®Ÿçš„ã§ãªã„ スã‚ーマを満ãŸã•ãªã„ã“ã¨ã¯ã€å¿…ãšã—も悪ã„ã“ã¨ã§ã¯ãªã„ 機械å¦ç¿’モデルã«å…¥åŠ›ã•ã‚Œã‚‹ã®ã¯ã€æ§‹é€ 化データã ã‘ã§ã¯ãªã„ 機械å¦ç¿’モデルã®ç‰¹å¾´é‡ã¯èŽ«å¤§ã«ãªã‚‹ã“ã¨ãŒã‚ã‚Šã€æ‰‹å‹•ã§ã™ã¹ã¦ã‚’書ã出ã™ã®ã¯ç¾å®Ÿçš„ã§ã¯ãªã„ 通常㮠Web サービスã¨ã¯ç•°ãªã‚Šã€æ©Ÿæ¢°å¦ç¿’ã§ã¯ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ãƒ†ã‚£ã‚¹ãƒˆãŒè¤‡é›‘㪠SQL を書ã„ã¦è†¨å¤§ãªç‰¹å¾´é‡ã‚’作り出ã™ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ ãŸã¨ãˆã°ã€éŽåŽ»ã® 1 ヶ月ã®è¡Œå‹•å±¥æ´ã‹ã‚‰è³¼å…¥äºˆæ¸¬ã‚’è¡Œã†å ´åˆã€éŽåŽ»ã«è¡Œã£ãŸè¡Œå‹• (ãƒã‚°ã‚¤ãƒ³ãªã©) ã®å›žæ•°ã‚’日毎ã«é›†è¨ˆã—ã€ç‰¹å¾´é‡ã¨ã™ã‚‹ã®ã¯ä¸€èˆ¬çš„ã§ã™ã€‚具体的ã«ã¯ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®ãƒã‚°ã‚¤ãƒ³å›žæ•°ã€è³¼å…¥å›žæ•°ã‚’日毎ã«é›†è¨ˆã—ã¦æ©Ÿæ¢°å¦ç¿’モ
GitHub - GoogleCloudPlatform/cloud-data-quality: Data Quality Engine for BigQuery
CloudDQ is a cloud-native, declarative, and scalable Data Quality validation Command-Line Interface (CLI) application for Google BigQuery. CloudDQ allows users to define and schedule custom Data Quality checks across their BigQuery tables. Data Quality validation results will be available in another BigQuery table of their choice. Users can then build dashboards or consume data quality outputs pro
Great Expectations: have confidence in your data, no matter what
Built for today’s data complexityModern data systems are powerful—but fragile. Data breaks, pipelines drift, and nobody wants to be the last to know. GX gives your team tools to: Validate critical data across your pipelines Share a common language for data quality Build trust across technical and business teams What you need, when you need itSet up in minutes. Scale with confidence. GX is a plat
Cloud Composerã«ã‚ˆã‚‹ãƒ‡ãƒ¼ã‚¿ãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ ～常ã«æ£ç¢ºãªãƒ‡ãƒ¼ã‚¿é›†è¨ˆã‚’実ç¾ã™ã‚‹ãŸã‚ã«ï½ž - ZOZO TECH BLOG
ã“ã‚“ã«ã¡ã¯ã€‚ECプラットフォーム部データエンジニアã®é è—¤ã§ã™ã€‚ç¾åœ¨ã€ç§ã¯æŽ¨è–¦åŸºç›¤ãƒãƒ¼ãƒ ã«æ‰€å±žã—ã¦ã€ãƒ‡ãƒ¼ã‚¿é›†è¨ˆåŸºç›¤ã®é‹ç”¨ã‚„DMP・広告ã¾ã‚ã‚Šã®ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ãªã©ã«å¾“事ã—ã¦ã„ã¾ã™ã€‚ 以å‰ã€ç§ãŸã¡ã®ãƒãƒ¼ãƒ ã§ã¯ã‚¯ã‚¨ãƒªç®¡ç†ã«Lookerã‚’å°Žå…¥ã™ã‚‹ã“ã¨ã§ã€ãƒ‡ãƒ¼ã‚¿ã‚¬ãƒãƒŠãƒ³ã‚¹ã‚’効ã‹ã›ãŸãƒ‡ãƒ¼ã‚¿é›†è¨ˆåŸºç›¤ã‚’実ç¾ã—ã¾ã—ãŸã€‚詳細ã¯ã€ä»¥å‰ç´¹ä»‹ã—ãŸãƒ‡ãƒ¼ã‚¿é›†è¨ˆåŸºç›¤ã«ã¤ã„ã¦ã¯ä»¥ä¸‹ã®éŽåŽ»è¨˜äº‹ã‚’ã”覧ãã ã•ã„。 techblog.zozo.com 本記事ã§ã¯ã€ãƒ‡ãƒ¼ã‚¿é›†è¨ˆåŸºç›¤ã«ã€Œãƒ‡ãƒ¼ã‚¿ãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã€ã®æ©Ÿèƒ½ã‚’åŠ ãˆã¦å¸¸ã«æ£ç¢ºãªãƒ‡ãƒ¼ã‚¿é›†è¨ˆã‚’è¡Œãˆã‚‹ã‚ˆã†ã«æ”¹è‰¯ã™ã‚‹æ‰‹æ®µã‚’ãŠä¼ãˆã—ã¾ã™ã€‚ データãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã¨ã¯ ãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³å°Žå…¥å¾Œã®ãƒ‡ãƒ¼ã‚¿é›†è¨ˆåŸºç›¤ ジョブãƒãƒƒãƒˆæ§‹ç¯‰ テンプレートã«ã‚ˆã‚‹åŠ¹çŽ‡çš„ãªDAGã®ä½œæˆ DAGé–“ã®ä¾å˜é–¢ä¿‚ã®è¨å®šæ–¹æ³• ãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³DAGã®ã‚¿ã‚¹ã‚¯æ§‹æˆ ã¾ã¨ã‚ データãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã¨ã¯ データãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã¨ã¯ãƒ‡ãƒ¼ã‚¿
ランã‚ング
ランã‚ング
ãŠçŸ¥ã‚‰ã›
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - TopQuadrant/shacl: SHACL API in Java based on Apache Jena
Automate data validation with DVT | Google Cloud Blog
GitHub - GoogleCloudPlatform/professional-services-data-validator: Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
How Google Uses Ibis for its Data Validation Tool (DVT)
Welcome to kedro-pandera’s documentation! | kedro-pandera 0.2.3 documentation
pandera documentation
Dive into TensorFlow Data Validation
Rules for Knowledge Graphs Rules
GitHub - veleda/shacl-masterclass: SHACL masterclass at Knowledge Connexions 2020, Knowledge Graph Conference 2021, 2022 & 2023.
Two approaches for data validation in ML production
Validating RDF Data
https://dl.acm.org/doi/pdf/10.1145/3399579.3399868
Capture Calendar | European Union, archived by Publications Office of the European Union | Archive-It Wayback Machine
{{#tags}}- {{label}}
{{/tags}}