[B! schemaEvolution] manboubirdã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

manboubird id:manboubird

schemaEvolutionã«é–¢ã™ã‚‹manboubirdã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (65)

${{author_name}}$

{{author_name}} {{created}}

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

{{#following_bookmarks}}

${{author_name}}$

{{author_name}} {{created}}

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

{{/following_bookmarks}}

{{/is_wiped}}

DBã‚¹ã‚ãƒ¼ãƒžå¤‰æ›´ç®¡ç†ãƒ„ãƒ¼ãƒ« sqldef ã‚’è©¦ã—ã¦ã¿ãŸ - Qiita
1. sqldef ã¨ã¯ sqldef ã¯ "The easiest idempotent MySQL/PostgreSQL/SQLite3/SQL Server schema management by SQL." ã¨è¬³ã£ã¦ã„ã‚‹DBã‚¹ã‚ãƒ¼ãƒžå¤‰æ›´ç®¡ç†ãƒ„ãƒ¼ãƒ«ã§ã™ã€‚ é€šå¸¸ã®é–‹ç™ºã«ãŠã„ã¦ DDL æ–‡ã‚’ç®¡ç†ã™ã‚‹å ´åˆã€ç’°å¢ƒã‚’ï¼‘ã‹ã‚‰ä½œã‚‹ã‚ˆã†ã« CREATE TABLE æ–‡ãªã©æ–°è¦ä½œæˆ DDL æ–‡ã‚’æº–å‚™ã™ã‚‹ã¨å…±ã«ã€æ—¢ã«ä½œæˆæ¸ˆã¿ã®ç’°å¢ƒã§ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’å¤‰æ›´ã™ã‚‹ãŸã‚ã« ALTER TABLE æ–‡ãªã©å·®åˆ†é©ç”¨ DDL æ–‡ã‚’æº–å‚™ã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ã“ã®ï¼’ç¨®é¡žã® DDL æ–‡ã‚’äºŒé‡ç®¡ç†ã—ãªã„ã¨ã„ã‘ãªã„ã¨ã„ã†ã®ã¯ DBA ã«ã¨ã£ã¦ã¯é ã®æ‚©ã¾ã—ã„å•é¡Œã§ã—ãŸã€‚ï¼ˆå·®åˆ†é©ç”¨ DDL æ–‡ã®ã¿æº–å‚™ã—ã€ï¼‘ã‹ã‚‰ç’°å¢ƒã‚’ä½œã‚‹å ´åˆã‚‚å…¨ã¦ã®å¤‰æ›´ã‚’é©ç”¨ã™ã‚‹ã¨ã„ã†æ‰‹ã‚‚ã‚ã‚Šã¾ã™ãŒâ€¦ï¼‰ sqldef ã‚’åˆ©ç”¨ã™ã‚‹ã¨ã€å¤‰æ›´é©ç”¨å…ˆ DB ã®ç¾åœ¨ã®çŠ¶æ³ã¨æ–°è¦ä½œæˆ
manboubird 2022/01/10
schemaManagement

sqldef

schemaEvolution

sql
ãƒªãƒ³ã‚¯
Delta Lake ã§ã®ã‚¹ã‚ãƒ¼ãƒžï¼ˆschemaï¼‰DB ã®é©ç”¨ãƒ»å±•é–‹ã¨ã¯
ãƒ‡ãƒ¼ã‚¿ãƒ–ãƒªãƒƒã‚¯ã‚¹ã® Notebook ã‚·ãƒªãƒ¼ã‚ºã‚’è©¦ã™ ãƒ‡ãƒ¼ã‚¿ã¯å¸¸ã«é€²åŒ–ã—ã€è“„ç©ã•ã‚Œã¦ã„ãã¾ã™ã€‚ç§ãŸã¡äººé–“ã®æ—¥ã€…ã®çµŒé¨“ã¨ä¼¼ã¦ã„ã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã€‚ç§ãŸã¡ã¯ã€è‡ªèº«ã®å‘¨ã‚Šã®ä¸–ç•Œã®å¤‰åŒ–ã«ã¤ã„ã¦ã„ããŸã‚ã«ã€å¸¸ã«æ–°ã—ã„ãƒ‡ãƒ¼ã‚¿ã‚’å–ã‚Šè¾¼ã¿ã€èªè˜ã—ã€ã¨ãã«ã¯ãã®ä¸ã‹ã‚‰æ–°ãŸãªæ¦‚å¿µã‚„è§£é‡ˆã‚’å¾—ã¾ã™ã€‚ã“ã®ã‚ˆã†ãªèªè˜ãƒ¢ãƒ‡ãƒ«ã¯ã€ã¾ã•ã«ãƒ†ãƒ¼ãƒ–ãƒ«ã®ã‚¹ã‚ãƒ¼ãƒžãã®ã‚‚ã®ã§ã™ã€‚ã©ã¡ã‚‰ã‚‚ã€æ–°ã—ãå¾—ã‚‹æƒ…å ±ã®åˆ†é¡žã¨å‡¦ç†ã®ã—ã‹ãŸã‚’æ±ºã‚ã‚‹å½¹å‰²ã‚’æŒã£ã¦ã„ã¾ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«ãŠã‘ã‚‹ã‚¹ã‚ãƒ¼ãƒžã¨ã¯ ï¼šãã‚‚ãã‚‚ã€Œã‚¹ã‚ãƒ¼ãƒžï¼ˆschemaï¼‰ã€ã¨ã¯ã€æ—¥æœ¬äººã«ã¨ã£ã¦ã‚‚é¦´æŸ“ã¿ã®ã‚ã‚‹ã€Œã‚¹ã‚ãƒ¼ãƒ ï¼ˆschemeï¼‰ã€ã¨ã„ã†è¨€è‘‰ã®æ´¾ç”Ÿèªžã§ã™ã€‚è¨ˆç”»ã‚„å›³ãªã©ã®æ„å‘³ã‚’æŒã¡ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹é–¢é€£ã ã‘ã§ãªãã€å“²å¦ã‚„å¿ƒç†å¦ã§ä½¿ã‚ã‚Œã¦ã„ã‚‹è¨€è‘‰ã§ã‚‚ã‚ã‚Šã¾ã™ã€‚ã“ã®è¨˜äº‹ã§èª¬æ˜Žã™ã‚‹ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‚¹ã‚ãƒ¼ãƒžï¼ˆDBã‚¹ã‚ãƒ¼ãƒžï¼‰ã¨ã¯ã€ç°¡å˜ã«è¨€ãˆã°ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã®æ§‹é€ ã‚„æ•´ç†ã®ä»•æ–¹ã®ã“ã¨ã§ã™ã€‚ç´°ã‹ãªå®šç¾©ã¯ã€ãƒ‡ãƒ¼ã‚¿
manboubird 2020/12/05
deltalake

dataLake

databricks

schemaManagement

schemaEvolution

schemaValidation

dataQuality
ãƒªãƒ³ã‚¯
API Versioning Has No "Right Way"
API versioning is a really difficult topic, and sometimes seen as a merely religious debate. Itâ€™s done differently at different companies, and different teams within different companies often vary. Some folks move from approach A to approach B, and when approach B solves their specific issues they act like approach B is "best" and nobody should ever use approach A. Thatâ€™s all well and good until t
manboubird 2020/07/04
schemaEvolution

api

restApi

versioning

design

designPattern

microServiceArchitecture
ãƒªãƒ³ã‚¯
Apache Iceberg - Apache Icebergâ„¢
What is Apache Icebergâ„¢? Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. Expressive SQL Iceberg supports flexible SQL commands to merge new data, update existing rows, an
manboubird 2020/07/02
apacheIceberg

schemaEvolution

serde
ãƒªãƒ³ã‚¯
How to use data version control (dvc) in a machine learning project
manboubird 2020/02/20
dataManagement

versioning

dvc

schemaEvolution
ãƒªãƒ³ã‚¯
p553-maddox.pdf
manboubird 2019/12/22
Decibel: The Relational Dataset Branching System

dataManagement

dataset

versioning

vldb

paper

schemaEvolution
ãƒªãƒ³ã‚¯
Building Machine Learning Pipelines
What Are Machine Learning Pipelines?Who Is This Book For?Why TensorFlow and TensorFlow Extended?Overview of the ChaptersConventions Used in This BookUsing Code ExamplesOâ€™Reilly Online LearningHow to Contact UsAcknowledgments Why Machine Learning Pipelines?When to Think About Machine Learning PipelinesOverview of the Steps in a Machine Learning PipelineData Ingestion and Data VersioningData Validat
manboubird 2019/11/24
book

machineLearning

tensorFlow

dataManagement

oreilly

etl

artificialIntelligence

versioning

schemaEvolution

validation
ãƒªãƒ³ã‚¯
PostgreSQL at Scale: Database Schema Changes Without Downtime
manboubird 2019/10/28
postgres

schemaEvolution
ãƒªãƒ³ã‚¯
How Netflix microservices tackle dataset pub-sub
By Ammar Khaku IntroductionIn a microservice architecture such as Netflixâ€™s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as they change over time. One example displaying the need for d
manboubird 2019/10/18
netflix
ãƒªãƒ³ã‚¯
Productionizing Machine Learning with Delta Lake
manboubird 2019/09/28
databricks

deltalake

dataLake

dataManagement

schemaManagement

deltalakes

schemaEvolution
ãƒªãƒ³ã‚¯
Open Sourcing Delta Lake
manboubird 2019/04/29
schemaEvolution

databricks

dataLake

deltalake

dataReliability

dataManagement

dataQuality

schemaManagement
ãƒªãƒ³ã‚¯
The Datum VEAâ€Šâ€”â€ŠValidate, Evolve and Anonymize your data with Data Schemas
manboubird 2019/03/18
validation

jsonSchema

schemaManagement

schemaEvolution
ãƒªãƒ³ã‚¯
ã‚¹ã‚ãƒ¼ãƒžãƒ¬ã‚¹ã‚«ãƒ©ãƒ ãƒŠãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã€ŒYosegiã€ã§å®Ÿç¾ã™ã‚‹ ã‚¹ã‚ãƒ¼ãƒžã®æŸ”è»Ÿæ€§ã¨å‡¦ç†æ€§èƒ½ã‚’ä¸¡ç«‹ã—ãŸãƒã‚°åŽé›†ã‚·ã‚¹ãƒ†ãƒ / Hadoop / Spark Conference Japan 2019 #hcj2019
ã‚¹ã‚ãƒ¼ãƒžãƒ¬ã‚¹ã‚«ãƒ©ãƒ ãƒŠãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã€ŒYosegiã€ã§å®Ÿç¾ã™ã‚‹ ã‚¹ã‚ãƒ¼ãƒžã®æŸ”è»Ÿæ€§ã¨å‡¦ç†æ€§èƒ½ã‚’ä¸¡ç«‹ã—ãŸãƒã‚°åŽé›†ã‚·ã‚¹ãƒ†ãƒ / Hadoop / Spark Conference Japan 2019 #hcj2019 Hadoop / Spark Conference Japan 2019 ã§ç™ºè¡¨ã•ã›ã¦é ‚ã„ãŸã€ ã€Œã‚¹ã‚ãƒ¼ãƒžãƒ¬ã‚¹ã‚«ãƒ©ãƒ ãƒŠãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã€ŒYosegiã€ã§å®Ÿç¾ã™ã‚‹ ã‚¹ã‚ãƒ¼ãƒžã®æŸ”è»Ÿæ€§ã¨å‡¦ç†æ€§èƒ½ã‚’ä¸¡ç«‹ã—ãŸãƒã‚°åŽé›†ã‚·ã‚¹ãƒ†ãƒ ã€ã®ç™ºè¡¨è³‡æ–™ã¨ãªã‚Šã¾ã™ã€‚
manboubird 2019/03/16
yahoo

yosegi

schemaManagement

schemaEvolution

slide

columnarFormat

logCollection
ãƒªãƒ³ã‚¯
DataFlow: Dealing with BigQuery schema change
manboubird 2019/03/12
schemaEvolution

schemaManagement

bigQuery

validation
ãƒªãƒ³ã‚¯
Data Versioning Â· EmilyGorcenski.com
manboubird 2019/03/09
dataManagement

model

machineLearning

schemaEvolution
ãƒªãƒ³ã‚¯
How we used DynamoDB Streams to visualize changes in frequently updated objects
manboubird 2019/02/25
schemaEvolution

schemaManagement

jsonPatch
ãƒªãƒ³ã‚¯
SQL is No Excuse to Avoid DevOps - ACM Queue
Everything Sysadmin - @YesThatTom December 12, 2018 Volume 16, issue 5 PDF SQL is No Excuse to Avoid DevOps Automation and a little discipline allow better testing, shorter release cycles, and reduced business risk. Thomas A. Limoncelli A friend recently said to me, "We can't do DevOps, we use a SQL database." I nearly fell off my chair. Such a statement is wrong on many levels. "But you don't und
manboubird 2019/01/14
schemaEvolution

schemaManagement

continuousDelivery

continuousIntegration

acmQueue
ãƒªãƒ³ã‚¯
Putting the Power of Kafka into the Hands of Data Scientists | Stitch Fix Technology â€“ Multithreaded
Over a year ago, my fellow data infrastructure engineers and I broke ground on a total rewrite of our event delivery infrastructure. Our mission was to build a robust, centralized data integration platform tailored to the needs of our Data Scientists. The platform would be fully self-service, so as to maximize the Data Scientistsâ€™ autonomy and give them complete control over their event data. Ulti
manboubird 2018/11/13
schemaManagement

schemaEvolution

stitchFix

dataPlatform
ãƒªãƒ³ã‚¯
https://cnr.sh/essays/kafka-change-data-capture-breaks-database-encapsulation/
- 1 user
- cnr.sh
- å¦ã³
manboubird 2018/11/13
changeDataCapture

Kafka

schemaEvolution
ãƒªãƒ³ã‚¯
Schema Evolution in Process Management Systems - DBIS EPub
manboubird 2018/10/09
schemaManagement

schemaEvolution

dataManagement

paper
ãƒªãƒ³ã‚¯
1 2 3 4 æ¬¡ã®ãƒšãƒ¼ã‚¸

ãŠçŸ¥ã‚‰ã›

ã‚‚ã£ã¨èªã‚€

å…¬å¼Twitter

@HatenaBookmark
ãƒªãƒªãƒ¼ã‚¹ã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“ã‚¹ã®ãŠçŸ¥ã‚‰ã›
@hatebu
æœ€æ–°ã®äººæ°—ã‚¨ãƒ³ãƒˆãƒªãƒ¼ã®é…ä¿¡

ã‚ãƒ¼ãƒœãƒ¼ãƒ‰ã‚·ãƒ§ãƒ¼ãƒˆã‚«ãƒƒãƒˆä¸€è¦§

jæ¬¡ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

kå‰ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

lã‚ã¨ã§èªã‚€

eã‚³ãƒ¡ãƒ³ãƒˆä¸€è¦§ã‚’é–‹ã

oãƒšãƒ¼ã‚¸ã‚’é–‹ã

è¨å®šã‚’å¤‰æ›´ã—ã¾ã—ãŸx