[B! Embulk] ji_kuã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

ji_ku id:ji_ku

Embulkã«é–¢ã™ã‚‹ji_kuã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (4)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Embulkã§MySQLã®ãƒ‡ãƒ¼ã‚¿ã‚’BigQueryã«ãƒãƒ¼ãƒ‰ã™ã‚‹ã®ã«ä¾¿åˆ©ãªãƒ„ãƒ¼ãƒ«ã®é–‹ç™º - é›»è„³å¾’ç„¶æ—¥è¨˜
Embulkã‚’åˆ©ç”¨ã—ã¦MySQLã®ãƒ‡ãƒ¼ã‚¿ã‚’BigQueryã«ãƒãƒ¼ãƒ‰ã™ã‚‹éš›ã«ä¾¿åˆ©ãªãƒ„ãƒ¼ãƒ«ã‚’é–‹ç™ºã—ãŸã®ã§ç´¹ä»‹ã§ã™ã€‚ ã€ŒSamidareã£ã¦ã„ã„ã¾ã™ï¼ã€€ã‚ˆã‚ã—ããŠé¡˜ã„ã—ã¾ã™ã€‚Embulkã®è¨å®šä½œæˆã¯ãŠä»»ã›ãã ã•ã„ï¼ã€ é–‹ç™ºçµŒç·¯ æ¥å‹™ã§ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã®è§£æžåŸºç›¤ã¨ã—ã¦BigQueryã‚’ä½¿ç”¨ã™ã‚‹ã“ã¨ã«ãªã£ãŸã®ã§ã™ãŒã€æœ¬ç•ªç’°å¢ƒã®MySQLã®ãƒ‡ãƒ¼ã‚¿ã‚‚BigQueryã§å‚ç…§ã§ãã‚‹ã‚ˆã†ã«ã™ã‚‹ã¨è‰²ã€…ã¨å¤¢ãŒè†¨ã‚‰ã‚€ã¨ã„ã†äº‹ã«æ°—ä»˜ãã¾ã—ãŸã€‚ Embulkã‚’åˆ©ç”¨ã™ã‚‹ã¨ç°¡å˜ã«MySQLã®ãƒ‡ãƒ¼ã‚¿ã‚’BigQueryã«ãƒãƒ¼ãƒ‰ã§ãã‚‹ã¨ã„ã†æƒ…å ±ã‚’å…¥æ‰‹ã—ãŸã®ã§æ¤œè¨¼ã—ãŸã¨ã“ã‚ã€ç¢ºã‹ã«ã“ã‚Œã¯æ¥½ãƒãƒ³ã ã¨ã„ã†çµè«–ã«è‡³ã‚Šã¾ã—ãŸã€‚ãŸã ã€å•é¡Œç‚¹ã¨ã—ã¦ãƒ†ãƒ¼ãƒ–ãƒ«ã®æ•°ã ã‘Embulkã®è¨å®šãƒ•ã‚¡ã‚¤ãƒ«ã‚’ä½œæˆã—ã€ãƒ†ãƒ¼ãƒ–ãƒ«æ¯Žã«Embulkã‚’å®Ÿè¡Œã—ãªã‘ã‚Œã°ãªã‚‰ãªã„ã¨ã„ã†ç‚¹ã§ã™ã€‚ å½“åˆã§ã‚‚BigQueryã«ãƒãƒ¼ãƒ‰ã—ãŸã„ãƒ†ãƒ¼ãƒ–ãƒ«æ•°ã¯20ä»¥ä¸Šã‚ã‚Šã€ä»Šå¾Œå¯¾è±¡ã®ãƒ†ãƒ¼ãƒ–ãƒ«ãŒå¢—ãˆ
ji_ku 2016/10/15
Embulk

BigQuery
ãƒªãƒ³ã‚¯
Embulkã‚’æ¥å‹™ã‚·ã‚¹ãƒ†ãƒ ã§ä½¿ã£ãŸè©± - ä»Šæ—¥ã‚‚ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°
èƒŒæ™¯ è‡ªåˆ†ã¯SIerã®ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§ã‚ã‚‹ã€‚ ã„ã‚ã„ã‚ãªãŠå®¢æ§˜ã®ã€ã„ã‚ã„ã‚ãªæ¥å‹™ã‚·ã‚¹ãƒ†ãƒ ã¨æ ¼é—˜ã™ã‚‹ã®ãŒãŠä»•äº‹ã§ã‚ã‚‹ã€‚ ã¾ãŸã€ä»Šã¯embulk-input-jdbcã¨ã‹embulk-output-jdbcã®ã‚³ãƒŸãƒƒã‚¿ã‚‚ã‚„ã£ã¦ã„ã‚‹ã€‚ æ¥å‹™ã‚·ã‚¹ãƒ†ãƒ ã¨RDBã¨ãƒ†ã‚ã‚¹ãƒˆãƒ•ã‚¡ã‚¤ãƒ« æ¥å‹™ã‚·ã‚¹ãƒ†ãƒ ã§ã¯ã€ãŸã„ã¦ã„RDBã‚’ä½¿ã£ã¦ã„ã‚‹ã€‚ ãã—ã¦ã€ã‚µãƒ¼ãƒé–“ã§ãƒ‡ãƒ¼ã‚¿ã‚’é€£æºã™ã‚‹ãŸã‚ã«ã€RDBã‹ã‚‰ãƒ†ã‚ã‚¹ãƒˆãƒ•ã‚¡ã‚¤ãƒ«ã«ã‚¨ã‚¯ã‚¹ãƒãƒ¼ãƒˆã—ãŸã‚Šã€ãƒ†ã‚ã‚¹ãƒˆãƒ•ã‚¡ã‚¤ãƒ«ã‚’RDBã«ã‚¤ãƒ³ãƒãƒ¼ãƒˆã—ãŸã‚Šã™ã‚‹ã“ã¨ãŒçµæ§‹ã‚ã‚‹ã€‚ ãã‚“ãªã¨ãã©ã†ã™ã‚‹ã‹ã¨è¨€ã†ã¨ã€RDBä»˜å±žã®ãƒ„ãƒ¼ãƒ«ã‚’ä½¿ã†ã“ã¨ãŒå¤šã„ã€‚ ä¾‹ãˆã°ã€Oracleã¸ã®ã‚¤ãƒ³ãƒãƒ¼ãƒˆã§ã‚ã‚Œã°SQL*Loaderã€MySQLã¸ã®ã‚¤ãƒ³ãƒãƒ¼ãƒˆã§ã‚ã‚Œã°mysqlimportã€ã¨ã„ã†ã‚ˆã†ã«ã€‚ RDBä»˜å±žã®ãƒ„ãƒ¼ãƒ«ã®å•é¡Œç‚¹ ã„ã‚ã„ã‚ãªãŠå®¢æ§˜ãŒã„ã¦ã€ã„ã‚ã„ã‚ãªã‚·ã‚¹ãƒ†ãƒ ãŒã‚ã‚‹ã®ã§ã€RDBã‚‚ã„ã‚ã„ã‚ã§ã‚ã‚‹ã€‚ ã¡ãªã¿ã«ã€ã†ã¡ã®ä¼šç¤¾ã§
ji_ku 2016/08/11
Embulk
ãƒªãƒ³ã‚¯
ã¿ã‚“ãªã®ã‚¦ã‚§ãƒ‡ã‚£ãƒ³ã‚°ã®ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®ç¾çŠ¶ â€” ã¿ã‚“ãªã®ã‚¦ã‚§ãƒ‡ã‚£ãƒ³ã‚°ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ãƒ–ãƒã‚°
ã“ã‚“ã«ã¡ã¯ã€ã¿ã‚“ãªã®ã‚¦ã‚§ãƒ‡ã‚£ãƒ³ã‚°ã®å°å®¤ (id:hogelog) ã§ã™ã€‚ ä»Šå›žã¯ã¿ã‚“ãªã®ã‚¦ã‚§ãƒ‡ã‚£ãƒ³ã‚°ã«ãŠã‘ã‚‹ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®ç¾çŠ¶ã«ã¤ã„ã¦ã”å ±å‘Šã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ ä¸‰è¡Œã¾ã¨ã‚ å¿™ã—ã„äººã®ãŸã‚ã«å…ˆã«çµè«–ã‚’æ›¸ãã¨ bricolage ã¨ embulk ã§ Redshift ã«é›†ã‚ã¦ re:dash ã§åˆ†æž ã§ã™ã€‚ ãƒ‡ãƒ¼ã‚¿åŽé›† ãƒ‡ãƒ¼ã‚¿åŽé›†ã¯ bricolage ã®ã‚¸ãƒ§ãƒ–ãƒãƒƒãƒˆæ©Ÿæ§‹ã‚’ç”¨ã„ã¦ bricolage ã®å„ç¨®ã‚¸ãƒ§ãƒ–ã‚„ embulk ã‚’é€£æºã•ã›ã€Redshift ã«ãƒ‡ãƒ¼ã‚¿ã‚’å–ã‚Šè¾¼ã‚“ã§ã„ã¾ã™ã€‚ å‚è€ƒã¾ã§ã« https://github.com/hogelog/dwh-example ã«ç°¡å˜ãªæ§‹æˆä¾‹ã‚’æº–å‚™ã—ã¾ã—ãŸã€‚ MySQL â†’ Redshift ã¿ã‚“ãªã®ã‚¦ã‚§ãƒ‡ã‚£ãƒ³ã‚° http://www.mwed.jp/ ã®ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¨ã—ã¦ã¯ MySQL ã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ã€‚ MySQL ã‹ã‚‰ Redshi
ji_ku 2016/06/03
Embulk

åˆ†æž
ãƒªãƒ³ã‚¯
ä¸¦åˆ—ãƒ‡ãƒ¼ã‚¿è»¢é€ãƒ„ãƒ¼ãƒ«ã€ŽEmbulkã€ãƒªãƒªãƒ¼ã‚¹ï¼ - Blog by Sadayuki Furuhashi
ã“ã‚“ã«ã¡ã¯ã€‚å¤æ©‹ã§ã™ã€‚ å…ˆæ—¥ã®*1 ãƒ‡ãƒ¼ã‚¿è»¢é€ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢å‹‰å¼·ä¼šã§ã€æ–°ã—ã„ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ãƒ„ãƒ¼ãƒ« Embulk ã‚’ãƒªãƒªãƒ¼ã‚¹ã—ã¾ã—ãŸã€‚ Embulk, an open-source plugin-based parallel bulk data loader from Sadayuki Furuhashi Embulk ã¯ã€ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªãƒã‚°åŽé›†ã§ã¯å¸¸è˜ã¨ãªã£ãŸ fluentd ã®ãƒãƒƒãƒç‰ˆã®ã‚ˆã†ãªãƒ„ãƒ¼ãƒ«ã§ã€ãƒ•ã‚¡ã‚¤ãƒ«ã‚„ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‹ã‚‰ãƒ‡ãƒ¼ã‚¿ã‚’å¸ã„å‡ºã—ã€åˆ¥ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚„ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«ãƒãƒ¼ãƒ‰ã™ã‚‹ãŸã‚ã®ã‚³ãƒ³ãƒ‘ã‚¯ãƒˆãªãƒ„ãƒ¼ãƒ«ã§ã™ã€‚ fluentd ã¨åŒæ§˜ã«ãƒ—ãƒ©ã‚°ã‚¤ãƒ³åž‹ã®ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã‚’æŽ¡ç”¨ ã—ã¦ã„ã‚‹ãŸã‚ã€Rubyã‚„Javaã§ç°¡å˜ãªã‚³ãƒ¼ãƒ‰ã‚’æ›¸ãã“ã¨ã§ã€æ§˜ã€…ãªãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã‚„ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã«å¯¾å¿œã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ä¸€æ–¹ã§ fluentd ã¨ã¯ç•°ãªã‚Šã€é«˜é€Ÿæ€§ã‚„ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³åˆ¶å¾¡ã€ã‚¹ã‚ãƒ¼ãƒžã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ã®ãƒãƒª
ji_ku 2015/02/18
Embulk
ãƒªãƒ³ã‚¯
1