ããã«ã¡ã¯ã@Civitaspo ã§ããå æ¥12/15 ã® Embulk Meetup Tokyo#2 ã§ã¯ããEmbulkã«è¶³ããªã5ã¤ã®ã³ããã¨ããå§å©è °ãªã¿ã¤ãã«ã§çºè¡¨ããã¦ããã ãã¾ããããããã§è²ããªæ¹ã¨embulkã«ã¤ãã¦è°è«ã§ããè¯ãMeetupã«ãªãã¾ãããå¼ãã§ãã ãã£ã @frsyuki ã¨ãè£ããæ¯ãã¦ä¸ãã£ã @sonots ã«ã¯ãã®å ´ããåããã¦æè¬ã®æãä¼ãããã¨æãã¾ãããããã¨ããããã¾ããã ãã¦ãä»æ¥ã¯ embulk java-filter plugin ã®ãã¹ãã®æ¸ãæ¹ ã«ã¤ãã¦è¨äºãæ¸ããã¨æãã¾ãã ããã¨ãã 話ãç°¡åã«ããããã«java-filter pluginã®ãã¹ãã®æ¸ãæ¹ã«focusãã¦ãã¾ãããä»ã®typeã®java pluginã§ãå¿ç¨ã§ããå 容ã ã¨æã£ã¦ãã¾ãã EmbulkEmbedã使ã£ããã¹ããæ¸ãã¾ããä»åã¯ç´¹ä»ãã¾ããã
ãã°åéåºç¤ã½ããã¦ã§ã¢ ãFluentdãããEmbulkããéä¿¡ãã£ãªã¢åãã½ãªã¥ã¼ã·ã§ã³ã¨ãã¦ãä¸è±é»æ©ã¤ã³ãã©ã¡ã¼ã·ã§ã³ã·ã¹ãã ãºæ ªå¼ä¼ç¤¾ã«æä¾ ç±³å½ãã¬ã¸ã£ã¼ãã¼ã¿ç¤¾(以ä¸ããã¬ã¸ã£ã¼ãã¼ã¿ã)ã¯ãéä¿¡ãã£ãªã¢åãã®ãã°åéåºç¤ã½ããã¦ã¨ã¢ãFluentdãããã³ãEmbulkãã®å°å ¥ã»éç¨ãµãã¼ããä¸è±é»æ©ã¤ã³ãã©ã¡ã¼ã·ã§ã³ã·ã¹ãã ãºæ ªå¼ä¼ç¤¾ï¼æ³¨1ã以ä¸ãMDISãï¼ã«æä¾ãã¦ã¾ããã¾ãã®ã§ããç¥ãããããã¾ãã ãFluentdãããã³ãEmbulkãã¯ããã¬ã¸ã£ã¼ãã¼ã¿ãéçºãããªã¼ãã³ã½ã¼ã¹ã®ãã°åéåºç¤ã½ããã¦ã§ã¢ã§ãããFluentdãã¯ããã°ãã¼ã¿ãã¹ããªã¼ãã³ã°ã§ã¢ãããã¼ãããæ©è½ãæãããEmbulkãã¯ä»ã®ãã¼ã¿ãã¼ã¹ã«ãããã¼ã¿ãªã©ããããã§åãè¾¼ãã½ããã¦ã§ã¢ã§ãããããããä¼æ¥ãä¿æãã大éã®ãã¼ã¿ãå¹çããåéã»ç®¡çãããã¨ãã§ãããã¨ããã大è¦æ¨¡W
ããã«ã¡ã¯ã ã¤ã³ãã©ã¨ã³ã¸ãã¢ã®æä¸ã§ãã ããã¼ãã©ã¯ã¼ãã®ã¤ã³ãã©ãã¼ã ã¯ããµã¼ãã¹ã«é¢ããã¤ã³ãã©ãããèªç¤¾ã®ä½æ¥ç°å¢ãéçºç°å¢ãããã«ã¯ãµã¼ãã¹ã®ã¤ã³ãã©ã®ä¸ã§ãç©ççãªãã®ããOSã»ããã«ã¦ã§ã¢ã»ã¢ããªã±ã¼ã·ã§ã³ã®ã¡ã³ããã³ã¹ã»ãã«ãã»ãªãªã¼ã¹ã»éç¨ã¾ã§å¹ åºãé¢ä¸ãã¦ãã¾ãã ä»åã¯Google Cloud Platformã®BigQueryãæ´»ç¨ãã¦ã¢ã¯ã»ã¹ãã°ã®åæç°å¢ãæ§ç¯ããæã®è©±ãç´¹ä»ãã¾ãã ãã®è¨äºã«æ¸ãããäº ãã¼ã¿åæåºç¤ã¨ãã¦BigQueryã使ç¨ãã話㨠ãã¼ã¿éãä¾ç¤ºããªãã使ç¨ãéå§ããæã®ãã©ãã«ã·ã¥ã¼ãã¨ããã©ã¼ãã³ã¹ã«ã¤ãã¦ç´¹ä»ããã ãã¼ã¿ç§»è¡ã®ã³ãããã¾ãå«ããªããæ¸ãã¦ããã BigQueryãæ¡ç¨ãã訳 ããã¼ãã©ã¯ã¼ãã®å®¶è¨ç°¿ã¯350ä¸äººä»¥ä¸ã®ã客æ§ã«å©ç¨ããã ãã ã¢ã¯ã»ã¹ãã°ã¯æ¥ã 2.500ä¸ä»¶ç¨åº¦å¢ãã¦ããã¾ãã ãµã¼ãã¹éå§ãã
Digdag ã Apache License 2.0 ã®å ã§ãªã¼ãã³ã½ã¼ã¹åããã¾ãããï¼ ãã試ããã â¦ï¼ ä»ããã«ã§ãï¼ https://t.co/Uzc4a5GLCe ããã¥ã¡ã³ãï¼https://t.co/PF8wy5KHlnâ Sadayuki Furuhashi (@frsyuki) 2016å¹´6æ15æ¥ Digdagãå æ¥ãªãªã¼ã¹ãããã®ããã£ããã«ãã¼ã¿åæåºç¤æ§ç¯ã®å¤¢ãè¦ãã ä»åã¯ãGoogle Cloud Platformï¼ä»¥ä¸ãGCPï¼ã®ã³ã¹ããå¯è¦åãã¤åæå¯è½ã«ãã¦ã¿ã¦ãã¾ãã¯Digdagã®ä½¿ç¨æã試ãã¦ã¿ããã¨ã«ããã äºåç¥è Digdagã¨ã¯ Workload Automation Systemã§ããã以ä¸ã®è¨äºã詳ããã Embulkã¨Digdagã¨ãã¼ã¿åæåºç¤ã¨ åæ£ã¯ã¼ã¯ããã¼ã¨ã³ã¸ã³ãDigDagãã®å®è£ at Tokyo RubyKaigi
2. WHO AM I ? ⢠Toru Takahashi (@nora96o) ⢠Treasure Data, Inc. ⢠Support Engineering Manager ⢠ã¡ã¼ã«ã«ãã£ããã«ãããã°æ¸ããããã³ã¼ããæ¸ãããã ⢠http://qiita.com/toru-takahashi ⢠æ°ã¥ãã¨ã社ä¼äººï¼å¹´ç®ã«çªå ¥ã»ã»ã» 2 3. 質åã§ãï¼ â¢ Treasure Data ã ç¥ã£ã¦ããäººã¯ ï¼ â¢ Fluentd ã ç¥ã£ã¦ããäººã¯ ï¼ â¢ Embulkãç¥ã£ã¦ãã人ã¯ï¼ ⢠Digdagãèãããã¨ããã人ã¯ï¼ ⢠ã¤ã³ãã© / åæåºç¤ ãæ®æ®µããéç¨ãéçºããã¦ãã人ã¯ï¼ ⢠iOS / Android / Unity / ããã³ãã¨ã³ãã®äººã¯ï¼ 3
ããã«ã¡ã¯ãæè¿Embulkï¼ã¨ã³ãã«ã¯ï¼ã調ã¹ã¦ããå·å´ã§ãã ã¿ã¤ãã«ã®ãDigDagãï¼ãã£ã°ãã°ï¼ *1ã¨èãã¦âã¡ãã³ã³ãâã ã³ãæãæµ®ãã¹ãæ¹ã¯ãã£ã¨ã¢ã©ãµã¼ä»¥ä¸ã§ããï¼ ä»åã¯Embulkã使ãå§ããã¨ç´é¢ããã§ããã課é¡ã«ã¤ãã¦æ¸ãã¦ããããã¨æãã¾ãã ããã¾ã§ã«æ¸ããEmbulkã¨ã³ããªã¼ [Embulk] Embulkã«ã¤ãã¦ã®å人çãªã¾ã¨ã [Embulk] guessæ©è½ã試ãã¦ã¿ãã追è¨ããã [Embulk] guessæ©è½ã試ãã¦ã¿ã Redshiftç·¨ [Embulk] ã¿ã¤ã ã¹ã¿ã³ãã®ã«ã©ã ã追å ãã [Embulk] ã¿ã¤ã ã¹ã¿ã³ãã®ã«ã©ã ã追å ãã add_timeãã©ã°ã¤ã³ç·¨ [Embulk] guessæ©è½ã試ãã¦ã¿ã (ãã¼ãã«ãåå¨ããå ´å) Embulkã使ãã¯ãããã¨... Embulkã¯ãã¼ã¿ãã¡ã¤ã«ã®ä¸èº«ãDBã«ãã¼ããããããªç¨éã§
ããã«ã¡ã¯ãã¿ããªã®ã¦ã§ãã£ã³ã°ã®å°å®¤ (id:hogelog) ã§ãã ä»åã¯ã¿ããªã®ã¦ã§ãã£ã³ã°ã«ããããã¼ã¿åæåºç¤ã®ç¾ç¶ã«ã¤ãã¦ãå ±åããã¦ããã ãã¾ãã ä¸è¡ã¾ã¨ã å¿ãã人ã®ããã«å ã«çµè«ãæ¸ã㨠bricolage 㨠embulk 㧠Redshift ã«éã㦠re:dash ã§åæ ã§ãã ãã¼ã¿åé ãã¼ã¿åé㯠bricolage ã®ã¸ã§ããããæ©æ§ãç¨ã㦠bricolage ã®å種ã¸ã§ãã embulk ãé£æºãããRedshift ã«ãã¼ã¿ãåãè¾¼ãã§ãã¾ãã åèã¾ã§ã« https://github.com/hogelog/dwh-example ã«ç°¡åãªæ§æä¾ãæºåãã¾ããã MySQL â Redshift ã¿ããªã®ã¦ã§ãã£ã³ã° http://www.mwed.jp/ ã®ãã¼ã¿ãã¼ã¹ã¨ãã¦ã¯ MySQL ãå©ç¨ãã¦ãã¾ãã MySQL ãã Redshi
ãã«ã¯ãã¼ãæ©è½ 1ã¤ã®è¨å®ãã¡ã¤ã«ã§è¤æ°ã¸ã§ããå®è¡ãã Running multiple jobs using one config file · Issue #167 · embulk/embulk · GitHub ä¾ãã° users.csv 㨠histories.csv ã®2ã¤ã®ãã¡ã¤ã«ããããããPostgreSQLã«ãã users 㨠histories ã®2ã¤ã®ãã¼ãã« ã«ãã¼ããããã¨ãããããªã¦ã¼ã¹ã±ã¼ã¹ã«å¯¾å¿ããæ©è½ã è¨å®ãã¡ã¤ã«ã®æ§æã¯issueã«æ¸ãã¦ããããã«ãdefault: ã«æ¸ã並ã¹ãè¨å®ã«å¯¾ãã¦ãjobs: ã«æ¸ããè¨å®ããã¼ã¸ãããã®ãå®éã®è¨å®ãã¡ã¤ã«ã¨ãã¦å®è¡ãã¦ããæ¹æ³ã§è¯ããããããããfliters: ã¯é åãªã®ã§ãdefault: ã«æ¸ããã filters: ã« jobs: ã«æ¸ããã filters: ãã©ããã¼ã¸ãããããã¾ãè¯
æ¦è¦ ãã©ã¼ããã¿ã¼ãã©ã°ã¤ã³ï¼java-formatterï¼ã¯ãèªã¿è¾¼ã¾ãããã¼ã¿ããã¡ã¤ã«åºåç¨ã«æ´å½¢ãããã©ã°ã¤ã³ã ï¼ä¾ãã°CSVãã¡ã¤ã«ã¨ãã¦ãã¼ã¿ãè¡ã»ã«ã©ã ãã¨ã«å解ããï¼ ããã¸ã§ã¯ãã®ä½æ Javaãã¼ãµã¼ãã©ã°ã¤ã³ã¯ä»¥ä¸ã®ã³ãã³ãã§ããã¸ã§ã¯ããä½æããã $ embulk new java-formatter poi-excel æå¾ã®å¼æ°ã®ãpoi-excelãã¯ããã©ã°ã¤ã³åã表ãã ããã§ããembulk-formatter-poi_excelãã¨ãããã£ã¬ã¯ããªã¼ãä½ããããã®ä¸ã«ã½ã¼ã¹é¡ä¸å¼ãçæãããã ããã¸ã§ã¯ããä½æããããã¾ã以ä¸ã®ã³ãã³ããå®è¡ããã $ cd embulk-formatter-poi_excel $ ./gradlew package çæããããã©ã°ã¤ã³ã®ä¸æ ¸ã¯ãsrc/main/javaã®ä¸ã®FormatterPluginã¯
Embulk v0.8.3 Usage: embulk [-vm-options] <command> [--options] Commands: mkbundle <directory> # create a new plugin bundle environment. bundle [directory] # update a plugin bundle environment. run <config.yml> # run a bulk load transaction. cleanup <config.yml> # cleanup resume state. preview <config.yml> # dry-run the bulk load without output and show preview. guess <partial-config.yml> -o <outp
Java ã®ã³ã¼ãã¯æµç³ã« vim ã§æ¸ããã㯠IDE ã使ã£ãã»ããããããªãã¨æã£ãã®ã§è©¦ãã¦ã¿ãã¡ã¢ã対象㯠Java 㪠Embulk ã®ãã©ã°ã¤ã³ã®éçºã§ãJRuby ãªãã©ã°ã¤ã³ã®éçºã®å ´åã¯ã¾ã 試ãã¦ããªããã¨ããã vim ã§ããããªã¨æã£ã¦ãã¦è©¦ãã¦ããªãã è£å®å©ãã¦ãã¬ã¼ã¯ãã¤ã³ããè²¼ããããã«ãªã£ãã®ã§æé«ã IntelliJ IDEA ã®ã¤ã³ã¹ãã¼ã« çç¥ãã¾ããhttps://www.jetbrains.com/idea/ ã¸ã©ããã JAVA_HOME ã調ã¹ã¦ããå¿ è¦ãããã®ã§ãMac ã®å ´å ã®ããã«ãã¦èª¿ã¹ã¦ããã¨è¯ãã ããã¸ã§ã¯ãã®èªã¿è¾¼ã¿ IntelliJ ãèµ·åãã¦ãOpen ãé¸ã³ããã©ã°ã¤ã³ã®ãã£ã¬ã¯ããªãéã㨠Gradle ããã¸ã§ã¯ãã¨ãã¦èªèãããã¯ããªã®ã§ããã¨ã¯ä»¥ä¸ã®ããã«è¨å®ããã¨è¯ãï¼åºæ¬çã«ã¯ããã ãã Use auto-
List<String> columnNames = task.getColumns(); ImmutableList.Builder<Column> builder = ImmutableList.builder(); int i = 0; for (String columnName : columnNames) { for (Column inputColumn: inputSchema.getColumns()) { if (inputColumn.getName().equals(columnName)) { Column outputColumn = new Column(i++, columnName, inputColumn.getType()); builder.add(outputColumn); break; } } } Schema outputSchema = n
ã©ã³ãã³ã°
ã©ã³ãã³ã°
ãç¥ãã
ãªãªã¼ã¹ãé害æ å ±ãªã©ã®ãµã¼ãã¹ã®ãç¥ãã
ææ°ã®äººæ°ã¨ã³ããªã¼ã®é ä¿¡
å¦çãå®è¡ä¸ã§ã
j次ã®ããã¯ãã¼ã¯
kåã®ããã¯ãã¼ã¯
lãã¨ã§èªã
eã³ã¡ã³ãä¸è¦§ãéã
oãã¼ã¸ãéã
{{#tags}}- {{label}}
{{/tags}}