[B! BigQuery] nobusueã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

nobusue id:nobusue

BigQueryã«é–¢ã™ã‚‹nobusueã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (16)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Streaming Cassandra at WePay - Part 1
Historically, MySQL had been the de-facto database of choice for microservices at WePay. As WePay scales, the sheer volume of data written into some of our microservice databases demanded us to make a scaling decision between sharded MySQL (i.e. Vitess) and switching to a natively sharded NoSQL database. After a series of evaluations, we picked Cassandra, a NoSQL database, primarily because of its
nobusue 2019/07/31
Cassandra3ã®CDCæ©Ÿèƒ½ã‚’åˆ©ç”¨ã—ã¦DebeziumçµŒç”±ã§BigQueryã«æµã™

cassandra

Debezium

Kafka

BigQuery
ãƒªãƒ³ã‚¯
Hadoopã‚’BQã«ãƒžã‚¤ã‚°ãƒ¬ã—ã‚ˆã†ã¨ã—ã¦ã‚‹è©±
5. ãƒªã‚¯ãƒ«ãƒ¼ãƒˆãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚ºã®ç«‹ä½ç½® Infrastructure /Security Big Data Solutions Techno logy R&D Systems Development Recruit Holdings Recruit Career Recruit Sumai Company Recruit Lifestyle Recruit Jobs Recruit Staffing Recruit Marketing Partners Staff service Holdings Recruit Administration Recruit Communications Business/ Service Function/ Support Project Management UXD/SEO Internet Marketing Recruit Techno logies
nobusue 2017/06/08
hadoop

BigQuery
ãƒªãƒ³ã‚¯
BigQueryã®æ–°ã—ã„æ–™é‡‘ä½“ç³»High Compute Queryã¯å»ƒæ¢ã•ã‚Œã¾ã—ãŸã€‚ - Qiita
High Compute Queryã¯2017å¹´11æœˆã«å»ƒæ¢ã«ãªã£ãŸã®ã§ã€ã“ã®è¨˜äº‹ã¯éŽåŽ»ã®æ€ã„å‡ºã§ã™ã€‚ https://cloud.google.com/bigquery/docs/release-notes?hl=en#november_14_2017 ä»Šå¾Œã¯BillingTierãŒ100ã‚’è¶…ãˆãªã„é™ã‚Šã¯ã€ç‰¹ã«æ–™é‡‘ãŒå¤‰ã‚ã‚‹ã“ã¨ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚ BillingTier100è¶…ãˆã¯ã‹ãªã‚Šã®ç„¡èŒ¶ã‚’ã—ãªã„ã¨å‡ºã¦ã“ãªã„ã®ã§ã€æ»…å¤šã«æ°—ã«ã™ã‚‹ã“ã¨ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚ ãŸã ã€å˜ç´”ãªãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã¨ã—ã¦ã€ä»¥ä¸‹ã®ãƒŽã‚¦ãƒã‚¦ã¯ã¾ã å½¹ã«ç«‹ã¡ã¾ã™ã€‚ High Compute Queryã«å‚™ãˆã‚‹ï¼Dremelã®æ°—æŒã¡ã«ãªã£ã¦è€ƒãˆã‚‹ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° ä»¥ä¸‹ã€éŽåŽ»ã®æ€ã„å‡ºè©±ã¨ãªã£ãŸå†…å®¹ BigQueryã®Queryæ–™é‡‘ã¯ä»Šã¾ã§ã©ã‚“ãªè¤‡é›‘ãªã‚¯ã‚¨ãƒªã‚’æ›¸ã„ã¦ã‚‚ã€ãƒ‡ãƒ¼ã‚¿ã‚’èªã¿è¾¼ã‚“ã å®¹é‡ã«å¯¾ã—ã¦æ–™é‡‘ãŒæ±ºå®šã•ã‚Œã¦ã„ã¾ã—ãŸ
nobusue 2016/12/26
GCP

BigQuery
ãƒªãƒ³ã‚¯
83å„„ãƒ¬ã‚³ãƒ¼ãƒ‰ã‚’ç§»è¡Œã—ã€æ—¥ã€…2,500ä¸‡ãƒ¬ã‚³ãƒ¼ãƒ‰ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’BigQueryã«è¨˜éŒ²ã—ã¦ã„ã‚‹è©±ï¼ˆã‚¤ãƒ³ãƒ•ãƒ©ç·¨ï¼‰ - Money Forward Developers Blog
ã“ã‚“ã«ã¡ã¯ã€‚ ã‚¤ãƒ³ãƒ•ãƒ©ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®æ‘ä¸Šã§ã™ã€‚ ãƒžãƒãƒ¼ãƒ•ã‚©ãƒ¯ãƒ¼ãƒ‰ã®ã‚¤ãƒ³ãƒ•ãƒ©ãƒãƒ¼ãƒ ã¯ã€ã‚µãƒ¼ãƒ“ã‚¹ã«é–¢ã‚ã‚‹ã‚¤ãƒ³ãƒ•ãƒ©ã‹ã‚‰ã€è‡ªç¤¾ã®ä½œæ¥ç’°å¢ƒã€é–‹ç™ºç’°å¢ƒã€ã•ã‚‰ã«ã¯ã‚µãƒ¼ãƒ“ã‚¹ã®ã‚¤ãƒ³ãƒ•ãƒ©ã®ä¸ã§ã‚‚ç‰©ç†çš„ãªã‚‚ã®ã‹ã‚‰OSãƒ»ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ãƒ»ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã®ãƒ¡ãƒ³ãƒ†ãƒŠãƒ³ã‚¹ãƒ»ãƒ“ãƒ«ãƒ‰ãƒ»ãƒªãƒªãƒ¼ã‚¹ãƒ»é‹ç”¨ã¾ã§å¹…åºƒãé–¢ä¸Žã—ã¦ã„ã¾ã™ã€‚ ä»Šå›žã¯Google Cloud Platformã®BigQueryã‚’æ´»ç”¨ã—ã¦ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã®åˆ†æžç’°å¢ƒã‚’æ§‹ç¯‰ã—ãŸæ™‚ã®è©±ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ ã“ã®è¨˜äº‹ã«æ›¸ã‹ã‚Œã‚‹äº‹ ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã¨ã—ã¦BigQueryã‚’ä½¿ç”¨ã—ãŸè©±ã¨ ãƒ‡ãƒ¼ã‚¿é‡ã‚’ä¾‹ç¤ºã—ãªãŒã‚‰ä½¿ç”¨ã‚’é–‹å§‹ã—ãŸæ™‚ã®ãƒˆãƒ©ãƒ–ãƒ«ã‚·ãƒ¥ãƒ¼ãƒˆã¨ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã«ã¤ã„ã¦ç´¹ä»‹ã™ã‚‹ã€‚ ãƒ‡ãƒ¼ã‚¿ç§»è¡Œã®ã‚³ãƒ„ã‚‚ã†ã¾ãå«ã‚ãªãŒã‚‰æ›¸ã„ã¦ã„ãã€‚ BigQueryã‚’æŽ¡ç”¨ã—ãŸè¨³ ãƒžãƒãƒ¼ãƒ•ã‚©ãƒ¯ãƒ¼ãƒ‰ã®å®¶è¨ˆç°¿ã¯350ä¸‡äººä»¥ä¸Šã®ãŠå®¢æ§˜ã«åˆ©ç”¨ã„ãŸã ãã€ ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã¯æ—¥ã€…2.500ä¸‡ä»¶ç¨‹åº¦å¢—ãˆã¦ãŠã‚Šã¾ã™ã€‚ ã‚µãƒ¼ãƒ“ã‚¹é–‹å§‹ã‹ã‚‰
nobusue 2016/08/03
BigQuery

Embulk
ãƒªãƒ³ã‚¯
ã‚°ãƒ¼ã‚°ãƒ«ã®ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚µãƒ¼ãƒ“ã‚¹ã€ŒBigQueryã€ãŒæ¨™æº–SQLã‚’ã‚µãƒãƒ¼ãƒˆ
æ³•äººå‘ã‘äº‹æ¥ã®åŠ é€Ÿã‚’å›³ã‚‹GoogleãŒç±³å›½æ™‚é–“6æœˆ2æ—¥ã€ã‚¯ãƒ©ã‚¦ãƒ‰ãƒ™ãƒ¼ã‚¹ã®ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚µãƒ¼ãƒ“ã‚¹ã€ŒBigQueryã€ã‚’å¼·åŒ–ã—ãŸã¨ç™ºè¡¨ã—ãŸã€‚å¾“æ¥ã®ãƒ“ãƒƒã‚¯ãƒ‡ãƒ¼ã‚¿ã‚µãƒ¼ãƒ“ã‚¹ã¨ã®äº’æ›æ€§æ”¹å–„ãªã©ãŒåŠ ã‚ã£ãŸã€‚ BigQueryã®å¼·åŒ–ã¨ã—ã¦ã€æ¨™æº–ã®SQLã‚’ã‚µãƒãƒ¼ãƒˆã—ãŸã€‚ã“ã‚Œã«åŠ ãˆã¦ã€é«˜åº¦ãªã‚¯ã‚¨ãƒªãƒ—ãƒ©ãƒ³ãƒ‹ãƒ³ã‚°ã¨æœ€é©åŒ–ã‚‚å¯èƒ½ã¨ãªã‚Šã€SQLã‚¹ãƒ†ãƒ¼ãƒˆãƒ¡ãƒ³ãƒˆã§ã®ä»»æ„ã®è¤‡é›‘ãªã‚µãƒ–ã‚¯ã‚¨ãƒªãŒã‹ã‘ã‚‰ã‚Œã‚‹ã‚ˆã†ã«ãªã£ãŸã€‚ã•ã‚‰ã«ã¯ã€æ—¥ã€æ™‚é–“ã€ã‚¢ãƒ¬ã‚¤ã€æ§‹é€ ä½“ãªã©å¹…åºƒã„åž‹ã‚·ã‚¹ãƒ†ãƒ ã€ãã‚Œã«Theta Joinã‚‚ã‚µãƒãƒ¼ãƒˆã—ãŸã€‚ Googleã¯ã¾ãŸã€ã‚¢ã‚¤ãƒ‡ãƒ³ãƒ†ã‚£ãƒ†ã‚£ã¨ã‚¢ã‚¯ã‚»ã‚¹ç®¡ç†ã®æ©Ÿèƒ½ã€ŒCloud IAMã€ã‚’BigQueryã§ã‚‚ãƒ™ãƒ¼ã‚¿ã¨ã—ã¦æä¾›ã™ã‚‹ã€‚ã“ã‚Œã«ã‚ˆã‚Šã€BigQueryãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆå‘ã‘ã®è¨±å¯ã‚’å®Œå…¨ã«è‡ªå‹•åŒ–ã§ãã€ç®¡ç†è€…ã¯è©³ç´°ã«ã‚»ã‚ãƒ¥ãƒªãƒ†ã‚£ãƒãƒªã‚·ãƒ¼ã‚’ä½œæˆã§ãã‚‹ã€‚ æœ€å¾Œã«ã€Googleã¯æ™‚é–“ãƒ™ãƒ¼ã‚¹ã®ãƒ†ãƒ¼ãƒ–ãƒ«ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ‹ãƒ³ã‚°æ©Ÿèƒ½ã‚‚å°Žå…¥ã—ãŸã€‚è¤‡
nobusue 2016/06/03
google

BigQuery
ãƒªãƒ³ã‚¯
Google BigQuery / Tableauã‚’ä½¿ã£ã¦ã¿ãŸ | DACã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒ–ãƒã‚°ï¼šã‚¢ãƒ‰ãƒ†ã‚¯ã‚‘ã³ã™ç•Œ
Tableauã‹ã‚‰Google BigQueryã¸æŽ¥ç¶šã—ã¦ã¿ã¾ã—ãŸã€‚ å¼Šç¤¾ã§åˆ©ç”¨ã—ã¦ã„ã‚‹TreasureDataã‹ã‚‰ãƒ‡ãƒ¼ã‚¿å‡ºåŠ›ã—ã¦BigQueryã¸ãƒãƒ¼ãƒ‰ã€Tableauã‹ã‚‰æŽ¥ç¶šã¾ã§å®Ÿéš›ã«è¡Œã£ãŸæ‰‹é †ã«ã¤ã„ã¦è¨˜è¼‰ã—ã¾ã™ã€‚ TreasureDataã‹ã‚‰AmazonS3ã¸ãƒ‡ãƒ¼ã‚¿å‡ºåŠ› ã¾ãšã€ãƒ‡ãƒ¼ã‚¿ãŒè“„ç©ã•ã‚Œã¦ã„ã‚‹TreasureDataã‹ã‚‰AmazonS3ã¸ãƒ•ã‚¡ã‚¤ãƒ«å‡ºåŠ›ã—ã¾ã™ã€‚ ã‚„ã‚Šæ–¹ã¯ç°¡å˜ã§ã€ã‚³ãƒ³ã‚½ãƒ¼ãƒ«ä¸Šã®â€Result Exportâ€ã‹ã‚‰â€AWS S3/Riak CSâ€ã‚’é¸æŠžã—ã¦ã€APIã‚ãƒ¼ã‚„å‡ºåŠ›å…ˆã®ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã‚’é¸æŠžã™ã‚‹ã ã‘ã§ã™ã€‚ Compressionã‚ªãƒ—ã‚·ãƒ§ãƒ³ã§â€gzâ€ã‚’æŒ‡å®šã—ãŸã®ã§ã™ãŒã€ã“ã‚ŒãŒä»‡ã«ãªã‚‹ã¨ã¯ã€‚ã€‚ã€‚è©³ç´°ã¯å¾Œè¿°ã—ã¾ã™ã€‚ ï¼ˆGoogleCloudStorageã¸ç›´æŽ¥å‡ºåŠ›ã§ããŸã‚‰å¬‰ã—ã„ãªã€ãªã‚“ã¦æ€ã„ã¤ã¤ï¼‰ AmazonS3ã‹ã‚‰GoogleCloudStorageã¸ãƒ‡ãƒ¼ã‚¿è»¢é€ Am
nobusue 2015/10/02
Tableau

BigQuery
ãƒªãƒ³ã‚¯
BigQueryã®èª²é‡‘ã€ç¯€ç´„ã—ã¾ã›ã‚“ã‹
13. èª²é‡‘ã®å¯¾è±¡ ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ - $0.020 per GB / month ã‚¯ã‚¨ãƒª - $5 per TB processed (scanned) ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ã‚¤ãƒ³ã‚µãƒ¼ãƒˆ - $0.01 per 100,000 rows until July 20, 2015. After July 20, 2015, $0.01 per 200 MB, with individual rows calculated using a 1 KB minimum size. æ³¨ç›®ã™ã¹ãã¯ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ 14. ç°¡å˜ãªä¾‹ 1TBã®ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’æ¯Žæ—¥ãƒãƒ¼ãƒ‰ã€ãã‚Œãžã‚Œã‚’ä¸€ãƒ¶æœˆé–“ä¿æŒ æ—¥ã€…ã®ãƒ‡ãƒ¼ã‚¿ã‚’æ¯Žæ—¥5å›žã‚¹ã‚ãƒ£ãƒ³ã—ã¦é›†è¨ˆ ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ : 1TB * 30 (tables) = $0.020 * 1000 * 30 = $600 ã‚¯ã‚¨ãƒª: 1TB * 5 (Queries) * 30 (days) = $750
nobusue 2015/09/30
GCP

BigQuery
ãƒªãƒ³ã‚¯
https://qiita.com/ryot_a/items/f5a4082edfb44287b968
nobusue 2015/09/30
BigQuery

node.js
ãƒªãƒ³ã‚¯
ã‚µãƒ¼ãƒ“ã‚¹çµ‚äº†ã®ãŠçŸ¥ã‚‰ã› - NAVER ã¾ã¨ã‚
ã‚µãƒ¼ãƒ“ã‚¹çµ‚äº†ã®ãŠçŸ¥ã‚‰ã› NAVERã¾ã¨ã‚ã¯2020å¹´9æœˆ30æ—¥ã‚’ã‚‚ã¡ã¾ã—ã¦ã‚µãƒ¼ãƒ“ã‚¹çµ‚äº†ã„ãŸã—ã¾ã—ãŸã€‚ ç´„11å¹´é–“ã€NAVERã¾ã¨ã‚ã‚’ã”åˆ©ç”¨ãƒ»ã”æ„›é¡§ã„ãŸã ãèª ã«ã‚ã‚ŠãŒã¨ã†ã”ã–ã„ã¾ã—ãŸã€‚
nobusue 2015/04/19
GCP

BigQuery
ãƒªãƒ³ã‚¯
èª°ã§ã‚‚ç°¡å˜ã«è¶…é«˜é€Ÿãªã‚¯ã‚¨ãƒªãŒã§ãã‚‹BigQueryã¨ã¯ï¼Ÿ
ã“ã®2ã¤ã®æŠ€è¡“ã¯ã€ã‚°ãƒ¼ã‚°ãƒ«ç‹¬è‡ªã®æŠ€è¡“ã¨ã„ã†ã‚ã‘ã§ã¯ãªã„ã€‚ã—ã‹ã—ã€ãƒãƒ¼ãƒ‰ã‚¦ã‚§ã‚¢ã‹ã‚‰æ§‹ç¯‰ã—ã¦ã„ã‚‹ã€æ—¢å˜ã®ã‚°ãƒ¼ã‚°ãƒ«ã®ã‚¯ãƒ©ã‚¦ãƒ‰æŠ€è¡“ã‚’æ´»ç”¨ã—ã€ãƒ‘ãƒ–ãƒªãƒƒã‚¯ãªã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã¨ã—ã¦æä¾›å¯èƒ½ãªãƒ¬ãƒ™ãƒ«ã®å®Ÿè£…ã«ãªã£ã¦ã„ã‚‹ç‚¹ãŒGoogle BigQueryã®å¼·ã¿ã¨ãªã£ã¦ã„ã‚‹ã€‚ BigQueryã®ç‰¹å¾´ ä»–ã®é¡žä¼¼ã‚µãƒ¼ãƒ“ã‚¹ã¨ã®æ¯”è¼ƒ å·¨å¤§ãƒ‡ãƒ¼ã‚¿ã‚’å‡¦ç†ã™ã‚‹æŠ€è¡“ã¨ã—ã¦ã¯ã€åŒã˜ã‚°ãƒ¼ã‚°ãƒ«ãŒä½¿ã£ã¦ããŸMapReduceã¨ã„ã†ã‚‚ã®ãŒã‚ã‚‹ã€‚MapReduceã¨BigQueryã‚’æ¯”ã¹ã‚‹ã¨ã€MapReduceãŒå·¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’å®‰å®šçš„ã«å‡¦ç†ã§ãã‚‹ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ãƒ¢ãƒ‡ãƒ«ã§ã‚ã‚‹ã“ã¨ã«å¯¾ã—ã€BigQueryã¯ã‚¢ãƒ‰ãƒ›ãƒƒã‚¯ã«ãƒˆãƒ©ã‚¤ï¼†ã‚¨ãƒ©ãƒ¼ã—ãªãŒã‚‰ã‚¯ã‚¨ãƒªã‚’å®Ÿè¡Œã™ã‚‹ã‚µãƒ¼ãƒ“ã‚¹ã§ã‚ã‚‹ã“ã¨ãŒç•°ãªã£ã¦ã„ã‚‹ã€‚ MapReduceã¯ã€éžæ§‹é€ åŒ–ãƒ‡ãƒ¼ã‚¿ã‚’ã€ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ãƒ¢ãƒ‡ãƒ«ã‚’é€šã—ã¦æ‰±ã†ã“ã¨ãŒã§ãã€å·¨å¤§ãªãƒ†ãƒ¼ãƒ–ãƒ«ã®çµåˆã‚„å·¨å¤§ãªå‡ºåŠ›çµæžœã®ã‚¨ã‚¯ã‚¹ãƒãƒ¼ãƒˆã‚‚å¯èƒ½ã§ã‚ã‚‹åŠé¢ã€å‡¦ç†æ™‚
nobusue 2015/04/19
GCP

BigQuery
ãƒªãƒ³ã‚¯
BigQuery ã‚’ã‚¹ã‚ãƒ¼ãƒžãƒ¬ã‚¹ã£ã½ãé‹ç”¨ã™ã‚‹æ–¹æ³• - Qiita
Deleted articles cannot be recovered. Draft of this article would be also deleted. Are you sure you want to delete this article?
nobusue 2015/04/19
GCP

BigQuery
ãƒªãƒ³ã‚¯
ã€Žã‚¢ãƒ‰ãƒ†ã‚¯ã‚¹ã‚ãƒ«ã‚¢ãƒƒãƒ—ã‚¼ãƒŸã€€ã‚«ãƒ©ãƒ ãƒŠãƒ¼ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹æ¤œè¨¼ã¾ã¨ã‚ã€
çš†æ§˜ã“ã‚“ã«ã¡ã¯ã€‚ ã‚¢ãƒ‰ãƒ†ã‚¯æœ¬éƒ¨ã‚«ãƒ©ãƒ ãƒ¼ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‚¼ãƒŸãƒãƒ¼ãƒ ã§ã™ã€‚ ä»Šå›žã®è¨˜äº‹ã§ã¯ã‚¼ãƒŸãƒãƒ¼ãƒ ãŒè¡Œã£ãŸæ¤œè¨¼çµæžœã«ã¤ã„ã¦ç™ºè¡¨ã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ ã¾ãŸã€ã“ã®è¨˜äº‹ã«ã¤ãã¾ã—ã¦ã¯ 11/12 ã«è¡Œã‚ã‚ŒãŸ db tech showcase Tokyo 2014 ã«ã¦ç™ºè¡¨ã•ã›ã¦é ‚ãã¾ã—ãŸå†…å®¹ã«ãªã‚Šã¾ã™ã€‚ ãƒ—ãƒ¬ã‚¼ãƒ³è³‡æ–™ã¯ã“ã¡ã‚‰ã«ã‚ãŒã£ã¦ã„ã¾ã™ã€‚ â€»è¿½è¨˜ Impala / Presto ã® File Format ã«ã¤ã„ã¦ã”æŒ‡æ‘˜ã‚’é ‚ãã¾ã—ãŸã®ã§ãƒ‡ãƒ¼ã‚¿ãƒãƒ¼ãƒ‰åŠã³ã¾ã¨ã‚ã®éƒ¨åˆ†ã«è¿½è¨˜ã—ã¾ã—ãŸã€‚ ã‚¢ãƒ‰ãƒ†ã‚¯ã‚¹ã‚ãƒ«ã‚¢ãƒƒãƒ—ã‚¼ãƒŸ ã‚«ãƒ©ãƒ ãƒŠãƒ¼ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹æ¤œè¨¼ã¾ã¨ã‚ç›®çš„ åºƒå‘Šã‚·ã‚¹ãƒ†ãƒ ã§ã¯å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã«å…¥ã‚Œã¦è§£æžã‚’è¡Œã„ã¾ã™ã€‚ å°è¦æ¨¡ã‹ã‚‰ä¸è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿ã¯RDBMSã§è¡Œãˆã¾ã™ãŒã€æ•°TBã‚’è¶…ãˆã‚‹ã¨ RDBMSä»¥å¤–ã®é¸æŠžè‚¢ã‚’æŽ¢ã•ãªã„ã¨ã„ã‘ã¾ã›ã‚“ã€‚ ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ç”¨ã®ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¯æ¯”è¼ƒè³‡æ–™ãŒå°‘ãªãã€ ã¾ãŸã€ã‚ã£ãŸã¨ã—ã¦ã‚‚æ¤œè¨¼
nobusue 2014/11/14
BigQueryã™ã’ã‡ãª

Redshift

Impala

Presto

BigQuery
ãƒªãƒ³ã‚¯
Google BigQueryã§ä»ŠæœŸã‚¢ãƒ‹ãƒ¡ã«é–¢ã™ã‚‹ãƒ„ã‚¤ãƒ¼ãƒˆã‚’åˆ†æžã™ã‚‹ãžã„ï¼ - Qiita
BigQueryã§ã‚¢ãƒ‹ãƒ¡ã«é–¢ã™ã‚‹ãƒ„ã‚¤ãƒ¼ãƒˆã‚’åˆ†æžã™ã‚‹ ã‚¢ãƒ‹ãƒ¡ãƒ„ã‚¤ãƒ¼ãƒˆã‚’è¨˜éŒ²ã—ãŸCSVã‚’BigQueryã«ä¸Šã’ã‚‹ ãƒ‡ãƒ¼ã‚¿å½¢å¼ã‚’é¸æŠžã™ã‚‹ CSV/JSON/AppEngine BackupãŒé¸ã¹ã¾ã™ãŒä»Šå›žã¯CSVã¨ã—ã¾ã™ã€‚ File Uploadã§ãƒ–ãƒ©ã‚¦ã‚¶ã‹ã‚‰ãƒ•ã‚¡ã‚¤ãƒ«ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ãƒ»ãƒ»ãƒ»ã¨ã„ããŸã„æ‰€ã§ã™ãŒãƒ•ã‚¡ã‚¤ãƒ«ãƒ–ãƒ©ã‚¦ã‚¶ã‹ã‚‰ã®ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã¯10Mãƒã‚¤ãƒˆã«åˆ¶é™ã•ã‚Œã¦ã„ã¾ã™ã€‚ 10Mãƒã‚¤ãƒˆä»¥ä¸Šã®ãƒ•ã‚¡ã‚¤ãƒ«ã¯Googleã®ã‚¯ãƒ©ã‚¦ãƒ‰ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã«ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ Google Cloud Storage ã«ãƒ‡ãƒ¼ã‚¿ã‚’ä¸Šã’ã‚‹ Cloud Storageã«Bucketsã‚’ä½œæˆã—ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã—ã¾ã™ã€‚ ã“ã¡ã‚‰ã§ã¯ãƒ–ãƒ©ã‚¦ã‚¶ã‹ã‚‰10Mä»¥ä¸Šã®ãƒ•ã‚¡ã‚¤ãƒ«ã®ã‚‚ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã§ãã¾ã™ãŒã€æ•°ç™¾Gãƒã‚¤ãƒˆãªã©ã®ã§ã‹ã„ãƒ•ã‚¡ã‚¤ãƒ«ã¯ãƒ–ãƒ©ã‚¦ã‚¶ãŒã‚¿ã‚¤ãƒ ã‚¢ã‚¦ãƒˆãªã©ã‚’èµ·ã“ã™ã®ã§Googleã®ç”¨æ„ã™ã‚‹ãƒ„ãƒ¼ãƒ«(gsutil
nobusue 2014/10/29
BigQuery
ãƒªãƒ³ã‚¯
ã¯ã˜ã‚ã¦ã® BQ GAS
gcpja night #28 ã§ã®ç™ºè¡¨è³‡æ–™ã§ã™ã€‚ ã‚µãƒ³ãƒ—ãƒ«ã‚³ãƒ¼ãƒ‰ã¯ã“ã¡ã‚‰: https://gist.github.com/hakobera/537b35971e3b698ae083
nobusue 2014/09/17
ã“ã‚Œã™ã”ã„

Google

BigQuery

GoogleAppsScript
ãƒªãƒ³ã‚¯
BigQuery Connector for Excelã‚’ä½¿ã£ã¦Excelã§è¶…é«˜é€Ÿã‚¯ã‚¨ãƒªå®Ÿè¡Œã‚’è¡Œã† - Qiita
Google BigQueryã¯è¶…é«˜é€Ÿã§ã‚¯ã‚¨ãƒªã‚’å®Ÿè¡Œã§ãã¦ä¾¿åˆ©ã§ã‚ã‚‹ãŒã€çµå±€ã®ã¨ã“ã‚Excelãƒ‡ãƒ¼ã‚¿ã§ãã ã•ã„ã€ã¿ãŸã„ãªã“ã¨ã‚’è¨€ã‚ã‚ŒãŸã‚Šã—ã¦é¢å€’ãªã“ã¨ãŒã‚ã‚‹ã€‚ã“ã†ã„ã£ãŸã¨ãã€Google BigQuery Connector for Excelã‚’ä½¿ç”¨ã™ã‚Œã°ã€Microsoft Excelã‹ã‚‰ç›´æŽ¥Google BigQueryã«ã‚¯ã‚¨ãƒªã‚’ç™ºè¡Œã—çµæžœã‚’å–å¾—ã™ã‚‹ã“ã¨ãŒå‡ºæ¥ã‚‹ã€‚ BigQuery Connector for Excel - Google BigQuery - Google Developers BigQuery Connector for Excelã‚’æº–å‚™ã™ã‚‹ BigQuery Connector for Excelã‚’ä½¿ç”¨ã™ã‚‹ã«ã¯ã€ãƒšãƒ¼ã‚¸ã«ã‚¢ã‚¯ã‚»ã‚¹ã—å°‚ç”¨ã®èªè¨¼ã‚ãƒ¼ã‚’ç™ºè¡Œã™ã‚‹å¿…è¦ãŒã‚ã‚‹ã€‚ Google BigQuery Connector for Excel BigQuery Conn
nobusue 2014/06/05
ã“ã‚Œã¯å¼·åŠ›ãƒ»ãƒ»ãƒ»

BigQuery

excel
ãƒªãƒ³ã‚¯
Googleã®è™Žã®åã€ŒBigQueryã€ã‚’Fluentdãƒ¦ãƒ¼ã‚¶ãƒ¼ãŒä½¿ã‚ãªã„ç†ç”±ãŒãªããªã£ãŸç†ç”± #gcpja - Qiita
From Fluentd Meetupã«è¡Œã£ã¦ãã¾ã—ãŸ ã“ã‚Œã‚’èªã‚“ã æ™‚ã€BigQueryã®æ¤œç´¢ã‚¹ãƒ”ãƒ¼ãƒ‰ã«ã¤ã„ã¦ã¡ã‚‡ã£ã¨è£œè¶³ã—ãŸããªã£ãŸã€‚ç¢ºã‹ã«Fluentd Meetupã®ãƒ‡ãƒ¢ã§ã¯9å„„ä»¶ã‚’7ç§’ç¨‹åº¦ã§æ¤œç´¢ã—ã¦ã„ãŸãŒã€BigQueryã®çœŸã®å®ŸåŠ›ã¯ã“ã‚Œã‚ˆã‚Š1ã€œ2ã‚±ã‚¿ä¸Šã ã‹ã‚‰ã ã€‚ã¡ã‚‡ã£ã¨æ‰‹å…ƒã§å°‘ã—å¤§ãã‚ã®ãƒ†ãƒ¼ãƒ–ãƒ«ã§è©¦ã—ã¦ã¿ãŸã‚‰ã€120å„„è¡Œã®æ£è¦è¡¨ç¾ãƒžãƒƒãƒä»˜ãé›†è¨ˆãŒ5ç§’ã§å®Œäº†ã—ãŸã€‚è«–ã‚ˆã‚Šè¨¼æ‹ ã§ã€ãƒ‡ãƒ¢ãƒ“ãƒ‡ã‚ªï¼ˆ1åˆ†16ç§’ï¼‰ã‚’ä½œã£ã¦ã¿ãŸï¼š From The Speed of Google BigQuery ã“ã‚Œã¯é€Ÿã™ãŽã‚‹ã€‚ä½•ã‹ã®ã‚¤ãƒ³ãƒã‚ã§ã‚ã‚‹ï¼ˆæœ€åˆã«ãƒ‡ãƒ¢ã‚’è¦‹ãŸæ™‚ãã†æ€ã£ãŸï¼‰ã€‚æ£è¦è¡¨ç¾ã‚’ã„ã‚ã„ã‚å¤‰ãˆã¦ã¿ã¦ã‚‚ã‚¹ãƒ”ãƒ¼ãƒ‰ã¯å¤‰ã‚ã‚‰ãªã„ã€‚ã¤ã¾ã‚Šã€ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ã‚’äº‹å‰æ§‹ç¯‰ã§ããªã„ã‚¯ã‚¨ãƒªã«å¯¾ã—ã¦ã“ã®ã‚¹ãƒ”ãƒ¼ãƒ‰ãªã®ã§ã‚ã‚‹ã€‚ ä¾¡æ ¼ã‚‚å®‰ã„ã€‚ã•ã™ãŒã«120å„„è¡Œã®ã‚¯ã‚¨ãƒªã¯1å›žã§200å††ã‚‚ã‹ã‹ã£ã¦æ°—è»½ã«å®Ÿè¡Œã§ããªã•ãã†ã§ã‚ã‚‹ãŒã€1.2å„„
nobusue 2014/05/19
ã“ã‚Œã¯è€³å¯„ã‚ŠãªãŠè©±

BigQuery

fluentd
ãƒªãƒ³ã‚¯
1