[B! emr] ishideoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

ishideo id:ishideo

emrã«é–¢ã™ã‚‹ishideoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

GitHub - dacort/demo-code: Bits of code I use during live demos
ishideo 2021/07/20
demo-code

demo

aws

emr

athena

notebooks

cluster

github

cloudformation
ãƒªãƒ³ã‚¯
S3ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã‚’AWSã®åˆ†æžã‚µãƒ¼ãƒ“ã‚¹ã‚’ä½¿ã£ã¦æœ‰åŠ¹æ´»ç”¨ã™ã‚‹ #AWS-06 #AWSSummit | DevelopersIO
æœ¬è¨˜äº‹ã¯ã€AWS Summit Japan 2021ã®ã‚»ãƒƒã‚·ãƒ§ãƒ³å‹•ç”»ã€ã€ŒAWS-06: è²¯ã‚ã‚‹ã ã‘ã˜ã‚ƒã‚‚ã£ãŸã„ãªã„ï¼AWS åˆ†æžã‚µãƒ¼ãƒ“ã‚¹ã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã®æœ‰åŠ¹æ´»ç”¨ã€ã®ãƒ¬ãƒãƒ¼ãƒˆè¨˜äº‹ã§ã™ã€‚ ã€Œãƒ‡ãƒ¼ã‚¿ã¯ã¨ã‚Šã‚ãˆãšS3ã«æºœã‚ã¦ãŠã‘ï¼ã€ã¨ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã£ã½ã„ã‚‚ã®ã¯ä½œã‚ŒãŸã‘ã©ä¸Šæ‰‹ãä½¿ã„ã“ãªã›ã¦ã„ãªã„æ–¹ã€ãã‚Œãªã‚Šã«ã„ã‚‰ã£ã—ã‚ƒã‚‹ã®ã§ã¯ãªã„ã§ã—ã‚‡ã†ã‹ï¼Ÿæœ¬ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ãœã²AWSã®åˆ†æžã‚µãƒ¼ãƒ“ã‚¹ã®æ´»ç”¨è¡“ã‚’ãŠã•ã‚‰ã„ã—ã¦ã¿ã¦ãã ã•ã„ã€‚ æ¦‚è¦ ãƒ‡ãƒ¼ã‚¿ã®æŒã¤åŠ›ã‚’æ´»ã‹ã™æ–¹æ³•ã¨ã—ã¦ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ãŒã‚ã‚Šã¾ã™ãŒã€ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ä¸Šã«ãƒ‡ãƒ¼ã‚¿ã¯è²¯ã¾ã£ã¦ããŸã‚‚ã®ã®ã€ã©ã†æœ‰åŠ¹æ´»ç”¨ã™ã‚Œã°ã„ã„ã‹æ‚©ã‚“ã çµŒé¨“ã¯ãªã„ã§ã—ã‚‡ã†ã‹ï¼Ÿãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã«å˜åœ¨ã™ã‚‹ãƒ‡ãƒ¼ã‚¿ã¨åˆ†æžãƒ„ãƒ¼ãƒ«ã¨çµ„åˆã›ã€æ´»ç”¨ã™ã‚‹æ–¹æ³•ã¨ã—ã¦ã€â€œãƒ¬ã‚¤ã‚¯ãƒã‚¦ã‚¹ã‚¢ãƒ—ãƒãƒ¼ãƒâ€ãŒã‚ã‚Šã¾ã™ã€‚æœ¬ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ã¯"ãƒ¬ã‚¤ã‚¯ãƒã‚¦ã‚¹ã‚¢ãƒ—ãƒãƒ¼ãƒ"ã®è€ƒãˆæ–¹ã‚’ç´¹ä»‹ã™ã‚‹ã¨å…±ã«ã€ã©ã®ã‚ˆã†ãªAWSã‚µãƒ¼ãƒ“ã‚¹ã‚’ç”¨ã„ã¦"ãƒ¬ã‚¤ã‚¯ãƒã‚¦ã‚¹ã‚¢ãƒ—
ishideo 2021/05/24
datalake

s3

aws

athena

gue

redshift

emr

lake-formation

classmethod
ãƒªãƒ³ã‚¯
AmazonAthenaã§æ—¥æ™‚ãƒãƒƒãƒå‡¦ç†ã‚’ä½œæˆ â€“ TECH4ALL
ishideo 2021/03/05
athena

emr

apachespark

batch

pyspark

s3

lambda

aws
ãƒªãƒ³ã‚¯
Athenaã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã®è¨è¨ˆ - Speee DEVELOPER BLOG
ã“ã‚“ã«ã¡ã¯ã€‚UZOUã®ãƒ—ãƒãƒ€ã‚¯ãƒˆé–‹ç™ºã‚’ã—ã¦ã„ã‚‹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®@kanga333ã§ã™ã€‚ UZOUã§ã¯åºƒå‘Šãƒ‡ãƒ¼ã‚¿ã®é›†è¨ˆã®ä¸€éƒ¨ã«Amazon Athenaã‚’æŽ¡ç”¨ã—ã¦ã„ã¾ã™ã€‚ ã“ã®è¨˜äº‹ã§ã¯UZOUã«ãŠã‘ã‚‹Athenaã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã®è¨è¨ˆã«ã¤ã„ã¦ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ å…¨ä½“æ§‹æˆ ãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã®å…¨ä½“æ§‹æˆã¯æ¬¡ã®ã‚ˆã†ã«ãªã£ã¦ã„ã¾ã™ã€‚ ä»¥å¾Œã¯ãã‚Œãžã‚Œã®ã‚³ãƒ³ãƒãƒ¼ãƒãƒ³ãƒˆã«ã¤ã„ã¦é †æ¬¡ç´¹ä»‹ã—ã¦ã„ãã¾ã™ã€‚ Fleuntdã«ã‚ˆã‚‹S3ã¸ã®é›†ç´„ UZOUã§ã¯ç‰¹ã«Fluentdã‚¢ã‚°ãƒªã‚²ãƒ¼ã‚¿ã®ã‚ˆã†ãªä¸ç¶™ã‚µãƒ¼ãƒã¯è¨ã‘ã¦ã„ã¾ã›ã‚“ã€‚åºƒå‘Šé…ä¿¡ã‚µãƒ¼ãƒã«å¸¸é§ã™ã‚‹FluentdãŒãƒã‚°ã‚’ç›´æŽ¥S3ã«ãƒ—ãƒƒãƒˆã—ã¦ã„ã¾ã™ã€‚ ä»¥ä¸‹ã¯Fluentdã®S3 outputéƒ¨åˆ†ã®è¨å®šã®ä¸€éƒ¨æŠœç²‹ã§ã™ã€‚ <buffer time> @type file timekey 60m </buffer> path example_table/dt=%Y%m%d/h
ishideo 2020/11/11
aws

athena

fluentd

s3

parquet

airflow

glue

emr

kenesis-data-firehose

terraform
ãƒªãƒ³ã‚¯
GitHub - commoncrawl/cc-pyspark: Process Common Crawl data with Python and Spark
ishideo 2020/10/23
commoncrawl

pyspark

python

apachespark

github

cc-pyspark

aws

emr

s3

warc
ãƒªãƒ³ã‚¯
å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«ã¤ã„ã¦ç¬¬2å›žï½žEMR(Hadoop)ã«ã¤ã„ã¦ã€ãªãœEMRãªã®ã‹ï½ž - Qiita
åˆã‚ã«ã€å‰å›žã‹ã‚‰å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã«ã¤ã„ã¦é€£è¼‰ã•ã›ã¦ã„ãŸã ãã¾ã—ãŸãŒã€ æ€ã£ãŸã‚ˆã‚Šå¤šãã®æ–¹ã«ã‚¹ãƒˆãƒƒã‚¯ã„ãŸã ãã€ã“ã®åˆ†é‡Žã®é–¢å¿ƒã®é«˜ã•ã‚’æ”¹ã‚ã¦å®Ÿæ„Ÿã—ã¦ãŠã‚Šã¾ã™ã€‚ ã‚ã‚Œã‚ã‚Œã‚‚ã€ã“ã®åˆ†é‡Žã®äº‹æ¥ã«åŠ›ã‚’å…¥ã‚Œã¦ãŠã‚Šã€ ç©æ¥µçš„ã«å–ã‚Šçµ„ã‚“ã§è¡Œãæ‰€å˜ã§ã”ã–ã„ã¾ã™ï¼ ãã‚Œã§ã¯ å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«ã¤ã„ã¦ç¬¬2å›ž ï½žEMR(Hadoop)ã«ã¤ã„ã¦ã€ãªãœEMRãªã®ã‹ï½ž ã”å˜ã˜ãªã„æ–¹ã¯ã€EMRã£ã¦ä½•ï¼Ÿã¨ã„ã†ã“ã¨ã«ãªã‚‹ã¨æ€ã†ã®ã§ è»½ãèª¬æ˜Žã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ EMRã¨ã¯ã€Amazon Web Serviceç¤¾ï¼ˆAWSï¼‰ãŒæä¾›ã—ã¦ã„ã‚‹ ã‚¯ãƒ©ã‚¦ãƒ‰ã§ã®Hadoopã®å®Ÿè¡Œç’°å¢ƒã§ã™ã€‚ æ£å¼åç§°ã¯ã€ŒAmazon Elastic MapReduceã€ã«ãªã‚Šã¾ã™ã€‚ å‡¦ç†å¯¾è±¡ãƒ‡ãƒ¼ã‚¿ã‚’AWSã®S3(ã‚¯ãƒ©ã‚¦ãƒ‰ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸)ã«é…ç½®ã—ã€ å¿…è¦ãªã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ãƒªã‚½ãƒ¼ã‚¹ã‚’ã‚’æŒ‡å®šã—ã¦å®Ÿè¡Œã™ã‚‹ã¨ã€ å¿…è¦ãªæ™‚ã«å¿…è¦ãªã ã‘ã€æ–‡å—é€šã‚Šã€ŒElasticã€ã«Hadoop
ishideo 2017/08/01
hadoop

emr

aws

mapreduce

qiita
ãƒªãƒ³ã‚¯
1