[B! kinesis] takaya030ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

takaya030 id:takaya030

kinesisã«é–¢ã™ã‚‹takaya030ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Apache Flinkã‚’ä½¿ã„Kinesisã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒ‡ãƒ¼ã‚¿ã‚’å‡¦ç†ã™ã‚‹
â€»ä¸Šè¨˜ã¯ã‚ãã¾ã§ç§ã®è¦–ç‚¹ã§ã®è©•ä¾¡ã§ã™ã€‚Akkaã®ç†Ÿç·´è€…ã§ã‚ã‚Œã°è©•ä¾¡ãŒå¤‰ã‚ã‚‹ã¨æ€ã„ã¾ã™ã€‚ Akkaã‚’ä½¿ã„ãŸã‹ã£ã¦ã¿ãŸã£ãŸã®ã§ã™ãŒã€å¦ç¿’ã‚³ã‚¹ãƒˆãŒä»–ã¨æ¯”è¼ƒã—ã¦ã‹ã‹ã£ã¦ã—ã¾ã†ãŸã‚ä»Šå›žã¯Flinkã‚’ä½¿ã†ã“ã¨ã«ã—ã¾ã—ãŸã€‚ ã¾ãŸAkkaã«è©³ã—ã„æ–¹ã‚ˆã‚Šã€ã€ŒAkkaã¯ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã¯ãªãã€ãƒ©ã‚¤ãƒ–ãƒ©ãƒªãªã®ã§ã€ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã¯Akkaã‚’ãƒ™ãƒ¼ã‚¹ã«ä½œã£ã¦ãã‚Œã€ã€€ã¨ã„ã†ç«‹ã¡ä½ç½®ã§ã‚ã‚Šã€ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã¨æ¯”è¼ƒã™ã‚‹ã¨ãƒˆãƒ¬ãƒ¼ã‚µãƒ“ãƒªãƒ†ã‚£ãªã©ã§è¦‹åŠ£ã‚Šã™ã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã¨æ•™ãˆã¦é ‚ãã¾ã—ãŸã€‚ å¾Œã§å¾Œè¿°ã—ã¾ã™ãŒã€Flinkã«ã‚‚å†…éƒ¨ã§AkkaãŒä½¿ã‚ã‚Œã¦ã„ã¦ã€Akkaãƒ™ãƒ¼ã‚¹ã§ä½œã‚‰ã‚Œã¦ãŠã‚Šã¾ã—ãŸã€‚ Akkaã¯Sparkã€ã¾ãŸã¯Flinkã‚ˆã‚Šã‚‚ä½Žãƒ¬ãƒ™ãƒ«ã®ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã§ã€ã‚ˆã‚Šå¤šãã®ãƒ‘ãƒ¯ãƒ¼ãŒã‚ã‚‹ãŒã€è€ƒæ…®ã™ã¹ãè€ƒæ…®äº‹é …ã‚‚å¤šããªã‚‹ã¨ã„ã†ã“ã¨ã ã¨ã„ã†ã“ã¨ã‹ãªã¨æ€ã„ã¾ã™ã€‚ Flinkã®å¼·ã¿ ãƒ‡ãƒ¼ã‚¿ã®ä¿è¨¼ Exactly once ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã¯æ¬ æã‚‚é‡è¤‡ã‚‚
takaya030 2023/08/31
kinesis

apache

flink
ãƒªãƒ³ã‚¯
Amazon Kinesisã®ãƒ‡ãƒ¼ã‚¿ã‚’td-agentã§å–ã‚Šå‡ºã™ - Qiita
Amazon Kinesisã«å…¥åŠ›ã•ã‚ŒãŸãƒ‡ãƒ¼ã‚¿ã‚’td-agentã§å–ã‚Šå‡ºã—ã¾ã™ã€‚ Amazon Kinesisã®ãƒ‡ãƒ¼ã‚¿ã®å–ã‚Šå‡ºã—ã«ã¯ã€kinpiraã•ã‚“ãŒä½œã£ãŸfluent-plugin-in-kinesisï¼ˆ http://qiita.com/kinpira/it ems/cefaf9050429325498aa ï¼‰ã¨ã„ã†ãƒ—ãƒ©ã‚°ã‚¤ãƒ³ã‚’ä½¿ã‚ã›ã¦é ‚ãã¾ã™ã€‚ ä»Šå›žã®ç’°å¢ƒ ä»Šå›žã¯ä»¥ä¸‹ã®ç’°å¢ƒã‚’ä½¿ã„ã¾ã™ã€‚ 1.Amazon Kinesis ãƒ»ã‚¹ãƒˆãƒªãƒ¼ãƒ åï¼štest-stream 2.Amazon EC2 ãƒ»Ubuntu 14.04 ãƒ»ã€Œ1ã€ã§ä½œæˆã—ãŸã‚¹ãƒˆãƒªãƒ¼ãƒ ã‚’æ“ä½œå¯èƒ½ãªIAM Roleã‚’ä»˜ä¸Ž ãƒ»AWS CLIã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« ä»¥é™ã®æ“ä½œã¯ã™ã¹ã¦ã€Œ2ã€ã§ä½œæˆã—ãŸEC2ä¸Šã§å®Ÿæ–½ã—ã¾ã™ã€‚ td-agentã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« ã¾ãšEC2ã«td-agentã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¾ã™ã€‚
takaya030 2021/10/30
fluentd

kept

kinesis
ãƒªãƒ³ã‚¯
fluentdã®plugin ã€Œin_kinesisã€ã‚’ä½œã£ã¦ã¿ãŸ - Qiita
#ã¯ã˜ã‚ã« ã¨ã‚ã‚‹ãã£ã‹ã‘ã§ã€AWS kinesisã‹ã‚‰logã‚’å–å¾—ã™ã‚‹ã€fluentdã®Input Pluginã€ in_kinesis ãªã‚‹ã‚‚ã®ã‚’ä½œã£ãŸã®ã§ã€ãã®è©±ã‚’æ›¸ãã¾ã™ã€‚ ã¡ãªã¿ã«ã€in_kinesisã€‘ã¯ã“ã¡ã‚‰ã«ãªã‚Šã¾ã™ã€‚ gemç‰ˆã¯ã“ã¡ã‚‰ã«ãªã‚Šã¾ã™ã€‚ #ãã£ã‹ã‘ ä»Šå›žã€ãªãœPluginã‚’ä½œã£ãŸã‹ã¨ã„ã†ã¨ã€ ä¼šç¤¾ã§æ–°ãŸã«ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãƒãƒˆãƒ«ã‚’å°Žå…¥ã™ã‚‹äº‹ã«ãªã‚Šã€ãã®logå‘¨ã‚Šã«æºã‚ã‚‹äº‹ã«ãªã£ãŸã®ãŒãã£ã‹ã‘ã§ã—ãŸã€‚ ãã“ã§ã€ä»Šã¾ã§é€šã‚Šã®ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã§è‰¯ã„ã®ã‹ã©ã†ã‹ã®æ¤œè¨Žã‹ã‚‰è¡Œã†äº‹ã«ã—ã¾ã—ãŸã€‚ ãã®æ™‚ã®logã®è¦ä»¶ã¯ã“ã‚“ãªæ„Ÿã˜ã§ã—ãŸã€‚ ã¨ã«ã‹ããƒã‚°ãŒBigQueryã«å…¥ã‚Œã°è‰¯ã„ã€‚ ãƒã‚°ã®æ•°ã®ä¸Šé™å€¤ãŒä¸æ˜Ž fluentdã®BigQuery pluginãŒæœ‰ã‚‹ã‹ã‚‰BigQueryã«å…¥ã‚Œã‚‹ã®ã¯ç°¡å˜ã ãï¼ ã¨æ€ã£ã¦ã„ãŸã®ã§ã™ãŒã€ ã“ã“ã§å•é¡Œã¨ãªã‚‹ã®ãŒã€Œãƒã‚°ã®æ•°ã®ä¸Šé™å€¤ãŒä¸æ˜Žã€â†ã“ã‚Œã§ã—ãŸ!!
takaya030 2021/10/30
fluentd

kept

kinesis
ãƒªãƒ³ã‚¯
Amazon Kinesis ãƒ©ã‚¤ãƒ–ãƒ©ãƒª Pythonç‰ˆã‚’ä½¿ã£ã¦ã¿ã‚‹ - ç¶š ã‚«ãƒƒã‚³ã®ä»˜ã‘æ–¹
å‰å›žã”ç´¹ä»‹ã—ãŸKCL for Python ã§ã™ãŒã€ã¡ã‚ƒã‚“ã¨ä½¿ã£ã¦ã¿ã¾ã—ãŸã€‚ ã¨è¨€ã„ã¾ã—ã¦ã‚‚ã€ãƒ–ãƒã‚°ã«ã‚ã‚‹é€šã‚Šã€ã‚µãƒ³ãƒ—ãƒ«ã‚’å‹•ã‹ã™ã ã‘ãªã®ã§ã™ãŒã€PyPiã§ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚‹ã¨ã€å‹æ‰‹ãŒé•ã†ã®ã§è‰²ã€…ãƒãƒžã‚Šã¾ã—ãŸã€ãŒä½•ã¨ã‹å‹•ä½œç¢ºèªãŒå‡ºæ¥ã¾ã—ãŸã€‚ ã§ã¯ã€é †ã‚’è¿½ã£ã¦èª¬æ˜Žã—ã¾ã™ã€‚ ã¾ãšã€sampleã«ã‚ã‚‹ãƒ–ãƒ„ã«ã¤ã„ã¦ã§ã™ãŒ - sample_kinesis_wordputter.py ã‚³ã‚¤ãƒ„ã¯Kinesisã«ãƒ‡ãƒ¼ã‚¿ã‚’æ”¾ã‚Šè¾¼ã‚€ã ã‘ã§ã™ã€‚å‚ç…§ã¨ã—ã¦ã¯boto.kinesisã‚’importã—ã¦ã„ã‚‹ã ã‘ãªã®ã§ã€KCLã¨ã¯ç›´æŽ¥é–¢ä¿‚ã”ã–ã„ã¾ã›ã‚“ã€‚ - sample_kclpy_app.py ã“ã‚ŒãŒä»Šå›žã®ãƒ¡ã‚¤ãƒ³ã§ã™ã€‚Kinesisã‹ã‚‰ãƒ‡ãƒ¼ã‚¿ã‚’å—ã‘å–ã‚Šã€å‡¦ç†ã—ã¾ã™ã€‚ä½•ã‚’ã—ã¦ã„ã‚‹ã®ã‹ã¯ã‚ã¨ã§ãã€‚ - sample.properties .propertiesãªã®ã§ã€ã‚ã‹ã‚‰ã•ã¾ã«Javaæ§˜ãŒä½¿ã†ã‚‚ã®ã§ã™ã‚ˆãï¼ã‚³ã‚¤ãƒ„ã¯æœ¬å®¶j
takaya030 2021/09/28
kept

kinesis

python
ãƒªãƒ³ã‚¯
KinesisClientLiblary(KCL)ã«ãŠã‘ã‚‹LATESTã¨TRIM_HORIZONã®æŒ™å‹•ã«ã¤ã„ã¦ - Qiita
AWS Kinesis Stream APIã§è¨˜è¼‰ã•ã‚Œã¦ã„ã‚‹Shard iterator typeã®ã†ã¡ã€KCL(Kinesis Client Liblary)ã«æ¨™æº–ã§ç”¨æ„ã•ã‚Œã¦ã„ã‚‹ã®ã¯ LATEST ã¨ TRIM_HORIZON ã§ã™ãŒã€ã“ã‚Œã®æŒ™å‹•ã«ã¤ã„ã¦æ°—ã«ãªã‚‹ç‚¹ãŒã‚ã£ãŸã®ã§èª¿æŸ»ã—ã¦ã¿ã¾ã—ãŸã€‚é€”ä¸ã®è¨³æ–‡ã¯é©å½“ã§ã™ã€‚ LATEST åŽŸç¾©ã§ã¯ ï¼ˆåŽŸæ–‡ï¼‰LATEST - Start reading just after the most recent record in the shard, so that you always read the most recent data in the shard. ï¼ˆè¨³ï¼‰LATEST - ã‚·ãƒ£ãƒ¼ãƒ‰å†…ã®æœ€æ–°ãƒ¬ã‚³ãƒ¼ãƒ‰ã®ç›´å¾Œã‹ã‚‰èªã¿å–ã‚Šã‚’é–‹å§‹ã—ã¾ã™ã€‚ãªã®ã§ã‚·ãƒ£ãƒ¼ãƒ‰å†…ã®æœ€æ–°ã®ãƒ‡ãƒ¼ã‚¿ã‚’å¸¸ã«èªã¿ã¾ã™ã€‚ ã¨ãªã£ã¦ã„ã¾ã™ã€‚ å®Ÿéš›ã®æŒ™å‹•ã¨ã—ã¦ã¯KCLã¯ã¾ãšDynam
takaya030 2021/09/28
kept

kinesis
ãƒªãƒ³ã‚¯
AWS kinesis GetRecords ã®æ£ã—ã„å‘¼ã³å‡ºã—æ–¹ - simotin13's message
AWSã®ã‚µãƒ¼ãƒ“ã‚¹ã®ï¼‘ã¤ã§ã‚ã‚‹Kinesis Streamã‚’ä½¿ã†æ©Ÿä¼šãŒã‚ã‚Šã¾ã—ãŸã€‚ APIã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿ã®ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã¨ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã‚’ã™ã‚‹éš›ã«ã„ãã¤ã‹æ°—ã‚’ä»˜ã‘ãªã„ã¨ã„ã‘ãªã„ç‚¹ãŒã‚ã£ãŸã®ã§æŒ™ã’ã¦ãŠããŸã„ã¨æ€ã„ã¾ã™ã€‚ Kinesisã®ç‰¹å¾´ Kinesisã®ç‰¹å¾´ã¯ãƒ‡ãƒ¼ã‚¿ã®ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ å‡¦ç†ãŒã§ãã‚‹ã“ã¨ã§ã‚¯ãƒ©ã‚¦ãƒ‰ã«ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã—ãŸãƒ‡ãƒ¼ã‚¿ã¯åŸºæœ¬çš„ã«ï¼’ï¼”æ™‚é–“ã®é–“ã‚¢ã‚¯ã‚»ã‚¹ã§ãã¾ã™ã€‚ RDBã¨é•ã£ã¦æ°¸ç¶šçš„ã«ä¿å˜ã•ã‚Œã‚‹ã‚ã‘ã§ã¯ãªãã€ã‚ãã¾ã§ã‚¹ãƒˆãƒªãƒ¼ãƒ ã«æµã—ãŸãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ä½•ã‚‰ã‹ã®è§£æžã‚’è¡Œã†ã¨ã„ã†ã®ãŒã‚µãƒ¼ãƒ“ã‚¹ã®ç”¨é€”ã®ã‚ˆã†ã§ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹API ã•ã¦ã€kinesisã¸ã®ãƒ‡ãƒ¼ã‚¿ã®ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã¯ã€ PutRecord PutRecords ã¨ã„ã†2ã¤ã®APIã«ã‚ˆã£ã¦è¡Œã„ã¾ã™ã€‚ ä¸¡è€…ã®é•ã„ã¯æ–‡å—é€šã‚Šï¼‘ã¤ã®ãƒ‡ãƒ¼ã‚¿ã‹è¤‡æ•°ãƒ‡ãƒ¼ã‚¿ã‹ã«ãªã‚Šã¾ã™ã€‚ ã¡ãªã¿ã«ã€PutRecordsã¯APIã®ä»•æ§˜ã‚’èªã‚€é™ã‚Š
takaya030 2021/09/28
kept

kinesis
ãƒªãƒ³ã‚¯
Kinesis Data Streamsã¨Lambdaã®é€£æºã§ã€å‡¦ç†æ¸ˆã¿ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ãŒé‡è¤‡ã—ã¦å®Ÿè¡Œã•ã‚Œã¾ã—ãŸ | DevelopersIO
å›°ã£ã¦ã„ãŸå†…å®¹ Kinesis Data Streamsã‚’ãƒˆãƒªã‚¬ãƒ¼ã«Lambdaã‚’å‹•ä½œã•ã›ã¦ã„ã¾ã™ã€‚ Lambdaã®ãƒˆãƒªã‚¬ãƒ¼ã‚’æœ‰åŠ¹ã‹ã‚‰ç„¡åŠ¹ã«åˆ‡ã‚Šæ›¿ãˆãŸå ´åˆã«ã€ æ—¢ã«å‡¦ç†æ¸ˆã¿ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ãŒã€é‡è¤‡ã—ã¦å®Ÿè¡Œã•ã‚Œã¾ã—ãŸã€‚ ãƒ»Kinesisã«put_recordå®Ÿæ–½ ãƒ»å…ˆé ã®ä½•ä»¶ã‹ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ã‚’Lambdaã§å®Ÿè¡Œ (A) ãƒ»update_event_source_mappingã§ãƒˆãƒªã‚¬ã‚’ç„¡åŠ¹åŒ– 10åˆ†å¾…æ©Ÿã—ãŸå¾Œã€ ãƒ»update_event_source_mappingã§ãƒˆãƒªã‚¬ã‚’æœ‰åŠ¹åŒ– ãƒ»å¾Œç¶šã®ä½•ä»¶ã‹ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ã‚’Lambdaã§å®Ÿè¡Œ (B) â†’ (A)ã§å‡¦ç†æ¸ˆã¿ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ãŒã€é‡è¤‡ã—ã¦å®Ÿè¡Œã•ã‚ŒãŸ é‡è¤‡ã—ã¦å‹•ä½œã™ã‚‹ç†ç”±ã‚’æ•™ãˆã¦ã„ãŸã ã‘ãªã„ã§ã—ã‚‡ã†ã‹ï¼Ÿ ã©ã†å¯¾å¿œã™ã‚Œã°ã„ã„ã®? Kinesis Data Streamsã¯ã€è¤‡æ•°ã®ã‚¢ãƒ™ã‚¤ãƒ©ãƒ“ãƒªãƒ†ã‚£ã‚¾ãƒ¼ãƒ³ã§ãƒ‡ãƒ¼ã‚¿ã‚’è¤‡è£½ã—ã¦ä¿æŒã—ã¦ã„ã¾ã™ã€‚ åˆ†æ•£åž‹ã®ç‰¹è³ªã‚’æŒã£ã¦ã„ã¾ã™ã®
takaya030 2021/09/28
kinesis

lambda
ãƒªãƒ³ã‚¯
1