[B! log] wkubotaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

wkubota id:wkubota

logã«é–¢ã™ã‚‹wkubotaã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ã€ãƒã‚°åˆ†é›¢ã€‘ ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’ DB ã«ä¿å˜ã—ã¦ã¯ã„ã‘ã¾ã›ã‚“
ã¯ã˜ã‚ã« TROCCO ã§ã¯ ETL ã‚¸ãƒ§ãƒ–ã‚„ã€dbt é€£æºã€ãƒ¯ãƒ¼ã‚¯ãƒ•ãƒãƒ¼ãªã©ã€æ§˜ã€…ãªã‚¸ãƒ§ãƒ–ã§å®Ÿè¡Œãƒã‚°ã‚’ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã§è¦‹ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ ã“ã‚Œã«ã‚ˆã‚Šã‚¨ãƒ©ãƒ¼æ™‚ã®ãƒˆãƒ©ãƒ–ãƒ«ã‚·ãƒ¥ãƒ¼ãƒˆã‚’ã‚¹ãƒ ãƒ¼ã‚ºã«è¡Œã†ã“ã¨ãŒã§ãã¾ã™ã€‚ ãã—ã¦ã€ã“ã®å®Ÿè¡Œãƒã‚°ã§ã™ãŒã€DB ã«ã‚ã‚‹ã‚¸ãƒ§ãƒ–ã®ãƒ¬ã‚³ãƒ¼ãƒ‰ã®1ã‚«ãƒ©ãƒ ã«æ›¸ãè¾¼ã¾ã‚Œã¦ã„ã¾ã—ãŸã€‚ ã“ã®ã‚ˆã†ã« TROCCO ã®å®Ÿè¡Œãƒã‚°ã®è¡¨ç¤ºæ©Ÿèƒ½ã¯ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã®ã‚¢ãƒ³ãƒãƒ‘ã‚¿ãƒ¼ãƒ³ã®ä¸Šã«æˆã‚Šç«‹ã£ã¦ã„ã¾ã™ã€‚ ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã¯ TEXT åž‹ã§ DB ã«ä¿å˜ã•ã‚Œã¦ã„ã‚‹ãŸã‚ã‚µã‚¤ã‚ºãŒå¤§ããã€ã¾ãŸ TROCCO ã®æˆé•·ã«å¿œã˜ã¦ã‚¸ãƒ§ãƒ–æ•°ã¯å¢—ãˆã‚‹ãŸã‚ã€ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚‚ã‚µãƒ¼ãƒ“ã‚¹ã®æˆé•·ã¨ã¨ã‚‚ã«ç„¡é™ã«æˆé•·ã—ã¦ã—ã¾ã„ã¾ã™ã€‚ ãŠã‹ã’ã•ã¾ã§ã‚¢ã‚«ã‚¦ãƒ³ãƒˆæ•°ã¯å¢—åŠ ã®ä¸€æ–¹ã§ã‚ã‚Šã€ãã‚Œã«ä¼´ã„ DDL ãŒé…ããªã£ãŸã‚Šã€ã¾ãŸ SELECT ã®ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ã‚‚å½“ç„¶è½ã¡ã‚‹ãŸã‚ã€INDEX ã«æ°—ã‚’ã¤ã‘ãŸã‚Šã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã—ãŸã€‚ SRE ã§ã¯ã“ã‚Œã«èª²
wkubota 2024/12/01
ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£

*ã‚ã¨ã§èªã‚€

log
ãƒªãƒ³ã‚¯
S3ã«ã‚ã‚‹ALBãƒã‚°ã®èª¿æŸ»ã¯Athenaã‚ˆã‚ŠDuckDBã®ã»ã†ãŒç°¡å˜ - road288ã®æ—¥è¨˜
AWSã®ALB(Application Load Balancer)ã®ãƒã‚°ã¯S3ã«ç½®ã‹ã‚Œã‚‹ãŒã€ã“ã®ä¸èº«ã‚’ã‚µã‚¯ãƒƒã¨èª¿ã¹ãŸã„ã¨ãã€Athenaã‚’ä½¿ã†æ–¹æ³•ãŒæ¨™æº–çš„ã§ã€ä¸‹è¨˜ã§æ¡ˆå†…ã•ã‚Œã¦ã„ã‚‹ã‚ˆã†ã«ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³å°„å½±(Partition Projection)ã§ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’ä½œã£ã¦Athenaã‹ã‚‰ã‚¯ã‚¨ãƒªã™ã‚‹ã€‚ ãƒ‘ãƒ¼ãƒ†ã‚£ã‚·ãƒ§ãƒ³å°„å½±ã‚’ä½¿ç”¨ã—ã¦ Athena ã§ ALB ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ç”¨ãƒ†ãƒ¼ãƒ–ãƒ«ã‚’ä½œæˆã™ã‚‹ - Amazon Athena ç§ã‚‚å¾“æ¥ã¯ãã®æ–¹æ³•ã‚’ä½¿ã£ã¦ã„ãŸãŒã€Athenaã¯ãƒ–ãƒ©ã‚¦ã‚¶ã‹ã‚‰ä½¿ã†ã¨å‹•ä½œãŒã‚‚ã£ã•ã‚Šã—ã¦ã„ã‚‹ã—ã€æ±ºã¾ã£ãŸã‚¯ã‚¨ãƒªã‚’1å›žãã‚Šå®Ÿè¡Œã—ã¦çµæžœã‚’å–å¾—ã—ãŸã„ã ã‘ã®ã¨ããªã‚‰ã¾ã ã—ã‚‚ã€æŽ¢ç´¢çš„ã«ã‚¯ã‚¨ãƒªã‚’ä½•ç™ºã‚‚å®Ÿè¡Œã—ãŸã„ã¨ãã«ã¯ä½¿ã„å‹æ‰‹ãŒæ‚ªã„ã€‚ æœ€è¿‘ä»–ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã§DuckDBã‚’ä½¿ã†ã‚ˆã†ã«ãªã£ã¦ã€ä½¿ã„å‹æ‰‹ã®è‰¯ã•ã«æ„Ÿå‹•ã—ã¦ã„ãŸãŒã€DuckDBã¯ALBã®ãƒã‚°ã‚’æŽ¢ç´¢çš„ã«èª¿ã¹ãŸã„ã¨ãã«ã‚‚ã‚ã£ã¡ã‚ƒä½¿ãˆã‚‹ã¨æ€ã£ãŸ
wkubota 2024/11/06
*ã‚ã¨ã§èªã‚€

log

db

è¨è¨ˆ
ãƒªãƒ³ã‚¯
ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã‚’ã©ã†ä½¿ãˆã°ã„ã„ã®ã‹ã£ã¦ç–‘å•ã€å…¨ã¦è§£æ±º
ã“ã‚Œã¯ãªã« ã©ã‚‚ã€ãƒ¬ãƒãƒ†ãƒƒã‚¯é–‹ç™ºéƒ¨ã®ã‚‚ã‚ŠãŸã§ã™ã€‚ ä»Šå›žã¯MySQLã§ã®ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã«ã¤ã„ã¦èª¿æŸ»ã—ã¦ã¾ã¨ã‚ã¾ã—ãŸã€‚ ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã¨ã„ãˆã°å¤ãã‹ã‚‰ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã®åŠ›å¼·ã„å‘³æ–¹ã¨ã„ã£ãŸãµã†ã«èªžã‚‰ã‚Œã‚‹ã“ã¨ã‚‚å¤šã„ã§ã™ãŒã€æœ€è¿‘ã¯ã‚¯ãƒ©ã‚¦ãƒ‰ã§ä½¿ãˆã‚‹ä¾¿åˆ©ãªãƒ„ãƒ¼ãƒ«ã‚‚ç”Ÿã¾ã‚Œã¦ã„ã¾ã™ã€‚ã“ã®è¨˜äº‹ã§ã¯ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã®ä¸€èˆ¬çš„ãªä½¿ã„æ–¹ã‚’ç´¹ä»‹ã™ã‚‹ã¨ã¨ã‚‚ã«ã€ä»–ã®ãƒ„ãƒ¼ãƒ«ã¨ã®æ¯”è¼ƒã‚„ã€ã©ã‚“ãªå ´é¢ã§ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ãŒå½¹ã«ç«‹ã¤ã®ã‹ã€ã¾ãŸå½¹ã«ç«‹ãŸãªã„å ´åˆã¯ã©ã‚“ãªãƒ„ãƒ¼ãƒ«ã‚’åˆ©ç”¨ã™ã‚‹ã“ã¨ãŒã§ãã‚‹ã®ã‹ã«ã¤ã„ã¦ã¾ã¨ã‚ã¾ã—ãŸã€‚ è¶³ã‚Šãªã„ã¨ã“ã‚ãªã©ã‚ã‚Œã°ãŠãŠã„ã«ãƒžã‚µã‚«ãƒªæŠ•ã’ã¦ã„ãŸã ã‘ã‚‹ã¨å¹¸ã„ã§ã™ã€‚ è¨˜äº‹ã®æµã‚Œ è¨˜äº‹ã®æµã‚Œ ã“ã®è¨˜äº‹ã¯ãã“ãã“é•·ã„ã®ã§ã€åˆã‚ã«è¨˜äº‹ã®æµã‚Œã‚’è§£èª¬ã—ã¾ã™ã€‚é©å®œèªã¿é£›ã°ã—ã¦ãã ã•ã„ã€‚ ãªãœã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ãªã®ã‹ ã“ã“ã§ã¯ãã‚‚ãã‚‚ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã‚’ãªãœç¢ºèªã—ãŸã„ã®ã‹ã¿ãŸã„ãªã¨ã“ã‚ã‚’èª¬æ˜Žã—ã¾ã™ ã‚¹ãƒãƒ¼ã‚¯ã‚¨ãƒªãƒã‚°ã®
wkubota 2024/09/08
*ã‚ã¨ã§èªã‚€

log

mysql
ãƒªãƒ³ã‚¯
ã‚‚ã£ã¨ log/slog ã‚’ä½¿ãŠã†
ã¯ã˜ã‚ã« ã“ã®è¨˜äº‹ã¯ Go ã‚¢ãƒ‰ãƒ™ãƒ³ãƒˆã‚«ãƒ¬ãƒ³ãƒ€ãƒ¼ 2023 ã®æœ€çµ‚æ—¥ 25 æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ çš†ã•ã‚“ log/slog ä½¿ã£ã¦ã¾ã™ã‹ã€‚ä¾¿åˆ©ãªã®ã§ãœã²ä½¿ã£ã¦ãã ã•ã„ã€‚ slog ã¯æ§‹é€ åŒ–ãƒã‚°ã‚’å‡ºåŠ›ã™ã‚‹ç‚ºã®ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã§ Go 1.21 ã§å°Žå…¥ã•ã‚Œã¾ã—ãŸã€‚ã“ã‚Œã¾ã§ã‚‚ zap ã‚„ zerolog ã¨ã„ã£ãŸã‚µãƒ¼ãƒ‰ãƒ‘ãƒ¼ãƒ†ã‚£è£½ã®ãƒã‚¬ãƒ¼ã‚’ä½¿ã†äº‹ã§æ§‹é€ åŒ–ãƒã‚°ã‚’å‡ºåŠ›ã™ã‚‹äº‹ãŒã§ãã¾ã—ãŸãŒã€æ§‹é€ åŒ–ãƒã‚°ã‚’å‡ºåŠ›ã™ã‚‹æ©Ÿèƒ½ãŒ Go ã®æ¨™æº–ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã«ãªã‚Šã¾ã—ãŸã€‚ slog ã¨ã¯ é€šå¸¸ã® log ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã¯ã€æ™‚åˆ»ã¨ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã®å˜ç´”ãªå‡ºåŠ›ã«ãªã‚Šã¾ã™ã€‚
wkubota 2023/12/25
*ã‚ã¨ã§èªã‚€

log

golang
ãƒªãƒ³ã‚¯
Fluentd å…¥é–€ ã€œé‹ç”¨ã«å¿…è¦ãªåŸºç¤ŽçŸ¥è˜ã€œ
æœ€è¿‘æ¥å‹™ã§ Fluentd ã‚’è§¦ã‚‹ã“ã¨ãŒå‡ºã¦ãã¦å…¥é–€ã—ãŸã‚“ã§ã™ãŒã€æœ€åˆã®ã†ã¡ã¯ãƒˆãƒ©ãƒ–ãƒ«ãŒèµ·ããŸæ™‚ã«ä½•ãŒèµ·ãã¦ã„ã‚‹ã®ã‹ã€ã©ã†å¯¾å‡¦ã—ãŸã‚‰è‰¯ã„ã®ã‹ãŒã•ã£ã±ã‚Šã‚ã‹ã‚‰ãªã‹ã£ãŸã®ã§ã€ã€ŒFluentd ã£ã¦ãƒã‚°ã®åŽé›†ã¨ã‹ã«ä½¿ã‚ã‚Œã‚‹ã‚„ã¤ã§ã—ã‚‡ï¼Ÿã€ç¨‹åº¦ã®çŸ¥è˜ã—ã‹ãªã‹ã£ãŸéŽåŽ»ã®è‡ªåˆ†ã«å‘ã‘ã¦ã€Œã¨ã‚Šã‚ãˆãšã“ã‚Œãã‚‰ã„ã¯çŸ¥ã£ã¦ãŠã‘ï¼ã€ã¨è¨€ã„ãŸã„å†…å®¹ã‚’ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ ãƒˆãƒ©ãƒ–ãƒ«ãŒèµ·ããŸæ™‚ã«ã©ã®å‡¦ç†ã§å•é¡ŒãŒèµ·ãã¦ã„ã‚‹ã®ã‹ç´ æ—©ãã‚³ãƒ¼ãƒ‰ã‚’è¿½ã†ã“ã¨ãŒã§ãã¦ã€ãƒ‡ãƒ¼ã‚¿ã®æ¶ˆå¤±ã‚’æœ€å°é™ã«æŠ‘ãˆã¤ã¤é©åˆ‡ã«å¯¾å‡¦ã§ãã‚‹ã‚ˆã†ã«ãªã‚‹ã“ã¨ã‚’ç›®çš„ã¨ã—ã¦ã„ã¾ã™ã€‚ ãªãŠã€ç¾æ™‚ç‚¹ã§æœ€æ–°ç‰ˆã® Fluentd v0.14.21 ã‚’å¯¾è±¡ã«ã—ã¦ã„ã¾ã™ã€‚ ã‚¢ã‚¸ã‚§ãƒ³ãƒ€ Getting Started Fluentd ã®ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ Processes Supervisor process Worker process Threads Input thread En
wkubota 2017/11/27
*ã‚ã¨ã§èªã‚€

log

ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°
ãƒªãƒ³ã‚¯
1