PagerDuty Incident Response Documentation
Home Being On-Call Before an Incident During an Incident After an Incident Crisis Response Training Additional Resources Getting Started On-Call Being On-Call Who's On-Call? Alerting Principles Before an Incident What is an Incident? Severity Levels Different Roles Call Etiquette Complex Incidents During an Incident During an Incident External Communication Guidelines Security Incident After an In
Update on software issue impacting certain customers
Following network disturbances in a number of Ericsson’s (NASDAQ:ERIC) customer networks, Ericsson has taken immediate action to minimize impact and support the restoration of services. During December 6, 2018, Ericsson has identified an issue in certain nodes in the core network resulting in network disturbances for a limited number of customers in multiple countries using two specific software v
[PDF] 08/25ã®é€šä¿¡éšœå®³æ¦‚説
08/25ã®é€šä¿¡éšœå®³æ¦‚説 Matsuzaki ‘maz’ Yoshinobu <maz@iij.ad.jp> maz@iij.ad.jp 1 観測ã•ã‚Œã¦ã„ã‚‹æ¦‚è¦ â€¢ 2017/08/25 12:22JSTé ƒ • AS15169ãŒä»–ASã®IPv4経路をトランジット開始 • â½‡é ƒæµé€šã—ãªã„ç´°ã‹ã„経路ãŒâ¼¤é‡ã«åºƒå ± • ã“ã‚Œã«ã‚ˆã‚Šãƒˆãƒ©ãƒ’ックã®å¸ã„è¾¼ã¿ãŒç™ºâ½£ • 国内ã®å„ASã§é€šä¿¡éšœå®³ã‚’検知 • 2017/08/25 12:33JSTé ƒ • AS15169ãŒãƒˆãƒ©ãƒ³ã‚¸ãƒƒãƒˆã—ã¦ã„ãŸçµŒè·¯ã‚’削除 maz@iij.ad.jp 2 観測ã•ã‚ŒãŸå•é¡Œã®BGPçµŒè·¯æ¦‚è¦ â€¢ 経路数 • 全体ã§ç´„11万経路 (⽇本分ãŒç´„25000経路) • /10ã‹ã‚‰/24ã¾ã§å¹…広ã„経路(åŠæ•°ç¨‹åº¦ãŒ/24) • 通常æµã‚Œã¦ã„ãªã„ç´°ã‹ã„経路ãŒå¤šã‹ã£ãŸ • AS PATHã¯æ¦‚ã “701 15169 <本æ¥ã®AS PATH>†• åºƒå ±å…ƒA
8月25æ—¥ã«ç™ºç”Ÿã—ãŸå¤§è¦æ¨¡é€šä¿¡éšœå®³ã‚’ã¾ã¨ã‚ã¦ã¿ãŸ - piyolog
2017å¹´8月25æ—¥12時éŽãŽã‚ˆã‚Šã€Webサイトã«ã¤ãªãŒã‚‰ãªã„ç‰ã®æŽ¥ç¶šéšœå®³ã¨ã¿ã‚‰ã‚Œã‚‹äº‹è±¡ãŒè¤‡æ•°ç™ºç”Ÿã—ã¾ã—ãŸã€‚ã¾ãŸåŒæ™‚間帯ã«NTTコミュニケーションズã€KDDIã‹ã‚‰éšœå®³å ±å‘ŠãŒç™ºè¡¨ã•ã‚Œã¦ã„ã¾ã™ã€‚ã“ã“ã§ã¯é–¢é€£æƒ…å ±ã‚’ã¾ã¨ã‚ã¾ã™ã€‚ インシデントタイムライン 日時 出æ¥äº‹ 8月25æ—¥ æ£åˆéŽãŽ GoogleãŒèª¤ã£ãŸçµŒè·¯æƒ…å ±ã‚’é€ä¿¡ã€‚ 〃 12時22分 NTTコミュニケーションズã®OCNãƒãƒƒã‚¯ãƒœãƒ¼ãƒ³ã§é€šä¿¡éšœå®³ç™ºç”Ÿã€‚ 〃 12時24分 KDDIã®ã‚µãƒ¼ãƒ“スã§é€šä¿¡éšœå®³ç™ºç”Ÿã€‚ 〃 ä»¥é™ æ—¥æœ¬å›½å†…ã®è¤‡æ•°ã®ã‚µãƒ¼ãƒ“スç‰ã§æŽ¥ç¶šéšœå®³ãŒç™ºç”Ÿã€‚ 〃 é€ä¿¡å¾Œ8分以内 GoogleãŒèª¤è¨å®šæƒ…å ±ã‚’ä¿®æ£ã€‚ 〃 12時39分 KDDIã®é€šä¿¡éšœå®³ãŒå¾©æ—§ã€‚ 〃 12時45分 NTTコミュニケーションズã®é€šä¿¡éšœå®³ãŒå¾©æ—§ã€‚ 〃 å¤•æ–¹é ƒ å„組織ã§ç™ºç”Ÿã—ãŸéšœå®³ãŒæ¦‚ã復旧。 8月26æ—¥ GoogleãŒèª¤è¨å®šã‚’èªã‚ã€è¬ç½ªã™ã‚‹å£°æ˜Žã‚’発表。 8月29æ—¥ ç·
ï¼»ç¶šå ±ï¼½OCNã®é€šä¿¡éšœå®³ã€ç±³ã‚°ãƒ¼ã‚°ãƒ«ã«ã‚ˆã‚‹èª¤ã£ãŸçµŒè·¯æƒ…å ±ã®å¤§é‡é€ä¿¡ãŒåŽŸå› ã‹
2017å¹´8月25æ—¥ã€NTTコミュニケーションズ（NTTコム）ã®ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆæŽ¥ç¶šã‚µãƒ¼ãƒ“ス「OCNã€ã§ç™ºç”Ÿã—ãŸé€šä¿¡éšœå®³ã«é–¢ã—ã¦ã€ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆé€šä¿¡é–¢é€£ã®è˜è€…ã¯èª¤ã£ãŸçµŒè·¯æƒ…å ±ãŒå¤§é‡ã«æµã‚ŒãŸã“ã¨ãŒåŽŸå› ã§ã¯ãªã„ã‹ã¨ã®è¦‹æ–¹ã‚’示ã—ãŸã€‚ã“ã“ã§ã„ã†çµŒè·¯æƒ…å ±ã¯ãƒ«ãƒ¼ã‚¿ãƒ¼ãŒBGP（Border Gateway Protocol）ã¨ã„ã†ãƒ—ãƒãƒˆã‚³ãƒ«ã‚’使ã£ã¦äº¤æ›ã™ã‚‹ã‚‚ã®ã 。 日本ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚¤ãƒ³ãƒ•ã‚©ãƒ¡ãƒ¼ã‚·ãƒ§ãƒ³ã‚»ãƒ³ã‚¿ãƒ¼ï¼ˆJPNIC）ã®å²¡ç”°é›…之æ°ã¯ã€NTTコムã¯è¤‡æ•°ã®çµ„ç¹”ã¨å¯¾ç‰ãªé–¢ä¿‚ã§ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã®çµŒè·¯æƒ…å ±ã‚’ã‚„ã‚Šå–ã‚Šã—ã¦ã„ã‚‹ãŒï¼ˆã“れを「ピアリングã€ã¨ã„ã†ï¼‰ã€ãã®ã†ã¡ã®ã‚る組織ãŒèª¤ã£ãŸçµŒè·¯æƒ…å ±ã‚’å¤§é‡ã«æµã—ãŸã®ã§ã¯ãªã„ã‹ã¨è©±ã™ã€‚ãã®çµæžœã€ã€ŒNTTコムを介ã—ã¦ã‚¤ãƒ³ã‚¿ãƒ¼ãƒãƒƒãƒˆã«æŽ¥ç¶šã—ã¦ã„ãŸä¼æ¥ã®ãƒ«ãƒ¼ã‚¿ãƒ¼ãŒã€å¤§é‡ã®çµŒè·¯æƒ…å ±ã‚’å—ã‘å–り高ã„è² è·ãŒã‹ã‹ã‚Šã€ä¸€éƒ¨ã¯ãƒ•ãƒªãƒ¼ã‚ºã—ãŸã‚ˆã†ãªçŠ¶æ…‹ã«é™¥ã‚‹ãªã©ã—ã¦é€šä¿¡éšœå®³ã«ã¤ãªãŒã£ãŸã®ã§ã¯ãªã„ã‹ã€ï¼ˆå²¡
GitLab.comãŒæ“作ミスã§æœ¬ç•ªãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹å–ªå¤±ã€‚5ã¤ã‚ã£ãŸã¯ãšã®ãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—手段ã¯å½¹ç«‹ãŸãšã€é ¼ã¿ã®ç¶±ã¯6時間å‰ã«å¶ç„¶å–ã£ãŸã‚¹ãƒŠãƒƒãƒ—ショット ï¼ Publickey
æžœãŸã—ã¦GitLab.comã§ä½•ãŒèµ·ããŸã®ã§ã—ょã†ã‹ï¼Ÿ ã“ã‚Œã¾ã§ã®çµŒç·¯ã‚’ã¾ã¨ã‚ã¾ã—ãŸã€‚ スパムã«ã‚ˆã‚‹ãƒˆãƒ©ãƒ•ã‚£ãƒƒã‚¯ã®ã‚¹ãƒ‘イクã‹ã‚‰ãƒ¬ãƒ—リケーションã®ä¸èª¿ã¸ GitLab.comã¯ä»Šå›žã®ã‚¤ãƒ³ã‚·ãƒ‡ãƒ³ãƒˆã«ã¤ã„ã¦ã®è©³ç´°ãªçµŒéŽã‚’「GitLab.com Database Incident - 2017/01/31ã€ã§å…¬é–‹ã—ã¦ã„ã¾ã™ã€‚ã¾ãŸã€ã‚‚ã†å°‘ã—æ•´ç†ã•ã‚ŒãŸæƒ…å ±ãŒãƒ–ãƒã‚°ã€ŒGitLab.com Database Incident | GitLabã€ã«ã‚‚掲載ã•ã‚Œã¦ã„ã¾ã™ã€‚ ã“れらã®ãƒ‰ã‚ュメントを軸ã«ã€ä¸»ãªã§ãã”ã¨ã‚’時系列ã«è¦‹ã¦ã„ãã¾ã—ょã†ã€‚ 1月31æ—¥16時（世界å”定時。日本時間2月1æ—¥åˆå‰8時）ã€YPæ°ï¼ˆYorick Peterseæ°ã¨æ€ã‚れる）ã¯PostgreSQLã®ãƒ¬ãƒ—リケーションをè¨å®šã™ã‚‹ãŸã‚ã«ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã®è«–ç†ã‚¹ãƒŠãƒƒãƒ—ショットを作æˆã€‚ã“ã‚ŒãŒã‚ã¨ã§å¤±ã‚ã‚ŒãŸãƒ‡ãƒ¼ã‚¿ã‚’æ•‘ã†å¹¸é‹ã«ã¤ãªãŒã‚Šã¾ã™ã€‚ 1月31æ—¥21時
GitLab.com database incident
GitLab.com database incident Yesterday we had a serious incident with one of our databases. We lost six hours of database data (issues, merge requests, users, comments, snippets, etc.) for GitLab.com. Update: please see our postmortem for this incident Yesterday we had a serious incident with one of our databases. We lost six hours of database data (issues, merge requests, users, comments, snippet
GitLab.com Database Incident - 2017/01/31
This incident affected the database (including issues and merge requests) but not the git repo's (repositories and wikis). Timeline (all times UTC): 2017/01/31 16:00/17:00 - 21:00 YP is working on setting up pgpool and replication in staging, creates an LVM snapshot to get up to date production data to staging, hoping he can re-use this for bootstrapping other replicas. This was done roughly 6 hou
GitHubãŒ1月28æ—¥ã®ã‚µãƒ¼ãƒ“ス障害ã®è©³ç´°ã‚’公開。åœé›»ã«ã‚ˆã‚Šå†…部ã®ChatOpsシステムもè½ã¡ã¦åˆæœŸå¯¾å¿œãŒå›°é›£ã«ã€‚Redisクラスタã®å¾©æ—§ã«æ™‚é–“
GitHubãŒ1月28æ—¥ã®ã‚µãƒ¼ãƒ“ス障害ã®è©³ç´°ã‚’公開。åœé›»ã«ã‚ˆã‚Šå†…部ã®ChatOpsシステムもè½ã¡ã¦åˆæœŸå¯¾å¿œãŒå›°é›£ã«ã€‚Redisクラスタã®å¾©æ—§ã«æ™‚é–“ å ±å‘Šã§ã¯ã€ã‚µãƒ¼ãƒ“ス障害ã¯GitHub社内ã®ChatOpsシステムも巻ã込んã§åˆæœŸå¯¾å¿œã«æ™‚é–“ãŒã‹ã‹ã£ã¦ã—ã¾ã£ãŸã“ã¨ã€ä¸€æ™‚çš„ãªåœé›»ãŒRedisクラスタã®éšœå®³ã‚’引ãèµ·ã“ã—ãŸãŸã‚ã€ãã®ç©¶æ˜Žã¨å¾©æ—§ãŒä½œæ¥ã®ä¸»ãªéƒ¨åˆ†ã ã£ãŸã“ã¨ãªã©ãŒèª¬æ˜Žã•ã‚Œã¦ã„ã¾ã™ã€‚ å ±å‘Šã®è¦ç‚¹ã‚’ã¾ã¨ã‚ã¾ã—ãŸã€‚ 内部ã®ChatOpsシステムも障害㫠GitHubã®ã‚µãƒ¼ãƒ“ス障害ã¯ã€ã™ã§ã«å ±å‘Šã•ã‚Œã¦ã„るよã†ã«ã€è‡ªç¤¾ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã«ãŠã‘る一時的ãªåœé›»ãŒæœ€åˆã®åŽŸå› ã§ã—ãŸã€‚ At 00:23am UTC on Thursday, January 28th, 2016 (4:23pm PST, Wednesday, January 27th) our primary data center experi
ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ï¼ˆJINSオンラインショップ）ã«é–¢ã™ã‚‹èª¿æŸ»çµæžœï¼ˆæœ€çµ‚å ±å‘Šï¼‰| JINS - 眼é¡ï¼ˆãƒ¡ã‚¬ãƒãƒ»ã‚ãŒã）
å¹³æˆ25年３月15日「ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ã«ã‚ˆã‚‹ãŠå®¢æ§˜æƒ…å ±æµå‡ºã®å¯èƒ½æ€§ã«é–¢ã™ã‚‹ãŠçŸ¥ã‚‰ã›ã¨ãŠè©«ã³ã€ã§ã”å ±å‘Šã„ãŸã—ã¾ã—ãŸå½“社オンラインショップã«å¯¾ã™ã‚‹ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ï¼ˆä»¥ä¸‹ã€ã€Œæœ¬ä»¶ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ã€ã¨ã„ã„ã¾ã™ã€‚）ã«ã‚ˆã‚Šã€ãŠå®¢æ§˜ã‚’ã¯ã˜ã‚ã¨ã™ã‚‹çš†æ§˜ã«å¤šå¤§ãªã‚‹ã”迷惑ãŠã‚ˆã³ã”心é…ã‚’ãŠã‹ã‘ã—ã¾ã—ãŸã“ã¨ã‚’æ·±ããŠè©«ã³ç”³ã—上ã’ã¾ã™ã€‚ 当社ã¯ã€äº‹æ¡ˆç™ºè¦šç›´å¾Œã‚ˆã‚Šã€é€æ¬¡æœ€æ–°æƒ…å ±ã‚’ã”å ±å‘Šã•ã›ã¦ã„ãŸã ã„ã¦ãŠã‚Šã¾ã—ãŸãŒã€ã“ã®åº¦ã€å½“社ã€å°‚門調査機関ãŠã‚ˆã³æƒ…å ±æ¼ãˆã„事故調査委員会ã«ã‚ˆã‚‹æœ¬ä»¶ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ã«é–¢ã™ã‚‹ä¸€é€£ã®èª¿æŸ»ãŒçµ‚了ã„ãŸã—ã¾ã—ãŸã®ã§ã€ä¸‹è¨˜ã®ã¨ãŠã‚Šã”å ±å‘Šã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ ãªãŠã€æœ¬ä»¶ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ã«é–¢ã™ã‚‹ãŠå®¢æ§˜ã¸ã®ã”対応ãŠã‚ˆã³å†ç™ºé˜²æ¢ã«å¯¾ã™ã‚‹å–り組ã¿ã«ã¤ã„ã¦ã¯å¼•ã続ãè¡Œã£ã¦ã¾ã„ã‚Šã¾ã™ãŒã€æœ¬ä»¶ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹ã«å¯¾ã™ã‚‹èª¿æŸ»çµæžœã«ã¤ãã¾ã—ã¦ã¯ã€ç‰¹æ®µã®å ±å‘Šäº‹é …ãŒç™ºç”Ÿã—ãªã„é™ã‚Šã€ä¸‹è¨˜ã‚’ã‚‚ã£ã¦æœ€çµ‚å ±å‘Šã¨ã•ã›ã¦ã„ãŸã ãã¾ã™ã€‚ 当社ã¯ã€è‡ªç¤¾ãŒé‹å–¶ã™ã‚‹ã€ŒJ
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
{{#tags}}- {{label}}
{{/tags}}