Caused by T350443? Database seems to have high lags

In T350445#9303791, @hgzh wrote:

Caused by T350443? Database seems to have high lags

Yes. It should recover any second now.

Noting that the edits are not lost, just due to replication lag, they can't be shown right now.

That’s a relieve.

In codfw it has already mostly recovered but it's not visible to Europe users as eqiad is still ongoing (and will be like this for another hour likely), you could potentially try to set the cookie to see codfw for now.

Maybe it is a related issue that the "Beantworten" tool on talk pages does not work when reacting to newer contributions. Contributions older than, say, three hours or so, you can answer normally with the "Beantworten" tool.

It's very very likely related, the eqiad replicas have been lagged for 2:40 minutes now.

They're increasing for now: https://replag.toolforge.org/

In T350445#9303916, @doctaxon wrote:

They're increasing for now: https://replag.toolforge.org/

Cloud replica replication will continue to grow to around ten hours (as they have more indirect replication than production so don't expect any new updates on cloud replicas until ~6:00 AM UTC), production has a different replication lag as it has less indirection to the primary.

The datacenter serving Europe will recover in roughly half an hour from now (up to an hour or so).

I would be glad about an outage report in next TechNews, thank you.