Works as expected. This is the effect on authevents logs:

Mentioned in SAL (#wikimedia-operations) [2025-06-09T13:31:29Z] <taavi@deploy1003> Finished scap sync-world: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]] (duration: 24m 30s)

Mentioned in SAL (#wikimedia-operations) [2025-06-09T13:21:13Z] <taavi@deploy1003> taavi, tgr: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-06-09T13:06:59Z] <taavi@deploy1003> Started scap sync-world: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]]

Change #1153363 merged by jenkins-bot:

[operations/mediawiki-config@master] logging: Allow sampling of Logstash logs

https://gerrit.wikimedia.org/r/1153363

Also, with sampling you can see changes in total volume. Without it, it just maxes out at 100/sec and to see if a code change affected the log volume you need filter on pseudo-random subsets (like an arbitrary wiki or user agent) which is a hassle and somewhat unreliable.

Thanks, that makes sense!

In T395967#10885676, @Krinkle wrote:

Can you say more explicitly what the intended gain is?

Sampling would be a step up from the much less controllable throttling mechanism of T395899, so we should fix that.

Change #1153626 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Use GetSecurityLogContext hook for goodpass/badpass logging

https://gerrit.wikimedia.org/r/1153626

Change #1153618 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Add global existence info on GetSecurityLogContext hook

https://gerrit.wikimedia.org/r/1153618

Change #1153363 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] logging: Allow sampling of Logstash logs

https://gerrit.wikimedia.org/r/1153363

@Krinkle found the config file where this is set up. I added some docs and filed T395967: Allow sampling of Logstash events about handling high-volume event streams in a better way; I think this can be closed.

@Tgr wrote in the **task description*:

seems to be based on the normalized_message field?

@Tgr wrote in the task description:

[…] exactly 90K events in 15 minutes (ie. 100/sec) but distributed in an uneven, sawtooth pattern of exactly 15K events in a roughly 1-minute window and then zero events for a slightly less than 1-minute or 2-minute interval (those two lengths seem to be alternating).
So maybe a combination of a 15K/60s cap and a 30K/300s cap? […]

A negative side effect of this is that the mwdebug Logstash dashboard becomes unreliable. It would be nice to exempt debug hosts from the cap, they don't have much log volume.

A check for normalized_message:"Central autologin attempt" in udp2log files:

tgr@mwlog1002:/srv/mw-log$ ack '^2025-06-03 00:20:' authevents.log > ~/authevents-2025-06-03-00-20.txt
tgr@mwlog1002:/srv/mw-log$ ack 'Central autologin attempt' ~/authevents-2025-06-03-00-20.txt | wc -l
44659

so that's about 800/sec. (And a massive waste of disk space - authevents really shouldn't be recorded at all. True for some other log channels as well, like poolcounter or xff.)

I skimmed codesearch results for 100/6000/30000 but did not see anything relevant. The rsyslog config has RateLimit.Burst=100 for MediaWiki but that doesn't really match the behavior.

This is intentional. The way it is meant to work is that uncaught errors or errors that otherwise means the request/process did not succeed in what it wanted to do (i.e. relevant operationally in terms of service health), that those go to the error channel.

We should also log whether the central and local account exists.

Change #1150677 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] [WIP] Unify logging of IP / user agent / etc on sensitive operations

https://gerrit.wikimedia.org/r/1150677

Change #1130543 had a related patch set uploaded (by Kosta Harlan; author: Máté Szabó):

[mediawiki/core@master] logger: Add client IP to logger context

https://gerrit.wikimedia.org/r/1130543

https://doc.wikimedia.org/ecs/ contains a spec for most of these fields (xff and geocookie are not in ECS), so I think we could follow the ECS guidance for these.

See also this patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1130543

Some examples of what's logged currently (per this search:

• WebProcessor (added to all logs): url, http_method, referrer, reqId, ip (not the real IP though)
• RateLimiter: name, ip
• action API feature usage: username, clientip, agent, referer
• AuthManager: user, clientip
• SpamRegexConstraint in EditPage: ip
• ChronologyProtector: clientIP, clientAgent
• SessionManager / SessionBackend: clientip, user, userAgent
• OATH: user / clientip
• goodpass/badpass in Wikimedia config: name, user group / privilege information, clientip, xff, ua, geocookie
• CentralAuth login: user, clientip, ua
• IPReputation: ip
• EmailAuth: user, ua, ip
• EmailAuth (in WikimediaEvents): user, ua, ip, group / LoginNotify / IPoid / geocookie / 2FA / activity info
• LoginNotify: name, ip / clientip, ua, xff, geocookie
• OAuth: user, clientip
• ConfirmEdit: user, clientip, ua
• StopForumSpam: user, clientip

I feel like this idea is mostly covered by the newly added "global log context": T142313: Add global information to debug logger context

Here's a very rough first approximation: https://logstash.wikimedia.org/goto/7c72bf2d02c84f4a56803129a1b16042

(wikitech-l announcement)

With @Tgr's patch, you can now add global information to debug logger context using:

LoggerFactory::getContext()->add(<array>)

or (within the lifetime of a scope variable):

$scope = LoggerFactory::getContext()->addScoped(<array>)