Out of 4637 test emails sent in December (from 18th to 31st), 100% of them were held correctly. The "Message content lost" did not appear.
So indeed, considering fixed.

This can probably be obtained from:
https://analytics.wikimedia.org/published/datasets/country_project_page/
https://analytics.wikimedia.org/published/datasets/country_project_page_historical/

It does look good so far.

I just found a weird behavior.
Just by reloading the page, I get different sets of held messages. It mostly shows messages hold on Nov 17, but sometimes there other, different ones, hold on Nov 18.

Yep, those file pages do not exist (as such, it got no license, etc.) Happen from time to time, no idea why. There were some bugs in the past.
However, these files were uploaded on 5 September 2024, so whatever bug is probably still there.

I am old enough to remember the days when only Special:Upload existed. The only way to upload any relatively large file was a server-side import.

I have processed those editrequests, so at least when a page contains multiple license templates, it will only need to be reparsed once.

In T380009#10329055, @revi wrote:

In T380009#10328760, @Platonides wrote:

In T380009#10326074, @eoghan wrote:

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

OIT?

Office IT, which is renamed to IT Services, is WMF dept handling employee account and their inboxes (which are handled by Google mailbox). (EDIT: I know this from my past employment at WMF, and it is probably usually-unseen group for almost all volunteers.)

(I've created meta redirect for Office IT and OIT.)

In T380009#10326074, @eoghan wrote:

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

@eoghan these messageids are of the form %Y%m%d%H%M%[email protected].T377045.wikimedia.es

I have been testing this by sending emails to be held. It is very clearly losing emails. There is a portion in the hour in which it loses held emails, with a slow drift:

In T117618#10081491, @ssingh wrote:

Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059423/comment/61d3b1a3_eebb1d4b/, @Vgutierrez's comment:

so we would be sending the header content twice, on both -Report-Only and enforcing flavors, according to turnilo, upload.wm.o gets 47M hits per day for URLs matching ^/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test). The content of the non-PDF CSP header is 328 bytes long so we would be increasing our outbound traffic in ~ 14GB per day.

We are still discussing this in Traffic but sharing here for posterity and easier discussion.

The bug for multiple mailing lists was fixed several years ago: https://gitlab.com/mailman/mailman/-/issues/955 (so, hopefully, the fix is included on our mailman version)

In T376267#10270792, @bd808 wrote:

Wikitech no longer knows a password for anyone until they use https://wikitech.wikimedia.org/wiki/Special:PasswordReset to establish one. Even if you have done this once since October 1st you can do it again.

In T240182#5825579, @Masumrezarock100 wrote:

No idea who is the operator of BanBot but it apparently posts (non)abusive and (non)trollish usernames to #wikimedia-stewards channel.

uploadwizard itself is not doing the stores, that goes through the intermediate filerepo layer. But, if MW code was consistently uploading only one copy, we would be seeing loads of this. Maybe a race condition, or a problem specific of this server.

@LSobanski: see the test ticket https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13245774

This message may not always be caused by what it says, since Gmail often also takes some other factors into account.

The description describes CP3 as used to «automatically prefetch top-ranked search results when the user views a Google search result page»

I don't think passing the generation to a background process and check within a loop every second if it was generated would be the proper way.

@grin my point is to configure the antispam (spamassassin) run by ticket.wikimedia.org to trust the single ip that chapter uses to forward mail to VRTS (well, assuming it can be trusted)

What Referer would be provided by such app? Would the requests from the app have a User-Agent identifying it? Which one?

If [email protected] forwards to [email protected], then example.net should be aware of that, so that it considers example.com as a valid relay. Otherwise, this happens, and is expected.
The outgoing ip of that chapter (or any such entity forwarding to vrts) should be added to trusted_networks, so that SpamAssassin processes it correctly, see https://cwiki.apache.org/confluence/display/SPAMASSASSIN/TrustedRelays

@Tommy_Kronkvist you may also PGP-sign a statement that you are indeed requesting this in T346513, signed by the PGP key you listed here: https://species.wikimedia.org/wiki/User:Tommy_Kronkvist/PGP so there's no need to share secrets in private pastes.

But it does fail if we also add --limit-rate 1M

The file can be curled down from all dcs:

for site in ulsfo eqiad codfw esams eqsin drmrs; do curl -o $site.pdf https://upload.wikimedia.org/wikipedia/commons/9/9f/ZHSY000097_%E5%AE%8B%E6%9B%B8%E4%B8%80%E7%99%BE%E5%8D%B7_%28%E6%A2%81%29%E6%B2%88%E7%B4%84_%E6%92%B0_%E5%AE%8B%E5%88%BB%E5%AE%8B%E5%85%83%E6%98%8E%E9%81%9E%E4%BF%AE%E6%9C%AC.pdf --connect-to ::upload-lb.$site.wikimedia.org ; done

Sorry @JTannerWMF, I had missed your comment. The user has replied informing that the image is now gone in his iOS app.

I can reproduce (using the snippet from T343795#9083114) the slowness (100% CPU) on chromium 90.0.4430.212-1~deb10u1 from Debian buster (15 May 2021), so it doesn't seem to be a bug recently introduced in Chrome.

You should probably file it at https://github.com/znuny/znuny/issues
I see no mention of a problem of znuny (or otrs) with iso-8859-8 / iso-8859-8-i mails.

What do you mean?

No problem connecting to commons.wikimedia.org from Germany.

¿Conseguisteis encontrar/arreglar el problema?

In T332815#8863252, @ashley wrote:

This, of course, has the rather obvious performance impact since now we're purging caches for the page on every vote...

This variant allows choosing the target url:

Ran

$ toolforge-jobs run ltsa-status --command /data/project/status/src/run.sh --image bullseye --schedule "1,16,31,46 * * * *"
$ toolforge-jobs run ltsa-pull --command /data/project/status/src/pull.sh --image bullseye --schedule "6 2,14 * * *"

Direct locations, both "File not found":

• https://upload.wikimedia.org/wikipedia/commons/c/cd/Living_on_the_Edge_of_the_World.jpg (Old)
• https://upload.wikimedia.org/wikipedia/commons/3/31/A_residential_cliff_area_in_the_Murree_Hills.jpg (New)

I also hope it's not something doing MITM into the https connection without you being aware. It seems unlikely, though, and listing the https certificate you see will let us discard that idea.

After some scanning, out of 270975 articles using ficha de persona, only 264 have that bad image (< 0.1%).

A null edit does fix it.

Aha,! They went through acl*sre-team I had been looking in the acl*security ones. Missing puzzle pieces found.
Thanks, @Dylsss

In T326752#8583804, @Zabe wrote:

Thats just a bit weird because then they would have been able to view protected tasks.

Error message received by a test account with no privileges

Not really. I was just testing it, and it is not. acl*security_sre is joinable by people in acl*security, so he must have already been in acl*security to be able to join there.

@Stevemunene is a member of acl*security_sre (he added himself?!)

I think there weren't so many people with CheckUser powers in the first year and a half. With that list, you would have:

Accounts with just 1 edit will probably be just . Even those that are experienced wikipedians (e.g. a cross-wiki user from another language) and would thus perfectly know how to edit their preferences and change the skin, would probably not bother changing the skin if they are just passing by for doing a single edit, even if they didn't like the skin. The cost of editing the preferences is actually larger than standing with it for their brief interaction. It's after some editing that it would pay back. Maybe worth checking for people that edited for at least N days.

Note that the ssh key to access wmcloud is a different one than the one for prod (ending in OLLJY vs YvU/Z), so you might be trying to log in there with the wrong key.

From the small portion, it would seem that the files were uploaded and when they were later deleted at MediaWiki, the files would be copied into the archive, and rmed from swift storage but in some cases the contents disappeared from swift but not the "filename".

I agree 20k is not money for this.

I can confirm the behavior.

It's funny, I would have probably fallocate --dig-holes <database> to reclaim the actual space from h2 behind their back.

Regarding the naming terminology, Special:CentralAuth uses "Registered:". e.g. https://en.wikipedia.org/wiki/Special:CentralAuth/Jimbo_Wales
(that page can actually be used to find out the edit count of an user in the current wiki, albeit it is an odd path)

(The Tangled Web is a 2012 by Michal Zalewski)

This seems a complaint that the diff of wiki pages misinterprets some cases as delete and readd instead of noting a new line was added in the middle. _In some cases_, since I think it does properly identifies them sometimes. But with no actual example, it's difficult to determine what's happening. Not that it will necessarily be fixable for all instances, diffing is not a simple task.

Both https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Thanks/+/593204 and https://gerrit.wikimedia.org/r/825339 look good and pretty straightforward. The default value of ThanksAllowedLogTypes isn't overriden in InitialiseSettings/CommonSettings. Yet, it is failing for a log entry which does have type rights/rights.

A bug in https://gerrit.wikimedia.org/r/825339 when implementing T191599 ?

The underlying API call returns:

I suspect it's a timeout at Varnish level, and it then got cached somewhere. I was always getting 200, even when asking every dc:

@Mentxuwiki: si reduzco el tamaño del viewport (ventana), la foto no se reduce (hace scrolling) pero el resto de componentes sí. Parece correcto. No veo por qué te desaprovecha la mitad del espacio disponible (entiendo que es ese el problema). ¿Qué estás haciendo?

In T316468#8192966, @Olea wrote:

Apuntar que también aparece un sombreado permanente en la primera solapa que no entiendo siquiera por qué ocurre.

What if someone wanted Taumatawhakatangi­hangakoauauotamatea­turipukakapikimaunga­horonukupokaiwhen­uakitanatahu or Taumatawhakatangihangakoauauotamateaturipukakapikimaungahoronukupokaiwhenuakitanatahu ? :-)

Cairo/Downtown showed: "Africa > North Africa > Egypt > <span class="mw-page-title-main">Lower Egypt</span> > Cairo > Cairo/Downtown"

You are not the first one to notice it. This is basically a duplicate of T122097

I think we should push for replacing http:// links with https:// if the external site is serving https. It removes a redirect from every reader (it adds up really easily, see T120085), and it's much more secure for the readers as well.

SQLite? I'm surprised it uses that, it may not be the best performant option for such a queue...

Changing the ordering (perhaps coupled with varnish redirecting all '?title=X&action=history' to the new '?action=history&title=X' to avoid old forms) would cause an stampede in that it's like no history page is cached. I would need planification, but (partially rolled-out per wikis) I think that could be acceptable. history page requests are much lower than to articles, and they are cheap to produce at MediaWiki layer

The SQL query uses querycachetwo (to find which users are 'active users'), which is not exposed in toolserver replicas (should it?), nor are 'gadget-%' up_property values (which would reveal the gadgets each one has enabled).
So this would have to run in the cluster.