Change #1152639 merged by jenkins-bot:

[mediawiki/extensions/Cite@master] Delete original selection when adding a subref

https://gerrit.wikimedia.org/r/1152639

https://zarcillo.wikimedia.org/login redirected me to IDS and logged in successfully. Fantastic!
Closing task, thank you!

Mentioned in SAL (#wikimedia-operations) [2025-06-03T07:25:24Z] <tchanders@deploy1003> Finished scap sync-world: Backport for [[gerrit:1142649|Assign IP auto-reveal rights to certain groups (T386492)]] (duration: 10m 39s)

Thanks @kevinbazira and @isarantopoulos for these details. Very useful information.

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1002 for host thanos-be2009.codfw.wmnet with OS bullseye

• Test suite will be added in T395795
• no-JS version is under development in T394885
• Passing in real data will be addressed in T395791

@Jhancock.wm thanks for this; I tried running puppet on thanos-be2009 and the problem was that /dev/sdi1 had an EFI partition on it (presumably either because it was shipped like that, or maybe the result of a previous failed isntall); I wiped that and puppet now runs to completion, so I'll try another reimage.

Mentioned in SAL (#wikimedia-operations) [2025-06-03T07:16:49Z] <tchanders@deploy1003> tchanders: Backport for [[gerrit:1142649|Assign IP auto-reveal rights to certain groups (T386492)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-06-03T07:14:45Z] <tchanders@deploy1003> Started scap sync-world: Backport for [[gerrit:1142649|Assign IP auto-reveal rights to certain groups (T386492)]]

Change #1142649 merged by jenkins-bot:

[operations/mediawiki-config@master] Assign IP auto-reveal rights to certain groups

https://gerrit.wikimedia.org/r/1142649

Change #1153030 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/alerts@master] gNMI: add target down alerting

https://gerrit.wikimedia.org/r/1153030

In T395304#10859676, @FCeratto-WMF wrote:

The application at prototype stage but it supports CAS natively, see https://zarcillo.wikimedia.org/docs#/default/cas_callback_cas_callback_get and https://gitlab.wikimedia.org/repos/data_persistence/zarcillo/-/blob/devel/zarcillo/webapp/routers/auth.py?ref_type=heads#L60

Change #1152680 merged by Muehlenhoff:

[operations/puppet@production] CAS: Add service definition for Zarcillo

https://gerrit.wikimedia.org/r/1152680

Change #1153026 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Prometheus: gnmic_target_up rewrite name to instance

https://gerrit.wikimedia.org/r/1153026

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host thanos-be2009.codfw.wmnet with OS bullseye executed with errors:

• thanos-be2009 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506021546_jhancock_3486738_thanos-be2009.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console thanos-be2009.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Mentioned in SAL (#wikimedia-operations) [2025-06-03T06:37:18Z] <marostegui> Decrease buffer size on clouddb1016:s8 T390954

Mentioned in SAL (#wikimedia-operations) [2025-06-03T06:30:27Z] <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1154.eqiad.wmnet with reason: Setting up x3 T390954

Mentioned in SAL (#wikimedia-operations) [2025-06-03T06:29:35Z] <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1016.eqiad.wmnet with reason: Setting up x3 T390954

Change #1153018 merged by Marostegui:

[operations/puppet@production] clouddb1016: Add x3

https://gerrit.wikimedia.org/r/1153018

Change #1153018 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1016: Add x3

https://gerrit.wikimedia.org/r/1153018

Migrated

This is what it'll look like:

Change #1152897 merged by Marostegui:

[operations/puppet@production] es1037: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1152897

This is done

Change #1152897 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1037: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1152897

Mentioned in SAL (#wikimedia-operations) [2025-06-03T05:27:19Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1037 T395867', diff saved to https://phabricator.wikimedia.org/P76876 and previous config saved to /var/cache/conftool/dbconfig/20250603-052719-marostegui.json

Change #1152891 merged by Marostegui:

[operations/dns@master] wmnet: Update es6-master alias

https://gerrit.wikimedia.org/r/1152891

Mentioned in SAL (#wikimedia-operations) [2025-06-03T05:26:16Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es1038 to es6 primary T395867', diff saved to https://phabricator.wikimedia.org/P76875 and previous config saved to /var/cache/conftool/dbconfig/20250603-052614-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-06-03T05:25:53Z] <marostegui> Starting es6 eqiad failover from es1037 to es1038 - T395867

Change #1152890 merged by Marostegui:

[operations/puppet@production] mariadb: Promote es1038 to es6 master

https://gerrit.wikimedia.org/r/1152890

Mentioned in SAL (#wikimedia-operations) [2025-06-03T05:23:54Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set es1038 with weight 0 T395867', diff saved to https://phabricator.wikimedia.org/P76874 and previous config saved to /var/cache/conftool/dbconfig/20250603-052353-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-06-03T05:22:51Z] <marostegui@deploy1003> Finished scap sync-world: Backport for [[gerrit:1152893|db-production.php: Disable writes on es6 (T395867)]] (duration: 13m 39s)