We had a chat with the team about this:

We are reviewing our tasks and this seems to be not relevant anymore in the context of MW on K8s, so we decided to close, but please re-open if you feel something is needed!

I updated the bullseye and bookworm netist images this morning but didn't know about upgrading the packages for the point release :(

I think I know why this is happening, didn't realize till now. Docker Reporter, the tool that scans the registry for images and calls debmonitor for them, has the following configs:

Follow up: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1070028 - Spicerack's setup.py needed to be updated to avoid build failures.

I tried to build 8.10.0 to make sure that it wasn't related to the 8.11.0 changes, and I get the same result.

In T370203#10103127, @bking wrote:

I appreciate everyone's feedback on the subject, and I want make clear that my team (Data Platform SRE) is firmly committed to open source principles, and that my colleagues spent a considerable amount of effort attempting to find an open-source alternative in accordance with the guiding principles.

Next steps:

Next steps:

A lot of cleanup has been done, so far it seems that the task can be closed. We have a better way to figure out what images are not supported by the docker reporter now, so it is also good.

After checking the JVM's committed vs used memory it seems to me that we allocate more than what we need on average, for a grand total of ~52GB (I think 48Gb of pre-allocated heap plus some extra GBs for the Metaspace).

Created https://grafana-rw.wikimedia.org/d/e0f6afe3-2aea-483d-9f5e-55f0cba9207f/puppetserver, didn't add all the metrics but it should be a good start to figure out what's happening.

Something very strange:

Thanks a lot for the task!

In T373526#10100630, @akosiaris wrote:

I see one problem with this approach. Teams change. Their names, their compositions, their email addresses and so on, especially during re-orgs and in the past we 've done a lot of those in the WMF. If this is to happen, really stable team names need to be chosen.

Created T371549 to move the Maintainer field of the production-images repo's control files to team-specific (where possible). Ideally this should be the info that docker-pkg will use to publish the label about who to contact.

Hi @MSantos! Ack thanks for the info, will keep it in mind when hopefully upgrading the maps nodes :) Any news about the Kartotherian depencency upgrade?

@hashar thanks a lot for the investigation! Would it be possible to have an option in the jjb config to allow Python jobs to specify the list of tox environments to execute? If so we could set the py39 ones for spicerack as special use case, and update them when the cumin nodes are upgraded.

In T373417#10096155, @Jhancock.wm wrote:

@Marostegui hey Papaul's on vacation this week. From what I remember that is a 10G issue. We started using this tag in the reimage script to keep this one from coming up.
--force-dhcp-tftp

Today I cleaned up some db nodes reported as debmonitor client failures while I was on holiday:

@mark @odimitrijevic thanks a lot for the explanations and the rationale, I am not 100% happy with the outcome but I will not oppose anymore :) In the future we'd love to get involved sooner (we == SRE team) rather than later in these kind of discussions, to help as much as possible and provide more options (if available).

@sbassett Hi! Please subscribe to the ops mailing list so you can get notified by these changes, usually we post a message there to warn users. In this case it is safe to deploy since it is just a rebuild to use a new OS (Bookworm) and skip Buster.

In T372472#10064694, @jhathaway wrote:

I'm a little surprised we haven't seen this on any bullseye hosts in production, but perhaps we haven't provisioned a new one host or added a user since we exceeded the length limit?

Left to do:

Currently blocked by T372485

Hi! I am Luca from the SRE Infrastructure Foundations team, I had a chat with Ben today about https://gerrit.wikimedia.org/r/c/operations/puppet/+/1062401 and the need to deploy this extra plugin.

Found another issue:

Filed https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/24 to fix a conftool issue:

@brouberol after T355550 do we have any plans to start testing the upgrade on kafka-test or similar? I can help if needed :)

Encountered an issue with the BMC's network config:

@Jhancock.wm Hi! Feel free to leave this host to me, still trying to figure out some stuff :(

Move done, tested the new "disarmed" pre-commit hook on puppetmaster1001 and a commit on puppetserver1001.

Sent an email to all SREs, the move will happen on Aug 12th 13:00 UTC.

Adding some ideas: https://docs.djangoproject.com/en/5.0/ref/models/querysets/#bulk-update

Buster and Bookworm rollouts done, no big issues registered. The only drawback is that due to the high volume of writes to the db (since we are changing the Debian version etc..) the UI gets not responsive for a bit, and we get some alarms. This is due to T371899.

Rolled out the change to the hadoop cluster, this is the only error that I got:

The issue is described in T371899. I proceeded anyway to upgrade both debmonitor server hosts, all good so far.

Fix for the dump-cloud-ip-ranges timer/unit rolled out, I also tried to do a manual puppet private commit and it worked. Let's wait for the next run of the timer to confirm that everything is working.

• The Netbox custom script for network provisioning is now asking for a mac address (for the mgmt interface), mandatory for each supermicro.
• spicerack's redfish module is now able to create admin users in the BMC (only for supermicro).

After a brainbounce with Joe on the SRE IRC channel, we noticed that the environment variables when running the post commit hook (in my local repro) contained GIT_INDEX_FILE pointing to the repo representing /srv/private, instead of /var/lib/git/etc... Simply unsetting the variable in the post-commit hook make the issue disappear.

I managed to repro the issue on a local docker container, and I can say that it is definitely the code of external_clouds_vendors that causes this. The repro is using only the post-commit hook to propagate the change between the two fake private repos.

It re-appened again, again only requestctl/ipblocks related staged content. Being staged meant that git pull worked, so it wouldn't have broken any regular commit happening under /srv/private (at least IIUC).

Tried to test the new debmonitor-server on debmonitor2003:

• changed sretest1001 /etc/hosts to point debmonitor.discovery.wmnet to debmonitor2003's ip
• dropped via spicerack the data on debmonitor for the host
• ran the debmonitor client on the host

From docker report (k8s images) set to work only with Bullseye+ images: