User Details
- User Since
- Jan 5 2016, 9:54 PM (451 w, 5 d)
- Availability
- Available
- LDAP User
- Unknown
- MediaWiki User
- LToscano (WMF) [ Global Accounts ]
Today
We had a chat with the team about this:
We are reviewing our tasks and this seems to be not relevant anymore in the context of MW on K8s, so we decided to close, but please re-open if you feel something is needed!
I updated the bullseye and bookworm netist images this morning but didn't know about upgrading the packages for the point release :(
I think I know why this is happening, didn't realize till now. Docker Reporter, the tool that scans the registry for images and calls debmonitor for them, has the following configs:
Follow up: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1070028 - Spicerack's setup.py needed to be updated to avoid build failures.
I tried to build 8.10.0 to make sure that it wasn't related to the 8.11.0 changes, and I get the same result.
Next steps:
Fri, Aug 30
Next steps:
A lot of cleanup has been done, so far it seems that the task can be closed. We have a better way to figure out what images are not supported by the docker reporter now, so it is also good.
Thu, Aug 29
Created https://grafana-rw.wikimedia.org/d/e0f6afe3-2aea-483d-9f5e-55f0cba9207f/puppetserver, didn't add all the metrics but it should be a good start to figure out what's happening.
Something very strange:
Thanks a lot for the task!
Wed, Aug 28
Created T371549 to move the Maintainer field of the production-images repo's control files to team-specific (where possible). Ideally this should be the info that docker-pkg will use to publish the label about who to contact.
Hi @MSantos! Ack thanks for the info, will keep it in mind when hopefully upgrading the maps nodes :) Any news about the Kartotherian depencency upgrade?
Tue, Aug 27
@hashar thanks a lot for the investigation! Would it be possible to have an option in the jjb config to allow Python jobs to specify the list of tox environments to execute? If so we could set the py39 ones for spicerack as special use case, and update them when the cumin nodes are upgraded.
Today I cleaned up some db nodes reported as debmonitor client failures while I was on holiday:
@mark @odimitrijevic thanks a lot for the explanations and the rationale, I am not 100% happy with the outcome but I will not oppose anymore :) In the future we'd love to get involved sooner (we == SRE team) rather than later in these kind of discussions, to help as much as possible and provide more options (if available).
@sbassett Hi! Please subscribe to the ops mailing list so you can get notified by these changes, usually we post a message there to warn users. In this case it is safe to deploy since it is just a rebuild to use a new OS (Bookworm) and skip Buster.
Mon, Aug 26
Left to do:
Fri, Aug 16
Currently blocked by T372485
Hi! I am Luca from the SRE Infrastructure Foundations team, I had a chat with Ben today about https://gerrit.wikimedia.org/r/c/operations/puppet/+/1062401 and the need to deploy this extra plugin.
Wed, Aug 14
Found another issue:
Filed https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/24 to fix a conftool issue:
@brouberol after T355550 do we have any plans to start testing the upgrade on kafka-test or similar? I can help if needed :)
Tue, Aug 13
Encountered an issue with the BMC's network config:
@Jhancock.wm Hi! Feel free to leave this host to me, still trying to figure out some stuff :(
Mon, Aug 12
Move done, tested the new "disarmed" pre-commit hook on puppetmaster1001 and a commit on puppetserver1001.
Thu, Aug 8
Wed, Aug 7
Sent an email to all SREs, the move will happen on Aug 12th 13:00 UTC.
Adding some ideas: https://docs.djangoproject.com/en/5.0/ref/models/querysets/#bulk-update
Buster and Bookworm rollouts done, no big issues registered. The only drawback is that due to the high volume of writes to the db (since we are changing the Debian version etc..) the UI gets not responsive for a bit, and we get some alarms. This is due to T371899.
Rolled out the change to the hadoop cluster, this is the only error that I got:
Tue, Aug 6
The issue is described in T371899. I proceeded anyway to upgrade both debmonitor server hosts, all good so far.
Fix for the dump-cloud-ip-ranges timer/unit rolled out, I also tried to do a manual puppet private commit and it worked. Let's wait for the next run of the timer to confirm that everything is working.
Mon, Aug 5
- The Netbox custom script for network provisioning is now asking for a mac address (for the mgmt interface), mandatory for each supermicro.
- spicerack's redfish module is now able to create admin users in the BMC (only for supermicro).
After a brainbounce with Joe on the SRE IRC channel, we noticed that the environment variables when running the post commit hook (in my local repro) contained GIT_INDEX_FILE pointing to the repo representing /srv/private, instead of /var/lib/git/etc... Simply unsetting the variable in the post-commit hook make the issue disappear.
I managed to repro the issue on a local docker container, and I can say that it is definitely the code of external_clouds_vendors that causes this. The repro is using only the post-commit hook to propagate the change between the two fake private repos.
Aug 2 2024
Aug 1 2024
It re-appened again, again only requestctl/ipblocks related staged content. Being staged meant that git pull worked, so it wouldn't have broken any regular commit happening under /srv/private (at least IIUC).
Tried to test the new debmonitor-server on debmonitor2003:
- changed sretest1001 /etc/hosts to point debmonitor.discovery.wmnet to debmonitor2003's ip
- dropped via spicerack the data on debmonitor for the host
- ran the debmonitor client on the host
Jul 31 2024
From docker report (k8s images) set to work only with Bullseye+ images: