On further thought, doing it properly, i.e. creating a k8s-aux cluster in codfw and placing it there, seems like a better idea. That's going to take more time than just moving benthos. @fgiunchedi this isn't urgent, correct?

FTR, I have reverted the buffer patch, as it shouldn't be necessary now that we have more partitions thanks to T369256, and I cannot rule it out as a possible cause for some of the "benthos is wedged" weirdness observed last week. That means we no longer know how much reserve capacity we have. In case of trouble, revert^2 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1055399 .

AFAICT, this is only caused by extremely long URLs that result in truncated (and thus invalid) JSON -- see also T368417. These URLs are not "real" user-generated traffic, and are not really useful. I will look into options for making them go away.

There were two unrelated causes for these:

Increasing batch size slightly improved the situation, very slowly clearing the backlog, suggesting that this was caused by some flavour of a performance issue:

Closing this and opening T368640, as what I'm seeing now is a tiny sub-problem of the original problem

I believe the errors are unrelated (they are due to T340935 and we've had bad messages before and they didn't cause the problem).

@DSmit-WMF Are you requesting SSH access too, or just tools?

@KFrancis can you please make sure @MunizaA's NDA is signed? Thank you!

@leila could you please confirm this access request? Thank you!

@cwylo Can you please confirm that you have read the Analytics Data Access User Responsibilities ? Thank you!

Needs approval from one of:

• Olja Dimitrjevic
• Dan Andreescu
• Will Doran
• Andreas Hoelzl
• Andrew Otto

Done, thanks a lot for the help @Papaul !

@AndyRussG since you're now with WMDE, we'd like to update your email, can you please let me know your @wikimedia.de email address?

@DMburugu can you please confirm that you have read the Analytics Data Access User Responsibilities? Thanks!

Done, though it's my first time doing clinic duty, so let me know if it doesn't work :D

As per the process at https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#WMDE_Group , we need confirmation from an engineering manager. @WMDE-leszek would you mind confirming this request please?

@AndyRussG Can you please confirm the new end date? I assume you meant July 30, 2024, correct?

wikikube-ctrl2003 looks happy, thanks for the help!

waiting for approval

Stalled on @Milimetric signing L3

In T365074#9898298, @MoritzMuehlenhoff wrote:

In T365074#9898255, @kamila wrote:

@Milimetric also, you provided an SSH key fingerprint, we need the public key. It should start with something like ssh-rsa AAAA...

There's an existing user already, this only needs to have the group added: L1920 in data.yaml

@Milimetric also, you provided an SSH key fingerprint, we need the public key. It should start with something like ssh-rsa AAAA...

@Milimetric I cannot find your signature on L3, could you please ensure that you have signed it?

New NICs seem to be happy including overnight network testing out of sheer paranoia, so calling it done 🎉

In T366205#9880294, @Papaul wrote:

@kamila your plan works for us as well, just depool and power the first server you want to move in your time zone and let us know which one and when we are on site in our time zone we will put in the 10G nic, move it to the new rack and do all the Netbox changes and hang it back to you for re-images. Once you are happy with it we can move to the next one.

I'm seeing more bad messages that look like an encoding problem:

@Papaul Thanks for the additional details!

@Papaul could you please let me know when would be a good time for you to do this? We don't have any specific time requirements, just that earlier would be nice. I would like to do it in two steps (first just 1 server, then the other 2) for capacity reasons. I am in CEST, so US mornings would work well for me, but I can also decom in advance and reimage the next day if you'd prefer to do it asynchronously, just let me know.

@VRiley-WMF the reimage of wikikube-ctrl1001 was finally successful, I want to run a few more tests due to having had network(?) problems during the reimage, but I _think_ we should be good to proceed next week, if that works for you. Tuesday would be the best day for me, but I can make other days work too. I believe that capacity-wise, we can move the two remaining machines at the same time if that's easier for you.

@VRiley-WMF Yes, that works, thank you!

@VRiley-WMF I am in UTC+2, so US mornings are best for me. Would Tuesday work for you? Thank you!

Based on some quick tests the image seems to be working \o/

For the record, I used sudo bash -c 'find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy1002.eqiad.wmnet/git_server: deploy2002.codfw.wmnet/"' (or the other way around) on all deployment servers to work around this.

Note also the increase in RX (but not really TX) traffic that coincides with these:

Inspired by some of the above:

mw1386 seems to be going fine now, so yes, we can close this. Sorry and thanks for finding the cause @Volans <3

I believe the above patch fixed it, so I'm closing this. I will reopen in case I see the race again.

All traffic is now going to k8s \o/

With Alex's patches, I ran 7 reimages and 20 reboots without the issue reappearing. It might be worthwhile to understand the issue better to see if that workaround is adequate, but as it appears to not be blocking us anymore, I'm closing this.

Note a few other things that have been tried:

• Updated firmware (UEFI + iDRAC) on mw1378, did not help.
• Appears to be related to the wdat_wdt watchdog driver (all affected hosts have that driver). Kernels >= 5.10.205-1 should have a related patch backported (https://patchwork.kernel.org/project/linux-watchdog/patch/1650874932-18407-3-git-send-email-liuxp11@chinatelecom.cn/), but this also happens on newer kernels.

Note that this is non-deterministic: the problem seems to happen more than half the time but far from always. So several reboots may be required to reproduce. Yay!

In T354413#9437404, @MoritzMuehlenhoff wrote:

Is this reproducible with every reboot or just some? One thing worth doing is to connect to the serial console an then issue a reboot over Cumin. Maybe we're seeing a kernel oops during the system tear down?

Additional findings:

• the watchdog: watchdog0: watchdog did not stop! message seems to be a red herring, it's always there
• the problem only occurs when running nohup reboot (which is what the cookbooks do), not when I run just reboot
• it cannot be reproduced on an insetup system with overlayfs in use

Note that we have tried updating the firmware: mw1388 is on new UEFI and iDRAC and still exhibits this problem.