User Details
- User Since
- Mar 16 2023, 2:18 PM (76 w, 3 d)
- Availability
- Available
- IRC Nick
- kamila_
- LDAP User
- Kamila Součková
- MediaWiki User
- KSoučková-WMF [ Global Accounts ]
Thu, Aug 29
Tue, Aug 27
Aug 2 2024
On further thought, doing it properly, i.e. creating a k8s-aux cluster in codfw and placing it there, seems like a better idea. That's going to take more time than just moving benthos. @fgiunchedi this isn't urgent, correct?
Jul 29 2024
Jul 23 2024
FTR, I have reverted the buffer patch, as it shouldn't be necessary now that we have more partitions thanks to T369256, and I cannot rule it out as a possible cause for some of the "benthos is wedged" weirdness observed last week. That means we no longer know how much reserve capacity we have. In case of trouble, revert^2 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1055399 .
Jul 17 2024
AFAICT, this is only caused by extremely long URLs that result in truncated (and thus invalid) JSON -- see also T368417. These URLs are not "real" user-generated traffic, and are not really useful. I will look into options for making them go away.
Jul 16 2024
Jul 15 2024
Jul 12 2024
There were two unrelated causes for these:
Jul 3 2024
Increasing batch size slightly improved the situation, very slowly clearing the backlog, suggesting that this was caused by some flavour of a performance issue:
Because I did not see any direct resource starvation on the host, my hunch was that this was something about the message delivery.
Jun 27 2024
Closing this and opening T368640, as what I'm seeing now is a tiny sub-problem of the original problem
Jun 26 2024
I believe the errors are unrelated (they are due to T340935 and we've had bad messages before and they didn't cause the problem).
Jun 24 2024
Jun 21 2024
@DSmit-WMF Are you requesting SSH access too, or just tools?
@leila could you please confirm this access request? Thank you!
@cwylo Can you please confirm that you have read the Analytics Data Access User Responsibilities ? Thank you!
Needs approval from one of:
- Olja Dimitrjevic
- Dan Andreescu
- Will Doran
- Andreas Hoelzl
- Andrew Otto
Done, thanks a lot for the help @Papaul !
Jun 19 2024
@AndyRussG since you're now with WMDE, we'd like to update your email, can you please let me know your @wikimedia.de email address?
@DMburugu can you please confirm that you have read the Analytics Data Access User Responsibilities? Thanks!
Done, though it's my first time doing clinic duty, so let me know if it doesn't work :D
Jun 18 2024
As per the process at https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#WMDE_Group , we need confirmation from an engineering manager. @WMDE-leszek would you mind confirming this request please?
@AndyRussG Can you please confirm the new end date? I assume you meant July 30, 2024, correct?
Jun 17 2024
wikikube-ctrl2003 looks happy, thanks for the help!
waiting for approval
Stalled on @Milimetric signing L3
@Milimetric also, you provided an SSH key fingerprint, we need the public key. It should start with something like ssh-rsa AAAA...
@Milimetric I cannot find your signature on L3, could you please ensure that you have signed it?
Jun 14 2024
New NICs seem to be happy including overnight network testing out of sheer paranoia, so calling it done 🎉
Jun 13 2024
Jun 12 2024
I'm seeing more bad messages that look like an encoding problem:
Jun 11 2024
@Papaul Thanks for the additional details!
Jun 10 2024
@Papaul could you please let me know when would be a good time for you to do this? We don't have any specific time requirements, just that earlier would be nice. I would like to do it in two steps (first just 1 server, then the other 2) for capacity reasons. I am in CEST, so US mornings would work well for me, but I can also decom in advance and reimage the next day if you'd prefer to do it asynchronously, just let me know.
Jun 6 2024
@VRiley-WMF the reimage of wikikube-ctrl1001 was finally successful, I want to run a few more tests due to having had network(?) problems during the reimage, but I _think_ we should be good to proceed next week, if that works for you. Tuesday would be the best day for me, but I can make other days work too. I believe that capacity-wise, we can move the two remaining machines at the same time if that's easier for you.
Jun 4 2024
@VRiley-WMF Yes, that works, thank you!
May 31 2024
@VRiley-WMF I am in UTC+2, so US mornings are best for me. Would Tuesday work for you? Thank you!
May 29 2024
Mar 27 2024
Mar 25 2024
Based on some quick tests the image seems to be working \o/
Mar 21 2024
For the record, I used sudo bash -c 'find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy1002.eqiad.wmnet/git_server: deploy2002.codfw.wmnet/"' (or the other way around) on all deployment servers to work around this.
Note also the increase in RX (but not really TX) traffic that coincides with these:
Feb 26 2024
Inspired by some of the above:
Feb 21 2024
Feb 19 2024
Feb 12 2024
Feb 7 2024
Feb 6 2024
mw1386 seems to be going fine now, so yes, we can close this. Sorry and thanks for finding the cause @Volans <3
Feb 5 2024
Feb 1 2024
Jan 23 2024
I believe the above patch fixed it, so I'm closing this. I will reopen in case I see the race again.
Jan 17 2024
All traffic is now going to k8s \o/
Jan 16 2024
Jan 15 2024
Jan 10 2024
With Alex's patches, I ran 7 reimages and 20 reboots without the issue reappearing. It might be worthwhile to understand the issue better to see if that workaround is adequate, but as it appears to not be blocking us anymore, I'm closing this.
Jan 8 2024
Note a few other things that have been tried:
- Updated firmware (UEFI + iDRAC) on mw1378, did not help.
- Appears to be related to the wdat_wdt watchdog driver (all affected hosts have that driver). Kernels >= 5.10.205-1 should have a related patch backported (https://patchwork.kernel.org/project/linux-watchdog/patch/1650874932-18407-3-git-send-email-liuxp11@chinatelecom.cn/), but this also happens on newer kernels.
Jan 5 2024
Note that this is non-deterministic: the problem seems to happen more than half the time but far from always. So several reboots may be required to reproduce. Yay!
Additional findings:
- the watchdog: watchdog0: watchdog did not stop! message seems to be a red herring, it's always there
- the problem only occurs when running nohup reboot (which is what the cookbooks do), not when I run just reboot
- it cannot be reproduced on an insetup system with overlayfs in use
Note that we have tried updating the firmware: mw1388 is on new UEFI and iDRAC and still exhibits this problem.