User Details
- User Since
- May 10 2021, 3:25 PM (172 w, 6 d)
- Availability
- Available
- IRC Nick
- topranks
- LDAP User
- Cathal Mooney
- MediaWiki User
- CMooney (WMF) [ Global Accounts ]
Fri, Aug 30
This all is working fine thank you @Jhancock.wm
Updated config is applied on asw2-ulsfo since yesterday and not showing signs of problems. Full validation the schedulers are working exactly as expected can be done at a later stage but all looks good and no reason to think they are not.
Thu, Aug 29
Hey @Jhancock.wm as discussed on irc I was hoping to get the main uplink for this server moved from asw-d4-codfw to lsw1-d4-codfw.
Wed, Aug 28
Nice work!
Thu, Aug 22
All work completed, no issues to report.
If anyone is rebooting lvs hosts and has to deal with this the below shell script will make sure a working route is in use:
#!/bin/bash set -x while : ; do V6_ROUTE=$(ip -6 route get fibmatch :: | awk '{print $1,$2,$3,$4,$5}') V6_DEF_INT=$(echo "$V6_ROUTE" | awk '{print $5}') if [[ $V6_DEF_INT == e* ]]; then break else sudo ip -6 route del $V6_ROUTE fi done
Wed, Aug 21
Work completed, no issues to report (although I had to downgrade the NIC firmware to 21.60.22.11 for PXEboot to start with the BCM57414 25G NIC and a 10G DAC, which we've seen before).
Working ok following update to the puppetdb import script. The issue was not actually caused by Netbox 4 changes, they were introduced some time back as part of the work on routed ganeti and this is the first time it was hit.
Path is now avoided again and packet loss is no longer observable:
cmooney@bast5004:~$ mtr -w -c 100 -b -z <removed> Start: 2024-08-21T11:13:30+0000 HOST: bast5004 Loss% Snt Last Avg Best Wrst StDev 1. AS14907 ae1-510.cr2-eqsin.wikimedia.org (2001:df2:e500:1:fe00::2) 0.0% 100 16.2 2.6 0.4 21.4 4.4 2. AS14907 ae0.cr3-eqsin.wikimedia.org (2001:df2:e500:fe05::1) 0.0% 100 0.8 1.1 0.4 31.0 3.5 3. AS??? 4637.sgw.equinix.com (2001:de8:4::4637:1) 59.0% 100 83.6 4.8 1.2 83.6 13.1 4. AS4637 2403:e800:508:300::36 0.0% 100 1.6 3.6 1.1 53.3 8.0 5. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0 6. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0 7. AS55836 2405:200:801:700::b48 1.0% 100 50.4 51.1 50.0 76.6 3.0 8. AS55836 2405:200:801:700::b48 8.0% 100 50.3 50.2 49.7 57.7 0.9 9. AS55836 2405:200:801:700::cc4 0.0% 100 51.0 51.1 50.8 51.6 0.2 10. AS55836 2405:200:393:eeee:20::1922 0.0% 100 53.0 53.3 52.7 62.9 1.5
Tue, Aug 20
@Jhancock.wm I've assigned new IPs for all these hosts on the private1-c-codfw and private1-d-codfw vlans instead. DNS and the switch port configs have been updated for them also. So you should be ok to reimage whenever.
The above patch will prevent this causing an issue when we follow the normal workflow - selecting a vlan 'type' (public/private) and allowing the Provision script to pick the right one for the rack.
Tue, Aug 13
Thu, Aug 8
Wed, Aug 7
Tue, Aug 6
It's probably best to manually flip the HA on the cloudgw/cloudnet nodes to the ones in rack C8 before we start. I just checked and the two nodes in rack D5 (cloudnet1006 and cloudgw1002) are - of course - the current live ones.
Just to update on the situation things remain stable since the changes earlier on.
cmooney@cloudsw1-d5-eqiad> show bgp summary | match "^[0-9]" 10.64.147.5 4264710003 912 986 0 33 6:55:09 Establ 10.64.147.7 4264710004 946 1035 0 32 7:10:21 Establ 10.64.147.12 64710 937 937 0 54 6:25:07 Establ 10.64.147.14 14907 987227 903873 0 0 41w4d 0:28:56 Establ 172.20.2.2 64605 983 926 0 42 7:08:48 Establ 172.20.2.4 64605 990 928 0 43 7:09:30 Establ 172.31.255.5 4264710003 912 939 0 36 6:55:05 Establ 172.31.255.7 4264710004 946 995 0 33 7:10:17 Establ 185.15.56.248 64710 841 836 0 54 6:24:39 Establ 208.80.154.212 14907 827730 869817 0 1 40w0d 3:17:52 Establ 2620:0:861:fe0a::1 14907 889734 903784 0 0 41w4d 0:29:07 Establ 2620:0:861:fe0b::1 64710 914 914 0 53 6:25:01 Establ 2620:0:861:fe0e::2 4264710003 912 977 0 33 6:54:58 Establ 2620:0:861:fe0f::2 4264710004 945 1030 0 32 7:10:13 Establ
I should say cloudcephosd1036 change I've not pushed to the switch - that will happen when we do a homer run after the planned reboot/upgrade (T371879)
Thanks guys, the second ports are now configured on the switches.
Jul 31 2024
Jul 30 2024
Work completed on this one on the network & LVS side.
Work on this one is completed, all that remains is to remove the old cross-rack links which are no longer being used.
Ultimately on these hosts those IPs are bridged onto the main interface, and both source and answer ARP requests on the wider (/27 and /64) networks they come from.
Jul 29 2024
Gonna close this one, I see hosts have been assigned to the new range and are reachable
cmooney@cumin1002:~$ ping pay-lb2001.mgmt.frack.codfw.wmnet PING pay-lb2001.mgmt.frack.codfw.wmnet (10.195.1.6) 56(84) bytes of data. 64 bytes from wmf11992.mgmt.frack.codfw.wmnet (10.195.1.6): icmp_seq=1 ttl=61 time=30.7 ms 64 bytes from wmf11992.mgmt.frack.codfw.wmnet (10.195.1.6): icmp_seq=2 ttl=61 time=30.8 ms