This all is working fine thank you @Jhancock.wm

Updated config is applied on asw2-ulsfo since yesterday and not showing signs of problems. Full validation the schedulers are working exactly as expected can be done at a later stage but all looks good and no reason to think they are not.

In T370475#10104012, @Jhancock.wm wrote:

@cmooney I got everything swapped around as requested and the new 1G link is on ge-0/0/43. I can confirm that it will not get in the way of the new servers. Please let me know if you need anymore assistance!

Hey @Jhancock.wm as discussed on irc I was hoping to get the main uplink for this server moved from asw-d4-codfw to lsw1-d4-codfw.

Nice work!

All work completed, no issues to report.

If anyone is rebooting lvs hosts and has to deal with this the below shell script will make sure a working route is in use:

#!/bin/bash
set -x 
while : ; do
  V6_ROUTE=$(ip -6 route get fibmatch :: | awk '{print $1,$2,$3,$4,$5}')
  V6_DEF_INT=$(echo "$V6_ROUTE" | awk '{print $5}')
  if [[ $V6_DEF_INT == e* ]]; then
    break
  else
    sudo ip -6 route del $V6_ROUTE
  fi
done

Work completed, no issues to report (although I had to downgrade the NIC firmware to 21.60.22.11 for PXEboot to start with the BCM57414 25G NIC and a 10G DAC, which we've seen before).

Working ok following update to the puppetdb import script. The issue was not actually caused by Netbox 4 changes, they were introduced some time back as part of the work on routed ganeti and this is the first time it was hit.

Path is now avoided again and packet loss is no longer observable:

cmooney@bast5004:~$ mtr -w -c 100 -b -z <removed>
Start: 2024-08-21T11:13:30+0000
HOST: bast5004                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS14907  ae1-510.cr2-eqsin.wikimedia.org (2001:df2:e500:1:fe00::2)   0.0%   100   16.2   2.6   0.4  21.4   4.4
  2. AS14907  ae0.cr3-eqsin.wikimedia.org (2001:df2:e500:fe05::1)         0.0%   100    0.8   1.1   0.4  31.0   3.5
  3. AS???    4637.sgw.equinix.com (2001:de8:4::4637:1)                  59.0%   100   83.6   4.8   1.2  83.6  13.1
  4. AS4637   2403:e800:508:300::36                                       0.0%   100    1.6   3.6   1.1  53.3   8.0
  5. AS???    ???                                                        100.0   100    0.0   0.0   0.0   0.0   0.0
  6. AS???    ???                                                        100.0   100    0.0   0.0   0.0   0.0   0.0
  7. AS55836  2405:200:801:700::b48                                       1.0%   100   50.4  51.1  50.0  76.6   3.0
  8. AS55836  2405:200:801:700::b48                                       8.0%   100   50.3  50.2  49.7  57.7   0.9
  9. AS55836  2405:200:801:700::cc4                                       0.0%   100   51.0  51.1  50.8  51.6   0.2
 10. AS55836  2405:200:393:eeee:20::1922                                  0.0%   100   53.0  53.3  52.7  62.9   1.5

@Jhancock.wm I've assigned new IPs for all these hosts on the private1-c-codfw and private1-d-codfw vlans instead. DNS and the switch port configs have been updated for them also. So you should be ok to reimage whenever.

The above patch will prevent this causing an issue when we follow the normal workflow - selecting a vlan 'type' (public/private) and allowing the Provision script to pick the right one for the rack.

In T370862#10035781, @Papaul wrote:

@cmooney links removed. You can resolve the task if nothing else needs to be done.

In T372781#10077810, @VRiley-WMF wrote:

@ayounsi I've checked the device and there doesn't seem to be any failure notifications (Physically anyway). Would it be possible to open up a RMA or Support ticket with Juniper?

In T371879#10049699, @Dzahn wrote:

We got paged at 20:19 UTC for "primary outbound port utilisation over 80%" on both cloudsw1-d5 and cloudsw1-f4 today.

In T371878#10045741, @cmooney wrote:

It's probably best to manually flip the HA on the cloudgw/cloudnet nodes to the ones in rack C8 before we start. I just checked and the two nodes in rack D5 (cloudnet1006 and cloudgw1002) are - of course - the current live ones.

It's probably best to manually flip the HA on the cloudgw/cloudnet nodes to the ones in rack C8 before we start. I just checked and the two nodes in rack D5 (cloudnet1006 and cloudgw1002) are - of course - the current live ones.

Just to update on the situation things remain stable since the changes earlier on.

cmooney@cloudsw1-d5-eqiad> show bgp summary | match "^[0-9]"    
10.64.147.5      4264710003        912        986       0      33     6:55:09 Establ
10.64.147.7      4264710004        946       1035       0      32     7:10:21 Establ
10.64.147.12          64710        937        937       0      54     6:25:07 Establ
10.64.147.14          14907     987227     903873       0       0 41w4d 0:28:56 Establ
172.20.2.2            64605        983        926       0      42     7:08:48 Establ
172.20.2.4            64605        990        928       0      43     7:09:30 Establ
172.31.255.5     4264710003        912        939       0      36     6:55:05 Establ
172.31.255.7     4264710004        946        995       0      33     7:10:17 Establ
185.15.56.248         64710        841        836       0      54     6:24:39 Establ
208.80.154.212        14907     827730     869817       0       1 40w0d 3:17:52 Establ
2620:0:861:fe0a::1       14907     889734     903784       0       0 41w4d 0:29:07 Establ
2620:0:861:fe0b::1       64710        914        914       0      53     6:25:01 Establ
2620:0:861:fe0e::2  4264710003        912        977       0      33     6:54:58 Establ
2620:0:861:fe0f::2  4264710004        945       1030       0      32     7:10:13 Establ

I should say cloudcephosd1036 change I've not pushed to the switch - that will happen when we do a homer run after the planned reboot/upgrade (T371879)

Thanks guys, the second ports are now configured on the switches.

Work completed on this one on the network & LVS side.

Work on this one is completed, all that remains is to remove the old cross-rack links which are no longer being used.

Ultimately on these hosts those IPs are bridged onto the main interface, and both source and answer ARP requests on the wider (/27 and /64) networks they come from.

Gonna close this one, I see hosts have been assigned to the new range and are reachable

cmooney@cumin1002:~$ ping pay-lb2001.mgmt.frack.codfw.wmnet 
PING pay-lb2001.mgmt.frack.codfw.wmnet (10.195.1.6) 56(84) bytes of data.
64 bytes from wmf11992.mgmt.frack.codfw.wmnet (10.195.1.6): icmp_seq=1 ttl=61 time=30.7 ms
64 bytes from wmf11992.mgmt.frack.codfw.wmnet (10.195.1.6): icmp_seq=2 ttl=61 time=30.8 ms