Right now this limit is not even 4GB.

Resharding completed, only pending 2 running purge screeen on ms-backup2001, 2002 for purging leftovers. backup1011 & backup2011 will have to be completented by backup1012 and backup2012 this Q.

1 more week left to finish the resharding.

5 million files left to recover!

taking that paste in note, thanks! :)

And this is the wiki distribution:

This is the API request I filed: T267365

Mentioned in SAL (#wikimedia-operations) [2024-05-22T15:01:41Z] <jynus> stopping eqiad mediabackups for cleaning up missing files T365607

Followup to T361087.

In any case, at this point I 'd prefer to do an in-place upgrade rather than a reimage, given how unreliable a reimage is and how impactful it can be for stateful services.

In T361087#9745154, @jcrespo wrote:

In T361087#9744977, @cmooney wrote:

In T361087#9744384, @jcrespo wrote:

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al


Debian 12 (bookworm) amd64 (Wikimedia edition)

                                              boot: 
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...
Boot failed: press a key to retry, or wait for reset...

Hmm. Not sure if we've seen this problem before. DHCP clearly worked as did the debian image download, but Linux failed to load for some reason.

@jcrespo the only difference was selecting bullseye rather than bookworm on the second attempt?

Yes. Check with @MoritzMuehlenhoff he did something to fix something, but not sure what, or if it applies here.

In T361087#9744977, @cmooney wrote:

In T361087#9744384, @jcrespo wrote:

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al


Debian 12 (bookworm) amd64 (Wikimedia edition)

                                              boot: 
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...
Boot failed: press a key to retry, or wait for reset...

Hmm. Not sure if we've seen this problem before. DHCP clearly worked as did the debian image download, but Linux failed to load for some reason.

@jcrespo the only difference was selecting bullseye rather than bookworm on the second attempt?

In T361087#9744384, @jcrespo wrote:

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al


Debian 12 (bookworm) amd64 (Wikimedia edition)

                                              boot: 
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...
Boot failed: press a key to retry, or wait for reset...

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bullseye completed:

• backup1005 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404251323_root_864747_backup1005.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status failed -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bullseye executed with errors:

• backup1005 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Checked BIOS boot parameters are back to normal
  • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" backup1005.eqiad.wmnet to get a root shellbut depending on the failure this may not work.

If booted into bullseye.

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bookworm executed with errors:

• backup1005 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" backup1005.eqiad.wmnet to get a root shellbut depending on the failure this may not work.

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host backup1005.eqiad.wmnet with OS bookworm

Will reimage soon.

We have received the PERC from Dell and I have just completed swapping it out. It now looks like the system can now see the PERC (previously, it wasn't). However, it does seem that the system will need to be rebuilt. @jcrespo would you be able to verify this? Thank you!

We have been able to get dell support on this unit. After sending over the logs for and they have reviewed it they suggested to update the BIOS and iDRAC. BIOS install went through fine. After completing the iDRAC update, it's not loading properly. Currently working with Dell to resolve this new issue that has been created.

Cloning speed for 133 GB / 28K objects:

# rclone copy -P backup2007:mediabackups/commonswiki/fff backup2011:mediabackups/commonswiki/
Transferred:      133.243 GiB / 133.243 GiB, 100%, 125.044 MiB/s, ETA 0s
Transferred:        28850 / 28850, 100%
Elapsed time:     14m24.4s

jynus merged https://gitlab.wikimedia.org/repos/sre/mediabackups/-/merge_requests/3

I tested the above patch and it solved the issue:

jynus opened https://gitlab.wikimedia.org/repos/sre/mediabackups/-/merge_requests/3

Opened ticket with dell in order to see what they could assist with since when first contacting them, it was on the day the warranty expired. Awaiting response from Dell

The new shard looking great:

Mentioned in SAL (#wikimedia-operations) [2024-03-27T18:54:07Z] <jynus> increasing volume size of backup2011 T334069