Change 933463 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge to remove unused cache servers

https://gerrit.wikimedia.org/r/933463

Mentioned in SAL (#wikimedia-cloud) [2023-06-28T09:54:41Z] <fabfur> removed (text|upload) instance references from wgCdnServersNoPurge (T327742)

We talked about this a bit in the Release-Engineering-Team team meeting on Wednesday—we discussed whether we had the ability to help. And I don't think we're well-positioned to do all this work as this encompasses a clone of most of Wikimedia's production footprint (where SRE has the most expertise).

I think one path forward here would be for multiple teams pitch-in to fix the instances they know how to fix.

Looking at the instance list, I do wonder if they're needed in beta (echostore?). A place to start might be a bunch of subtasks verifying:

1. Needed for beta? (i.e., critical to some team's workflow, if so: who?)
2. Ported to Bullseye in production? If not, any foreseeable blockers?
3. Can you help?

Other notes:

• Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.
• Production debian packages: T291916: Tracking task for Bullseye migrations in production may be a blocker in some cases deployment-prep depends on the same Debian packages used for production, and if they're not yet packaged for Bullseye, then there's no way to get a Bullseye image to run using production puppet code.

In T327742#9673367, @thcipriani wrote:

• Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.

The average instance on deployment-prep is ~ 2vCPU, 20GB disk, 4GB ram.

deployment-prep is almost at RAM quota and at CPU quota. Bumping those up to allow 5 new instance should allow folks to at least get started. So 10vCPU/20GB ram? Possible? It'll probably possible to reclaim some of that post-migration.

In T327742#9673515, @thcipriani wrote:

In T327742#9673367, @thcipriani wrote:

• Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.

The average instance on deployment-prep is ~ 2vCPU, 20GB disk, 4GB ram.

deployment-prep is almost at RAM quota and at CPU quota. Bumping those up to allow 5 new instance should allow folks to at least get started. So 10vCPU/20GB ram? Possible? It'll probably possible to reclaim some of that post-migration.

Added as subtask: T361477: requests to increase quotas deployment-prep

I have asked something on IRC's wikimedia-cloud about a way to speed up the process. In the puppet repo we have dist-upgrade.sh, a script that should be able to do all the safe steps to upgrade the OS in place. If it is ok to use it, we'd avoid the re-creation of new VMs with the issue of copying stateful data around (for example, Kafka/databases/etc..). The only downside would be for the Cloud team, that would need to fix the VM's metadata (like reported OS) after the dist-upgrade.

I have further increased the core quota for this project from 220 to 240.

In T327742#9812917, @elukey wrote:

I have asked something on IRC's wikimedia-cloud about a way to speed up the process. In the puppet repo we have dist-upgrade.sh, a script that should be able to do all the safe steps to upgrade the OS in place. If it is ok to use it, we'd avoid the re-creation of new VMs with the issue of copying stateful data around (for example, Kafka/databases/etc..). The only downside would be for the Cloud team, that would need to fix the VM's metadata (like reported OS) after the dist-upgrade.

There are a few reasons I don't like in-place upgrade.

One is the metadata issue that you mentioned: it's not as simple as changing a setting, it's more like rewriting history since 'what base image was used for this VM' is a question that's of use in various places. It used to be /very/ important as part of VM migration due to copy-on-write issues; I'd need to retest things to make sure that's still the case.

Another is that deployment-prep is meant to resemble production, and we mostly don't do in-place upgrades in production. For example with T361384, the old memc hosts were still running redis and had a ton of puppet and hiera leftover related to supporting redis. Only a new rebuild got us memc servers that actually resemble real production servers without a bunch of miscellaneous cruft.

And, of course, proper puppet design means that it isn't hard to rebuild hosts. If it /is/ hard we probably want to fix that :)

In T327742#9949679, @Andrew wrote:

In T327742#9812917, @elukey wrote:

I have asked something on IRC's wikimedia-cloud about a way to speed up the process. In the puppet repo we have dist-upgrade.sh, a script that should be able to do all the safe steps to upgrade the OS in place. If it is ok to use it, we'd avoid the re-creation of new VMs with the issue of copying stateful data around (for example, Kafka/databases/etc..). The only downside would be for the Cloud team, that would need to fix the VM's metadata (like reported OS) after the dist-upgrade.

There are a few reasons I don't like in-place upgrade.

One is the metadata issue that you mentioned: it's not as simple as changing a setting, it's more like rewriting history since 'what base image was used for this VM' is a question that's of use in various places. It used to be /very/ important as part of VM migration due to copy-on-write issues; I'd need to retest things to make sure that's still the case.

Ok thanks for the explanation!

Another is that deployment-prep is meant to resemble production, and we mostly don't do in-place upgrades in production. For example with T361384, the old memc hosts were still running redis and had a ton of puppet and hiera leftover related to supporting redis. Only a new rebuild got us memc servers that actually resemble real production servers without a bunch of miscellaneous cruft.

To be really fair, deployment-prep is really far from production, from its puppet config to the hosts/VM that runs the software. I can surely agree on the point that it is good every now and then to clean up stale/old packages, but I don't think it is strictly needed for deployment-prep (my 2c).

And, of course, proper puppet design means that it isn't hard to rebuild hosts. If it /is/ hard we probably want to fix that :)

I don't think that this has anything to do with Puppet design, it is just a matter of system complexity and the usability of deployment-prep and horizon/open-stack. In production we are able to reimage in place a node, with the possibility of keeping its data/state around, something that IIUC it is not possible on horizon/open-stack unless you use tricks like the OS in place dist-upgrade. This makes the creation of new VMs painful, since it is not just a matter of running puppet on the new node with the correct role, but also to keep any distributed system consistent with its ensemble (for example, Kafka/Cassandra/etc.. clusters). And in turn this makes the upgrade of deployment-prep something that it is done only when strictly needed, and from people that volunteer to do it. We have been talking about a real staging environment for production since I started working here, but we never really got anywhere close. The deployment-prep setup is incredibly useful but far from production, and we should find better and painless ways to keep it up with production that could encourage people to upgrade rather than avoiding it completely.

To be clear - I know that you are doing this work because it is needed and because moving away from old OSes is a good practice, not blaming you for the deployment-prep situation, just stating some thoughts out loud so we can discuss about them :)

Change #1053956 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] deployment-prep: replace deploy03 with deploy04

https://gerrit.wikimedia.org/r/1053956

Change #1053956 merged by Andrew Bogott:

[operations/puppet@production] deployment-prep: replace deploy03 with deploy04

https://gerrit.wikimedia.org/r/1053956