Even when it doesn't page, the increased lag causes bots that respect a reasonable maxlag to not be able to function. T368098#9914664

It just caused a page

Hi, this has caused ~12 alerts just since this weekend (https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-data-persistence/20240901.txt and https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-data-persistence/20240902.txt). The prefetch is fully done so that's not really the issue here. Can you do something about it?

This has caused another page in production T372961

@VirginiaPoundstone @lbowmaker who owns the decision on this task?

Change #856655 merged by Ryan Kemper:

[operations/puppet@production] snapshot: Remove absented cirrus dump job

https://gerrit.wikimedia.org/r/856655

In T352650#10051791, @dr0ptp4kt wrote:

• If I'm understanding correctly, people are thinking that we cutover from bare metal execution to use of container entrypoints for the jobs, and that we aren't necessarily running bare metal side-by-side with containers. This seems like a risk, at least if we consider the possibility that new stuff may be more prone to breakage and that such breakage could lead to lagged data delivery which can have a compounding effect in the infrastructure.

Following up on some discussions:

We just had another lag spike caused by dumps on enwiki: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-job=All&var-server=db1206&var-port=9104&viewPanel=6&from=1722603665043&to=1722604675864
It wasn't as impactful as before but it did cause some of our alerts to fire up as it went over a minute of lag, although it recovered quickly

To confirm understanding, did we have a leaning on whether the containerized version would pin to an older version of MediaWiki versus whether it would need to keep getting MediaWiki updates?

Just for the record, we met and discussed @Joe's proposal (this task's description) and were in general agreement that it's the best way forward. We have follow-up discussions to have and coordination to do, but we're aligned on the idea.

I think it will be quite a while before we are fully able to decom Dumps 1. This task will unblock SRE's MW to k8s migration, and will allow them to remove all the complicated puppet and scap code supporting the bare metal MW deployment.

Got it, thank you for the clarification!

In T365155#10000944, @PeterBowman wrote:

Hello. This is an empty text entry in wikidatawiki-20240701-pages-articles-multistream6.xml-p4469005p5969004.bz2 as downloaded from https://dumps.wikimedia.org/wikidatawiki/20240701/:

<page>
  <title>Q6157973</title>
  <ns>0</ns>
  <id>5952866</id>
  <revision>
    <id>2136476381</id>
    <parentid>2045646165</parentid>
    <timestamp>2024-04-24T18:13:17Z</timestamp>
    <contributor>
      <username>William Avery Bot</username>
      <id>2964320</id>
    </contributor>
    <comment>/* wbeditentity-update:0| */ Changing runeberg.org URLs to https (×7). See [[Wikidata:Requests_for_permissions/Bot/William_Avery_Bot_11|request for permission]]</comment>
    <origin>2136476381</origin>
    <model>wikibase-item</model>
    <format>application/json</format>
    <text bytes="90348" sha1="aimrgeqjzqp6d3oz5qc8awzr5nhi9da" />
    <sha1>aimrgeqjzqp6d3oz5qc8awzr5nhi9da</sha1>
  </revision>
</page>

Shouldn't this be already fixed with the deployed patch?

In T352650#10002138, @Milimetric wrote:

Sorry to ask this very basic question, but I found a bunch of others didn't know: how exactly is Dumps blocking the php 8 upgrade? Like, if we leave everything exactly as-is and just upgrade PHP, would it not run the way it is currently set up? On surface I see no big difference between the current setup and a containerized MW running on the same servers, so I'm curious about the nuance I'm missing here.

IIUC, the PHP 8 issue with be the same with containerized MW. I also don't know exactly what it is. :)

Sorry to ask this very basic question, but I found a bunch of others didn't know: how exactly is Dumps blocking the php 8 upgrade? Like, if we leave everything exactly as-is and just upgrade PHP, would it not run the way it is currently set up? On surface I see no big difference between the current setup and a containerized MW running on the same servers, so I'm curious about the nuance I'm missing here.

Hello. This is an empty text entry in wikidatawiki-20240701-pages-articles-multistream6.xml-p4469005p5969004.bz2 as downloaded from https://dumps.wikimedia.org/wikidatawiki/20240701/:

@Joe thanks for the ping. Just to keep you posted about our progress: Will is out this week but he and Data SRE are in conversation to see what we can accomplish. Should know more mid next week.

Looks like the dumps are starting to come back \o/

I can give a status update here, which I hope will be useful.