Hi, strangely, some of my MIDI uploads have not been transcoded (Quarry) e.g. in my list of Commons uploads, the first six are fine, but File:6-Z46B_set_class_on_C.mid and on are still not, after 3 days?

Seems to have stabilized:

Hmm, it's down under 4k entries but still high.

Hi, apologies that I don't know how to help, and not sure if it's the same problem; the manual intervention here worked for files I uploaded to Commons on Monday (e.g. File:3-5B set class on C.mid), but files I uploaded yesterday are still waiting, e.g. File:4-3 set class on C.mid. I made a quarry query for these.

The queue was shrinking since yesterday, but is climbing again since this morning. This doesn't make any sense. With those high res jobs disabled, we should have more than enough capacity to catch up, should we not ?

Ok, 1440p and 2160p transcodes are temporarily disabled for now until better fixes, and we did a kill of the old stuck processes. Might still take a bit to shake everything out; I'm trying to flush through all the missing audio.

I'm seriously considering bringing back my "chunked" scheme that would at least produce smaller, standalone jobs that encode say 10 seconds worth of video, then reassemble the final into a single video at the end. :P Main reason I haven't is that the logic needs to be able to handle missing chunks if individual ones time out or fail and that sounds like a pain, but it'll be a lot friendlier to the job queue infrastructure.

Funnily enough, thats exactly the kind of problem i was also concerned would pop up with the new k8s cluster T357309#9561624
Didnt know we were so susceptible to it on the old setup as well.

Looks like we've got a couple problems with high-res videos:

• a bunch of 4K videos got uploaded at once and they all queued up
• some of them are stuck! they should be timing out
• it's also possible the audio clips are going to the wrong queue, i have to double-check this

The list of "active" (may or may not actually be active) includes a number of 2160p high-res videos hitting since June 21. We've also gotten reports before about certain kinds of AV1 videos slowing down the input handling, which I haven't checked for.

I'm bulk-adding the missing audio transcodes which should force them to run through as fast as possible between other jobs, and hopefully will handle the prioritized queue split better.

Live system thinks it has 9,223 items queued on commons and requeue is throttling there for now.... occasionally it goes down an item and moves on.

Mentioned in SAL (#wikimedia-operations) [2024-06-25T16:23:55Z] <bvibber> running requeueTranscodes for missing audio files on commons (mwmaint1002) cf T368364

Batch requeueTranscodes failured on June 22 with this error:

Could be a backfill run but that shouldn't be interfering with anything... I'll check on it

So pretty early on the 20th around 01:20 (last Thursday) it started rising. So it wasn't the train, as the train arrived on Commons on Wednesday. In SAL, I only see db maintenance T367856 going on around that time.

See also T368333: No sound in MP4 video files downloaded from Commons (which is likely not related; still mentioning it for the paper trail).

I see a surge in backlog for WebVideoTranscodePrioritized

In T354042#9863716, @Wbm1058 wrote:

Please do open a new Phab for this, if appropriate.

I enhanced one of my bots to check for, and report, errors it encounters.

But maybe this is for RunJobs (which WMF doesn't use) rather than RunSingleJob (which I thought this task is mainly about). I very much like the idea of replacing Special:RunJobs with a (non-private) REST replacement with signed headers. Is that planned as part of this task?

• The patches under T365752 add generic support for private modules as a new feature in REST. It seems non-trivial to maintain and support in a generic way. A simpler approach might be a feature flag specific to JobRunner, in which case one could communicate disablement via an HTTP status code from RunSingleJob, rather than by being absent from the registry. This might make sense since as that way the manifest isn't variable by user, which would risk/complicate internals around caching, and e.g. generating Swagger specs as we'd need to be careful not to let the cache be infected by a jobrunner. For special pages and API modules, we don't vary their registration by permission. The pages that are installed are always existent, but they may return a permission error. Anyway, more than the registry aspect pro/con, my main question here is about cost/benefit of an early abstraction now (which delays the main work of this task, and means there will be multiple moving pieces when we deploy/test this) vs later if/when there is more than 1 user for it (which would contain the business logic in a single class, and allow us to move more rapidly).

Change #1033227 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] REST: add support for private modules

https://gerrit.wikimedia.org/r/1033227

Resolved as part of k8s migration

@hnowlan: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

Quick summary of a conversation with Timo:

• the primary protection mechansim should eb "off per default, enabled on internal cluster"
• to prevent 3rd frmo accidentially making this endpoint public , there should be a second line of defense, like a list of IPs
• signing requests, like SpecialRunJobs does, is a good mechanism as well. As long as the signature is created by MW, it's easy. It's inconvenient to try and generate a signature outside MW.
• relying on the host header for picking the wiki that needs to process a job should be fine, but we nede to actually start setting this header in changeprop
• we can probably just use the RunSingleJob REST enpoint that exists in the EventBus extension. Should we move it into core? 3rd parties are very unlinkely to ever need it.
• we may want to establish best practices for "private" endpoints in general

Notes to self:

• Why do we need rpc/RunSingleJob.php, if EventBus already has a REST handler for RunSingleJob?
• rpc/RunSingleJob.php uses the "database" field in the body to pick the wiki. Why are we not using the Host header, like we do for all other entry points?
• @akosiaris sais the host header should be present, per https://github.com/wikimedia/mediawiki-services-change-propagation/blob/master/config.jobqueue.wikimedia.yaml#L51
• @Jgiannelos found the job runner deployment charts: https://codesearch.wmcloud.org/search/?q=jobrunner_uri&files=&excludeFiles=&repos=
• @akosiaris dug up some relevant changes: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/370004 and here: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/369999

I fear I read that task, the way it is written at least, differently.

Marking this resolved as you just confirmed a big file upload going through correctly. Thanks for your help in debugging this!

Seems like this is fixed! We just had File:OBR_Hafignnover_6-2024.webm (4.88 GB) successfully uploaded (req-id: c1a5aab7-6e6b-4ef5-b20d-f9ddb095577f ). The job took 5 minutes 20 seconds to complete, so went beyond the previous 202 second limit.

In T249745#9723704, @Ottomata wrote:

For replicating state changes (T120242) [...]

Why though? Why is 99.9999% (or 99.999999% or 99.99%) not enough?

There is a "Why do we need this?" section in T120242's description. Let's keep this discussion there?

Replied at T120242#9726131

In T249745#9725953, @Ottomata wrote:

search index not getting updated in 0.001% of edits

Search is probably fine.

In T358308#9724785, @Bawolff wrote:

[...]
So it seems like two separate issues.

I guess sometimes the job runner pod gets terminated in the middle of a job. That would be fine if something like https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1008403 got merged)

request_terminate_timeout for mw-jobrunner should now be set to 86400, as it was on bare metal.

Change #1021427 merged by jenkins-bot:

[operations/deployment-charts@master] mw-debug: fix php.timeout

https://gerrit.wikimedia.org/r/1021427

search index not getting updated in 0.001% of edits

Mentioned in SAL (#wikimedia-operations) [2024-04-18T11:29:19Z] <cgoubert@deploy1002> Finished scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 (duration: 37m 04s)

Change #1021427 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-debug: fix php.timeout

https://gerrit.wikimedia.org/r/1021427

Mentioned in SAL (#wikimedia-operations) [2024-04-18T10:52:15Z] <cgoubert@deploy1002> Started scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308

Change #1021418 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Fix php.timeout

https://gerrit.wikimedia.org/r/1021418