[go: nahoru, domu]

Page MenuHomePhabricator

TMH should not make DB writes on HTTP GET for its on-the-fly transcode corrections
Closed, ResolvedPublic

Description

Expectation (writes <= 0) by MediaWiki::main not met:
query-m: UPDATE `transcode` SET transcode_time_error = 'X' [TRX#441a33]
#0 /srv/mediawiki/php-1.29.0-wmf.5/includes/libs/rdbms/TransactionProfiler.php(215): TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.29.0-wmf.5/includes/libs/rdbms/database/Database.php(959): TransactionProfiler->recordQueryCompletion()
#2 /srv/mediawiki/php-1.29.0-wmf.5/includes/libs/rdbms/database/Database.php(870): Database->doProfiledQuery()
#3 /srv/mediawiki/php-1.29.0-wmf.5/includes/libs/rdbms/database/Database.php(1542): Database->query()
#4 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php(840): Database->update()
#5 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php(766): WebVideoTranscode::getTranscodeState()
#6 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php(942): WebVideoTranscode::isTranscodeReady()
#7 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php(749): WebVideoTranscode::addSourceIfReady()
#8 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php(638): WebVideoTranscode::getLocalSources()
#9 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(512): WebVideoTranscode::getSources()
#10 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(299): TimedMediaTransformOutput->getMediaSources()
#11 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(194): TimedMediaTransformOutput->getHtmlMediaTagOutput()
#12 /srv/mediawiki/php-1.29.0-wmf.5/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(141): TimedMediaTransformOutput->getImagePopUp()
#13 /srv/mediawiki/php-1.29.0-wmf.5/includes/specials/pagers/ImageListPager.php(433): TimedMediaTransformOutput->toHtml()
#14 /srv/mediawiki/php-1.29.0-wmf.5/includes/pager/TablePager.php(192): ImageListPager->formatValue()
#15 /srv/mediawiki/php-1.29.0-wmf.5/includes/pager/IndexPager.php(442): TablePager->formatRow()
#16 /srv/mediawiki/php-1.29.0-wmf.5/includes/pager/TablePager.php(101): IndexPager->getBody()
#17 /srv/mediawiki/php-1.29.0-wmf.5/includes/specials/SpecialListfiles.php(58): TablePager->getFullOutput()
#18 /srv/mediawiki/php-1.29.0-wmf.5/includes/specialpage/SpecialPage.php(522): SpecialListFiles->execute()
#19 /srv/mediawiki/php-1.29.0-wmf.5/includes/specialpage/SpecialPageFactory.php(577): SpecialPage->run()
#20 /srv/mediawiki/php-1.29.0-wmf.5/includes/MediaWiki.php(283): SpecialPageFactory::executePath()
#21 /srv/mediawiki/php-1.29.0-wmf.5/includes/MediaWiki.php(851): MediaWiki->performRequest()
#22 /srv/mediawiki/php-1.29.0-wmf.5/includes/MediaWiki.php(512): MediaWiki->main()
#23 /srv/mediawiki/php-1.29.0-wmf.5/index.php(43): MediaWiki->run()
#24 /srv/mediawiki/w/index.php(3): include()
#25 {main}

Maybe a maintenance script could be run to fix this all at once?

Event Timeline

@aaron Is this still a problem, and if so how urgent?
If so, please ping someone in the Global Collab team.

Note related task filed at the same time, with possibly useful patch: T152847: Frequent DB writes on HTTP GET from PageTriage

Krinkle renamed this task from LQT on-the-fly DB corrections causing writes on HTTP GET to TMH should not make DB writes on HTTP GET for its on-the-fly transcode corrections.Mar 9 2021, 11:09 PM
Krinkle moved this task from Untriaged to Older on the Wikimedia-production-error board.

Still seen.

Page view example:

  • GET en.wikipedia.org /wiki/Timeline_of_<something>
Expectation (writes <= 0) by MediaWiki::main not met (actual: 1):
query-m: UPDATE `transcode` SET transcode_time_error = 'X')  [TRX#cefc77]
from /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/TransactionProfiler.php(378)
#0 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/TransactionProfiler.php(250): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer)
#1 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1416): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1298): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(2463): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(780): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(705): WebVideoTranscode::getTranscodeState(ForeignDBFile)
#9 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(887): WebVideoTranscode::isTranscodeReady(ForeignDBFile, string)
#10 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(689): WebVideoTranscode::addSourceIfReady(ForeignDBFile, array, string, array)
#11 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(574): WebVideoTranscode::getLocalSources(ForeignDBFile, array)
#12 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(582): WebVideoTranscode::getSources(ForeignDBFile)
#13 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(356): TimedMediaTransformOutput->getMediaSources()
#14 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(189): TimedMediaTransformOutput->getHtmlMediaTagOutput(array)
#15 /srv/mediawiki/php-1.36.0-wmf.30/includes/Linker.php(615): TimedMediaTransformOutput->toHtml(array)
#16 /srv/mediawiki/php-1.36.0-wmf.30/includes/Linker.php(395): Linker::makeThumbLink2(Title, ForeignDBFile, array, array, boolean, string)
#17 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(5368): Linker::makeImageLink(Parser, Title, ForeignDBFile, array, array, boolean, string, integer)
#18 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(2627): Parser->makeImage(Title, array, LinkHolderArray)
#19 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(2374): Parser->handleInternalLinks2(string)
#20 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(1580): Parser->handleInternalLinks(string)
#21 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(644): Parser->internalParse(string)
#22 /srv/mediawiki/php-1.36.0-wmf.30/includes/content/WikitextContent.php(375): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#23 /srv/mediawiki/php-1.36.0-wmf.30/includes/content/AbstractContent.php(591): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#24 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#25 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#26 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#27 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#28 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#29 /srv/mediawiki/php-1.36.0-wmf.30/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#30 /srv/mediawiki/php-1.36.0-wmf.30/includes/poolcounter/PoolWorkArticleView.php(137): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#31 /srv/mediawiki/php-1.36.0-wmf.30/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#32 /srv/mediawiki/php-1.36.0-wmf.30/includes/page/ParserOutputAccess.php(260): PoolCounterWork->execute()
#33 /srv/mediawiki/php-1.36.0-wmf.30/includes/page/Article.php(762): MediaWiki\Page\ParserOutputAccess->getParserOutput(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreCacheRecord, integer)
#34 /srv/mediawiki/php-1.36.0-wmf.30/includes/page/Article.php(575): Article->generateContentOutput(User, ParserOptions, integer, OutputPage, array)
#35 /srv/mediawiki/php-1.36.0-wmf.30/includes/actions/ViewAction.php(74): Article->view()
#36 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(532): ViewAction->show()
#37 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(316): MediaWiki->performAction(Article, Title)
#38 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(944): MediaWiki->performRequest()
#39 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(548): MediaWiki->main()
#40 /srv/mediawiki/php-1.36.0-wmf.30/index.php(53): MediaWiki->run()
#41 /srv/mediawiki/php-1.36.0-wmf.30/index.php(46): wfIndexMain()
#42 /srv/mediawiki/w/index.php(3): require(string)
#43 {main}

Other examples:

  • GET commons.wikimedia.org /w/index.php?title=Special:NewFiles&mediatype%5B0%5D=VIDEO&wpFormIdentifier=specialnewimages
  • GET commons.wikimedia.org /wiki/Category:Videos_of_<something>
Expectation (writes <= 0) by MediaWiki::main not met (actual: 1):
query-m: UPDATE `transcode` SET transcode_time_error = 'X')  [TRX#595834]
trace
from /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/TransactionProfiler.php(378)
#0 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/TransactionProfiler.php(250): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer)
#1 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1416): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1298): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/Database.php(2463): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(780): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(705): WebVideoTranscode::getTranscodeState(LocalFile)
#9 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(887): WebVideoTranscode::isTranscodeReady(LocalFile, string)
#10 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(689): WebVideoTranscode::addSourceIfReady(LocalFile, array, string, array)
#11 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/WebVideoTranscode/WebVideoTranscode.php(574): WebVideoTranscode::getLocalSources(LocalFile, array)
#12 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(582): WebVideoTranscode::getSources(LocalFile)
#13 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(356): TimedMediaTransformOutput->getMediaSources()
#14 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(247): TimedMediaTransformOutput->getHtmlMediaTagOutput(array)
#15 /srv/mediawiki/php-1.36.0-wmf.30/extensions/TimedMediaHandler/includes/TimedMediaTransformOutput.php(186): TimedMediaTransformOutput->getImagePopUp()
#16 /srv/mediawiki/php-1.36.0-wmf.30/includes/gallery/TraditionalImageGallery.php(167): TimedMediaTransformOutput->toHtml(array)
#17 /srv/mediawiki/php-1.36.0-wmf.30/includes/specials/pagers/NewFilesPager.php(193): TraditionalImageGallery->toHTML()
#18 /srv/mediawiki/php-1.36.0-wmf.30/includes/pager/IndexPager.php(617): NewFilesPager->getEndBody()
#19 /srv/mediawiki/php-1.36.0-wmf.30/includes/specials/SpecialNewFiles.php(144): IndexPager->getBody()
#20 /srv/mediawiki/php-1.36.0-wmf.30/includes/specialpage/SpecialPage.php(645): SpecialNewFiles->execute(NULL)
#21 /srv/mediawiki/php-1.36.0-wmf.30/includes/specialpage/SpecialPageFactory.php(1405): SpecialPage->run(NULL)
#22 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(310): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#23 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(944): MediaWiki->performRequest()
#24 /srv/mediawiki/php-1.36.0-wmf.30/includes/MediaWiki.php(548): MediaWiki->main()
#25 /srv/mediawiki/php-1.36.0-wmf.30/index.php(53): MediaWiki->run()
#26 /srv/mediawiki/php-1.36.0-wmf.30/index.php(46): wfIndexMain()
#27 /srv/mediawiki/w/index.php(3): require(string)
#28 {main}
Krinkle triaged this task as High priority.Mar 9 2021, 11:39 PM
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.

Raising priority since it affects parsing during page views.

Just to clarify, this means that this work should be moved into a job, not a deferredupdate, correct ? As it is a GET, not a POST ?

A post-send deferred would be fine too I suppose. Depends on what's easier to maintain and how complex the state is to recreate and how much/many there would be to change.

If it's literally a single set by primary key that's easy to defer, then a post-send callback seems fine. If it's more involved then it might be more reliable to queue a job instead. The delay of a job would make it important to ensure not only that the write no-ops safely, but also that it won't undo a different write that happened meanwhile.

Minor post-send deferred writes from GET are fine if it's rare.

Nothing should be writing to the database from TimedMediaHandler in page views, as far as I know. I think there used to be something years ago but if it's still there it needs to be removed. Is this still a problem that anyone is aware of?

Oof, yeah that explains it. I'll add it to my pile of fixes for this week. :D

I think we can just take this bit out safely, it wasn't reliable and the requeue scripts have to handle the case where a row just got left anyway. :D Patch coming...

Change 973833 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@master] Don't change transcode rows during read operations

https://gerrit.wikimedia.org/r/973833

Change 973833 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Don't change transcode rows during read operations

https://gerrit.wikimedia.org/r/973833

Krinkle awarded a token.

Change 973790 had a related patch set uploaded (by Brion VIBBER; author: Brion VIBBER):

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Don't change transcode rows during read operations

https://gerrit.wikimedia.org/r/973790

Change 973790 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@wmf/1.42.0-wmf.4] Don't change transcode rows during read operations

https://gerrit.wikimedia.org/r/973790

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:21:25Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:22:45Z] <urbanecm@deploy2002> urbanecm and brion and dani: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] synced to the testservers (https://wikitech.wiki

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:32:02Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:973808|Undeploy pilot survey on metawiki (T349854)]], [[gerrit:973790|Don't change transcode rows during read operations (T152851)]], [[gerrit:973791|Fixes to requeueTranscodes to make it easier to batch-fill (T68722)]], [[gerrit:973793|Only include completed transcodes in .m3u8 playlist (T350996)]] (duration: 10m 37s)