php tests/phpunit/phpunit.php tests/phpunit/languages/LanguageIntegrationTest.php Using PHP 7.4.26 PHPUnit 8.5.21 by Sebastian Bergmann and contributors. ............................................................... 63 / 690 ( 9%) ............................................................... 126 / 690 ( 18%) ............................................................... 189 / 690 ( 27%) ............................................................... 252 / 690 ( 36%) ............................................................... 315 / 690 ( 45%) ............................................................... 378 / 690 ( 54%) ............................................................... 441 / 690 ( 63%) ............................................................... 504 / 690 ( 73%) ............................................................... 567 / 690 ( 82%) ..........FFFFF................................................ 630 / 690 ( 91%) ............................................................ 690 / 690 (100%) Time: 4.04 seconds, Memory: 175.00 MB There were 5 failures: 1) LanguageIntegrationTest::testGetNamespaces with data set "Default configuration" ('en', array(), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Project', 'Project_talk')) Failed asserting that two arrays are equal. --- Expected +++ Actual @@ @@ 15 => 'Category_talk' 4 => 'Project' 5 => 'Project_talk' + 828 => 'Module' + 829 => 'Module_talk' + 2600 => 'Topic' ) /Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092 /Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456 === Logs generated by test case [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"} [wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false} [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one [] === 2) LanguageIntegrationTest::testGetNamespaces with data set "Custom project NS + extra" ('en', array('Wikipedia', array('Borderlands', 'Borderlands_talk')), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Wikipedia', 'Wikipedia_talk', 'Borderlands', 'Borderlands_talk')) Failed asserting that two arrays are equal. --- Expected +++ Actual @@ @@ 5 => 'Wikipedia_talk' 100 => 'Borderlands' 101 => 'Borderlands_talk' + 828 => 'Module' + 829 => 'Module_talk' + 2600 => 'Topic' ) /Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092 /Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456 === Logs generated by test case [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"} [wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false} [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one [] === 3) LanguageIntegrationTest::testGetNamespaces with data set "Custom project NS and talk + extra" ('en', array('Wikipedia', 'Wikipedia_drama', array('Borderlands', 'Borderlands_talk')), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Wikipedia', 'Wikipedia_drama', 'Borderlands', 'Borderlands_talk')) Failed asserting that two arrays are equal. --- Expected +++ Actual @@ @@ 5 => 'Wikipedia_drama' 100 => 'Borderlands' 101 => 'Borderlands_talk' + 828 => 'Module' + 829 => 'Module_talk' + 2600 => 'Topic' ) /Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092 /Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456 === Logs generated by test case [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"} [wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false} [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one [] === 4) LanguageIntegrationTest::testGetNamespaces with data set "Ukrainian default" ('uk', array(), array('Медіа', 'Спеціальна', 'Обговорення', 'Користувач', 'Обговорення_користувача', 'Файл', 'Обговорення_файлу', 'MediaWiki', 'Обговорення_MediaWiki', 'Шаблон', 'Обговорення_шаблону', 'Довідка', 'Обговорення_довідки', 'Категорія', 'Обговорення_категорії', '', 'Project', 'Обговорення_Project')) Failed asserting that two arrays are equal. --- Expected +++ Actual @@ @@ 0 => '' 4 => 'Project' 5 => 'Обговорення_Project' + 828 => 'Модуль' + 829 => 'Обговорення_модуля' + 2600 => 'Тема' ) /Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092 /Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456 === Logs generated by test case [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"} [wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false} [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache::isExpired(uk): cache missing, need to make one [] [localisation] [debug] LocalisationCache::recache: got localisation for uk from source [] === 5) LanguageIntegrationTest::testGetNamespaces with data set "Ukrainian custom NS" ('uk', array('Вікіпедія'), array('Медіа', 'Спеціальна', 'Обговорення', 'Користувач', 'Обговорення_користувача', 'Файл', 'Обговорення_файлу', 'MediaWiki', 'Обговорення_MediaWiki', 'Шаблон', 'Обговорення_шаблону', 'Довідка', 'Обговорення_довідки', 'Категорія', 'Обговорення_категорії', '', 'Вікіпедія', 'Обговорення_Вікіпедії')) Failed asserting that two arrays are equal. --- Expected +++ Actual @@ @@ 0 => '' 4 => 'Вікіпедія' 5 => 'Обговорення_Вікіпедії' + 828 => 'Модуль' + 829 => 'Обговорення_модуля' + 2600 => 'Тема' ) /Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092 /Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456 === Logs generated by test case [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"} [wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false} [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache using store LCStoreNull [] [localisation] [debug] LocalisationCache::isExpired(uk): cache missing, need to make one [] === FAILURES! Tests: 690, Assertions: 888, Failures: 5. You should really speed up these slow tests (>50ms)... 1. 218ms to run LanguageIntegrationTest:testGetNamespaceAliasesReal 2. 54ms to run LanguageIntegrationTest:testGetNamespaces with data set "Ukrainian default" 3. 50ms to run LanguageIntegrationTest:testFormatNum with data set #42
Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
phpunit: Use submap assertion for LanguageIntegrationTest | mediawiki/core | master | +1 -1 |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | None | T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min | |||
Open | pmiazga | T50217 Speed up MediaWiki PHPUnit build by running integration tests in parallel | |||
Resolved | kostajh | T297078 PHPUnit tests should pass when run on their own | |||
Resolved | Krinkle | T297079 LanguageIntegrationTest.php fails when run on its own |
Event Timeline
Change 743909 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/core@master] phpunit: Use submap assertion for LanguageIntegrationTest
Change 743909 merged by jenkins-bot:
[mediawiki/core@master] phpunit: Use submap assertion for LanguageIntegrationTest
I think loosening this test is fine (as the patch does), however it's not obvious to me that there isn't a DI-bug lurking here.
@kostajh If I understand correctly, you're observing that when a non-default extension with extra namespaces is installed, and then running this test in isolation, the extension's namespaces end up in the actual output but not the expected output. And when running all tests, the extra namespaces are both expected and actually there?
That seems like a general mismatching problem that could affect production runtime as well if we're inconsistently involving global state. Or it might be an issue with the test not properly injecting or considering the same service for both sides of the assertion.
For this one case, it seems we're lucky since the global state is only adding data, not changing or removing any, but we have plenty of features and use cases where we do. I suspect we have already solved the general case here as otherwise we'd be seeing these failures more often, which makes me think there's something wrong with this test?
Right
And when running all tests, the extra namespaces are both expected and actually there?
Maybe. One motivation in T297287: More verbose / prettier output for PHPUnit is to print out the tests being run, because I'm not sure if that test is executed by the default php tests/phpunit/phpunit invocation. I mean, I think it is, but can't be totally sure. If it is being run alongside all the other tests, then some other test must be modifying global state in such a way that it is removing extra namespaces when they are accessed in this test.
That seems like a general mismatching problem that could affect production runtime as well if we're inconsistently involving global state. Or it might be an issue with the test not properly injecting or considering the same service for both sides of the assertion.
Right
For this one case, it seems we're lucky since the global state is only adding data, not changing or removing any, but we have plenty of features and use cases where we do. I suspect we have already solved the general case here as otherwise we'd be seeing these failures more often, which makes me think there's something wrong with this test?
Certainly possible, yes. There are NamespaceInfoTest failures noted in T297078#7548882, and there's probably something about how we are setting up the NamespaceInfo service and overriding it that is not working as intended.