LanguageIntegrationTest.php fails when run on its own
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kostajh
	Dec 6 2021, 9:01 AM

Description

php tests/phpunit/phpunit.php tests/phpunit/languages/LanguageIntegrationTest.php

Using PHP 7.4.26
PHPUnit 8.5.21 by Sebastian Bergmann and contributors.

...............................................................  63 / 690 (  9%)
............................................................... 126 / 690 ( 18%)
............................................................... 189 / 690 ( 27%)
............................................................... 252 / 690 ( 36%)
............................................................... 315 / 690 ( 45%)
............................................................... 378 / 690 ( 54%)
............................................................... 441 / 690 ( 63%)
............................................................... 504 / 690 ( 73%)
............................................................... 567 / 690 ( 82%)
..........FFFFF................................................ 630 / 690 ( 91%)
............................................................    690 / 690 (100%)

Time: 4.04 seconds, Memory: 175.00 MB

There were 5 failures:

1) LanguageIntegrationTest::testGetNamespaces with data set "Default configuration" ('en', array(), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Project', 'Project_talk'))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
     15 => 'Category_talk'
     4 => 'Project'
     5 => 'Project_talk'
+    828 => 'Module'
+    829 => 'Module_talk'
+    2600 => 'Topic'
 )

/Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092
/Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456
=== Logs generated by test case
[objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
[wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false}
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
===

2) LanguageIntegrationTest::testGetNamespaces with data set "Custom project NS + extra" ('en', array('Wikipedia', array('Borderlands', 'Borderlands_talk')), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Wikipedia', 'Wikipedia_talk', 'Borderlands', 'Borderlands_talk'))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
     5 => 'Wikipedia_talk'
     100 => 'Borderlands'
     101 => 'Borderlands_talk'
+    828 => 'Module'
+    829 => 'Module_talk'
+    2600 => 'Topic'
 )

/Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092
/Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456
=== Logs generated by test case
[objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
[wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false}
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
===

3) LanguageIntegrationTest::testGetNamespaces with data set "Custom project NS and talk + extra" ('en', array('Wikipedia', 'Wikipedia_drama', array('Borderlands', 'Borderlands_talk')), array('Media', 'Special', '', 'Talk', 'User', 'User_talk', 'File', 'File_talk', 'MediaWiki', 'MediaWiki_talk', 'Template', 'Template_talk', 'Help', 'Help_talk', 'Category', 'Category_talk', 'Wikipedia', 'Wikipedia_drama', 'Borderlands', 'Borderlands_talk'))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
     5 => 'Wikipedia_drama'
     100 => 'Borderlands'
     101 => 'Borderlands_talk'
+    828 => 'Module'
+    829 => 'Module_talk'
+    2600 => 'Topic'
 )

/Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092
/Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456
=== Logs generated by test case
[objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
[wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false}
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
===

4) LanguageIntegrationTest::testGetNamespaces with data set "Ukrainian default" ('uk', array(), array('Медіа', 'Спеціальна', 'Обговорення', 'Користувач', 'Обговорення_користувача', 'Файл', 'Обговорення_файлу', 'MediaWiki', 'Обговорення_MediaWiki', 'Шаблон', 'Обговорення_шаблону', 'Довідка', 'Обговорення_довідки', 'Категорія', 'Обговорення_категорії', '', 'Project', 'Обговорення_Project'))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
     0 => ''
     4 => 'Project'
     5 => 'Обговорення_Project'
+    828 => 'Модуль'
+    829 => 'Обговорення_модуля'
+    2600 => 'Тема'
 )

/Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092
/Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456
=== Logs generated by test case
[objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
[wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false}
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache::isExpired(uk): cache missing, need to make one []
[localisation] [debug] LocalisationCache::recache: got localisation for uk from source []
===

5) LanguageIntegrationTest::testGetNamespaces with data set "Ukrainian custom NS" ('uk', array('Вікіпедія'), array('Медіа', 'Спеціальна', 'Обговорення', 'Користувач', 'Обговорення_користувача', 'Файл', 'Обговорення_файлу', 'MediaWiki', 'Обговорення_MediaWiki', 'Шаблон', 'Обговорення_шаблону', 'Довідка', 'Обговорення_довідки', 'Категорія', 'Обговорення_категорії', '', 'Вікіпедія', 'Обговорення_Вікіпедії'))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
     0 => ''
     4 => 'Вікіпедія'
     5 => 'Обговорення_Вікіпедії'
+    828 => 'Модуль'
+    829 => 'Обговорення_модуля'
+    2600 => 'Тема'
 )

/Users/kostajh/src/mediawiki/w/tests/phpunit/languages/LanguageIntegrationTest.php:2092
/Users/kostajh/src/mediawiki/w/tests/phpunit/MediaWikiIntegrationTestCase.php:456
=== Logs generated by test case
[objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
[wfDebug] [debug] LanguageClassesTestCase::setUp could not extract a language name out of LanguageIntegrationTest failling back to 'en' {"private":false}
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache using store LCStoreNull []
[localisation] [debug] LocalisationCache::isExpired(uk): cache missing, need to make one []
===

FAILURES!
Tests: 690, Assertions: 888, Failures: 5.


You should really speed up these slow tests (>50ms)...
 1. 218ms to run LanguageIntegrationTest:testGetNamespaceAliasesReal
 2. 54ms to run LanguageIntegrationTest:testGetNamespaces with data set "Ukrainian default"
 3. 50ms to run LanguageIntegrationTest:testFormatNum with data set #42

Details

	Subject	Repo	Branch	Lines +/-
	phpunit: Use submap assertion for LanguageIntegrationTest	mediawiki/core	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min
Open	pmiazga	T50217 Speed up MediaWiki PHPUnit build by running integration tests in parallel
Resolved	kostajh	T297078 PHPUnit tests should pass when run on their own
Resolved	Krinkle	T297079 LanguageIntegrationTest.php fails when run on its own

Event Timeline

kostajh created this task.Dec 6 2021, 9:01 AM

Restricted Application added a subscriber: Base. · View Herald TranscriptDec 6 2021, 9:01 AM

Change 743909 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] phpunit: Use submap assertion for LanguageIntegrationTest

https://gerrit.wikimedia.org/r/743909

gerritbot added a project: Patch-For-Review.Dec 6 2021, 9:04 AM

Change 743909 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Use submap assertion for LanguageIntegrationTest

https://gerrit.wikimedia.org/r/743909

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.12; 2021-12-06).Dec 6 2021, 1:00 PM

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2021, 1:10 PM

I think loosening this test is fine (as the patch does), however it's not obvious to me that there isn't a DI-bug lurking here.

@kostajh If I understand correctly, you're observing that when a non-default extension with extra namespaces is installed, and then running this test in isolation, the extension's namespaces end up in the actual output but not the expected output. And when running all tests, the extra namespaces are both expected and actually there?

That seems like a general mismatching problem that could affect production runtime as well if we're inconsistently involving global state. Or it might be an issue with the test not properly injecting or considering the same service for both sides of the assertion.

For this one case, it seems we're lucky since the global state is only adding data, not changing or removing any, but we have plenty of features and use cases where we do. I suspect we have already solved the general case here as otherwise we'd be seeing these failures more often, which makes me think there's something wrong with this test?

Krinkle closed this task as Resolved.Dec 6 2021, 3:33 PM

Krinkle claimed this task.

In T297079#7549996, @Krinkle wrote:

I think loosening this test is fine (as the patch does), however it's not obvious to me that there isn't a DI-bug lurking here.

@kostajh If I understand correctly, you're observing that when a non-default extension with extra namespaces is installed, and then running this test in isolation, the extension's namespaces end up in the actual output but not the expected output.

Right

And when running all tests, the extra namespaces are both expected and actually there?

Maybe. One motivation in T297287: More verbose / prettier output for PHPUnit is to print out the tests being run, because I'm not sure if that test is executed by the default php tests/phpunit/phpunit invocation. I mean, I think it is, but can't be totally sure. If it is being run alongside all the other tests, then some other test must be modifying global state in such a way that it is removing extra namespaces when they are accessed in this test.