Description
Details
Event Timeline
Although not an absolute blocker, we think that T123537: Generate wikitext-based and query-based language models for TextCat should be done before this. Maybe it should block this?
I don't think T123537 is a blocker for doing A/B tests on enwiki. We have query-based models for the languages relevant to enwiki, so we don't need the wikitext-based models.
Change 268048 had a related patch set uploaded (by EBernhardson):
A/B/C test of control vs textcat vs accept-lang textcat
I was looking at this after a comment Stas made about Italian, and I realized that the set of languages currently in LM-query under TextCat is not the ideal one for this test.
Portuguese and Japanese are missing, which are minor issues, because there are not many Portuguese or Japanese queries on enwiki. Hebrew, Armenian, Georgian, Tamil, and Telegu are present—they won't do much, but they won't hurt.
However, French and German are present, and they both tend to get many more false positives than true positives. Not a ton, but they will bring the overall performance down on enwiki.
As I understand it, the PHP version of TextCat doesn't yet have the ability to specify/limit languages, other than by what's in the requested directory.
Do we want to patch LM-query/ before the A/B test?
The primary outstanding question for this task is how to measure the effectiveness of the test. This was discussed briefly in a sprint planning meeting today, but @EBernhardson and @mpopov didn't come to a conclusion. @mpopov will schedule a meeting to discuss this. Marking as stalled until that's done.
Met & discussed: https://lists.wikimedia.org/pipermail/discovery/2016-April/001043.html
Erik, Trey, David, Kevin, and I met this morning to discuss how we're going to handle data collection for the upcoming TextCat test. A big problem in this particular case is that the system wasn't designed/engineered in a way that's conducive for cross-wiki logging / session tracking. And recently we even lost the ability to use the referrer info to see which page the user came from when visiting another wiki page when going between wikis. (I was told this was done for user privacy reasons.)
Erik said he had recently implemented a click event in the TestSearchSatisfaction2 schema that we might be able to hook into to measure clickthrough rate for users who are eligible for TextCat language detection & get shown results in the language their non-English query probably is written in. Whether we use this and how much we rely on this particular method of measuring whether TextCat is successful (beyond just measuring how it impacts the zero results rate) depends on the validation of the click events and how they compare to page visit events (which cannot be fired in an interwiki context).
We also discussed an alternative approach which uses web requests with the caveat being that if a user is selected for the test once, they'll be selected every time. So if a particular IP+UA combination is part of the test and performs 2 million searches (as is sometimes the case), then we'll have to do some very careful filtering which will also exclude some completely valid use cases (a computer lab in a school or a country with only 2 public IP addresses). But we're shooting for being able to use TestSearchSatisfaction2
Will add validation of click events in TestSearchSatisfaction2 (T132706) as a blocker.
After discussion the way we are running this test has slightly changed. The provided patch above does a backend only test which doesn't include as much data as we would like to analyze with. Will re-work to run a test using our frontend search satisfaction schema.
Change 287674 had a related patch set uploaded (by EBernhardson):
Allowing triggering user tests from query parameter
@mpopov What additional metrics should we collect into the satisfaction schema for users in the textcat test?
Some i'm thinking might be useful:
- # of interwiki results provided
- boolean indicating if click event was interwiki or not
These might be unnecessary though, we mostly are just looking at if the click through more or not?
Change 287677 had a related patch set uploaded (by EBernhardson):
Add textcat subtest
Change 287674 merged by jenkins-bot:
Allowing triggering user tests from query parameter
Change 288313 had a related patch set uploaded (by EBernhardson):
Adjust textcat data collection for AB test
This should now be ready to go. Ideally we want to ship this in the thurs afternoon swat window
Change 288465 had a related patch set uploaded (by EBernhardson):
Adjust textcat data collection for AB test
Change 288509 had a related patch set uploaded (by EBernhardson):
Add textcat subtest
Change 268048 merged by jenkins-bot:
A/B/C test of control vs textcat vs accept-lang textcat
Mentioned in SAL [2016-05-12T23:54:13Z] <dereckson@tin> Synchronized php-1.28.0-wmf.1/extensions/CirrusSearch/includes/CirrusSearch.php: Adjust textcat data collection for AB test (T121542) (duration: 00m 26s)