Specific issues where an Analytics dataset has incorrect, missing, or malformed data or shows an anomaly which might be caused by such data. Not for general work on data quality processes or monitoring.
(Project tag requested in T362839.)
Details
Details
Description
Yesterday
Yesterday
CodeReviewBot added a project to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report: Patch-For-Review.
Mon, Aug 12
Mon, Aug 12
Krinkle added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Hm.. it seems the "Other" bucket has grown slightly larger than our predictions of 0.26% prediction at T342267#9998984. That could be fine, but wanted to share it in case it's surprising:
Milimetric added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
the new graphs are up. The pivot transformation failed for all the browser family reports, so I'm still fixing that. But, for example, we can now compare these two:
gerritbot added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Change #1062044 merged by Milimetric:
[analytics/refinery@master] Remove scripts related to old hive version
gerritbot added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Change #1062044 had a related patch set uploaded (by Milimetric; author: Milimetric):
[analytics/refinery@master] Remove scripts related to old hive version
Fri, Aug 9
Fri, Aug 9
Milimetric added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
The backfill job should be done sometime this weekend, and I'll rerun the weekly job then.
Wed, Aug 7
Wed, Aug 7
Milimetric added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
For reference, I cleared all the tasks that this dag ran, and that will refresh data for 2 years. We can decide then if we want to do the full history:
Milimetric added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Status update on this: the new job is running, I'm going to keep it here until we vet the data. But new data should start showing up right away, and we can compare dashboards side by side and day by day:
gerritbot added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Change #1049281 merged by Milimetric:
[analytics/refinery@master] Implement new way to aggregate browser statistics
Mon, Aug 5
Mon, Aug 5
gerritbot added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Change #1059914 merged by Milimetric:
[analytics/analytics.wikimedia.org@master] Add temporary dashboard pointing to old data
gerritbot added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Change #1059914 had a related patch set uploaded (by Milimetric; author: Milimetric):
[analytics/analytics.wikimedia.org@master] Add temporary dashboard pointing to old data
Thu, Aug 1
Thu, Aug 1
Dusan_Krehel updated the task description for T370108: Missed pageview data over API.
Dusan_Krehel updated the task description for T370108: Missed pageview data over API.
Wed, Jul 31
Wed, Jul 31
Mon, Jul 29
Mon, Jul 29
Milimetric moved T342267: Investigate surprising "10% Other" portion of Analytics Browsers report from Sign Off to To Deploy on the Data Products (Data Products Sprint 16) board.
great, moving this to get deployed. Steps will be:
Krinkle added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
👍 This is great!
Snwachukwu added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
Next steps would be to rerun any affected downstream jobs.
Snwachukwu added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
So I reran mediawiki_history_denormalize airflow dag to re-generate the snapshot for 2024-06 and also reran mediawiki_history_check_denormalize. I did a check using the same query @nshahquinn-wmf ran. We don't have any more duplicates.
Fri, Jul 26
Fri, Jul 26
gerritbot added a comment to T371099: No longer use removed cuc_actiontext column in analytics/refinery.
Change #1057221 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):
[analytics/refinery@master] Don't select cuc_actiontext from cu_changes for sqoop
Thu, Jul 25
Thu, Jul 25
Snwachukwu added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
So I digged further by looking at the airflow job to see if it ran twice for any reason and I think I found the culprit. The job indeed ran twice.
The first run reported this error which we've seen before in this ticket T342911. Skein job ran and failed with this error:
Snwachukwu added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
I tried to rerun the job for one of the small wiki_db (tetwiki) with duplicate revision record. using this command:
Wed, Jul 24
Wed, Jul 24
DVrandecic moved T364872: Unique devices per country spikes on wikifunctions from To triage to No current plans / External on the Abstract Wikipedia team board.
Mon, Jul 22
Mon, Jul 22
Milimetric added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
- wmf.mediawiki_history: duplicate revision/create records indeed exist, some have 4 copies and some 2 copies but all spot-checked duplicates come in even numbers
- wmf_raw.mediawiki_revision: does not show the same duplication
- analytics mysql replicas: the pages those revisions belong to were moved and had some delete/restore and delete/revision actions in the logging table
- cloud replicas: agrees with analytics replicas
Ottomata updated subscribers of T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
Fri, Jul 19
Fri, Jul 19
Milimetric moved T342267: Investigate surprising "10% Other" portion of Analytics Browsers report from Code Review / Tech Input to To Deploy on the Data Products (Data Products Sprint 16) board.
ok, moving to ready to deploy. I'm going to ping @Krinkle one more time for data review. I executed this as I was testing and the results are available in milimetric.browser_general_test. You can query this like this:
Jdforrester-WMF added a project to T364872: Unique devices per country spikes on wikifunctions : Abstract Wikipedia team.
[…]
mpopov added a comment to T364872: Unique devices per country spikes on wikifunctions .
select normalized_host.project_class, ip, count(1) as view_count from pageview_actor where year = 2024 and month = 6 and day = 28 and http_status = '301' and agent_type = 'user' and uri_path = '/w/index.php' and regexp_like(uri_query, 'title=Special%3AGlobalUsage') and (is_redirect_to_pageview or is_pageview) group by 1, 2 order by view_count desc limit 1000
mpopov added a comment to T364872: Unique devices per country spikes on wikifunctions .
It also means there's nobody to ask to fix the behavior. I believe this requires engineering help from DPE.
mpopov added a comment to T364872: Unique devices per country spikes on wikifunctions .
@Mayakp.wiki: Special:GlobalUsage comes from Extension:GlobalUsage (GlobalUsage), which is a volunteer-authored extension.
Thu, Jul 18
Thu, Jul 18
mforns added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Heya @Milimetric, sorry for taking so long to review this.
I left a comment and a +1, I think that the code looks great and that we can deploy this 👍
This new query is so cool! Kudos :-)
Wed, Jul 17
Wed, Jul 17
Tue, Jul 16
Tue, Jul 16
VirginiaPoundstone moved T359004: arywiki view stats too low for agent = user? from Radar (other teams) to To be discussed on the Data Products board.
Mon, Jul 15
Mon, Jul 15
Jul 12 2024
Jul 12 2024
Milimetric added a comment to T342267: Investigate surprising "10% Other" portion of Analytics Browsers report.
Ok, sent updated code, it's fast now due to a CACHE statement, but that doesn't change the query plan which is still absolutely nuts, check this out:
Jul 11 2024
Jul 11 2024
nshahquinn-wmf added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
This may help in diagnosing the problem: looking at the snapshot, the number of duplicates is not uniform across event_timestamp. There are almost none until 2014, and then the number generally increases until the most recent month.