User Details
User Details
- User Since
- Apr 16 2015, 4:17 PM (485 w, 2 d)
- Availability
- Available
- LDAP User
- Neil Shah-Quinn (WMF)
- MediaWiki User
- Neil Shah-Quinn (WMF) [ Global Accounts ]
Yesterday
Yesterday
nshahquinn-wmf moved T362594: Update the editor_month table with an Airflow job from Waiting on others to Doing on the Movement-Insights board.
nshahquinn-wmf moved T362595: Update the new_editors table with an Airflow job from Waiting on others to Doing on the Movement-Insights board.
nshahquinn-wmf updated the task description for T314541: Update the active_editors table with an Airflow job.
Fri, Aug 2
Fri, Aug 2
nshahquinn-wmf changed Source Repo from https://github.com/wikimedia-research/canonical-data to https://gitlab.wikimedia.org/repos/movement-insights/canonical-data on Analytics-Canonical-Data.
nshahquinn-wmf renamed T273197: Update all Wmfdata-Python run functions to have consistent API from Update run functions to accept a filepath as well as a string for the SQL command to Update all Wmfdata-Python run functions to have consistent API.
nshahquinn-wmf added a comment to T369325: [MI 3] Investigate trends in movement metrics.
This week, I:
- Had exploratory conversations with Jan Eissfeldt, Becky Maung, Isaac Johnson, Nino Hemmer, Kinneret Gordon, Marshall Miller, and Sonja Perry
nshahquinn-wmf updated the task description for T371651: Update Movement Insights tables and movement metrics code to accomodate temporary users.
Thu, Aug 1
Thu, Aug 1
nshahquinn-wmf added a project to T371560: REQUEST: A useful namespace_canonical_name column in wmf_raw.mediawiki_project_namespace_map: Analytics-Canonical-Data.
This would definitely be very useful.
Wed, Jul 31
Wed, Jul 31
@matej_suchanek thank you very much! I didn't realize that and that does seem to be a very big part of the puzzle.
Tue, Jul 30
Tue, Jul 30
nshahquinn-wmf added a comment to T363125: sustainability of wikitech.wikimedia.org.
Sat, Jul 27
Sat, Jul 27
nshahquinn-wmf added a comment to T371157: Some new users do not have account creation log events.
It's possible this is expected behavior, but I couldn't find any documentation saying so.
Fri, Jul 26
Fri, Jul 26
nshahquinn-wmf added a comment to T369325: [MI 3] Investigate trends in movement metrics.
Weekly updates:
- Had exploratory conversations with Irene Florez, Leila Zia, Kate Zimmerman, Maryana Pinchuk, Sam Patton, Jaime Anstee, and Zack McCune
Thu, Jul 25
Thu, Jul 25
nshahquinn-wmf moved T314541: Update the active_editors table with an Airflow job from Planned Next 2 weeks to Doing on the Movement-Insights board.
nshahquinn-wmf moved T369327: Choose topics for first two trend investigations from Planned Next 2 weeks to Doing on the Movement-Insights board.
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
I suspect that this tracking task is no longer useful.
Wed, Jul 24
Wed, Jul 24
Tue, Jul 23
Tue, Jul 23
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
Mon, Jul 22
Mon, Jul 22
nshahquinn-wmf moved T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy from Done to Needs Review on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.
nshahquinn-wmf added a comment to T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy.
@BTullis that makes sense!
nshahquinn-wmf added a parent task for T370713: Upgrade to Pyspark ≥ 3.5: T370705: Upgrade to Pandas ≥ 2 in Conda-Analytics.
nshahquinn-wmf added a subtask for T370705: Upgrade to Pandas ≥ 2 in Conda-Analytics: T370713: Upgrade to Pyspark ≥ 3.5.
nshahquinn-wmf added a parent task for T370712: Upgrade to Pyspark ≥ 3.4: T370710: Upgrade to Numpy ≥ 1.24 in Conda-Analytics.
nshahquinn-wmf moved T362595: Update the new_editors table with an Airflow job from Doing to Waiting on others on the Movement-Insights board.
Sat, Jul 20
Sat, Jul 20
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
Fri, Jul 19
Fri, Jul 19
nshahquinn-wmf added a comment to T369325: [MI 3] Investigate trends in movement metrics.
I've been doing some initial exploratory work:
- conversations with members of Movement Insights and Morten Warncke-Wang
- looking through recent movement metrics output and brainstorming ideas
- scheduling stakeholder conversations (I have six scheduled for the coming week)
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
Thu, Jul 18
Thu, Jul 18
nshahquinn-wmf renamed T365211: Standardize time range of Wikicharts charts from Wikicharts charts have a fixed start date to Standardize time range of Wikicharts charts.
nshahquinn-wmf moved T370196: Retire the old movement metrics intermediate tables from Backlog to Code improvements on the Movement-Metrics board.
Wed, Jul 17
Wed, Jul 17
nshahquinn-wmf triaged T370225: Provide recommendations to Data Engineering on Airflow documentation improvements as Medium priority.
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
Tue, Jul 16
Tue, Jul 16
nshahquinn-wmf triaged T370196: Retire the old movement metrics intermediate tables as Medium priority.
nshahquinn-wmf lowered the priority of T362595: Update the new_editors table with an Airflow job from High to Medium.
nshahquinn-wmf lowered the priority of T362594: Update the editor_month table with an Airflow job from High to Medium.
nshahquinn-wmf lowered the priority of T362593: Update the content_interactions table with an Airflow job from High to Medium.
nshahquinn-wmf lowered the priority of T333225: Migrate the movement_metrics ETL jobs to Airflow from High to Medium.
nshahquinn-wmf lowered the priority of T314541: Update the active_editors table with an Airflow job from High to Medium.
nshahquinn-wmf updated the task description for T362593: Update the content_interactions table with an Airflow job.
nshahquinn-wmf updated the task description for T362595: Update the new_editors table with an Airflow job.
nshahquinn-wmf placed T361143: Update the main unique devices documentation up for grabs.
nshahquinn-wmf removed a project from T362594: Update the editor_month table with an Airflow job: Patch-For-Review.
nshahquinn-wmf updated the task description for T362594: Update the editor_month table with an Airflow job.
Mon, Jul 15
Mon, Jul 15
nshahquinn-wmf updated the task description for T362594: Update the editor_month table with an Airflow job.
nshahquinn-wmf moved T362594: Update the editor_month table with an Airflow job from Doing to Waiting on others on the Movement-Insights board.
I figured out the anomaly, and I'm waiting until it's fixed to do another backfill of the table: T369851.
nshahquinn-wmf moved T370090: Make the movement-metrics repo a Python package from Backlog to Code improvements on the Movement-Metrics board.
Fri, Jul 12
Fri, Jul 12
Clean up work is never done, but we've finished this batch!
nshahquinn-wmf closed T362130: Clean up the movement-metrics codebase, a subtask of T359207: Improve the delivery of the movement movements (SDS 2.6.2), as Resolved.
nshahquinn-wmf added a comment to T369325: [MI 3] Investigate trends in movement metrics.
No update.
Thu, Jul 11
Thu, Jul 11
nshahquinn-wmf added a comment to T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
This may help in diagnosing the problem: looking at the snapshot, the number of duplicates is not uniform across event_timestamp. There are almost none until 2014, and then the number generally increases until the most recent month.
nshahquinn-wmf renamed T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions from NEW BUG REPORT Mediawiki_history contains duplicate rows for many revisions to NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions.
nshahquinn-wmf triaged T369851: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions as High priority.
Wed, Jul 10
Wed, Jul 10
nshahquinn-wmf added a comment to T361329: Convert Wikicharts to use regularly-calculated metrics and canonical data instead of static files.
MR 21, which is in review, removes the unused files. After it's merged, we can resolve this.
nshahquinn-wmf closed T368218: Move the code from the calculate notebook into a Python module, a subtask of T362130: Clean up the movement-metrics codebase, as Resolved.
nshahquinn-wmf closed T368218: Move the code from the calculate notebook into a Python module as Resolved.
@Hghani's MR has been merged!
nshahquinn-wmf moved T365211: Standardize time range of Wikicharts charts from Backlog to Visualization on the Movement-Metrics board.
nshahquinn-wmf moved T365086: Fetch content counts from AQS using internal API endpoint from Backlog to Code improvements on the Movement-Metrics board.
nshahquinn-wmf moved T365511: Wikicharts content gap charts should use same style as other charts from Backlog to Visualization on the Movement-Metrics board.
nshahquinn-wmf moved T368217: Ensure consistent formatting of the movement-metrics codebase from Backlog to Code improvements on the Movement-Metrics board.
nshahquinn-wmf moved T368218: Move the code from the calculate notebook into a Python module from Backlog to Code improvements on the Movement-Metrics board.
nshahquinn-wmf moved T368995: Move metrics in wikicharts/resources/data to metrics folder from Backlog to Code improvements on the Movement-Metrics board.
nshahquinn-wmf moved T369396: Remove chart data from monthly report notebook from Backlog to Code improvements on the Movement-Metrics board.
Tue, Jul 9
Tue, Jul 9
nshahquinn-wmf added a comment to T362594: Update the editor_month table with an Airflow job.
I ran the second backfill, but ran into another weird anomaly 😩
Sat, Jul 6
Sat, Jul 6
nshahquinn-wmf updated the task description for T369396: Remove chart data from monthly report notebook.
nshahquinn-wmf updated the task description for T369396: Remove chart data from monthly report notebook.
nshahquinn-wmf updated the task description for T369396: Remove chart data from monthly report notebook.
Jul 5 2024
Jul 5 2024
nshahquinn-wmf updated the task description for T362594: Update the editor_month table with an Airflow job.
nshahquinn-wmf updated subscribers of T315024: Creating a Spark session causes a torrent of log spam.
By chance, I discovered that there is a quieter_spark_log4j.properties file in analytics-refinery. It's not quiet enough for our purposes, but we could use it as the base for a configuration we like. Refinery is also deployed automatically to HDFS, so that's a good way to make it widely available.
nshahquinn-wmf renamed T369327: Choose topics for first two trend investigations from Choose topic for first trend investigation to Choose topics for first two trend investigations.
nshahquinn-wmf moved T369325: [MI 3] Investigate trends in movement metrics from Incoming to [FY24/25 Q1-Q2] - Movement Insights on the Movement-Insights board.