@KCVelaga_WMF the event is logged when the "view translation" step is displayed. That means the event will be logged even if the translation is not loaded properly. I believe that for now, it's more important to track the user's behaviour within the tool (i.e. which screens the user visits, in which order etc), than to measure how often the translation works properly. Of course, the latter is also important but I guess we'll need a new event for that. What do you think?

@mforns thanks for taking this up and sharing your thoughts.

Some additional notes:

• For any data related to false positive reports, unless there is a consistent format or some kind of template used, it will hard to scale this across wikis in an automated way.
• For "Potential revert rate average based on revert scores at various thresholds", we can show mean and mode values based on preceeding six months of data.

The report of the analysis is available at this link.
To view, please download the HTML locally and open it with any web browser. I will publish it publicly after privacy review.

There is currently no plan for this. We will observe this as a guardrail, and then consider running a test if see any significant changes.

@abi_ @Wangombe As we tracking several others steps of user flow (apart from just MinT usage itself), I didn't specify mint in the stream name. We can keep it generic for translate extension, which can also used if we expand in the future.

@jsn.sherman I am planning do this on priority during the week of Aug 19. I will on leave on from 8 to 16 Aug.

@PWaigi-WMF This is engineering work, I will do the QA after that, which should be a separate task.

• I missed mentioning pages in the original description, I added a query for that.
• I think excluding editors in 10+ editors seems reasonable, we're excluding about 14 editors and all of LangCom.
• We should remove bots, added conditions for that.
• Also, I realized the initial TSV doesn't have the time spent on incubator data, so I split the output into monthly metrics and project info (including language directionality).

@ngkountas That's a good idea, thanks for prioritizing this. Does it ensure that this event is only logged once the translation is loaded/viewed? We may want to avoid logging this event is cases where the translation is not loaded properly or broken due to some or the other issue.

@CMyrick-WMF I think N=165 who editors across 5+ languages is still quite a high number of editors to exclude. I wonder if we should switch to a defined list or consider some other criteria. Some other options consider are:

I have updated the plan to reflect the status for various fiscal years. I will discuss with @Samwalton9-WMF once he is back about when and how to plan the rest of remaining evaluations.

@jsn.sherman @DMburugu @SonjaPerry @Scardenasmolinar We initially discussed to do this after the June snapshot of revert risk scores become available, which they have now. As we are in the third week of July already, I think it is better to wait until the end of the month, and do the analysis based on the whole month of July. As Automoderator was enabled on trwiki on June 26, for June, we only have about 3-4 days of data to work with, which may not give us the full picture we need. Let me know what you all think.

New task for the FY 24-25 at T370439: [Request] Support needed to understand baseline satisfaction levels for moderator tools

• The first draft of the measurement plan is available for review on this document.
  • The final evaluation method is yet o be decided but that shouldn't be a blocker for the initial steps such as selection of wikis.
• Separate sub-tasks to be created for steps such as selection of wikis, variables for clustering etc.

I can confirm that click events for user selecting an article are being logged. We have ~30+ events so far.

@VirginiaPoundstone that's great, thank you.

@VirginiaPoundstone What is reasonable estimate from your side? We can plan things accordingly for future instrumentation.

@VirginiaPoundstone There is a no fixed timeline, but the sooner the better. For now and until we have this, we will be capturing the required data as a JSON blob in action_context.

@abi_ @Wangombe I have made changes to the instrumentation spec based on last week's discussion. Please review and let me know if you have any feedback.

Summary of activity report until Jul 14

Sorry, the assignment got changed due to a trigger mistakenly.

@Pginer-WMF Yes, not all events will use all the fields. If the target is a machine translation (like in case of MinT for Readers), then target_id is not relevant.

I re-estimated the activity levels based on scores for first four months of 2024. The averages have changed slightly, but an important addition is the mode (most frequent), which is what we should expect to see on most days, and then some spikes.

@Pginer-WMF @ngkountas @abi_ @Wangombe I have listed the data that we usually need to capture for source & target. Please share if there is something else that I might be missing. We can evolve the schema later as well.