User story
As the Growth team, I want to understand the impact of adding a new Section-Level structured task, because I want to know if we should continue to rollout this task, rollback, or make improvements.
As the Structured Data team, I want to understand the impact of adding a new Section-Level structured task, because I need to add details to the SDAW report (due September 1, 2023).
Not an A/B test
Similarly as we've done previously for Add a Link in T286816 and Article-level Add an Image in T311531, we're interested in understanding the effect of the section-level "add an Image" task. Unlike those other two tasks, we do not plan to run the section-level "add an image" task in an A/B test. There are a few reasons for choosing a different approach:
- The task will be very similar to original "add an image" task, so results are likely to be similar.
- We don't want to expose this as the first task suggested to newcomers, since we see it as a task that newcomers should "level up" to once they are familiar with the article-level "add an image" task.
- Since this won't be a task provided to all newcomers, it would be a challenge to gather enough data to come to a statistically significant conclusion.
- Due to the shift in focus for the upcoming Annual Plan, it's unlikely we will be able to devote time to iterating on improvements based on experiment findings. Instead we will try to iterate more quickly on feedback from user testers and Growth pilot wiki feedback.
Rather than running a lengthy A/B test, we will instead set Leading indicators prior to release and monitor these on an ongoing basis.
Leading indicators
We define the following leading indicators and plans for action:
- Task completion rate. We compare this to the completion rate of the article-level Add an Image task and we expect it to be roughly similar. If it is significantly lower, we run a funnel analysis to identify the drop-off point (unless it's the caption stage as described below).
- Caption stage bounce rate. Similar criteria as for the Task completion rate.
- Revert rates. Similar criteria as for the Task completion rate, but we also plan to monitor the revert rates for the copy edit and Add a Link tasks for comparison.
In addition, we plan to monitor the following measurements to get a sense of the adoption rate of the task:
- Number of users who have the task selected.
- Number of tasks initiated.
- Number of tasks completed.
Acceptance Criteria
- Share Section-Level Image leading indicator measurement plans on Mediawiki
- Share summary of findings on Mediawiki with the Growth team