Below are some notes based on my initial review of existing Citoid instrumentation:

What Citoid-related instrumentation is currently in place? What kinds of questions can we use this instrumentation to help us answer?

VisualEditorFeatureUse includes a number of events that track user engagement with citoid including clicks to open the feature, generate a citation, and insert a citation. These are documented in the VEFU data dictionary and listed below:

• • Window-open-from-command to window-open-from-check: User opens citoid or it is opened after user interaction with reference check dialog.
  • automatic-generate : The "generate" button was clicked to generate a citation based on a URL (or DOI or ISBN)
  • automatic-generate-fail[1]: Automatic citation generation failed and error message is displayed.
  • automatic-generate-fail-searchResults[1]: Automatic citation generation failed because no search results were found for the provided input.
  • automatic-generate-fail-network[1]: Automatic citation generation failed due to a network error while attempting to retrieve the necessary information.
  • automatic-insert: The "insert" button was clicked to insert a Citoid-generated citation into the page
  • Panel-switch: The user switched between tabs or panes within a dialog. For example, switching between “manual” and “automatic”
  • Dialog-Manual-choose: choose to insert manual citation
  • Resuse-choose: The user selected an existing citation to insert again.
• Note: Once a Citoid-generated reference is inserted, it becomes a normal reference and editing it would fall under cite.

[1] It looks like these three automatic-fail- events were recently instrumented by WMDE in T363292.

Other Instrumentation:

• The logstash dashboard created in T368802 can be used to view Citoid requests over time and top requested domains. Data can be viewed by response codes so you can determine the rate at which requests are failing.
• There's also a Citoid grafana dashboard that shows requests over time by input type and HTTP status as well as some performance tracking (latency, memory, CPU)

Of the Research questions documented in T368988, which – if any – are we NOT currently able to answer using the instrumentation that is currently in place?

We should be able to answer most all research questions as currently defined using existing instrumentation. I outlined events and available existing instrumentation in this spec.

The only question that might require additional instrumentation is the following:

Of the citations people attempt to use Citoid to generate, what percentage of these attempts are for various content types (.mov, .pdf, url, etc.)?

This is not currently instrumented. We do know the requested domains from Citoid after we get a response from the server.

Note: If we add additional user workflows to the citoid feature such as offering an action they can take from within Citoid’s error state to cite a source. We will need to add additional instrumentation to track this engagement (likely within VE).

Hi from the Technical Wishes dev team. I just wanted to cross link our recent analytics work in the WMDE-References-FocusArea that's mainly around T362347: Log events for some simple interactions in the VE cite dialogs and the superset dashboard for Citoid usage at https://superset.wikimedia.org/superset/dashboard/518/?native_filters_key=o1vk2wECKCNertKma5Q5UXnFr6-cCk5e_wirjEAd01PYCOEdAadlO3IpkCtKB_OR maybe there's something of use.