@revi @Trizek-WMF @PPham @Dyolf77_WMF @Urbanecm -- we're ready to start on this task, like we discussed in our meetings this week. Let's evaluate the first 50 article on each of our sheets. Please post any questions here. And when you're finished, please post a comment here leaving any overall notes or reactions to the algorithm.

Hopefully you all aren't that far: because I'm lazy, I built a script that generates a page that shows both page title and the image in one place. You can view your own generated file at https://people.wikimedia.org/~urbanecm/growth-team/image-evaluation-aug-2020-T260857/. Hope this helps!

Some general comments after finishing the evaluation:
Most of the suggestions are on animals/flowers, astronomical bodies, biographies and geographical locations in my case:

• Animals/flowers: Mostly wrong suggestions as it suggests image of species in the same genus or family but they are different species.
• Astronomical bodies: This is hard to evaluate since my knowledge about this subject is narrow, but in my impression none in this group is a "2".
• Biographies: Has a high rate of correct suggestion, perhaps because you can just match the name of the article with file's name
• Geographical locations: has the highest rate of correct suggestion, same reason with biographies maybe.

If you want to confirm whether it fits or not, you have to actually follow the file to other languages, read the articles, google search the subject in some cases... overall most of the evaluation would require more than two clicks. Some of them would not be suitable for newcomers - it could be hard for them to navigate through different languages and projects (wikidata, wikispecies, wikicommons, etc.).

Of course it is a promising idea, but let's not forget that the newcomer is not as dedicated to Wikipedia as us, so perhaps they won't want to waste so much time clicking around, but instead want a more immediate and direct result/edit? But this type of suggestion is definitely valuable to us experienced users lol.

Finished the evaluation of ar.wiki suggestions. Suggestions can be improved by:

• hiding low quality/resolution images (they are used as icons),
• hiding images with DR (Deletion Request) tags,
• looking for local images in Other files section (especially for translatable maps/diagrams).

In T260857#6402309, @Urbanecm_WMF wrote:

Hopefully you all aren't that far: because I'm lazy, I built a script that generates a page that shows both page title and the image in one place. You can view your own generated file at https://people.wikimedia.org/~urbanecm/growth-team/image-evaluation-aug-2020-T260857/. Hope this helps!

Thank you Martin! This is really helpful!

Concerning the samples we have to review, after a meeting with Revi and Phuong, we all agreed on the lack of diversity. For instance, Phuong and I have a lot of asteroids, Phuong also says that she has a lot of animals. It doesn't seems to be well balanced. @Miriam, is it normal to have this lack of diversity?

In T260857#6412679, @Trizek-WMF wrote:

In T260857#6402309, @Urbanecm_WMF wrote:

Hopefully you all aren't that far: because I'm lazy, I built a script that generates a page that shows both page title and the image in one place. You can view your own generated file at https://people.wikimedia.org/~urbanecm/growth-team/image-evaluation-aug-2020-T260857/. Hope this helps!

Thank you Martin! This is really helpful!

I'm glad you like it! A code author is always happy to hear someone used it and liked it ;).

Concerning the samples we have to review, after a meeting with Revi and Phuong, we all agreed on the lack of diversity. For instance, Phuong and I have a lot of asteroids, Phuong also says that she has a lot of animals. It doesn't seems to be well balanced. @Miriam, is it normal to have this lack of diversity?

Maybe I'm highly tolerant to lack of diversity, but I don't feel any lack of diversity in my set of images.

Hi all. A quick note that Miriam will be back at work on 2020-09-07 and will likely not be able to respond to your questions until then.

@Trizek-WMF I'm done with the first 50 items. Let me know if there is anything more I should do.

Is it possible to refine by section title to illustrate a section? For instance, an article about a city in which there is a "tourism" section: is the recommendation tool able to find a picture of the local beach (for example) for this section?

@revi @Urbanecm -- could you please put your overall comments on the results here?

@Miriam -- we are finished evaluating this round of image recommendations. For six languages, we evaluated 50 random matches. The results are in the sheets in this workbook: https://docs.google.com/spreadsheets/d/120ux_OPnqGWwrufgAvoBFBqDiPGquK4Xgd4UevLFuu0/edit#gid=383843253. There is a graph of results in the "Summary" tab.

We classified the matches into these classifications:

Here are the toplines:

• Depending on the wiki, 20-40% of matches were 2s. We have to talk about what accuracy level we think is minimal for the newcomer experience.
• The number of 1s calls into question many design challenges: how much information can we give users to investigate these matches? For instance, if it's the article on Albert Einstein in Arabic Wikipedia, and the suggested image is of Albert Einstein's childhood home, and the name and description of that image are in German, how can the user determine that it's a good match, and not just a random house?
• Depending on the wiki, -2s could be up to 30% of the matches. These seem to be caused largely by this phenomenon: there is an unillustrated article about a specific butterfly species (or asteroid or whatever). Some other wiki has an article about that butterfly, and it has a "butterflies" navbox at the bottom, which uses a certain butterfly image. All butterfly articles in that wiki therefore have that one butterfly image at the bottom, and so it is erroneously recommended for the wrong species.
• The specific sheets for each language contain many notes that will help us refine this algorithm. I recommend reading through all of them to discover patterns that can be addressed.
• There are also comments above, in this task, listing some clear areas for improvement from the people who did the evaluation.

Here is the graph from the "Summary" tab:

@MMiller_WMF: Hi, the Due Date set for this open task is more than two months ago.
Can you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks.