@Gimenadelrioriande: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

Some of the target webpages are returning a "non applicable target webpage" error, which complicates collaborators evaluating the tests and writing templates.

Thanks for the reminder @Aklapper!

Nidiah moved this task from To do to Done on the Web2Cit-Research board.

@Nidiah: Hi, if there is nothing else to do in this task, could you please set the task status to resolved? Thanks a lot!

Some examples of manual data with wikicode:

Added normalization for citation template names in section 2.3 Citation template extraction summary in notebook .

Our script is using this column from the list of citation templates already.

URLs having schemes other than http or https or private hosts were excluded.

Added https://phabricator.wikimedia.org/tag/web2cit-research/ to User-Agent

Thanks, Nidia. I would add Web2Cit-Research somewhere, considering that the script may be used in the future, after the project grant has ended. I think it's OK to keep the other information too.

Note that @Kerry_Raymond has found other URLs that are failing with Citoid. She has listed them here, some of which have been listed in our list of problematic URLs as well.

As discussed with @Nidiah, ignoring URLs that don't start with http or https is pending.

I have been using http://caicyt-conicet.gov.ar/; mailto:nidiahernandez@conicet.gov.ar, I can change it for the the proyect's page in MetaWiki or "Web2Cit-Research" is better?

Hi @Aklapper, sorry for the delayed response, I was out of town for a couple of weeks. Thank you for your reminder and for the link!

@Nidiah: Hi! This task has been assigned to you a while ago. Could you maybe share an update? Do you still plan to work on this task, or do you need any help? Thanks!

Thanks, @Mvolz! We will probably be doing this around the end of March. As mentioned, we will properly identify our requests with a custom user agent. Hopefully we won't cause any disruptions, but please let us know in case we do!

In T301510#7745430, @diegodlh wrote:

I've been looking into Citoid API request rate limits.

We access the Citoid API through Wikmedia's RESTBase proxy. I found two 429 HyperSwitch errors for exceeded request rates: https://www.mediawiki.org/wiki/HyperSwitch/errors/rate_exceeded and https://www.mediawiki.org/wiki/HyperSwitch/errors/request_rate_exceeded

Here it says that there is a global limit of up to 200 requests per second, but that individual endpoints may have specific limits. However, the Citoid API documentation doesn't seem to say anything about it.

On the other hand, I found this thread where @Mvolz mentions a "1000/10s (100/s long term, with 1000 burst)" limit.

She also refers to how long requests take and timeouts, but I'm not sure what she means. How does time to response affect request rate limit? Say we make 1000 requests at t=0s of which only 500 have returned a response at t=10s, can we make another 1000-request batch now? Or do pending requests count against our request rate limit?

I've been looking into Citoid API request rate limits.

Maybe also check and ignore some of the cases discussed by email, that would be valid URLs, but that would return error 400 from Citoid:

• the URL scheme is not http or https
• the host is a private IP address

Sure, at the current rhythm, that would save us 22h for T301510.

Indeed, URLs ending in .pdf return 404 error. But they represent only 6.35% of our corpus.

URLs are validated before request using validators.url(url) (see doc: https://validators.readthedocs.io)