Wikipedia:Link rot/URL change requests
This page is for requesting modifications to URLs, such as marking dead or changing to a new domain. Some bots are designed to fix link rot; they can be notified here. These include InternetArchiveBot and WaybackMedic. This page can be monitored by bot operators from other language wikis since URL changes are universally applicable.
finlex.fi
editThis section is pinned and will not be automatically archived. |
Finlex.fi URLs aren't dead but for some reason InternetArchiveBot keeps adding archived URLs for them. This was brought up at meta:User_talk:InternetArchiveBot#Finlex.fi_URLs_aren't_dead a month ago: Bot's edits: [1], [2], [3]. Some URLs it tagged as dead but are actually working: [4], [5], [6].
Those finlex.fi URLs that now have both a working URL and an archive URL should be tagged with the |url-status=live
tag, and could someone try to tell IABot that Finlex is live? Thanks. 2001:14BA:9C94:9A00:E866:DADA:1085:E3D9 (talk) 09:28, 17 March 2024 (UTC)
- Just noticed that this same issue is being discussed at fi.wikipedia: fi:Wikipedia:Kahvihuone_(tekniikka)#Botti_hakee_arkistosta_kumottuja_lakeja 2001:14BA:9C94:9A00:E866:DADA:1085:E3D9 (talk) 09:41, 17 March 2024 (UTC)
- The site has a "Are you human?" check box (CloudFlare). This is causing the bot to think it's a dead site. I logged into iabot.org and changed the domain to "Subscription" status and that will cause the bot to avoid this domain, it won't set live or dead. My bot WaybackMedic has capabilities to bypass CloudFlare. I can try to process this domain and see what happens. My bot also has a feature "make live" ie. convert a citation from dead to live state. Unfortunately my bot only works on English Wikipedia. I'll let you know what happens. -- GreenC 15:13, 17 March 2024 (UTC)
- Unfortunately, this site has maximum security enabled, none of my tools can get through. It started happening in late January 2024. I don't know what to do because no bot is able to determine if a link is live or dead. And no archive service such as WaybackMachine is able to archive a page. Only humans can get through, and they need to solve a captcha. It might be worthwhile waiting to see if they relax security in the future, since this is a recent development. -- GreenC 00:40, 19 March 2024 (UTC)
- @GreenC: Before this section gets archived and if it's easy/fast to check, can you check if this is still the case, i.e. that the site still has the maximum security enabled and no tool/bot can get through? Thank you. 85.76.109.152 (talk) 06:21, 2 June 2024 (UTC)
When going to [7] it still asks "Are you human?" with the CloudFlare security tag at the bottom. This is a feature of CloudFlare service, clients have the option to enable, it's the highest level of security. I'm not aware of a tool that can bypass. What I will do is set a reminder in 6 months to check again and post the results here. I use W-Ping which posts a reminder in the watchlist at whatever time in the future with a custom message. -- GreenC 16:06, 2 June 2024 (UTC)
- @GreenC: Before this section gets archived and if it's easy/fast to check, can you check if this is still the case, i.e. that the site still has the maximum security enabled and no tool/bot can get through? Thank you. 85.76.109.152 (talk) 06:21, 2 June 2024 (UTC)
- Unfortunately, this site has maximum security enabled, none of my tools can get through. It started happening in late January 2024. I don't know what to do because no bot is able to determine if a link is live or dead. And no archive service such as WaybackMachine is able to archive a page. Only humans can get through, and they need to solve a captcha. It might be worthwhile waiting to see if they relax security in the future, since this is a recent development. -- GreenC 00:40, 19 March 2024 (UTC)
- The site has a "Are you human?" check box (CloudFlare). This is causing the bot to think it's a dead site. I logged into iabot.org and changed the domain to "Subscription" status and that will cause the bot to avoid this domain, it won't set live or dead. My bot WaybackMedic has capabilities to bypass CloudFlare. I can try to process this domain and see what happens. My bot also has a feature "make live" ie. convert a citation from dead to live state. Unfortunately my bot only works on English Wikipedia. I'll let you know what happens. -- GreenC 15:13, 17 March 2024 (UTC)
RateTheRef.net
editThe website RateTheRef.net seems to have been usurped by a Thai gambling site. I don't know how many pages this affects, or whether the old content has been archived, but I figured someone ought to be told. DavidKVT (talk) 21:21, 18 March 2024 (UTC)
Done User:DavidKVT: Thank you. Added to the JUDI list for a batch job later: Special:Diff/1207703597/1214769148 -- GreenC 01:26, 21 March 2024 (UTC)
cinestaan.com
editIt looks like the site is dead as I cannot find it on Google search, and an article is error 503. Check this out too. Kailash29792 (talk) 11:10, 23 March 2024 (UTC)
- 2,243 pages. Offline since December 2023. I can do this. -- GreenC 14:16, 23 March 2024 (UTC)
Done Checked 2,243 pages. Edited 2,206 pages. Added 2,371 archive URLs all WaybackMachine. Added 312
{{dead link}}
tags. Added 255|url-status=dead
for existing archive URLs previously set live. Updated IABot database so changes will propagate to 300+ other wiki language sites. -- GreenC 16:54, 24 March 2024 (UTC)
Bumping thread. GreenC 19:27, 4 June 2024 (UTC) -- GreenC 19:27, 4 June 2024 (UTC)
symetratour.com
editHello. The Symetra Tour has been renamed to The Epson Tour. Their links have been subsquently moved. Here is the new format:
Some links can not be converted such as [8] this link because the event is no longer held. Other links like this one needs the word symetra changed to epson in order to work like this to that. I fixed some already. 91 links under http and 95 under https currently to fix. Thanks! MrLinkinPark333 (talk) 01:42, 12 May 2024 (UTC)
- 116 pages. -- GreenC 20:42, 3 June 2024 (UTC)
Done - Checked 116 pages and edited 116 pages. Moved 109 links to a new URL. Switched 1
|url-status=dead
to live. Added 78 archive URLs (77 Wayback).- -- GreenC 22:42, 3 June 2024 (UTC)
ECI - Election Commission of India
editThe ECI has changed links for a lot of election results on their site. e.g. [9] to [10]. -MPGuy2824 (talk) 11:43, 14 May 2024 (UTC)
- 4,700 pages -- GreenC 20:40, 3 June 2024 (UTC)
User:MPGuy2824: The "old." links are not working https://old.eci.gov.in/assembly-election/ae-2021-tamilnadu/ although they were, it exists at Wayback [11] .. hopefully a temporary outage. I'll recheck in a week or ping me if you see it change before then. -- GreenC 23:01, 3 June 2024 (UTC)
- It looks like geofencing, as the link works for me (in India). Let's wait a week as you suggest. -MPGuy2824 (talk) 05:49, 4 June 2024 (UTC)
- There is a new BRFA at Wikipedia:Bots/Requests for approval/BaranBOT 2. – DreamRimmer (talk) 13:00, 8 June 2024 (UTC)
britannica.co.kr
editThis was brought to my attention through Special:Diff/1224405115. The following hostname should be marked as dead and set to the archived urls given that they are no longer serving any content and being redirected to the company's corp site, or simply dead:
- *.britannica.co.kr
– robertsky (talk) 10:56, 18 May 2024 (UTC)
- 53 pages. -- GreenC 20:44, 3 June 2024 (UTC)
Done - Checked 53 pages and edited 20 pages. Added 4
{{dead link}}
. Added 16 archive URLs (8 Wayback). -- GreenC 00:46, 4 June 2024 (UTC)
South Asia Analysis Group
editwww.southasiaanalysis.org
- domain has been usurped. not sure if it's used anywhere other than Major non-NATO ally (where I already fixed the cite template). thanks, Kdroo (talk) 22:14, 23 May 2024 (UTC)
Done - added to WP:JUDI for later processing: Special:Diff/1225704304/1225804735 -- GreenC 20:44, 26 May 2024 (UTC)
nfl.com
editHello. I found that URLs under the http://www.nfl.com/news/story/ format are either broken or redirect to a new URL:
- URLs with only numbers are broken, and might have an archived copy.
- URLs with a numbers and letters string might redirect to the new URL. This redirect works
- Some URLs with a number/letters string don't work and need converting with the article name in the URL: This URL should go here
- Some URLs with numbers/letters and article name might redirect to new URLs: This is now here.
9000+ links under http and 100+ links under https Thanks! MrLinkinPark333 (talk) 20:22, 26 May 2024 (UTC)
Done - Checked 3,402 pages and edited 3,236 pages. Moved 6,863 links to a new URL. Added 32
{{dead link}}
. Switched 72|url-status=dead
to live. Switched 241|url-status=live
to dead. Added 1,043 archive URLs (1,007 Wayback). Changed 888 citation metadata fields. -- GreenC 18:32, 4 June 2024 (UTC)
donjohnsonbigband.com
editThis domain seems to have been usurped: in 2020, it was still a normal band site https://web.archive.org/web/20201202185840/http://www.donjohnsonbigband.com/[usurped] vs since 2021 it's "DJ Son Band - Rock Music Review" https://web.archive.org/web/20211115153056/https://www.donjohnsonbigband.com/[usurped].
New official URL for the band is https://www.donjohnsonbigband.fi/ TuukkaH (talk) 22:10, 28 May 2024 (UTC)
Done Amused the usurpers interpreted "donjohnson" as "DJ Son" ie. Don John Son. Or maybe a computer algorithm, stupid AI. Well, I added it to WP:JUDI for future processing:Special:Diff/1225804735/1226183097 and the URL is in one article, Support de Microphones, which I sortafixed.Special:Diff/1171308429/1226183692 -- GreenC 01:35, 29 May 2024 (UTC)
deccanchronicle.com
editDeccan Chronicle: Many 2010s articles like this are dead. Kailash29792 (talk) 05:02, 2 June 2024 (UTC)
Done - Checked 8,059 pages and edited 3,532 pages. Moved 3,217 links to a new URL. Added 81
{{dead link}}
. Switched 334|url-status=dead
to live. Switched 208|url-status=live
to dead. Added 742 archive URLs (694 Wayback). Changed 1,018 citation metadata fields. -- GreenC 22:35, 5 June 2024 (UTC)
cnlbr.org
editOld path of "www.cnlbr.org/Portals/.../pagename
" moved to "irp.cdn-website.com/33d0c3d0/files/uploaded/pagename
"
-- BX (talk) 20:58, 2 June 2024 (UTC)
- BX, can you clarify. For example, old URL http://www.cnlbr.org/Portals/0/Hero/Herbert_Rap_Dixon.pdf goes to ? -- GreenC 21:00, 3 June 2024 (UTC)
- @GreenC: The old path after "Portals/" varied, however the new path has no variables. So for your example, the new path is https://irp.cdn-website.com/33d0c3d0/files/uploaded/Herbert_Rap_Dixon.pdf It's basically just cutting the last "pagename" from the old path and pasting it to the the new prefix, if that makes sense. Rgdrs. --BX (talk) 04:03, 4 June 2024 (UTC)
- Got it, didn't realize "33d0c3d0" is a static string. 138 pages. -- GreenC 19:13, 4 June 2024 (UTC)
- User:BX: There were edge cases in about 30 URLs. Needed to convert "%20%20" to "%20". And in some, changing ".pdf" to "-2020.pdf" - After those changes, I was able to convert all to live links. I made metadata changes eg, changing
|publisher=cnlbr.org
to|publisher=Center for Negro League Baseball Research
, because supposed to use names vs. domains. Anything that was previous marked dead and had an archive URL, I changed the primary URL to the live version and set|url-status=live
and kept the original archive URL. -- GreenC 01:37, 7 June 2024 (UTC)
- User:BX: There were edge cases in about 30 URLs. Needed to convert "%20%20" to "%20". And in some, changing ".pdf" to "-2020.pdf" - After those changes, I was able to convert all to live links. I made metadata changes eg, changing
- Got it, didn't realize "33d0c3d0" is a static string. 138 pages. -- GreenC 19:13, 4 June 2024 (UTC)
- @GreenC: The old path after "Portals/" varied, however the new path has no variables. So for your example, the new path is https://irp.cdn-website.com/33d0c3d0/files/uploaded/Herbert_Rap_Dixon.pdf It's basically just cutting the last "pagename" from the old path and pasting it to the the new prefix, if that makes sense. Rgdrs. --BX (talk) 04:03, 4 June 2024 (UTC)
- BX, can you clarify. For example, old URL http://www.cnlbr.org/Portals/0/Hero/Herbert_Rap_Dixon.pdf goes to ? -- GreenC 21:00, 3 June 2024 (UTC)
Done Checked 140 pages and edited 140 pages. Moved 321 links to a new URL. Switched 17
|url-status=dead
to live. Changed 185 citation metadata fields. -- GreenC 01:37, 7 June 2024 (UTC)- Wow, thanks User:GreenC. The work you and your bot do is invaluable to keeping this place working. Thanks again! Rgrds. --BX (talk) 04:04, 7 June 2024 (UTC)
- Thank you! Your appreciation helps to keep this going. -- GreenC 14:59, 7 June 2024 (UTC)
- Wow, thanks User:GreenC. The work you and your bot do is invaluable to keeping this place working. Thanks again! Rgrds. --BX (talk) 04:04, 7 June 2024 (UTC)
google.com/hostednews
editSoft-404s and 404s. 5,300 pages. -- GreenC 20:35, 3 June 2024 (UTC)
Done - Checked 5,322 pages and edited 4,351 pages. Converted 1 templates. Removed 2
{{dead link}}
templates. Added 1,739{{dead link}}
. Switched 707|url-status=live
to dead. Added 3,633 archive URLs (2,179 Wayback). Changed 176 citation metadata fields. -- GreenC 15:42, 7 June 2024 (UTC)
cinestaan.com makelive
editMysteriously the site is back and working, per this. Maybe the dead links can be reassessed? Kailash29792 (talk) 04:14, 4 June 2024 (UTC)
- Previous: Wikipedia:Link_rot/URL_change_requests#cinestaan.com -- GreenC 19:28, 4 June 2024 (UTC)
- I changed the domain status from "Permadead" to "Permalive" in iabot.org --- for the moment the bot won't convert links to dead automatically. For Enwiki, Medic has a "makelive" function which I could apply to any link responding with status 200. -- GreenC 19:37, 4 June 2024 (UTC)
- It checked every link, any that are 200 it converted to a live link. -- GreenC 01:37, 8 June 2024 (UTC)
Done - Checked 2,242 pages and edited 2,033 pages. Moved 2,360 links to a new URL. Removed 153
{{dead link}}
templates. Added 29 {{dead link}}
. Switched 1,943 |url-status=dead
to live. Added 6 archive URLs (6 Wayback). Changed 65 citation metadata fields.
google.com/patents
edit2,700 pages. -- GreenC 20:51, 7 June 2024 (UTC)
- The way GreenC bot is handling these by replacing them with half-broken archive.today links is problematic. The bot's activity on this should be paused, the changes made so far should be reverted, and someone should write a bot/script which properly fixes the URLs to current working versions. –jacobolus (t) 16:49, 8 June 2024 (UTC)
- My reply here Special:Diff/1227943344/1227944480. I agree that it's a good idea to switch archived URLs to live URLs, and my bot can do that. But I need to know what the live URL is. And your not providing information how to figure that out. Currently, the bot is repairing a completely broken non-functioning URL with an archive URL. I understand the archive URL is incomplete, but at least better than a completely dead URL. If there is a way to determine the live URL, I can replace the archive URL with the live URL. -- GreenC 17:01, 8 June 2024 (UTC)
- Looks like the Patent ID is in the title of the archive.today page eg. for [12]:
<title>Patent US417831 - ARTISTS EASEL - Google Patents</title>
from which can be generated https://patents.google.com/patent/US417831A .. although I am unclear about "A", how to determine. -- GreenC 17:50, 8 June 2024 (UTC) - Early on Google made up a new identifier for every patent. More recently they have sensibly figured out how to use the patent number itself. I think the A is optional; it's just the form of URL that turned up when I did a search for a couple of these specific patents. You can see how https://patents.google.com/patent/US640792A and https://patents.google.com/patent/US640792 give the same result. –jacobolus (t) 21:48, 8 June 2024 (UTC)
- The archive roll back is done. Edited 731 articles and 1,468 citations. Example: Special:Diff/1227937185/1228131980 and Special:Diff/1227937170/1228131968. There are 64 links with no patent number the list is available here Wikipedia:Link rot/Cases/Googlepatents in case you or anyone want to research. Optionally update that page with the patent numbers and I'll update wiki via bot. -- GreenC 16:54, 9 June 2024 (UTC)
- The linkrot ones can probably be figured out by scraping a wayback page. E.g. the first one is here, from which we can find patent number 2612994, so the current google patent link would be https://patents.google.com/patent/US2612994. –jacobolus (t) 17:00, 9 June 2024 (UTC)
- The archive roll back is done. Edited 731 articles and 1,468 citations. Example: Special:Diff/1227937185/1228131980 and Special:Diff/1227937170/1228131968. There are 64 links with no patent number the list is available here Wikipedia:Link rot/Cases/Googlepatents in case you or anyone want to research. Optionally update that page with the patent numbers and I'll update wiki via bot. -- GreenC 16:54, 9 June 2024 (UTC)
- Looks like the Patent ID is in the title of the archive.today page eg. for [12]:
- My reply here Special:Diff/1227943344/1227944480. I agree that it's a good idea to switch archived URLs to live URLs, and my bot can do that. But I need to know what the live URL is. And your not providing information how to figure that out. Currently, the bot is repairing a completely broken non-functioning URL with an archive URL. I understand the archive URL is incomplete, but at least better than a completely dead URL. If there is a way to determine the live URL, I can replace the archive URL with the live URL. -- GreenC 17:01, 8 June 2024 (UTC)
Done - Checked 2,702 pages and edited 2,502 pages. Moved 2,891 links to a new URL. Removed 1
{{dead link}}
templates. Added 256{{dead link}}
. Switched 3|url-status=dead
to live. Switched 11|url-status=live
to dead. Added 1,539 archive URLs (222 Wayback). Changed 2 citation metadata fields. (NOTE: these stats are outdated due to the archive roll back in a later pass, which removed 1,468 archive URLs) -- GreenC 16:54, 9 June 2024 (UTC)- Thanks! –jacobolus (t) 16:58, 9 June 2024 (UTC)
google.com/culturalinstitute
edit666 pages -- GreenC 20:56, 7 June 2024 (UTC)
Done - Checked 675 pages and edited 670 pages. Moved 489 links to a new URL. Added 63
{{dead link}}
. Switched 1|url-status=dead
to live. Switched 3|url-status=live
to dead. Added 166 archive URLs (163 Wayback). -- GreenC 00:50, 10 June 2024 (UTC)
google.com/finance
edit562 pages -- GreenC 20:59, 7 June 2024 (UTC)
Done - Checked 562 pages and edited 513 pages. Converted 1 templates. Moved 365 links to a new URL. Added 40
{{dead link}}
. Switched 7|url-status=dead
to live. Switched 6|url-status=live
to dead. Added 150 archive URLs (127 Wayback). Changed 1 citation metadata fields. -- GreenC 04:01, 10 June 2024 (UTC)
google.com/doodles
edit900 pages -- GreenC 21:40, 7 June 2024 (UTC)
Done - Checked 907 pages and edited 904 pages. Moved 960 links to a new URL. Added 4
{{dead link}}
. Switched 9|url-status=dead
to live. Added 21 archive URLs (10 Wayback). -- GreenC 04:20, 11 June 2024 (UTC)
chennaionline.com
editThe site is working properly per this, but there are still many pre-2020 dead links like this. Kailash29792 (talk) 09:52, 12 June 2024 (UTC)
Done - Checked 893 pages and edited 147 pages. Moved 8 links to a new URL. Added 19
{{dead link}}
. Switched 49|url-status=live
to dead. Added 80 archive URLs (70 Wayback). Changed 24 citation metadata fields. -- GreenC 18:30, 13 June 2024 (UTC)
angelfire.com
editThis book, Hook, James; Franck, Dave; Austin, Steve (1982). An Aid to Collecting Selected Council Shoulder Patches with Valuation. has within it a link. Yes, I know it's from angelfire. Can I ask that: www
- User:Evrik - if you don't mind, I'll use this request to process the entire angelfire.com domain which needs to be done anyway, checking for link rot. It will include a trap for https://www.angelfire.com/tx6/patch/cspbook.html to replace with https://scouttrader.org/csiguidebook.shtml .. it is in about 90 pages.
- 4,794 pages -- GreenC 15:59, 13 June 2024 (UTC)
- It may http and not https --evrik (talk) 16:14, 13 June 2024 (UTC)
- You know, the citation is for a book authored by Hook, Franck and Austin (1982). The source link is for a book published by Ellis, Jones and Austin (2003). The replacment link is for a book published by Austin and Keasey (2013). There are many editions and authors. Maybe more. If someone is citing the 1982 edition on page 52, and we change the link to the 2013 edition, it will be a wrong page number. I think this needs to be done with more care - or a consensus discussion. I don't want to be in the position of asked to undo the changes, which is time consuming. -- GreenC 01:50, 15 June 2024 (UTC)
- It may http and not https --evrik (talk) 16:14, 13 June 2024 (UTC)
Done - Checked 4,890 pages and edited 4,317 pages. Moved 4,750 links to a new URL. Removed 1
{{dead link}}
templates. Added 58{{dead link}}
. Switched 109|url-status=dead
to live. Switched 13|url-status=live
to dead. Added 379 archive URLs (339 Wayback). Changed 521 citation metadata fields. -- GreenC 01:50, 15 June 2024 (UTC)
mhc-macris.net
editI didn't check very many of the pages here (first 2 out of 3,685) [13], but the current links are dead, and changing "Details.aspx?" to "details?" fixes them (with them redirecting to a version with a lower case ID). Another improvement is using https instead of http.
For instance in College of the Holy Cross, which as of right now citation 28's url is "http://mhc-macris.net/Details.aspx?MhcId=WOR.K" which just redirects to the mhc-macris home page, but if it's changed to "https://mhc-macris.net/details?MhcId=WOR.K" it redirects to "https://mhc-macris.net/details?mhcid=wor.k" which has the desired content.
There are another handful of URLs needed to be changed sprinkled amongst this search, but some of them are archive links. GrapesRock (talk) 20:52, 19 June 2024 (UTC)
- OK. I'll check every link, in case there are any other soft-404 issues. 3,836 pages. thanks. -- GreenC 23:54, 19 June 2024 (UTC)
Question: GrapesRock, I have run into a problem. The site is using a bot blocker system that I don't recognize and have tried various methods to get around unsuccessfully. The only way is a "blind move" ie. changing the URL without verifying the new URL exists and/or works. This is potentially dangerous because sites frequently do not migrate every URL to the new scheme. Another method is treat every URL containing "Details.aspx" as a dead link, and add an archive URL. It depends on how reliable the archive.org links are (they may have the same problem saving pages due to bot blocker) vs. how consistent the site was in migrating to the new scheme. If you want to manual spot check to see which of these methods looks better that would be helpful in deciding which course to take. I also emailed the site admins on the off chance they are willing to temporarily whitelist my IP. -- GreenC 17:48, 20 June 2024 (UTC)
- I looked at the top 10 pages with Details.aspx in them, and everyone where I changed it to "details", it redirected to the correct page.
- As for the archive, all the archives I've seen before July 7, 2022 are okay, but on that date and after it is inconsistently marked
- List of National Historic Landmarks in Massachusetts
- http://mhc-macris.net/Details.aspx?MhcId=WSP.211 where the corresponding archive with one capture (on July 7, 2022), https://web.archive.org/web/20220707050042/http://mhc-macris.net/Details.aspx?MhcId=WSP.211 showed okay (no 404s or redirects reported), but didn't display anything
- http://mhc-macris.net/Details.aspx?MhcId=SAL.1126 where the July 7, 2022 showed as okay, but didn't ever load the content https://web.archive.org/web/20220707051738/http://mhc-macris.net/Details.aspx?MhcId=SAL.1126. Earlier archives such as https://web.archive.org/web/20150513134101/http://mhc-macris.net/Details.aspx?MhcId=SAL.1126 work fine
- etc.
- Holyoke, Massachusetts
- This archive for 2023, shows both the archives as blue even though they're 301 redirects.
- List of National Historic Landmarks in Massachusetts
- So, probably if the only archive that exists is from July 7, 2022 or later, it shouldn't be used, but it should be safe to add any archive links from before then.
- This attributes section seems to indicate that the mhcid does not change since its uniquely assigned. Also, the MACRIS home page says "Each historic property or area in the MACRIS database will have an MHC ID assigned to it. The MHC ID in Search Results is linked to a Details screen". This makes me think it's safe to change the URL as long as it's of the format "mhc-macris.net/Details.aspx?MhcId=[THE ID]" since the ID will still be in the MACRIS database and will be linked to a details screen GrapesRock (talk) 18:41, 20 June 2024 (UTC)
- Alright I have it programmed for blind search-replace. I'll wait a till Friday, and see if they respond about the IP. Mean time can start on the other one below. -- GreenC 22:26, 20 June 2024 (UTC)
- Soft-redirect rule:
newurl = "https://mhc-macris.net/Details.aspx?MhcId=WOR.K" if newurl ~ "mhc-macris[.]net/Details[.]aspx[?]MhcId[=]": subs("Details.aspx?", "details?", newurl) if match(newurl, "(?i)[?]MhcId=[^$]*[^$]*", d) > 0: subs(d, tolowerAscii(d), newurl) newurl == "https://mhc-macris.net/details?mhcid=wor.k"
Done - Checked 3,720 pages and edited 3,716 pages. Moved 4,419 links to a new URL. Removed 36
{{dead link}}
templates. Switched 39|url-status=dead
to live. Added 3 archive URLs (2 Wayback). Changed 22 citation metadata fields. -- GreenC 16:36, 22 June 2024 (UTC)- @GrapesRock@GreenC FWIW, I created {{MACRIS}} a few years back after they changed their url scheme for the second time in a short period. That allows all MACRIS links using the template to be updated with a single edit to the template. I'm not sure if it's worth mass conversion, but wanted to make you both aware. Pi.1415926535 (talk) 20:14, 22 June 2024 (UTC)
- Those templates are OK the problem is they don't account for some percentage of links that were not migrated to the new URL scheme. It assumes all or nothing, in practice is rarely the case - some links get left behind becoming dead URLs. By using standard citation templates, bots like this can check the links and add archives or
{{dead link}}
tags on a per-URL basis. Otherwise the bot would need to be specially programmed for the custom template, and there are thousands of custom templates making it impractical. In this case, the site is bot protected so it really is all or nothing so the template (for now) is not a problem. -- GreenC 04:40, 23 June 2024 (UTC)
- Those templates are OK the problem is they don't account for some percentage of links that were not migrated to the new URL scheme. It assumes all or nothing, in practice is rarely the case - some links get left behind becoming dead URLs. By using standard citation templates, bots like this can check the links and add archives or
- @GrapesRock@GreenC FWIW, I created {{MACRIS}} a few years back after they changed their url scheme for the second time in a short period. That allows all MACRIS links using the template to be updated with a single edit to the template. I'm not sure if it's worth mass conversion, but wanted to make you both aware. Pi.1415926535 (talk) 20:14, 22 June 2024 (UTC)
atlantaintownpaper.com
editThis source has been moved to roughdraftatlanta.com. For instance in George Floyd protests, currently there is the url https://atlantaintownpaper.com/2020/05/mayor-police-chief-denounce-anarchists-and-terrorists-who-destroyed-city-curfew-begins-at-9-p-m/ and the source has been moved to https://roughdraftatlanta.com/2020/05/30/mayor-police-chief-denounce-anarchists-and-terrorists-who-destroyed-city-curfew-begins-at-9-p-m/ (and there's no redirect).
Both
- https://roughdraftatlanta.com/atlantaintownpaper/2020/05/mayor-police-chief-denounce-anarchists-and-terrorists-who-destroyed-city-curfew-begins-at-9-p-m/
- https://roughdraftatlanta.com/2020/05/mayor-police-chief-denounce-anarchists-and-terrorists-who-destroyed-city-curfew-begins-at-9-p-m/
Redirect to the proper site GrapesRock (talk) 15:22, 20 June 2024 (UTC)
- OK. This is a "Soft-redirect", where a page exists at a new URL but a redirect is missing (versus a soft-404, where the redirect exists but goes to a wrong page). I can fix Soft-redirects, when there is foreknowledge like you helpfully discovered. It also has a "Redirect" element so Soft-redirect --> Redirect --> Destination. 78 pages. -- GreenC 16:28, 20 June 2024 (UTC)
Done - Checked 78 pages and edited 75 pages. Moved 79 links to a new URL. Added 1
{{dead link}}
. Switched 13|url-status=dead
to live. Added 3 archive URLs (3 Wayback). -- GreenC 02:38, 21 June 2024 (UTC)- Soft-redirect rule:
subs("atlantaintownpaper.com", "roughdraftatlanta.com", newurl)
- Soft-redirect rule:
Ooh, cool! Thanks for the explanation on a piece of terminology, it's always fun to learn new words/concepts (and of course thanks for moving all the stuff). GrapesRock (talk) 14:03, 21 June 2024 (UTC)
- I made a glossary WP:LINKROT#Glossary of terminology it can get complicated. -- GreenC 15:51, 22 June 2024 (UTC)
clatl.com
editRedirects to creativeloafing.com and soft-404s - 379 pages -- GreenC 14:38, 21 June 2024 (UTC)
Done - Checked 379 pages and edited 365 pages. Moved 389 links to a new URL. Added 21
{{dead link}}
. Switched 37|url-status=dead
to live. Switched 8|url-status=live
to dead. Added 108 archive URLs (84 Wayback). Changed 63 citation metadata fields. -- GreenC 15:47, 21 June 2024 (UTC)
- Soft-404 rule by URL: If a redirect contains:
(?i)(page[+]not[+]found|page%20not%20found)
- Soft-404 rule by page title: If a page title contains:
(?i)^[ ]*search([ ]*[|][ ]*Creative Loafing)?[ ]*$
- Soft-404 rule by page content: If a page contains:
Content is needed
- Soft-404 rule by URL: If a redirect contains:
stat.kg
editThe URL of the National Statistical Comitee of the Kyrgyz Republic changed from stat.kg to stat.gov.kg, everything else stayed the same. The links lead to 404, e. g. in Chaek MarcelloIV (talk) 11:04, 24 June 2024 (UTC)
Done Checked 780 pages and edited 769 pages. Moved 800 links to a new URL. Added 1
{{dead link}}
. Switched 7|url-status=dead
to live. Added 3 archive URLs (3 Wayback). Changed 19 citation metadata fields.
- Soft-redirect rule:
subs("stat.kg", "stat.gov.kg", newurl)
- Soft-redirect rule:
mtv.com
editAll mtv.com/news links have broke according to https://variety.com/2024/digital/news/mtv-news-website-archives-pulled-offline-1236047163/. Looks like we have several thousand references. --Nintendofan885T&Cs apply 22:52, 24 June 2024 (UTC)
(edit conflict) Variety is reporting that 20 years of MTV News archives have been pulled. A few I've tested seem to support that:
- https://www.mtv.com/news/3r5xfl/dungeons-and-dragons-arena-of-war
- https://www.mtv.com/news/olxhhg/joe-manganiello-dungeons-dragons-movie
- https://www.mtv.com/news/bhktbo/dungeons-and-dragons-online-character-creator-official-released
Thanks! Sariel Xilo (talk) 23:09, 24 June 2024 (UTC)
FYI looks like mtv.com is marked as permalive on IABot so it will ignore the links on other wikis (as GreenC bot only processes enwiki) --Nintendofan885T&Cs apply 23:23, 24 June 2024 (UTC)
- WaybackMedic can edit the IABot database, changing target links to permadead, which then propagate to the other wikis, via IABot. I will also process all MTV links on enwiki as normal, and see what other soft-404 rules might be discovered, which can also be applied to the IABot database. -- GreenC 01:30, 25 June 2024 (UTC)
- 20,263 pages mtv.com/*
- Soft-404 Rules: When a redirect matches the regex, and original URL does not match. Example for rule #1: [14] redirects to [15]
([.]|[/])mtv[.]com([.](tw|br|au))?(/(music|#))?/news/?
([.]|[/])mtv[.]com([.](tw|br|au))?/?$
([.]|[/])paramountshop[.]com/?$
([.]|[/])mtv[.]com([.](tw|br|au))?/[?]xrs=PPM-18-10caf1c/?$
([.]|[/])mtv[.]com([.](tw|br|au))?/category/ffftn1/style/?$
([.]|[/])paramountplus[.]com/brands/mtv/#ftag=PPM-18-10caf1c/?$
([.]|[/])mtvema[.]com(/en-us)?/?$
([.]|[/])mtv[.]com([.](tw|br|au))?/videos/?$
- For Enwiki: Checked 19,690 pages and edited 18,113 pages. Moved 319 links to a new URL. Removed 2
{{dead link}}
templates. Added 787{{dead link}}
. Switched 11,106|url-status=live
to dead. Added 27,134 archive URLs (25,337 Wayback).- More work to do, about 700 pages missed due to Cirrus vs. SQL search incongruities. -- GreenC 01:10, 28 June 2024 (UTC)
- Done. Checked 643 pages and edited 192 pages. Moved 92 links to a new URL. Added 14
{{dead link}}
. Switched 4|url-status=dead
to live. Switched 51|url-status=live
to dead. Added 54 archive URLs (44 Wayback). -- GreenC 15:54, 1 July 2024 (UTC)
- Done. Checked 643 pages and edited 192 pages. Moved 92 links to a new URL. Added 14
- More work to do, about 700 pages missed due to Cirrus vs. SQL search incongruities. -- GreenC 01:10, 28 June 2024 (UTC)
- For IABot database updates: est. 3-4 days -- GreenC 00:23, 28 June 2024 (UTC)
- Done. Checked about unique 51,000 links. Over 98% are hard and soft dead. Uploaded the information to IABot which will be changing the links across 300+ wikis. -- GreenC 15:57, 1 July 2024 (UTC)
Done with mtv.com -- GreenC 15:58, 1 July 2024 (UTC)
- @GreenC: Can you also check the following sites found through the certificate (don't match the above regex) that have seemingly also killed their news sections and have similar redirect-to-homepage deadlinks looking at Special:LinkSearch?
- mtv.pl (mtv.pl/newsy)
- mtv.co.uk (mtv.co.uk/news)
Thanks! --Nintendofan885T&Cs apply 19:21, 1 July 2024 (UTC)
- User:Nintendofan885, thanks. New sections below. -- GreenC 14:17, 2 July 2024 (UTC)
apps.ehsni.gov.uk
editLooks like we have a soft-redirect from http://apps.ehsni.gov.uk/ambit/Details.aspx?MonID=8572 to https://apps.communities-ni.gov.uk/NISMR-PUBLIC/Details.aspx?MonID=8572. Checking a smattering of links from List of castles in Ireland this seems to redirect to the proper place consistently (i.e. the few links I've checked, changing "http://apps.ehsni.gov.uk/ambit" to "https://apps.communities-ni.gov.uk/NISMR-PUBLIC" has worked). GrapesRock (talk) 17:49, 25 June 2024 (UTC)
- Hi User:GrapesRock: Looks like these exist on 4 pages. Can you repair them? It will be a lot easier than programming a fix. -- GreenC 16:16, 1 July 2024 (UTC)
- Yup, done. For the future, is there any value for posterity in adding posts here for links that only have a smattering of pages or should I just fix 'em? GrapesRock (talk) 16:50, 1 July 2024 (UTC)
- It's hard to say because it depends what work is involved making the fix. I've seen cases where 5 pages can take a long time to figure out manually and better done by bot. To setup the bot, compile, generate a list of target pages, run the bot, check for errors, upload diffs .. it's like 10 or 15 minutes for a small run. If you can do it faster than that manually, go for it. But even for simple cases, if it's more than around 20 pages don't hesitate to ask for bot help. -- GreenC 18:36, 1 July 2024 (UTC)
- Yup, done. For the future, is there any value for posterity in adding posts here for links that only have a smattering of pages or should I just fix 'em? GrapesRock (talk) 16:50, 1 July 2024 (UTC)
Done -- GreenC 05:05, 16 August 2024 (UTC)
kp.by
editLooks like there's a soft-redirect from kp.by to kp.ru links, such as "https://www.kp.by/daily/27084/4156223/" in Victory Day (9 May) being dead, but "https://www.kp.ru/daily/27084/4156223/" working GrapesRock (talk) 18:26, 25 June 2024 (UTC)
- 117 pages -- GreenC 16:22, 1 July 2024 (UTC)
Done - Checked 121 pages and edited 116 pages. Moved 149 links to a new URL. Removed 33
{{dead link}}
templates. Added 2{{dead link}}
. Switched 10|url-status=dead
to live. Added 7 archive URLs (4 Wayback). -- GreenC 19:38, 1 July 2024 (UTC)
dailynews.gov.bw
editURLs of the form "http://www.dailynews.gov.bw/news-details.php?nid=23359" that I've checked soft-redirect to "https://dailynews.gov.bw/news-detail/23359" (example from Gladys Olebile Masire) GrapesRock (talk) 20:58, 25 June 2024 (UTC)
newurl = <url> subs("http://", "https://", newurl) # normal site # http://www.dailynews.gov.bw/news-details.php?nid=23359 # https://dailynews.gov.bw/news-detail/23359 if newurl ~ "www[.]dailynews[.]gov[.]bw/news-details[.]php[?]nid=": subs("www.dailynews.gov.bw", "dailynews.gov.bw", newurl) subs("/news-details.php?nid=", "/news-detail/", newurl) # mobile site # http://www.dailynews.gov.bw/mobile/news-details.php?nid=10829&flag= # https://dailynews.gov.bw/news-detail/10829 if newurl ~ "www[.]dailynews[.]gov[.]bw/mobile/news-details[.]php[?]nid=": subs("www.dailynews.gov.bw", "dailynews.gov.bw", newurl) subs("/mobile/news-details.php?nid=", "/news-detail/", newurl) subs("&flag=", "", newurl)
- Soft-404: if redirect URL is
https://dailynews.gov.bw/page-not-found
Done - Checked 91 pages and edited 71 pages. Moved 111 links to a new URL. Removed 20
{{dead link}}
templates. Added 6{{dead link}}
. Switched 22|url-status=dead
to live. Added 9 archive URLs (7 Wayback). -- GreenC 21:58, 1 July 2024 (UTC)
blackcountryhistory.org
editFrom the page 1980, "http://blackcountryhistory.org/collections/getrecord/GB149_P_915/" soft redirects to "https://www.blackcountryhistory.org/collections/getrecord/GB149_P_915/" (adding a "www." to the start fixes the dead link, and might as well upgrade security). GrapesRock (talk) 17:33, 26 June 2024 (UTC)
- 123 pages -- GreenC 22:52, 1 July 2024 (UTC)
- User:GrapesRock - the website has CloudFlare protection at maximum level ("Click box if you are human"). My bot is unable to verify if the new URL is working. Given the low number of links, and simplicity of the move (adding "www" to the domain), I will go ahead and do a "blind move" ie. without verifying. -- GreenC 23:21, 1 July 2024 (UTC)
Done - Checked 130 pages and edited 106 pages. Moved 133 links to a new URL. Removed 1
{{dead link}}
templates. Switched 28|url-status=dead
to live. Added 5 archive URLs (2 Wayback). -- GreenC 23:38, 1 July 2024 (UTC)
cmt.com
editCountry Music Television, sister company of MTV. Paramount cost cutting: https://www.savingcountrymusic.com/cmt-mtvs-eradication-of-editorial-content-is-a-catastrophe/
6,278 pages GreenC 02:17, 27 June 2024 (UTC)
- Enwiki - Checked 6,244 pages and edited 3,242 pages. Moved 47 links to a new URL. Removed 2
{{dead link}}
templates. Added 310{{dead link}}
. Switched 1|url-status=dead
to live. Switched 616|url-status=live
to dead. Added 5,635 archive URLs (5,327 Wayback). Changed 220 citation metadata fields. -- GreenC 14:05, 3 July 2024 (UTC) - IABot DB - Checked and updated 6,831 URLs.
Done -- GreenC 22:44, 3 July 2024 (UTC)
cc.com
editComedy Central. More Paramount cost cutting: https://www.hollywoodreporter.com/business/business-news/comedy-central-website-daily-show-clips-wiped-out-1235933345/amp/ -- GreenC 15:03, 27 June 2024 (UTC)
909 pages -- GreenC 15:11, 27 June 2024 (UTC)
Soft-404 when a link redirects to one of these:
([.]|[/])cc[.]com/[?]xrs=PPM-18-10caf1[chd]/?$
([.]|[/])paramountplus[.]com/shows/tosh-0/#ftag=PPM-18-10caf1d/?$
([.]|[/])paramountplus[.]com/shows/the-daily-show/#ftag=PPM-18-10caf1d/?$
([.]|[/])paramountplus[.]com/brands/comedy-central/#ftag=PPM-18-10caf1d/?$
([.]|[/])southpark[.]cc[.]com/seasons/south-park/?$
([.]|[/])cc[.]com/fan-hub/the-daily-show[?]xrs=PPM-18-10caf1d/?$
([.]|[/])southpark[.]cc[.]com/?$
([.]|[/])southpark[.]cc[.]com/wiki/Main/?$
Results:
- Enwiki - Checked 918 pages and edited 782 pages. Moved 135 links to a new URL. Removed 2
{{dead link}}
templates. Added 121{{dead link}}
. Switched 18|url-status=dead
to live. Switched 136|url-status=live
to dead. Added 656 archive URLs (638 Wayback). Changed 131 citation metadata fields. - IABot DB Checked 2,338 links of which modified 1,675. Changes will propagate to 300+ wikis via IABot.
Done -- GreenC 19:03, 4 July 2024 (UTC)
tvland.com
editTV Land. More Paramount cost cutting: https://www.hollywoodreporter.com/business/business-news/comedy-central-website-daily-show-clips-wiped-out-1235933345/amp/ -- GreenC 15:08, 27 June 2024 (UTC)
100 pages -- GreenC 15:12, 27 June 2024 (UTC)
Soft-404 when a link redirects to:
paramountplus[.]com/browse/#ftag=PPM-18-10caf1i/?$
Results:
- Enwiki: Checked 106 pages and edited 42 pages. Moved 17 links to a new URL. Added 1
{{dead link}}
. Switched 2|url-status=live
to dead. Added 22 archive URLs (20 Wayback). Changed 8 citation metadata fields. - IABot DB: Checked 184 links and modified 172
Done -- GreenC 00:41, 5 July 2024 (UTC)
zdnet.com
edit"it went through major changes that broke many of its old links" - https://www.msn.com/en-us/news/technology/the-internet-s-memory-is-under-threat/ar-BB1oRToN
5,100 pages -- GreenC 19:01, 27 June 2024 (UTC)
Soft-404s are any redirects that match this:
zdnet[.]com/topic/(virtualization|microsoft|enterprise-software|apple|networking|education|storage|computing|hardware|social-media|government|smartphones|innovation|open-source|security|smb|collaboration|google|big-data)/?$
Bot results:
- Enwiki: Checked 5,148 pages and edited 2,442 pages. Moved 2,325 links to a new URL. Removed 2
{{dead link}}
templates. Added 52{{dead link}}
. Switched 32|url-status=dead
to live. Switched 116|url-status=live
to dead. Added 639 archive URLs (554 Wayback). Changed 403 citation metadata fields. - IABot DB: Checked 9,883 links and modified 2,629. Changes will propagate to 300+ wikis.
Done -- GreenC 02:24, 6 July 2024 (UTC)
mtv.pl
editParamount site per #mtv.com: mtv.pl (mtv.pl/newsy) 35 pages ---- GreenC 14:14, 2 July 2024 (UTC)
Done - Checked 35 pages and edited 19 pages. Added 3
{{dead link}}
. Added 39 archive URLs (39 Wayback). Changed 2 citation metadata fields. -- GreenC 02:46, 6 July 2024 (UTC)
mtv.co.uk
editParamount site per #mtv.com: mtv.co.uk (mtv.co.uk/news) 1,700 pages -- GreenC 14:15, 2 July 2024 (UTC)
Bot results:
- Enwiki - Checked 1,741 pages and edited 1,503 pages. Moved 942 links to a new URL. Removed 1
{{dead link}}
templates. Added 22{{dead link}}
. Switched 61|url-status=dead
to live. Switched 228|url-status=live
to dead. Added 755 archive URLs (688 Wayback). Changed 280 citation metadata fields.
- IABot DB - Checked 2,616 unique links and fixed 1,550 which will update on 300+ wikis
Done -- GreenC 21:59, 6 July 2024 (UTC)
catholicnewsagency.com
editThis page "http://www.catholicnewsagency.com/new.php?n=15190" soft-redirects to "http://www.catholicnewsagency.com/news/15190" which redirects to "https://www.catholicnewsagency.com/news/15190/cardinal-rouco-opens-cause-of-canonization-for-spanish-couple" (example from Opus Dei) GrapesRock (talk) 18:10, 2 July 2024 (UTC)
GrapesRock: In Charles of Sezze I found: http://www.catholicnewsagency.com/saint.php?n=416 .. do you think it soft-redirects? Also http://www.catholicnewsagency.com/document.php?n=147 in Maria Gabriella Sagheddu, and https://www.catholicnewsagency.com/martyrology_entry.php?n=596 in List of Catholic saints, and http://www.catholicnewsagency.com/resource.php?n=409 in Natural marriage, and http://www.catholicnewsagency.com/column.php?n=1360 in Mark Templeton (trombonist). There are probably more. -- GreenC 22:22, 6 July 2024 (UTC)
- The saint one seems to exist here https://www.catholicnewsagency.com/saint/st-charles-of-sezze-416 , so there's still that 416. I don't see an algorithmic way to figure out what belongs between saint/ and 416
- Can't find a newer location for http://www.catholicnewsagency.com/document.php?n=147
- Ditto on the martyrology link
- Ditto on the resource link
- The column link soft-redirects to https://www.catholicnewsagency.com/column/51359
- Seems like adding 49999 to the number works, at least in the example above and http://www.catholicnewsagency.com/column.php?n=2930 (from The Fault in our Stars) soft-redirecting to http://www.catholicnewsagency.com/column/52929
- GrapesRock (talk) 23:05, 6 July 2024 (UTC)
- Thanks. -- GreenC 01:28, 7 July 2024 (UTC)
Bot results:
- Enwiki - Checked 2,004 pages and edited 1,352 pages. Moved 1,754 links to a new URL. Removed 9
{{dead link}}
templates. Added 5{{dead link}}
. Switched 70|url-status=dead
to live. Switched 5|url-status=live
to dead. Added 101 archive URLs (90 Wayback). Changed 75 citation metadata fields. -- GreenC 01:28, 7 July 2024 (UTC) - IABot DB - Checked 3,445 unique links and changed 432 which will propagate across 300+ wikis.
Done -- GreenC 14:13, 7 July 2024 (UTC)
ew.com
editHello, Old URLs for Entertainment Weekly that mainly consist of numbers don't work. These links can be sorted into multiple categories:
- 1) Links that already have an archived copy in the article: this link at Don't Tell Me (Avril Lavigne song) is here.
- 2) Links that can be moved over to a new URL: this should go to that for Buckshot_LeFonque_(album). This has /ew/ removed, while adding date and title in the URL.
- 3) Links at a new URL but not in the date/title format: this is now here for Heavy Competition.
- 4) Links that can't be moved over to a new URL: this link at Fast Times at Barrington High needs an archived copy.
Since the new URLs don't always match the date/title format, I would like the broken links to be focused on first Then, if any of the archived links have a working URL in the date/title format, they could be converted over. Any links that don't have a matching date/title format can keep the archived URL because I can't predict the new URL.
- http://ew.com/ew/ 18
- https://ew.com/ew/ 188
- http://www.ew.com/ew/ 6000
- https://www.ew.com/ew/ 3154
These numbers include ones that are already fixed, such at the above link at Don't Tell Me. Thanks! MrLinkinPark333 (talk) 18:54, 4 July 2024 (UTC)
For #2 this is one of those dead redirects that eventually leads to the answer. It's a multi step process:
- Convert http://www.ew.com/ew/article/0,,302934,00.html to https://redir.ew.com/ew/article/0,,302934,00.html
- Run this which finds a redirect saved in the Wayback Machine:
wget -q -O- 'https://web.archive.org/cdx/search/cdx?url=https://redir.ew.com/ew/article/0,,302934,00.html&MatchType=prefix' | awk -v u="https://redir.ew.com/ew/article/0,,302934,00.html" '/text\/html 30[12]/{a[++i]=$2}END{print "https://web.archive.org/web/" a[i] "/" u}'
- With the answer from #2 run this:
curl -ILs 'https://web.archive.org/web/20160606115016/https://redir.ew.com/ew/article/0,,302934,00.html' | /usr/local/bin/awk '/^[ ]*[Ll]ocation:/{sub("^[ ]*[Ll]ocation:[ ]*https?://web[.]archive[.]org/web/[0-9]{14}id_/", "", $0); a[++i]=$0}END{print a[i]}'
- Which produces: https://web.archive.org/web/20151030104409/http://www.ew.com/article/1994/07/15/buckshot-lefonque .. from which extract the answer: http://www.ew.com/article/1994/07/15/buckshot-lefonque
For #3 same as #2, although in this example it leads to a soft-404.
For #4 same as #2. There are over 100,000 URLs in the wayback machine to redir.ew.com so there should be a good chance of finding, though not for this example. -- GreenC 22:27, 7 July 2024 (UTC)
- If that's easier than extracting the title/date, that could work. Didn't know this would be a complex change. Maybe more URLs could be fixed than I thought, even the ones that already have archived copies! MrLinkinPark333 (talk) 22:39, 7 July 2024 (UTC)
- Well, like in https://ew.com/ew/article/0,,302934,00.html there is no title/date in the URL string, and the page itself is dead, so the only way is search the Wayback Machine for old redirects. Unless you have another idea. It does appear to be working pretty well so far getting a lot, it's only slow because of the multiple I/O steps, and large number of links to process. -- GreenC 00:55, 8 July 2024 (UTC)
- I mean title/date in the citation would be added to the URL string like at buckshot. Then again, there's not always dates nor does the title always match the URL like for Heavy Competition. Searching https://web.archive.org/web/*/http://www.ew.com/article/2009/04/17/* for Heavy Competition doesn't give an archived updated URL. MrLinkinPark333 (talk) 20:37, 8 July 2024 (UTC)
- Ah yeah that method will be very hit or miss because it depends on the title in Wiki matching exactly the title in the URL. It worked for a two-word title, but I suspect for longer titles the URL drops words and punctuation. Maybe with a lot of experimenting. It also depends on the date being available. For now, I am doing the above old-redirects-in-wayback method which is getting a lot of positive results. -- GreenC 15:13, 9 July 2024 (UTC)
- I mean title/date in the citation would be added to the URL string like at buckshot. Then again, there's not always dates nor does the title always match the URL like for Heavy Competition. Searching https://web.archive.org/web/*/http://www.ew.com/article/2009/04/17/* for Heavy Competition doesn't give an archived updated URL. MrLinkinPark333 (talk) 20:37, 8 July 2024 (UTC)
- Well, like in https://ew.com/ew/article/0,,302934,00.html there is no title/date in the URL string, and the page itself is dead, so the only way is search the Wayback Machine for old redirects. Unless you have another idea. It does appear to be working pretty well so far getting a lot, it's only slow because of the multiple I/O steps, and large number of links to process. -- GreenC 00:55, 8 July 2024 (UTC)
- Yesterday there was a multirack hardware outage at archive.org that may take some time to fully recover, some of the services I need for this work are running slow or intermittent. -- GreenC 14:37, 8 July 2024 (UTC)
- MrLinkinPark333, I need to scale back the number of parallel processes to 2 because it's slow and tying up my rate-limited slots at the archive. This works but it will be running a long time. This way I can move on to other work. It will remain in "working" mode for a while, could be weeks not sure. Diff uploads will be in intermittent batches. -- GreenC 15:13, 9 July 2024 (UTC)
- No worries. It is a big request after all. MrLinkinPark333 (talk) 20:49, 9 July 2024 (UTC)
- MrLinkinPark333, I need to scale back the number of parallel processes to 2 because it's slow and tying up my rate-limited slots at the archive. This works but it will be running a long time. This way I can move on to other work. It will remain in "working" mode for a while, could be weeks not sure. Diff uploads will be in intermittent batches. -- GreenC 15:13, 9 July 2024 (UTC)
Bot results:
- NOTE: List of URL discoveries for future reference, or in case anyone needs it.
Enwiki: Done in 5 chunks, total was 8,673 pages
- Checked 1,000 pages and edited 804 pages. Moved 454 links to a new URL. Removed 1
{{dead link}}
templates. Added 4{{dead link}}
. Switched 164|url-status=dead
to live. Switched 97|url-status=live
to dead. Added 359 archive URLs (290 Wayback). Changed 106 citation metadata fields.
- Checked 1,000 pages and edited 910 pages. Moved 604 links to a new URL. Removed 1
{{dead link}}
templates. Added 5{{dead link}}
. Switched 286|url-status=dead
to live. Switched 78|url-status=live
to dead. Added 333 archive URLs (274 Wayback). Changed 113 citation metadata fields.
- Checked 2,220 pages and edited 1,761 pages. Moved 1,172 links to a new URL. Removed 2
{{dead link}}
templates. Switched 457|url-status=dead
to live. Switched 170|url-status=live
to dead. Added 724 archive URLs (581 Wayback). Changed 180 citation metadata fields.
- Checked 2,220 pages and edited 1,731 pages. Moved 1,126 links to a new URL. Removed 1
{{dead link}}
templates. Added 3{{dead link}}
. Switched 450|url-status=dead
to live. Switched 145|url-status=live
to dead. Added 736 archive URLs (657 Wayback). Changed 184 citation metadata fields.
- Checked 2,233 pages and edited 1,734 pages. Moved 1,187 links to a new URL. Removed 4
{{dead link}}
templates. Added 5{{dead link}}
. Switched 490|url-status=dead
to live. Switched 167|url-status=live
to dead. Added 724 archive URLs (619 Wayback). Changed 198 citation metadata fields.
IABot DB: Checked 14,591 unique links and changed 13,886 which will propagate across 300+ wikis via IABot
Done -- GreenC 05:27, 15 July 2024 (UTC)
pac-12.com
editHello. Pacific 12 Conference links are not working. Some can be moved to a new URL while others cant:
- /article/ link that has a new URL: this is now here - /article/ is swapped to /news/, and adds aspx to the end for John Ross (American football).
- If the day number starts with a 0, the 0 gets removed.as well like this is now here for N'Keal Harry.
- However, this does not always work. Changing this to that gives a 404 for 2018–19 Pac-12 Conference men's basketball season. The article is also missing from their archives.
Miscellaneous links that do not work that are not under the /article/ format include /content/, and /event/.
Thanks! MrLinkinPark333 (talk) 19:17, 5 July 2024 (UTC)
- Enwiki: Checked 879 pages and edited 825 pages. Moved 736 links to a new URL (per soft-redirect rules given above). Added 72
{{dead link}}
. Switched 9|url-status=dead
to live. Switched 45|url-status=live
to dead. Added 1,261 archive URLs (1,235 Wayback). Changed 135 citation metadata fields. - IABot DB: Checked 1,193 unique links and updated 890. Will propagate across 300+ wikis via IABot.
-- GreenC 04:35, 10 July 2024 (UTC)
- Looking through the remainder links, I see the third link above now redirects here. The date was wrong in the old URL.Therefore, I think this could be revisited later to see if any more links have been moved. I don't think it's worth going through them again now as I've only found that one so far today. MrLinkinPark333 (talk) 22:56, 10 July 2024 (UTC)
- Hmm that exposes a logic flaw in my bot. If there is a soft-redirect defined, it checks the status and when it fails, it assumes the URL is dead. In this case, the soft-redirect failed but a hard redirect existed. It adds overhead to check again so I never do, I am presuming the original link must be dead. As always, can't presume. The question is the overhead worth it - yes for this site, maybe not for others. It's worth retrying a set of articles to see what happens -- GreenC 01:04, 11 July 2024 (UTC)
- The link wasn't working on Friday but it is today. This makes sense as when I asked Pac-12 about the broken links on Friday, they said not all of the links were redirecting while they're reworking the site. MrLinkinPark333 (talk) 01:40, 11 July 2024 (UTC)
- Hmm that exposes a logic flaw in my bot. If there is a soft-redirect defined, it checks the status and when it fails, it assumes the URL is dead. In this case, the soft-redirect failed but a hard redirect existed. It adds overhead to check again so I never do, I am presuming the original link must be dead. As always, can't presume. The question is the overhead worth it - yes for this site, maybe not for others. It's worth retrying a set of articles to see what happens -- GreenC 01:04, 11 July 2024 (UTC)
- ω Awaiting site to be reworked -- GreenC 23:34, 11 July 2024 (UTC)
tampabay.com
editBoth /blogs/soundcheck
and /blogs/the-buzz-florida-politics
extensions seem to be removeable from the link to achieve a soft-redirect.
Examples:
- http://www.tampabay.com/blogs/soundcheck/xxxtentacion-announces-free-show-at-the-orpheum-in-tampa-this-saturday/2335772 soft-redirects to https://www.tampabay.com/xxxtentacion-announces-free-show-at-the-orpheum-in-tampa-this-saturday/2335772/ (from XXXTentacion)
- http://www.tampabay.com/blogs/the-buzz-florida-politics/rubio-comes-out-in-support-of-medical-marijuana-but-not-ballot/2190709 soft-redirects to https://www.tampabay.com/rubio-comes-out-in-support-of-medical-marijuana-but-not-ballot/2190709/ (from Marco Rubio)
GrapesRock (talk) 00:58, 7 July 2024 (UTC)
6,800 pages -- GreenC 16:58, 10 July 2024 (UTC)
Bot results:
- Enwiki: Checked 6,846 pages and edited 2,329 pages. Moved 2,533 links to a new URL. Removed 3
{{dead link}}
templates. Added 83{{dead link}}
. Switched 214|url-status=dead
to live. Switched 84|url-status=live
to dead. Added 944 archive URLs (751 Wayback). Changed 164 citation metadata fields. - IABot DB: Checked 9,980 unique links and updated 3,366 which will propagate across 300+ wikis via IABot
Started a thread about this at User talk:GreenC bot#Tampabay.com. ▶ I am Grorp ◀ 00:43, 12 July 2024 (UTC)
Done -- GreenC 04:35, 12 July 2024 (UTC)
bet.com
editBlack Entertainment Television (Paramount) - 2,100 pages -- GreenC 02:01, 7 July 2024 (UTC)
Soft-404 redirect rules:
bet-awards(/(nominees|performers|photos))?/?$
soul-train-awards(/nominees)?/?$
hip-hop-awards(/(nominees|photos|videos))?/?$
shows(/soul-train-awards)?/?$
vertical(/(jo1ilh/celebrity|o2fii9/news))?/?$
topic(/betexperience)?/?$
Bot results:
- Enwiki: Checked 2,143 pages and edited 1,522 pages. Moved 1,190 links to a new URL. Added 57
{{dead link}}
. Switched 36|url-status=dead
to live. Switched 71|url-status=live
to dead. Added 620 archive URLs (584 Wayback). Changed 263 citation metadata fields. - IABot DB: Checked 3,087 unique links and changed 1,164 which will propagate across 300+ wikis via IABot
Done -- GreenC 04:22, 13 July 2024 (UTC)
abclocal.go.com
editSoft-redirects that I found:
- kfsn -> abc30.com
- kgo -> abc7news.com
- /story?section=news/local/los_angeles -> abc7news.com
- kabc -> abc7.com
- kgo -> abc7.com
- wabc -> abc7ny.com
- wls -> abc7chicago.com
- wtvd -> abc11.com
- ktrk -> abc13.com
- http://abclocal.go.com/ktrk/story?section=news/local&id=8700136 soft-redirects to https://abc13.com/archive/8700136/ (from BP)
These ABC affiliates don't seem to work: wpvi, wjrt, wtvg
I'm sure that there's other ones that didn't appear in the first two pages of searching abclocal.go.com (and I can test them if you link them). I found the correct domain by searching the thing directly after .com/ (such as ktrk) on the internet and selecting the corresponding ABC affiliate (and found the Elton John one by searching the title given in the WP article).
(and a question: is this page just for moving *dead* links? I think all pages with espn.go.com in the domain now are either dead or redirect normally) GrapesRock (talk) 16:27, 7 July 2024 (UTC)
- Looks like some wpvi links actually do redirect, such as http://abclocal.go.com/wpvi/story?section=news/politics&id=6038619 to https://6abc.com/archive/6038619/ (from Dana Redd).
- Some, however, do not, such as http://abclocal.go.com/wpvi/story?section=entertainment&id=4498224 (from Elton John). GrapesRock (talk) 18:33, 7 July 2024 (UTC)
- To answer your question, the bot can/will also move redirects that it encounters. Assuming they pass any soft-404 rules. -- GreenC 19:24, 7 July 2024 (UTC)
- GrapesRock, good discovery. The bot checked all abclocal.go.com links on enwiki, it didn't find any additional affiliates. It attempted the soft-redirects per rules above. If not live, it added an archive URL. One thing I did not check, it's possible the new soft-redirect URL when it returns 404 it previously worked and thus has an archive URL, but the one's I manually checked either don't exist or end up soft-404 (redirect to home page), so I didn't look for those. I did check wjrt (abc12.com) and wtvg (13abc.com) for any working soft-redirects. -- GreenC 21:52, 13 July 2024 (UTC)
Bot results:
- Enwiki: Checked 1,857 pages and edited 1,422 pages. Moved 1,389 links to a new URL. Removed 13
{{dead link}}
templates. Added 74{{dead link}}
. Switched 650|url-status=dead
to live. Switched 18|url-status=live
to dead. Added 301 archive URLs (184 Wayback). Changed 30 citation metadata fields. - IABot DB: Checked 3,938 unique links and updated 3,889 which will propagate to 300+ wikis via IABot
Done -- GreenC 02:32, 14 July 2024 (UTC)
espn.go.com
editThe pages redirect to the espn.com domain per https://www.niemanlab.org/2016/08/espn-com-has-finally-replaced-espn-go-com-and-a-tweet-about-google-seo-may-be-part-of-why/ (though not all do).
28,843 pages GrapesRock (talk) 20:52, 7 July 2024 (UTC)
Soft-404 rules: over 100 - contact me for the list.
Bot results:
- Enwiki: done in segments of 3k to 8k pages
- (#1 to 5000) Checked 5,000 pages and edited 4,715 pages. Moved 11,901 links to a new URL. Removed 7
{{dead link}}
templates. Added 115{{dead link}}
. Switched 275|url-status=dead
to live. Switched 149|url-status=live
to dead. Added 1,219 archive URLs (887 Wayback). Changed 411 citation metadata fields. - (#13001 to 16000) Checked 3,000 pages and edited 2,830 pages. Moved 6,940 links to a new URL. Removed 3
{{dead link}}
templates. Added 50{{dead link}}
. Switched 166|url-status=dead
to live. Switched 63|url-status=live
to dead. Added 660 archive URLs (415 Wayback). Changed 260 citation metadata fields. - (#6001 to 13000) Checked 8,000 pages and edited 7,684 pages. Moved 17,056 links to a new URL. Removed 12
{{dead link}}
templates. Added 289{{dead link}}
. Switched 465|url-status=dead
to live. Switched 549|url-status=live
to dead. Added 3,661 archive URLs (2,989 Wayback). Changed 596 citation metadata fields. - (#1 to 5000 + #13001 to 16000) - reprocessed with new code updates: Checked 8,000 pages and edited 1,923 pages. Moved 2 links to a new URL. Switched 16
|url-status=dead
to live. Switched 529|url-status=live
to dead. Added 1,791 archive URLs (1,791 Wayback). Changed 546 citation metadata fields. - (#16001 to 22000) Checked 6,000 pages and edited 5,614 pages. Moved 12,433 links to a new URL. Removed 5
{{dead link}}
templates. Added 134{{dead link}}
. Switched 311|url-status=dead
to live. Switched 326|url-status=live
to dead. Added 2,610 archive URLs (2,017 Wayback). Changed 543 citation metadata fields. - (#1 to 8691) - different set: Checked 8,691 pages and edited 3,534 pages. Moved 2 links to a new URL. Switched 60
|url-status=dead
to live. Switched 476|url-status=live
to dead. Added 1,119 archive URLs (1,043 Wayback). Changed 1,633 citation metadata fields. - (#22001 to 28891) Checked 6,897 pages and edited 6,462 pages. Moved 14,724 links to a new URL. Removed 7
{{dead link}}
templates. Added 205{{dead link}}
. Switched 307|url-status=dead
to live. Switched 455|url-status=live
to dead. Added 3,393 archive URLs (1,776 Wayback). Changed 592 citation metadata fields.
- (#1 to 5000) Checked 5,000 pages and edited 4,715 pages. Moved 11,901 links to a new URL. Removed 7
- IABot DB: Processed 73,000 unique URLs and updated 16,000 in the database which will propagate to 300+ wikis via IABot. Note that an addition 30,000 links were not processed due to intractable problems with this domain and the Wayback Machine that is taking too long.
Done - This was a complex domain. It contains redirects, soft-redirects, soft-404s, crunchy-404s, and a lot of content drift (see WP:LINKROT#Glossary). The signals are in the URLs, page titles, and page body. I was able to discover over 100 rules, but many more in the "crunchy" realm went unaddressed, there are probably thousands. ESPN is a massive site. I focused on espn.go.com and it took over 10 days .. the espn.com domain is at least 7 times larger ie. 70 days of processing (est). It's beyond available resources and time it would probably require a team working full time for weeks, building a knowledge-base of how the ESPN site(s) are structured and changes over time. Then keeping up with new changes in the future. Unfortunately ESPN is not the only domain "too big to maintain" correctly. -- GreenC 20:22, 24 July 2024 (UTC)
social.techcrunch.com
editThere's server errors on pages in this domain, and removing "social" from the domain fixes them (and based on the archives of the two links they've had this server error since at least October 2023)
Examples (from Instagram):
- https://social.techcrunch.com/2021/05/04/instagram-adds-a-captions-option-for-stories-and-soon-reels/
- https://social.techcrunch.com/2020/11/14/this-week-in-apps-conservative-apps-surge-instagram-redesigned-tiktok-gets-ghosted/
1,818 pages GrapesRock (talk) 21:45, 7 July 2024 (UTC)
Done - Checked 1,812 pages and edited 1,809 pages. Moved 2,650 links to a new URL. Removed 336
{{dead link}}
templates. Added 3 {{dead link}}
. Switched 213 |url-status=dead
to live. Added 34 archive URLs (16 Wayback). Changed 1 citation metadata fields. -- GreenC 04:12, 26 July 2024 (UTC)
blogs.msdn.com
editThere are ~100 pages with URLs like http://blogs.msdn.com/b/oldnewthing/archive/2004/09/02/224672.aspx. These are dead but can be fixed by extracting the date from the URL and rewriting it as https://devblogs.microsoft.com/oldnewthing/20040902-00 ("-00" is static), which points to a list of blog posts on that day. Often there's only one, in which case it's unambiguous, but occasionally there's two or more and you need to disambiguate them somehow. Comparing the citation title with the article title is one possibility. (The destination URL for that instance is https://devblogs.microsoft.com/oldnewthing/20040902-00/?p=37983 with a numeric ID that can't be found any way I know of). * Pppery * it has begun... 23:02, 8 July 2024 (UTC)
- User:Pppery, I will do this, but the disambiguation based on
|title=
, probably not. Trying to read and match citation information breaks the model of the bot, it would require a special parser, can be difficult and error prone, and it's probably not that many pages (a percentage of 100 pages), could be done manually by someone faster and more accurately. I could probably parse the HTML to see when there are multiple blog items and log those URLs so we know which articles and URLs need checking. -- GreenC 05:16, 14 July 2024 (UTC)
Done Checked 626 pages and edited 46 pages. Moved 34 links to a new URL. Switched 4
|url-status=dead
to live. Switched 3 |url-status=live
to dead. Added 13 archive URLs (2 Wayback).
- * Pppery *, the bot converted 34 URLs. Of those, 10 were converted to the index page with multiple blog entries; they can remain as-is or someone can manually convert them to the exact page, listed above. There were another 5 dead links at devblogs.microsoft.com/oldnewthing. There are still some URLs in search, these are prior archive URLs now set live, or the 5 dead ones. -- GreenC 18:13, 26 July 2024 (UTC)
- I've disambiguated the ones that need disambiguating. * Pppery * it has begun... 23:34, 26 July 2024 (UTC)
billboard.com/bbcom
editHello. Billboard URLs with /bbcom/ are broken. I found this redirects to that for Because You Left. However, the rest of links I tried do not redirect.The numbers in the URL also don't match so it's not a simple find and replace.
- HTTPS - 300.
- HTTP - ~5600.
- HTTP without www - ~220
- HTTPS without www - 25
These include links not in mainspace and ones that are already archived. Thanks! MrLinkinPark333 (talk) 23:47, 14 July 2024 (UTC)
- I checked WaybackMachine for Ghost redirects, like with ew.com, but no ghosts. I guess the solution will be to archive (and any working redirects). -- GreenC 05:18, 15 July 2024 (UTC)
Done - Checked 1,937 pages and edited 1,163 pages. Moved 26 links to a new URL. Removed 1
{{dead link}}
templates. Added 634 {{dead link}}
. Switched 8 |url-status=dead
to live. Switched 55 |url-status=live
to dead. Added 649 archive URLs (639 Wayback). Changed 207 citation metadata fields. -- GreenC 22:12, 26 July 2024 (UTC)
- Many of these URLs first died over 15 years ago, possibly 20 or more, thus there are gaps in archival coverage as evidenced by "Added 634
{{dead link}}
". It was pre-Wayback and pre-Archive.today automatic archiving. -- GreenC 22:19, 26 July 2024 (UTC)
- Many of these URLs first died over 15 years ago, possibly 20 or more, thus there are gaps in archival coverage as evidenced by "Added 634
pqasb.pqarchiver.com
editHello again. There are a ton of pqarchiver links broken. These articles can be found at ProQuest but fall into 2 categories:
- 1. URLs with /doc/ can be converted into ProQuest URLs. this is now here for Here, My Dear. Some URLs have a different source type, such as converting this goes here for 1916 Michigan Agricultural Aggies football team and ends in Historical Newspapers. Template:ProQuest can help with this. ProQuest 565997669 points to this and will redirect to the right link for the Aggies article.
- 2. URLs that don't have /doc/ can't be converted as the new URLs don't match the number. Such as this is now here for 1974–75 Buffalo Sabres season.These need regular archives as I can't predict what the number will be in the new URL..
Please note that not all of these links are in articlespace. Also, some of these already have archived copies. Since it's so much, I don't mind if only the /doc/ URLs are focussed on and the other URLS dealt with later. Thanks again! MrLinkinPark333 (talk) 01:55, 17 July 2024 (UTC)
12,618 pages. Need to process all, to filter on the "/doc/" set, so I'll check the others for redirects, 404s, and https, hopefully not too many problems. If it gets messy will consider just focusing on /doc/ -- GreenC 22:34, 26 July 2024 (UTC)
MrLinkinPark333: I'm not sure which is better: archive or live link. The archive version has more information. They are both paywalls. My instinct is to leave the /doc/ pages as is, and add archives if not already. The entire pqasb.pqarchiver.com is dead. I'll start adding archives, and can go back for the /doc/ pages if decided. -- GreenC 05:06, 27 July 2024 (UTC)
- I think the /doc/ ones should be converted over since live links are available. ProQuest is available via The Wikipedia Library, so they are accessible without an account. The link for Here, My Dear in #1 works without a paywall. I don't know why some are and some are not. If they are converted, then perhaps each link can have added |url-access=subscription |via=ProQuest. MrLinkinPark333 (talk) 13:49, 27 July 2024 (UTC)
- The 'here my dear' gives a paywall, for me, you may be logged in somehow, try incognito. If you really think it should be done I'll do it. I don't want to get into adding new fields because that can get really complicated, I'll leave that for another bot that knows how to deal with conflicts with other fields and so on, like identifiers. Citation bot might have rules for it. -- GreenC 14:42, 27 July 2024 (UTC)
- Well, this may solve itself. They have rate limiting enabled via CloudFlare which is blocking my bot, checking the new link is live. I tried various techniques to get around but none work. I could do a blind move, search-replace the URL and not test, but with 4,000 links, the odds of problems are high, and I don't know what the error rate is. I can provide a list of the URLs, if anyone wants to work on testing headers return status 200. -- GreenC 17:02, 27 July 2024 (UTC)
- What I've found is that any links with /doc/ seems to match up with the new URL. Though I can't predict the source type. Otherwise, URLs without /doc/ don't seem to match up. See for instance this with that for Cultural impact of Mariah Carey. MrLinkinPark333 (talk) 18:28, 27 July 2024 (UTC)
- Source type is added automatically by the server side only need https://www.proquest.com/docview/283783667 works. Why don't we do this. I'll just manually check 50 URLs and see what the error rate is. -- GreenC 18:48, 27 July 2024 (UTC)
- ..checked 50 and error rate is 0%. That gives me more confidence to move forward with a blind move. -- GreenC 18:53, 27 July 2024 (UTC)
- Sounds good. Much easier to let the sourcetype be added automatically in the URL when viewing the link without it. MrLinkinPark333 (talk) 19:09, 27 July 2024 (UTC)
- ..checked 50 and error rate is 0%. That gives me more confidence to move forward with a blind move. -- GreenC 18:53, 27 July 2024 (UTC)
- Source type is added automatically by the server side only need https://www.proquest.com/docview/283783667 works. Why don't we do this. I'll just manually check 50 URLs and see what the error rate is. -- GreenC 18:48, 27 July 2024 (UTC)
- What I've found is that any links with /doc/ seems to match up with the new URL. Though I can't predict the source type. Otherwise, URLs without /doc/ don't seem to match up. See for instance this with that for Cultural impact of Mariah Carey. MrLinkinPark333 (talk) 18:28, 27 July 2024 (UTC)
- Well, this may solve itself. They have rate limiting enabled via CloudFlare which is blocking my bot, checking the new link is live. I tried various techniques to get around but none work. I could do a blind move, search-replace the URL and not test, but with 4,000 links, the odds of problems are high, and I don't know what the error rate is. I can provide a list of the URLs, if anyone wants to work on testing headers return status 200. -- GreenC 17:02, 27 July 2024 (UTC)
- The 'here my dear' gives a paywall, for me, you may be logged in somehow, try incognito. If you really think it should be done I'll do it. I don't want to get into adding new fields because that can get really complicated, I'll leave that for another bot that knows how to deal with conflicts with other fields and so on, like identifiers. Citation bot might have rules for it. -- GreenC 14:42, 27 July 2024 (UTC)
Enwiki in two batches:
- Batch 1: Checked 6,000 pages and edited 4,582 pages. Moved 1,716 links to a new URL ("doc" moves). Removed 58
{{dead link}}
templates. Added 754{{dead link}}
. Switched 453|url-status=dead
to live. Switched 443|url-status=live
to dead. Added 4,978 archive URLs (2,501 Wayback). - Batch 2: Checked 6,638 pages and edited 5,656 pages. Moved 1,708 links to a new URL ("doc" moves). Removed 86
{{dead link}}
templates. Added 845{{dead link}}
. Switched 338|url-status=dead
to live. Switched 180|url-status=live
to dead. Added 7,407 archive URLs (3,278 Wayback).
IABot db: Modified 54,000 URLs. Uploaded new archive URLs. Set status to dead. Unable to do the "/doc/" moves because IABot does not currently support URL moves. The changes will propagate to 300+ wikis.
Done -- GreenC 16:15, 30 July 2024 (UTC)
ieee.org
editMost of these (search link) are broken and can be replaced.
E.g.
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=933500&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F2%2F20203%2F00933500
can be replaced with
https://ieeexplore.ieee.org/document/933500/
as long as the first link is 404 and the second link resolves as 200.
The proper ID is written in the string arnumber=933500
Jonatan Svensson Glad (talk) 01:33, 19 July 2024 (UTC)
- OK. This is a soft-redirect, thank you for the information. I'll check the entire ieee.org domain to also look for soft-404s, and redirects. WP:LINKROT#Glossary. 8,800 pages. -- GreenC 16:26, 30 July 2024 (UTC)
Found about twenty soft-404 rules.
Enwiki done in two batches:
- Batch 1: Checked 1,000 pages and edited 501 pages. Moved 337 links to a new URL. Added 106
{{dead link}}
. Switched 4|url-status=dead
to live. Switched 11|url-status=live
to dead. Added 166 archive URLs (112 Wayback). Changed1,335citation metadata fields [bug in program, unsure the actual number] - Batch 2: Checked 7,898 pages and edited 3,943 pages. Moved 2,804 links to a new URL. Removed 3
{{dead link}}
templates. Added 575{{dead link}}
. Switched 19|url-status=dead
to live. Switched 78|url-status=live
to dead. Added 1,654 archive URLs (1,454 Wayback). Changed12,927citation metadata fields [bug in program, unsure the actual number]
IABot database: Checked ~25,000 links. Modified about 2,500. Changes will propagate to 300+ wikis.
Done -- GreenC 15:28, 31 July 2024 (UTC)
goo.gl
editGoogle has announced it will no longer support goo.gl shortened links after 25 August 2025. We have quite a lot of these in use on the Wiki at present. It may be necessary to look at a project/bot to replace them with the lengthy URLs. Stifle (talk) 07:52, 19 July 2024 (UTC)
- Google URL Shortener is about the service. https://developers.googleblog.com/en/google-url-shortener-links-will-no-longer-be-available/ also says: "Starting August 23, 2024, goo.gl links will start displaying an interstitial page for a percentage of existing links notifying your users that the link will no longer be supported after August 25th, 2025 prior to navigating to the original target page." PrimeHunter (talk) 08:56, 19 July 2024 (UTC)
- Thank you. I'll triage this after ESPN, above, which will be a couple more days at least. URL shortening was supposedly disallowed on Enwiki. The interstitial page could interfere with the bot, I'll start migrating sooner than later. -- GreenC 15:32, 19 July 2024 (UTC)
- meta:Spam blacklist says
\bgoo\.gl\b(?!/maps\b).*
, allowing goo.gl/maps. I guess the rationale was that it can only redirect to Google pages which aren't blacklisted so it cannot be used to bypass a blacklist entry, and it was probably assumed that Google would keep it working as long as they keep the target online. I think there was a time where Google itself gave goo.gl/maps links without the user asking for url shortening, so it would have been annoying if goo.gl/maps was blacklisted. PrimeHunter (talk) 20:34, 19 July 2024 (UTC)
- Ah thanks for the background that explains the distribution of links. -- GreenC 05:30, 20 July 2024 (UTC)
- meta:Spam blacklist says
- Note from Google: "In the event the interstitial page is disrupting your use cases, you can suppress it by adding the query param “si=1” to existing goo.gl links."
- Thank you. I'll triage this after ESPN, above, which will be a couple more days at least. URL shortening was supposedly disallowed on Enwiki. The interstitial page could interfere with the bot, I'll start migrating sooner than later. -- GreenC 15:32, 19 July 2024 (UTC)
- insource:goo.gl insource:/([.]|\/)goo[.]gl/ = 4,000+ pages.
- Over 90% are to goo.gl/maps, and about 350 to images.google.com where they are incorrectly/uselessly in the
|image=
field of infoboxen. -- GreenC 17:40, 19 July 2024 (UTC)
- Over 90% are to goo.gl/maps, and about 350 to images.google.com where they are incorrectly/uselessly in the
PrimeHunter or Stifle: Question: do you know anything about Google shortened URLs https://g.co/kgs/P8bHo7 ? insource:g.co insource:/\/\/g[.]co/ = 303 pages. Any reason not to expand these? -- GreenC 04:40, 25 July 2024 (UTC)
- I don't, but they should be expanded as a general good thing anyway. Stifle (talk) 07:35, 25 July 2024 (UTC)
- Many of them are used inappropriately for Google searches as references, for example:
- Macken, John (2008). "The Autonomy Theme in the Church Dogmatics: Karl Barth and his Critics". Archived from the original on 2024-02-19. Retrieved 2021-04-21.
- The link redirects to https://www.google.com/search?kgmid=/m/0c4k5ns&hl=en-US&q=The+autonomy+theme+in+the+Church+dogmatics+John+Macken&kgs=f8767d431dbb6421&shndl=0&source=sh/x/kp&entrypoint=sh/x/kp. It would be better if somebody is willing to improve them manually but it takes time. PrimeHunter (talk) 10:02, 25 July 2024 (UTC)
- Yeah URL shortening hides the true nature of dodgy links which go unnoticed. I think it's a step forward to expand them so someone can more easily discover them. -- GreenC 20:21, 25 July 2024 (UTC)
Done
- for goo.gl -- Checked 4,024 pages and edited 3,970 pages. Moved 5,198 links to a new URL. Added 11
{{dead link}}
. Added 79 archive URLs (12 Wayback).
- There are still some unconverted, insource:goo.gl insource:/([.]|\/)goo[.]gl/ shows 140 pages remain. Reasons include: bot inability to parse; marked with a
{{dead link}}
(cites should be removed or refactored); embedded in an archive URL.
- There are still some unconverted, insource:goo.gl insource:/([.]|\/)goo[.]gl/ shows 140 pages remain. Reasons include: bot inability to parse; marked with a
- for g.co -- Checked 303 pages and edited 295 pages. Added 45 archive URLs (0 Wayback). Changed 3 citation metadata fields.
ghostarchive.org
editWeb archive provider GhostArchive is dead as of July 19. insource:ghostarchive insource:/ghostarchive[.]org/ = 66,000 pages to be converted/deleted. -- GreenC 21:20, 19 July 2024 (UTC)
- I just tried two links [16] from England national football team and [17] from YouTube and they both seemed to work? GrapesRock (talk) 03:14, 20 July 2024 (UTC)
- Same, GhostArchive works fine for me... don't know why GreenC is/was having trouble. Nex 🌐 📰 leave a message 04:05, 20 July 2024 (UTC)
- Interesting! https://ghostarchive.org/ doesn't work .. and this admittedly hearsay post that said it is defunct, with a specific reason: Special:Diff/1230523321/1235539197. If anyone finds more information, let me know. I won't do anything until the situation is understood. -- GreenC 05:01, 20 July 2024 (UTC)
- Looks like I'm still able to archive. For instance, I just archived https://example.com here.
- Edit: looks like 10 hours ago people were having similar issues on lostmediawiki forum, though they don't give an explanation. GrapesRock (talk) 06:38, 20 July 2024 (UTC)
- Can you access https://ghostarchive.org/ ? -- GreenC 17:55, 20 July 2024 (UTC)
- Yeah. I'd presume it's still down for you? GrapesRock (talk) 17:57, 20 July 2024 (UTC)
- Yes, down. It appears to be local to my Firefox browser, works on a different browser. The oddity is other people reported an outage. Maybe something with SSL certificate or DNS. -- GreenC 19:45, 20 July 2024 (UTC)
- Yeah. I'd presume it's still down for you? GrapesRock (talk) 17:57, 20 July 2024 (UTC)
- Can you access https://ghostarchive.org/ ? -- GreenC 17:55, 20 July 2024 (UTC)
- Interesting! https://ghostarchive.org/ doesn't work .. and this admittedly hearsay post that said it is defunct, with a specific reason: Special:Diff/1230523321/1235539197. If anyone finds more information, let me know. I won't do anything until the situation is understood. -- GreenC 05:01, 20 July 2024 (UTC)
- Same, GhostArchive works fine for me... don't know why GreenC is/was having trouble. Nex 🌐 📰 leave a message 04:05, 20 July 2024 (UTC)
Not done - false alarm -- GreenC 01:48, 21 July 2024 (UTC)
- Note: Ghostarchive keeps getting down at random times, i am thisisanusername123 on that forum
- The website showed "Unknown domain" for me Notrealname1234 (talk) 15:01, 22 July 2024 (UTC)
- It did for me also, I suspect something got corrupted with SSL (https) certificates, or cookies, or DNS cache, and then stuck in the local browser cache. If it happens again try clearing browser cache or a different browser. There are intermittent problems with WebCite and Archive.today also that effect some users and not others. -- GreenC 16:12, 22 July 2024 (UTC)
hp.vector.co.jp
edithttps://cohost.org/gosokkyu/post/6918235-heads-up-jp-web-arc
Seems like that web hosting service is shutting down, there are about 31 links in enwiki, there are possibly more at jawiki. Notrealname1234 (talk) 15:06, 22 July 2024 (UTC)
- There are 159 links in jawiki. Notrealname1234 (talk) 15:27, 22 July 2024 (UTC)
Notrealname1234: Thank you for the notification. They are deleting all pages December 20, 2024. IABot has registered 133 unique URLs across 300+ wikis including jawiki. IABot has been disabled on jwiki since early 2023 and no idea when it will return. Well, I can do this on enwiki, and update the 133 URLs in IABot, which will save them on jawiki whenever it is enabled. They are still live but I'll treat them as dead. Might be a few weeks (above work ahead). -- GreenC 17:25, 22 July 2024 (UTC)
Done on enwiki and IABot database (133 unique links). -- GreenC 01:20, 1 August 2024 (UTC)
- Thanks! Notrealname1234 (talk) 23:40, 1 August 2024 (UTC)
forumromanum.org
editWP currently has 291 links beginning with "http://www.forumromanum.org/"; but the siteowner deleted the site, and the domain name has fallen into the hands of a scammer. (Do not go that link! you will get a browser freeze with a scam message telling you to call a phone number etc.)
Fortunately, the site — a very valuable source of classical Latin texts, often not to be found elsewhere online — was backed up at WebArchive, and as far as I can tell all its pages can be reached by a simple substitution in the URL. The global-replace pattern is:
replace
by
I have first-hand experience of the scammer splash screen; and, separately, confirmation of the removal of the site by David Camden the original site owner himself. For privacy, I'm of course not posting his e‑mail or details, but I can be reached by e‑mail to confirm privately if need be.
Bill (talk) 20:29, 22 July 2024 (UTC)
- No problem, have done this before per procedure WP:USURPURL. Sounds like a nasty malware site I have increased the priority. -- GreenC 20:49, 22 July 2024 (UTC)
- On behalf of many people out there, thank you! Bill (talk) 20:52, 22 July 2024 (UTC)
- https://urlscan.io/result/81c76ba4-c427-49d1-bd4c-e446e3d16cc7/
- Seems like it's a parking page now. Notrealname1234 (talk) 21:15, 22 July 2024 (UTC)
- oh wait, still a malware attack. Notrealname1234 (talk) 21:18, 22 July 2024 (UTC)
- On behalf of many people out there, thank you! Bill (talk) 20:52, 22 July 2024 (UTC)
Done in a JUDI batch with 27 other domains usurped: Special:Diff/1236485623/1236486118. -- GreenC 00:11, 25 July 2024 (UTC)
- Nifty! and nice detective work on your part, fixing 27 other problems as well! Bill (talk) 14:06, 29 July 2024 (UTC)
popmani.se
editPopmani.se has been usurped by a casino advertising/SEO spam operation AlexandraAVX (talk) 17:45, 23 July 2024 (UTC)
- Added to the usurpation queue: Special:Diff/1236087736/1236249622 -- GreenC 17:52, 23 July 2024 (UTC)
Done -- GreenC 00:12, 25 July 2024 (UTC)
slate.msn.com
editHello. slate.msn.com doesn't work. These have archived redirects and also working redirects. Here are examples:
- For Raging Cow, changing this to that by removing msn from the URL redirects to the new link here.
- If that doesn't work, I've seen archived redirects. This goes here for Peter Maass. Removing the archive from the URL makes a redirect to this new URL.
- Redirects also exist without msn. This link goes here for Amazon Theater. Removing the archived part redirects to the new URL here.
~300 links. URLs such as fray.state.msn.com or cagle.slate.msn.com would need regular archives. These links also include ones not in Articlespace, such as talk pages. Thanks! MrLinkinPark333 (talk) 18:57, 26 July 2024 (UTC)
- In the third example, this returns a header status 200 and no redirect information, so curl can't see the redirect. It's being redirected by JavaScript. Hopefully an edge case. -- GreenC 23:53, 1 August 2024 (UTC)
- The bot got it right anyway: Special:Diff/1225902007/1238080300 - it followed the logic of the first example and that worked. Same with the second example, it followed the logic of the first example and it worked. -- GreenC 01:04, 2 August 2024 (UTC)
I was able to convert 53 URLs, and not convert 10:
- Albert Gore Sr. ---- http://slate.msn.com/ebooks/Sons%20George%20W.%20Bush%20and%20Al%20Gore.htm
- George W. Bush ---- http://politics.slate.msn.com/Features/bushisms/bushisms.asp
- No Fly List ---- http://slate.msn.com/id/2113157/fr/rss/
- Security theater ---- http://slate.msn.com/id/2113157/fr/rss/
- Godzilla 2000 ---- http://slate.msn.com/default.aspx?id=88714
- Charlie's Angels (2000 film) ---- http://slate.msn.com/default.aspx?id=92656
- List of films featuring giant monsters ---- http://slate.msn.com/default.aspx?id=88714
- Homeobox protein CDX-2 ---- http://slate.msn.com/id/2110670/fr/rss/
- List of monster movies ---- http://slate.msn.com/default.aspx?id=88714
Done Checked 103 pages and edited 53 pages. Moved 53 links to a new URL. Removed 1
{{dead link}}
templates. Switched 41|url-status=dead
to live. Added 2 archive URLs (2 Wayback). Changed 4 citation metadata fields.
This was a twister if you see anything I missed let me know. search, it might take time for the search cache to reflect the edits. -- GreenC 00:55, 2 August 2024 (UTC)
- For No Fly List, making the link into here (without fr/ss) works as a redirect to there. For Godzilla 2000, making the link into here works as a redirect to there (by removing default and change id= to /id/). No luck with Albert Gore Sr. George W. Bush at Slate doesn't match the article either, so it could be left archived. MrLinkinPark333 (talk) 01:32, 2 August 2024 (UTC)
- OK. If you want to adjust those manually it won't make sense to program and run the bot for these edge cases. -- GreenC 01:44, 2 August 2024 (UTC)
- Fair enough. MrLinkinPark333 (talk) 02:25, 2 August 2024 (UTC)
- OK. If you want to adjust those manually it won't make sense to program and run the bot for these edge cases. -- GreenC 01:44, 2 August 2024 (UTC)
- @GreenC: The bot now changes perfectly fine refs that were properly waybacked and marked as 'dead'. This is pointless. In fact, I would argue it's worse. See this edit at Pokémon. This is the waybacked page from slate.msn.com. This is the new page from slate.com.
- When I wrote the paragraph in question, I purposely chose the waybacked old page, because the new page is filled with ads and has a very annoying floating, picture-in-picture video that automatically starts playing when the page loads.
- On a positive note, the ad blocker not only blocks this, but also busts through the "You seem to have an ad blocker" message. So the ad blocker does work here. But not everyone has an ad blocker installed. - Manifestation (talk) 09:18, 2 August 2024 (UTC)
- I understand. Yeah this is murky territory because if we are using the Wayback Machine to intentionally bypass a website, that otherwise has live content available, it is undermining traffic to the website, and traffic is why websites exist. In response, there is nothing stopping Slate from making a takedown request at Wayback. The entire domain would be taken down, leaving us with no archives even for legitimately dead links (except archive.today who do not honor most take down requests). This is not hypothetical it is happening more frequently. Anyway, I didn't remove the archive URL, and it can be flipped back to dead status, the bot won't reprocess the domain anytime in the foreseeable future. -- GreenC 13:35, 2 August 2024 (UTC)
msnbc.msn.com
editHello again. Msnbc.msn.com links don't work. Some have redirects that work while other's dont. Please note that they redirect to NBC News links. This falls under two categories:
- URLs with IDs
- Changing this to that makes a working redirect for Latrobe Brewing Company.
- Changing this to that redirects to a 404 for Disappearance of Lisa Stebic.
- Sometimes removing parts of the URL will create a valid redirect. For instance, making this Today MSNBC link into that redirects here for Kieron Williamson.
- URLs without IDs: URLs with dates such as this don't work. In this case, it already has an archived URL at 2012 Leap Day tornado outbreak.
~12,500 links. Not all of these are in mainspace. MrLinkinPark333 (talk) 21:10, 28 July 2024 (UTC)
- MrLinkinPark333, for the first two examples, "this" and "that" are the same URL (copy paste typo). I'll need the "that" URL you discovered works. -- GreenC 01:15, 2 August 2024 (UTC)
- Whoops. That is supposed to be here. MrLinkinPark333 (talk) 01:17, 2 August 2024 (UTC)
- OK it's a soft-redirect -> redirect -> destination: Any URL that contains "/id/", extract the ID and convert to "https://www.msnbc.com/id/{id}/" -- thus
http://today.msnbc.msn.com/id/43584191/ns/today-today_people/t/monaco-palace-releases-guest-list-royal-wedding/
converts tohttps://www.msnbc.com/id/43584191/
.. then follow the redirect to https://www.nbcnews.com/id/wbna43584191 -- GreenC 03:56, 2 August 2024 (UTC)
- OK it's a soft-redirect -> redirect -> destination: Any URL that contains "/id/", extract the ID and convert to "https://www.msnbc.com/id/{id}/" -- thus
- Whoops. That is supposed to be here. MrLinkinPark333 (talk) 01:17, 2 August 2024 (UTC)
Enwiki:
- Checked 3,616 pages and edited 1,288 pages. Converted 1 templates. Moved 725 links to a new URL. Removed 4
{{dead link}}
templates. Added 291{{dead link}}
. Switched 661|url-status=dead
to live. Switched 20|url-status=live
to dead. Added 182 archive URLs (132 Wayback). Changed 213 citation metadata fields.
IABot DB:
- About 17,000 links. Updated about 12,500 links which will propagate to 300+ wikis via IABot. -- GreenC 01:51, 3 August 2024 (UTC)
Done
smmsport.com
editSmmsport.com appears to have been usurped by an online gambling operation masquerading as the original site. Some links, such as [18] and [19], appear to still work and are intact with their original content, while others return 404 errors. But anything linked from the home page is fake. --Paul_012 (talk) 11:09, 29 July 2024 (UTC)
- User:Paul 012: 400+ pages. I'm not seeing gambling pages. Can you find examples? -- GreenC 15:27, 29 July 2024 (UTC)
- They're somewhat insidiously inserted into the first top navigation menu. [20] for example is a link farm advertising gambling sites. --Paul_012 (talk) 15:34, 29 July 2024 (UTC)
- Ahh I see. This is somewhat unusual case of WP:USURPSOURCE. Probably we need an edit filter to prevent editors from adding more links they believe are legitimate, but actually insidious spam (ie. MediaWiki_talk:Spam-blacklist#Proposed_additions). And the existing links usurped by WaybackMedic (ie. this URLREQ). As the primary discoverer, can you make the Spam Blacklist request? -- GreenC 16:54, 29 July 2024 (UTC)
- I added it to the usurpation queue for WaybackMedic Special:Diff/1236486118/1237406269 -- GreenC 16:58, 29 July 2024 (UTC)
- Thanks. I'm not sure about blacklisting, as their old articles could still be useful references. Also, upon closer look, it seems the situation looks more like a hijacking rather than usurpation? Checking the Wayback Machine, the last good version of the home page was archived on 2023-08-13, before the site went down and showed a domain for sale notice. It came back on 2024-06-15, appearing mostly the same as it last did, but by the next archival on 2024-07-02 the gambling links had been inserted into the navigation menu, and the articles linked from the home page had been altered to show a date of 23 May 2024. --Paul_012 (talk) 14:27, 30 July 2024 (UTC)
- The spam blacklist prevents adding new links. Since they appear to have legitimate content, this is a problem editors unknowingly adding new links into Wikipedia, that they found with Google or whatever. It is a classic case of WP:USURPSOURCE. It really needs to be blocked. The old links will be kept and converted to usurped ie. changed to archive URLs, and the source URL no longer hot linked. -- GreenC 14:45, 30 July 2024 (UTC)
- Block request: MediaWiki_talk:Spam-blacklist#smmsport.com -- GreenC 15:54, 31 July 2024 (UTC)
- The spam blacklist prevents adding new links. Since they appear to have legitimate content, this is a problem editors unknowingly adding new links into Wikipedia, that they found with Google or whatever. It is a classic case of WP:USURPSOURCE. It really needs to be blocked. The old links will be kept and converted to usurped ie. changed to archive URLs, and the source URL no longer hot linked. -- GreenC 14:45, 30 July 2024 (UTC)
- Thanks. I'm not sure about blacklisting, as their old articles could still be useful references. Also, upon closer look, it seems the situation looks more like a hijacking rather than usurpation? Checking the Wayback Machine, the last good version of the home page was archived on 2023-08-13, before the site went down and showed a domain for sale notice. It came back on 2024-06-15, appearing mostly the same as it last did, but by the next archival on 2024-07-02 the gambling links had been inserted into the navigation menu, and the articles linked from the home page had been altered to show a date of 23 May 2024. --Paul_012 (talk) 14:27, 30 July 2024 (UTC)
- They're somewhat insidiously inserted into the first top navigation menu. [20] for example is a link farm advertising gambling sites. --Paul_012 (talk) 15:34, 29 July 2024 (UTC)
On hold for next JUDI run. -- GreenC 05:01, 16 August 2024 (UTC)
businessinsider.com.au
edithttps://www.businessinsider.com.au/coronavirus-us-has-worlds-biggest-outbreak-topping-china-2020-3?r=US&IR=T soft-redirects to https://www.businessinsider.com/coronavirus-us-has-worlds-biggest-outbreak-topping-china-2020-3?r=US&IR=T (with the referral ?r being optional) (from Timeline of the COVID-19 pandemic in the United States (2020)). Simply removing the .au generalizes to all the businessinsider.com.au links that I checked.
Per The Sydney Morning Herald, they "will no longer produce editorial content for Insider/BI and there will not be a BIAUS website", so I think it's safe to assume these links are not gonna come back to this domain.
821 pages GrapesRock (talk) 16:05, 30 July 2024 (UTC)
- Checked 823 pages and edited 739 pages. Moved 570 links to a new URL. Removed 3
{{dead link}}
templates. Added 9{{dead link}}
. Switched 81|url-status=dead
to live. Switched 38|url-status=live
to dead. Added 214 archive URLs (204 Wayback). Changed 22 citation metadata fields.
fortblissbugle.com
editfortblissbugle.com has been usurped by a gambling website. One example is http://fortblissbugle.com/german-air-force-train-at-fort-bliss/ from Fort Bliss.
While, this claims that it's moved to an army website, that website's news archive only goes back to October 24, 2019, a week before the fortblissbugle went offline. Just searching a handful of titles, I can't find anywhere where individual stories are hosted.
46 pages GrapesRock (talk) 18:53, 30 July 2024 (UTC)
- Page says JUDIKING88 at the top. Judi is Indonesian for gambling. Part of the global judi empire. Added to WP:JUDI for later usurpation Special:Diff/1237845244/1238103384 -- GreenC 04:19, 2 August 2024 (UTC)
On hold for next JUDI run. -- GreenC 05:01, 16 August 2024 (UTC)
nbcnews.com/id
editHello. NBC News links with /id/ in the URL redirect to new links. For example, this goes here for General Electric. However, this not always work:
- Keeping only the id number sometimes makes a valid redirect: changing this to that goes to here for Chicken or the egg.
- However, keeping only the id in the URL doesn't always work. Making this into that redirects to a 404 for Legality of euthanasia. The new URL is here and does not match up. I think it would be better to find archived copies for these pages that redirects to 404s as I can't predict the new URL.
- Also, at times links will give a "Something Went Wrong" error but still work after refreshing the page. This happened to me after changing this to the new URL for David Yalof.
~7250. Any links with /id/wbna after the above msnbc request above can be ignored as they will be already fixed.
Thanks! MrLinkinPark333 (talk) 00:18, 31 July 2024 (UTC)
- User:MrLinkinPark333, for the "Something Went Wrong", I tried the example and it never loads after repeat refresh. A header check returns "HTTP/1.1 500 Internal Server Error". 500 is a generic error code when no more specific error code is available. I tried with a proxy sock IP (VPN) and it returns 206, which is sort of like saying it's a partial shipment, only one data segment arrived, more typical of large data files or video files. These are weird responses both are rare. The archive version (few days ago) is of a normal news article. I think the conservative solution is treat them as dead for now until NBC works out whatever went wrong. I'll test and see what percentage are like this. -- GreenC 02:18, 3 August 2024 (UTC)
- 24% of the links are "Something Went Wrong". 1,767 out of 7,423 .. the others converted successfully. Retries after hours pause makes no difference. Now the proxy does not work either. I don't have much option but consider them dead links. If this problem lifts in the future it can be reprocessed (note to self: find links in project nbcnewscom.0001-8263 with "grep 'Went Wrong' syslog"). -- GreenC 14:55, 3 August 2024 (UTC)
- I didn't realize so many of them would not work. It makes sense to have archived copies now, even if temporarily. MrLinkinPark333 (talk) 15:59, 3 August 2024 (UTC)
- 24% of the links are "Something Went Wrong". 1,767 out of 7,423 .. the others converted successfully. Retries after hours pause makes no difference. Now the proxy does not work either. I don't have much option but consider them dead links. If this problem lifts in the future it can be reprocessed (note to self: find links in project nbcnewscom.0001-8263 with "grep 'Went Wrong' syslog"). -- GreenC 14:55, 3 August 2024 (UTC)
- Enwiki: Checked 8,263 pages and edited 6,637 pages. Converted 1 templates. Moved 5,660 links to a new URL. Removed 2
{{dead link}}
templates. Added 387{{dead link}}
. Switched 50|url-status=dead
to live. Switched 320|url-status=live
to dead. Added 2,072 archive URLs (1,979 Wayback). Changed 230 citation metadata fields.
Done -- GreenC 16:01, 3 August 2024 (UTC)
Can this be run in Tewiki?
edit@User:GreenC, In Tewiki, we have more than 10,400 pages in the category CS1 errors: archive-url. Almost 99% of these are "timestamp mismatch" errors. Can you plesase run WaybackMedic_2.5 to correct the error in these pages? Thank you. __ Chaduvari (talk) 15:59, 31 July 2024 (UTC)
- Ahh. I'd like to, but I am not setup for other wikis very difficult. The CS1 error: archive-url is across most wikis. Let me think about it because it's a growing problem. It might be I can process, but only some English-language templates like
{{cite web}}
that use English-language parameters like|archive-url=
. GreenC 19:25, 31 July 2024 (UTC)- Hi GreenC, in tewiki, this template, like many others, use English parameters and templates only. This policy was kept to ensure future compatibility. Thanks. __ Chaduvari (talk) 09:29, 12 August 2024 (UTC)
- User:Chaduvari, I could try some tests for Telugu Wiki. Can you help me get bot flag permissions for User:GreenC bot? I don't know where to start to ask permission. -- GreenC 18:19, 12 August 2024 (UTC)
- @GreenC, you can raise the request at te:వికీపీడియా:Bot/Requests for approvals.__ Chaduvari (talk) 23:40, 12 August 2024 (UTC)
- I made a request for approval. -- GreenC 02:30, 13 August 2024 (UTC)
- @GreenC, you can raise the request at te:వికీపీడియా:Bot/Requests for approvals.__ Chaduvari (talk) 23:40, 12 August 2024 (UTC)
- User:Chaduvari, I could try some tests for Telugu Wiki. Can you help me get bot flag permissions for User:GreenC bot? I don't know where to start to ask permission. -- GreenC 18:19, 12 August 2024 (UTC)
- Hi GreenC, in tewiki, this template, like many others, use English parameters and templates only. This policy was kept to ensure future compatibility. Thanks. __ Chaduvari (talk) 09:29, 12 August 2024 (UTC)
onlinelibrary.wiley.com
editAll links (that I have checked) starting with https://onlinelibrary.wiley.com/store/ seems to be dead. Replacing them to start with https://onlinelibrary.wiley.com/doi/ instead, seem to make those links to work (example).
Perhaps more URLS to Wiiley with other paths has died but can be saved if replacing the path (in above example /store/
) with /doi/
. Mind checking? Jonatan Svensson Glad (talk) 23:29, 31 July 2024 (UTC)
Jonatan Svensson Glad, the site uses CloudFlare bot protection. I can't verify if the new URL is live/dead or redirects. Because there are so few, and this seems like it should work, I'll do a blind move. Worst case, I can change them back to /store/. -- GreenC 18:30, 3 August 2024 (UTC)
- Checked 111 pages and edited 111 pages. Moved 123 links to a new URL. Removed 9
{{dead link}}
templates. Switched 3|url-status=dead
to live. Added 1 archive URLs (1 Wayback).
Done -- GreenC 18:49, 3 August 2024 (UTC)
Jonatan Svensson Glad: On a related note, I spot-checked the edits, and in all cases they were part of citation templates where there was a |doi=
parameter that also goes to the same target. Given these |url=
point to the content via their DOI, cite-template docs advise against including the URL at all. There are about 16k links to wiley.com/doi URLs and some do not have separate DOI fields, so it would be a harder bot task to fix them. DMacks (talk) 19:10, 3 August 2024 (UTC)
- Can Citation bot fix these? I recall it removed URLs when there is a duplicate identifier URL, but it was also controversial in some way, and can't recall how it settled. -- GreenC 19:18, 3 August 2024 (UTC)
- If there is a PMC link (which is open access) or
|doi-access=free
, then Citation bot removes the URL to some specific domains but not all, unsure which specific domains though. This since, the title will use the PMC or free DOI ink instead. Jonatan Svensson Glad (talk) 19:39, 4 August 2024 (UTC)
- If there is a PMC link (which is open access) or
gameinformer.com
edithttps://www.gameinformer.com
& https://gameinformer.mydigitalpublication.com
- Kotaku just highlighted that GameStop killed Game Informer. Looks like the articles are redirecting to the front page farewell message. For example, these two sources are dead (both are archived):
- https://www.gameinformer.com/2020/11/05/dragon-age-4-theory-solas-red-lyrium-and-blight-ambitions (format for website articles)
- https://gameinformer.mydigitalpublication.com/publication/?i=824318 (format for the magazines)
Thanks! Sariel Xilo (talk) 17:55, 2 August 2024 (UTC)
- Just doubled checked that magazine example and while it was archived a few times, the magazine doesn't appear to load & just shows a spinning waiting icon. So those might be total dead links if the Internet Archive copies don't work. Sariel Xilo (talk) 18:13, 2 August 2024 (UTC)
- I was about to post this website to here. Notrealname1234 (talk) 21:47, 2 August 2024 (UTC)
- Sariel Xilo, I guess it won't matter for gameinformer.mydigitalpublication.com because there are only 2 pages .. gameinformer.com has over 6,000 pages. -- GreenC 23:09, 2 August 2024 (UTC)
I'm assuming every link in the domain is functionally dead. I'm not verifying that assumption, because they use JavaScript redirects, which I can't detect, thus every page appears to be status 200 (live) which is actually a soft-404 to an end-of-life page. If after the bot is done anyone sees a problem with a link still live but marked dead, I can investigate and redo those links. -- GreenC 01:06, 4 August 2024 (UTC)
- Checked 6,484 pages and edited 4,349 pages. Added 58
{{dead link}}
. Switched 3,867|url-status=live
to dead. Added 3,182 archive URLs (3,024 Wayback). Changed 75 citation metadata fields.
- Thanks! Sariel Xilo (talk) 17:16, 4 August 2024 (UTC)
- User:Sariel Xilo, I forgot to load IABot's database with archive URLs. I did set the domain status to "permadead" at iabot.org, but IABot can't discover archive.today links which make up a sizeable portion of available archives. Once finished the Highway Administration site below I'll return to this. There are 3,400 unique URLs. -- GreenC 20:04, 4 August 2024 (UTC)
- Added to IABot database.
Done -- GreenC 14:50, 5 August 2024 (UTC)
fhwa.dot.gov
editLinks to many, but not all, pages under http://www.fhwa.dot.gov/environment, http://www.fhwa.dot.gov/planning/, and http://www.fhwa.dot.gov/hep10, are dead.
http://www.fhwa.dot.gov/reports/routefinder/ is also a redirect. RajanD100 (talk) 19:30, 3 August 2024 (UTC)
- Well, their 404 page is misconfigured to return status 200 (live), example. I'll need to download every URL and web scrape for key words. This kind of basic problem with website management portends other more difficult ones. There are 3,000 pages (articles) on Wikipedia with this domain. -- GreenC 17:53, 4 August 2024 (UTC)
Enwiki in two batches:
- Batch 1: Checked 1,000 pages and edited 738 pages. Moved 718 links to a new URL. Added 3
{{dead link}}
. Switched 12|url-status=dead
to live. Switched 11|url-status=live
to dead. Added 196 archive URLs (191 Wayback). Changed 76 citation metadata fields. - Batch 2: Checked 2,000 pages and edited 1,579 pages. Moved 1,582 links to a new URL. Added 6
{{dead link}}
. Switched 28|url-status=dead
to live. Switched 19|url-status=live
to dead. Added 483 archive URLs (469 Wayback). Changed 179 citation metadata fields.
IABot DB:
- Checked about 2,000 unique URLs and modified about 400 which will propagate to 300+ wikis via IABot.
Done -- GreenC 17:34, 5 August 2024 (UTC)
ts.fi
editI noticed that some of the Turun Sanomat URLs result in a 404 error, but they seem to be easily fixable:
- Ostrobothnians:
https://www.ts.fi/puheenvuorot/1073936480/Suomen+heimojen+peruspiirteet
gives a 404, but if you remove everything after the number ID, including the last/
(as highlighted in red in the previous URL), the URL works again:https://www.ts.fi/puheenvuorot/1073936480
- Night Visions (film festival):
http://www.ts.fi/kulttuuri/1073969115/Night+Visions+laajeneekolmipaivaiseksi
->http://www.ts.fi/kulttuuri/1073969115
- Jukka Kalso:
https://www.ts.fi/urheilu/1073750270/Soinisen+paa+kestaa
->https://www.ts.fi/urheilu/1073750270
There are approximately a hundred of these: 116 results (probably includes some false positives, i.e. archived URLs). --JAAqqO (talk) 22:49, 4 August 2024 (UTC)
- There is more, for example http://www.ts.fi/uutiset/talous/590113/Artekille+myos+Littoisten+Korhosen+tehdas becomes http://www.ts.fi/uutiset/590113 -- GreenC 17:53, 5 August 2024 (UTC)
- More: https://www.ts.fi/urheilu/jalkapallo/liiga/1207968074/Interin+hyokkaaja+debytoi+liigassa+vanhaa+seuraansa+vastaan --> https://www.ts.fi/urheilu/1207968074
- In one case out of 75, did not work: http://www.ts.fi/mielipiteet/paakirjoitukset/1073950477/Odotettu+fuusio+selkeyttaaSuomen+telakoiden+tilannetta -- GreenC 18:03, 5 August 2024 (UTC)
Enwiki: Checked 452 pages and edited 321 pages. Moved 315 links to a new URL. Added 15 {{dead link}}
. Switched 29 |url-status=dead
to live. Added 31 archive URLs (23 Wayback). Changed 92 citation metadata fields.
Done -- GreenC 19:07, 5 August 2024 (UTC)
- That was fast, thank you. I checked about 50 affected articles on my watchlist, and all the new ts.fi URLs now work in those articles. However, I noticed one problematic edit, but I believe I found the rest of the erroneous edits, as they all appeared to be URLs with unusual characters (colons, semicolons, question marks, commas): edit #1, #2, #3, #4, #5. I found working URLs for them by checking the edit histories (except for this one that seems to be permanently dead), so everything should be good now. Thanks again. --JAAqqO (talk) 20:52, 5 August 2024 (UTC)
- Ah yes those URLs I came across and intentionally re-routed to the home page because they were redirecting there anyway as soft-404s (WP:LINKROT#Glossary) and they looked like errors anyway. These are in fact soft-redirects, which requires foreknowledge or search and discovery to determine the correct destination. -- GreenC 23:36, 5 August 2024 (UTC)
cdc.gov
editCDC recently overhauled their website. Many links now have this interstitial saying the page has moved while linking to the new one. For example: https://www.cdc.gov/niosh/topics/motorvehicle/ -- in defiance of standards, that URL returns a 404 instead of a 301
-- GreenC 18:15, 5 August 2024 (UTC)
On hold - pending how to retrieve the redirect URL. -- GreenC 18:41, 5 August 2024 (UTC)
uk.businessinsider.com
editThis link from Antony Jenkins doesn't work unless you remove the uk from the url:
- https://uk.businessinsider.com/barclays-antony-jenkins-fintech-startup-10x-future-technologies-core-banking-2016-10
- E
- https://www.businessinsider.com/barclays-antony-jenkins-fintech-startup-10x-future-technologies-core-banking-2016-10
Bonus Person (talk) 17:09, 8 August 2024 (UTC)
- Enwiki: Checked 653 pages and edited 638 pages. Moved 697 links to a new URL. Added 2
{{dead link}}
. Switched 36|url-status=dead
to live. Switched 5|url-status=live
to dead. Added 20 archive URLs (18 Wayback). Changed 5 citation metadata fields. - IABot: set domain to permadead
Done -- GreenC 04:07, 13 August 2024 (UTC)
cartoonnetwork.com
edithttps://www.cartoonnetwork.com
is dead & now redirects "to a landing page on Max" per Variety. Just under 250 articles use it as a source: 247 results. Sariel Xilo (talk) 16:22, 9 August 2024 (UTC)
- Enwiki: Checked 262 pages and edited 120 pages. Added 1
{{dead link}}
. Switched 36|url-status=live
to dead. Added 85 archive URLs (80 Wayback). Changed 60 citation metadata fields. - IABot: set to permadead
Done -- GreenC 04:46, 13 August 2024 (UTC)
articles.latimes.com
editHello. This is a big request. URLs with articles.latimes.com either redirect to the new URL or don't work:
- Working redirect: this to that at Carleton Varney
- Not working redirect: this should go to here for The Man from Earth. These redirects that don't redirect could use archive copies for the time being as the date format doesn't match the above URL at Carleton Varney.
- Broken redirects: this gives a redirect to the Los Angeles Times's main page for Cedars-Sinai Medical Center. It's a search URL and not an URL to a specific article.
60000 with HTTP/HTTPS. Any non articlespace links can be filtered out. Thank you very much! MrLinkinPark333 (talk) 23:58, 12 August 2024 (UTC)
- MrLinkinPark333: It looks like *.latimes.com is 96,000 pages and articles.latimes.com is 37,000. I could focus on articles.latimes.com (which is a significant project) but I wonder about the other 2/3rds. Are they redirecting also? Maybe I should do articles.* right now to keep the size manageable. -- GreenC 02:00, 13 August 2024 (UTC)
- From the sample checks at Summer Olympic Games, Tampa Bay Buccaneers and 2020 Summer Olympics for latimes.com, these look fine and don't need new URLs. If that changes in the future, I could file a separate request later. MrLinkinPark333 (talk) 22:40, 13 August 2024 (UTC)
- OK great. This job is going fast because the LAT has an exceptionally clean site, rapid response, few dead links. It's mostly just finding the redirect and replacing. I'm happy with how well ghost redirect discovery is working, now noted in the statistics (along with soft-404 stats) starting with this run. -- GreenC 16:05, 14 August 2024 (UTC)
- From the sample checks at Summer Olympic Games, Tampa Bay Buccaneers and 2020 Summer Olympics for latimes.com, these look fine and don't need new URLs. If that changes in the future, I could file a separate request later. MrLinkinPark333 (talk) 22:40, 13 August 2024 (UTC)
Enwiki in multiple batches:
- Batch 1: Checked 3,000 pages and edited 2,935 pages. Moved 4,152 links to a new URL. Resolved 68 ghost redirects. Resolved 25 soft-404s. Removed 2
{{dead link}}
templates. Added 8{{dead link}}
. Switched 149|url-status=dead
to live. Switched 14|url-status=live
to dead. Added 166 archive URLs (142 Wayback). Changed 13 citation metadata fields. - Batch 2: Checked 7,000 pages and edited 6,859 pages. Moved 9,663 links to a new URL. Resolved 143 ghost redirects. Resolved 53 soft-404s. Removed 1
{{dead link}}
. Added 21{{dead link}}
. Switched 314|url-status=dead
to live. Switched 34|url-status=live
to dead. Added 372 archive URLs (276 Wayback). Changed 36 citation metadata. - Batch 3: Checked 26,845 pages and edited 26,251 pages. Moved 36,856 links to a new URL. Resolved 481 ghost redirects. Resolved 195 soft-404s. Removed 5
{{dead link}}
. Added 89{{dead link}}
. Switched 1,289|url-status=dead
to live. Switched 128|url-status=live
to dead. Added 1,329 archive URLs (1,106 Wayback). Changed 140 citation metadata.
IABot: does not support URL moves, redirects are working the bot will consider links live.
Done -- GreenC 20:43, 14 August 2024 (UTC)
prweb.com
editHello. Some links on the prweb.com website are now dead. This article from Nancy O'Dell, along with this one from Meryl Streep and this article from Birmingham, all lead to a 404 redirect. 2,952 articles use it as a source. I think we should have the dead links looked at. Lord Sjones23 (talk - contributions) 22:41, 13 August 2024 (UTC)
- Enwiki: Checked 2,993 pages and edited 2,721 pages. Moved 1,274 links to a new URL. Resolved 4 soft-404s. Removed 1
{{dead link}}
. Added 89{{dead link}}
. Switched 14|url-status=dead
to live. Switched 131|url-status=live
to dead. Added 1,503 archive URLs (1,386 Wayback). Changed 224 citation metadata.
- IABot DB: Updated about 3,000 unique links which will propagate to 300+ wikis via IABot
Done -- GreenC 03:31, 16 August 2024 (UTC)
emporis.com
editLast processed Sept 2022. Many {{dead links}}
added. Since then, archive.today added archives, previously unavailable: Special:Diff/1220029968/1240218179. Re-process cites with dead links (emporis3.auth) -- GreenC 05:38, 14 August 2024 (UTC)
- The domain is technically usurped (ie. Emporis). Has 6,000 pages. Will fix in three steps: 1. add archive URLs on enwiki, as a normal dead domain. 2. Same with IABot DB. 3. Later, usurpify everything in a WP:JUDI batch. -- GreenC 03:50, 16 August 2024 (UTC)
- Step 1: Enwiki: Checked 5,979 pages and edited 1,550 pages. Added 430
{{dead link}}
. Switched 265|url-status=live
to dead. Added 1,412 archive URLs (136 Wayback). Changed 1,569 citation metadata. - Step 2: IABot DB: Checked 24,000 links. Updated 23,520 links (set permadead and added new archive URLs). Changes will propagate to 300+ wikis via IABot.
- Step 3: Enwiki: usurpify via JUDI batch
On hold for next run. -- GreenC 15:26, 17 August 2024 (UTC)
caspianenvironment.org
editShows a page which relates to Car finance in Australia! I believe I have found and changed all instances in the Articlespace, but placed here in case not! Big Blue Cray(fish) Twins (talk) 16:05, 15 August 2024 (UTC)
- Thanks Big Blue Cray(fish) Twins: that's a usurped site. I added it to the list Special:Diff/1239726206/1240485275 .. it will get special handling during a future batch job. -- GreenC 16:18, 15 August 2024 (UTC)
On hold for next JUDI run. -- GreenC 04:59, 16 August 2024 (UTC)
bcsportshalloffame.com
editHello. I was looking through the Judi list and saw that bcsportshalloffame.com is there. These links could be converted over to their new url at bcsportshall.com/honoured_member/ Here are examples:
- Generally, any individual inductee's url is the person's first and last name. Here is now there for Andrea Neil.
- Last names at birth are included in the urls. Here is now there for Claire Lovett.
- (M) is ignored for team inductees. For instance, here is now there for Mark Evans (rower).
Just over 100 links. If any don't convert over, let me know and I'll fix them. Thanks! MrLinkinPark333 (talk) 19:50, 16 August 2024 (UTC)
- Hmm.. I've never done something like this. It will require de-usurping, like parsing and removing
{{usurped}}
. It's an inevitable situation as old usurped domains are migrated to a new working domain. It's probably more complicated than it seems. Will take a look. -- GreenC 16:53, 17 August 2024 (UTC)
erenow.com
editUsurped by gambling (e.g. https://erenow.com/postclassical/the-fears-of-henry-iv-the-life-of-englands-king from Wars of the Roses). For pretty clear cut cases like this, can I just add it to the WP:JUDI list directly?
Only 20 pages. GrapesRock (talk) 15:35, 17 August 2024 (UTC)
- Yes, please! -- GreenC 16:41, 17 August 2024 (UTC)