User talk:Trappist the monk: Difference between revisions
→developer-discouraged: new section |
→Semi-protected edit request on 10 April 2021: new section Tags: Reverted Mobile edit Mobile web edit |
||
Line 586: | Line 586: | ||
Regarding: {{tq|I don't know what to think about {{tq|developer discouraged|q=yes}}. Outside of the RFC, is there such a thing?}}<br />No, I literally just made it up. I'm not exactly wedded to the term, and I justed wanted people to understand what the exact status of these parameters were (to replace the old definition of deprecated but also describe something where support won't ever necessarily be removed).<br />I was exactly thinking it was going to be used for maintenance category names (figured it would be something like {{tq|Category:CS1 maint: nonhyphenated parameter}}), but that's not really for me to weigh in on. –<span style="font-family:CG Times, times">[[User:MJL|<span style="color:black">MJL</span>]] [[User talk:MJL|‐'''Talk'''‐]]<sup>[[WP:WikiProject Connecticut|☖]]</sup></span> 18:14, 7 April 2021 (UTC) |
Regarding: {{tq|I don't know what to think about {{tq|developer discouraged|q=yes}}. Outside of the RFC, is there such a thing?}}<br />No, I literally just made it up. I'm not exactly wedded to the term, and I justed wanted people to understand what the exact status of these parameters were (to replace the old definition of deprecated but also describe something where support won't ever necessarily be removed).<br />I was exactly thinking it was going to be used for maintenance category names (figured it would be something like {{tq|Category:CS1 maint: nonhyphenated parameter}}), but that's not really for me to weigh in on. –<span style="font-family:CG Times, times">[[User:MJL|<span style="color:black">MJL</span>]] [[User talk:MJL|‐'''Talk'''‐]]<sup>[[WP:WikiProject Connecticut|☖]]</sup></span> 18:14, 7 April 2021 (UTC) |
||
== Semi-protected edit request on 10 April 2021 == |
|||
{{edit semi-protected|User talk:Monkbot|answered=no}} |
|||
[[Special:Contributions/114.125.180.230|114.125.180.230]] ([[User talk:114.125.180.230|talk]]) 05:06, 10 April 2021 (UTC) |
Revision as of 05:06, 10 April 2021
Archives
|
Index 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 |
This page has archives. Sections older than 7 days may be automatically archived by Lowercase sigmabot III. |
| ||
them at the article's talk page so that everyone who has an interest in the article may participate.
To stop Monkbot, add a message to its talk page. Comments and questions about Monkbot are welcome here. |
Module:Smallem
Hello! :)
As you know, lately I've been trying to write a script capable of making Smallem to autoupdate. To make that process easier, it would be good to have some anchors on its module results to allow for easier regexing. For example, in the results generated here: {{#invoke:Smallem|lang_lister|list=y}}, it would be good to have the list of language codes start and end with some specific strings. Given your experience, what could be the best approach on this thing, considering a good graphical solution and the easiest way to be regexed?
The same could be said about the list of regexes generated. That list, as you know, comes with an added problem that sometimes it generates this warning: skipped: got: 𐌲𐌿𐍄𐌹𐍃𐌺; from: got.wiki, which theoretically could come accompanied by other similar warnings in the future. Those warnings pollute the list of commands and whatever anchors we choose for them should also take care of leaving the warnings out. - Klein Muçi (talk) 22:49, 4 March 2021 (UTC)
- Something human readable? So, for example, you might do something like
begin_code_list
...end_code_list
andbegin_regex_list
...end_regex_list
. - —Trappist the monk (talk) 23:31, 4 March 2021 (UTC)
- Maybe something only alphanumerical? Underlines may add an extra layer of difficulty in regex, no? - Klein Muçi (talk) 23:39, 4 March 2021 (UTC)
- Why would you think that? Underscores don't have special meaning in regex.
- —Trappist the monk (talk) 23:56, 4 March 2021 (UTC)
- Yes but lately I'm scared of using symbols (or even plain whitespaces) because even when regex is not a problem, they start interfering with Linux commands and you end up Google-ing for ages for escape mechanisms. But as far as I know, underscores are not generally used in Linux commands either so... - Klein Muçi (talk) 23:59, 4 March 2021 (UTC)
- So don't include underscores. Or make the delimiter strings all uppercase (linux is case sensitive, typically lowercase). Or make the delimiter strings distinctly non-English.
- —Trappist the monk (talk) 00:13, 5 March 2021 (UTC)
- Yes but lately I'm scared of using symbols (or even plain whitespaces) because even when regex is not a problem, they start interfering with Linux commands and you end up Google-ing for ages for escape mechanisms. But as far as I know, underscores are not generally used in Linux commands either so... - Klein Muçi (talk) 23:59, 4 March 2021 (UTC)
- Maybe something only alphanumerical? Underlines may add an extra layer of difficulty in regex, no? - Klein Muçi (talk) 23:39, 4 March 2021 (UTC)
Maybe a camel case in English BeginCodeList
, etc. ? Even though that would make it a bit hard to read but it's not like we'd spend too much time reading it anyway. Albanian strings were the first that came into my mind but we have everything in English there and I wanted to preserve that. - Klein Muçi (talk) 00:24, 5 March 2021 (UTC)
- Ok, so do that.
- —Trappist the monk (talk) 01:16, 5 March 2021 (UTC)
- As usual, can you do it? Without wanting to bother you. I'm reluctant to experiment with Lua now to find the correct space and all in regard to the generated results. - Klein Muçi (talk) 01:27, 5 March 2021 (UTC)
- sq:Përdoruesi:Trappist the monk/Livadhi personal
- —Trappist the monk (talk) 12:34, 5 March 2021 (UTC)
- Thanks a lot! Can (should?) the delimiters be without a newline separation? I believe it would make for an easier data extraction at bash, even though I'm not sure how that would look like with the regex list. - Klein Muçi (talk) 12:49, 5 March 2021 (UTC)
- As usual, can you do it? Without wanting to bother you. I'm reluctant to experiment with Lua now to find the correct space and all in regard to the generated results. - Klein Muçi (talk) 01:27, 5 March 2021 (UTC)
Here's what I get as data at the moment:
CodeListBegin</p><div about="#mwt1">aa, ab, ace, ady, ... zh-tw, zh-yue, zu</div><p about="#mwt1">CodeListEnd
This is coming from me getting the HTML file of this page and grepping between CodeListBegin and CodeListEnd. We want only the codes. I can, of course, not include the endpoints but the problem remains with those command lines in-between. Can they be "fixed" somehow? Or are they necessary part of the module? Sadly, that kind of defeats the purpose of having those strings as delimiters. - Klein Muçi (talk) 13:27, 5 March 2021 (UTC)
- Really? When I look at the source for sq:Përdoruesi:Trappist the monk/Livadhi personal, I see:
CodeListBegin</p><div>aa, ab, ace, ady, ... zh-tw, zh-yue, zu</div><p>CodeListEnd
- How is it that smallem works at all with the unordered list markup that formats the regex list? For the same page I see this:
<div class="div-col columns column-width" style="column-width:30em"> <ul><li>(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Abasinisch(\s*[\|\}])", r"\1abq-latn\2"),</li> <li>(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Abasinisch(\s*[\|\}])", r"\1abq\2"),</li> ... <li>(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)힌디어(\s*[\|\}])", r"\1hi\2"),</li></ul> </div>
- I wrapped the code list with
<div>...</div>
tags so that it would be somewhat similar to the regex list which has always been wrapped with<div>...</div>
tags and which (apparently) smallem has been able to read and use. - We could, I suppose, change sq:Moduli:Smallem to accept a parameter, perhaps
|plain=yes
that would instruct the module to render the lists without the 'prettifying' markup and whitespace that make the lists human-readable. But, the keywords would change to include a colon just because humans will read the lists in their machine-readable forms:CodeListBegin:
...:CodeListEnd
RegexListBegin:
...:RegexListEnd
- producing:
CodeListBegin:aa,ab,ace,ady,...zh-tw,zh-yue,zu:CodeListEnd
- and:
RegexListBegin:(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Abasinisch(\s*[\|\}])", r"\1abq-latn\2"),(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Abasinisch(\s*[\|\}])", r"\1abq\2"),...(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)힌디어(\s*[\|\}])", r"\1hi\2"),:RegexListEnd
- —Trappist the monk (talk) 14:23, 5 March 2021 (UTC)
- Nice catch! That's because it doesn't. Until now the update of its source code has been done manually by me. I copy the regex lines by simply copy-pasting them from whatever wiki page they're generated after being rendered. Now I'm trying to make the process automatic. To do that, I'm utilizing curl in getting data from a wiki page (Smallem's personal sandbox) and according to the instructions mentioned here, getting the HTML format was the best solution. The only other one being the source one, which brought up way more extra data than the HTML option. I wish there was a way to get only the plain text we see rendered on wiki pages, without any commands but... The first step was to get the code list and utilize them in batches of 50 items to get the regex lists. Then I'd have to work on getting the regex lists and concatenate them together. Of course, both steps need to be done in a row in a cycle until the code list is exhausted. But given that I was dealing with cleaning up the code list data now, I made up my mind to worry later about the regex lists. So what you just noticed was a problem that it was on the to do list to be fixed. If I succeed in making the process automatic, we can afford to lose some esthetic elements (so far, the results look promising). So I guess you can go on with your suggestion? Even though I'd like to somehow retain the plain list (without bulletpoints) format of the regexes considering their sheer size. But I'm forced to accept whatever I can get. I like the idea of having a switch in our hands though to, well, switch, from machine to human readable form.
- Everything would be perfect like it currently is if I could just get the plain text somehow but I know of no way in doing that. :( - Klein Muçi (talk) 15:22, 5 March 2021 (UTC)
- sq:Përdoruesi:Trappist the monk/Livadhi personal
- —Trappist the monk (talk) 16:17, 5 March 2021 (UTC)
- Works very good except for... Can we add back 1 single whitespace after commas? That would be the perfect formatting bash wise I believe. - Klein Muçi (talk) 20:48, 5 March 2021 (UTC)
I was able to utilize the regexes and create a loop to be able to get the whole list eventually. The loop works as intended but I'm running onto a strange problem. I keep getting edit conflicts every once in a while even though no one is editing that page other than me with these requests. You need to send the latest id revision of the page you're trying to edit with every request you send and I've written a function to get it and send it automatically. Sometimes it glitches, it doesn't update fast enough, or... I don't know really but it will make it appear as running unto an edit conflict, basically with itself (just because it's trying to edit an older version of the page). This would happen even when sending requests manually (before creating the loop) but then it wouldn't cause any harm because I could just re-send the request and it would get fixed. The problem now is that I send 5 codes per request and if the edit conflict happens, those 5 codes are lost and the loop goes on with the other 5 codes, which means that even 1 edit conflict would corrupt the whole end results. I tried making Smallem sleep after every request (started with 5 seconds per request) and more edits got through before running into conflicts so I tried incrementing the sleeping time. Eventually I got into a point when incrementing it wouldn't help anymore (reached 10 minutes per request) and the conflicts were rare but still persisted in a, what appears to be, random manner. You could get 5 requests go through and then get 1-2 conflicts in a row, for example. At this point, I don't really know what's causing them. Maybe some problem in regard to Wikipedia's caches or job queues... Do you have any experience with this kind of problem? I've been reading the bot help pages here and the ones on Mediawiki in regard to API requests but... - Klein Muçi (talk) 10:18, 7 March 2021 (UTC)
- I guess that I'm confused. It appears to me that you are attempting to recreate the regexes at a rate far in excess of the rate-of-change of the code list. The language codes are updated only occasionally so it isn't necessary to recreate the regexes until the code list changes. Shouldn't you just create the regexes once and let smallem use that set until sometime, maybe months from now, MediaWiki changes the language codes?
- —Trappist the monk (talk) 12:19, 7 March 2021 (UTC)
- Yes, that's what I do. Every few months, I go and work with module Smallem and recreate the regexes. Now I want Smallem to go by itself to that module and recreate the regexes by itself, again, every few months. Now, when I do it manually, I feed around 50 codes at the module in one go, get the results and repeat until the list is exhausted. Smallem can't work with 50 codes in one go (one request) because it can't get that much of information in 1 request so, by trial and error, I found out that it must work with 5 codes at a time (request). So, considering we have around 330 codes, Smallem needs to send 66 POST requests (to add codes) and 66 GET requests (to get the regexes from those codes) in a row to complete the update, when the time comes (every 3 months, for example) [66*5=330] . This is done by creating a loop which, to put it simply, works like this: Gets the total list of codes and sends 1 request with 5 of them, deletes 5 codes from the list, save the regexes for the 5 codes sent, repeats until the list is exhausted. The problem is that every 5-6 requests, it encounters an edit conflict, for unknown reasons. No one other than Smallem is editing that page. What I described in details above. This doesn't cause the loop to stop which continues deleting those 5 codes from the list and going on with sending another request, so the results from those 5 codes, whatever they were, will be missing in the end. This is the problem I'm explaining. The big number of requests is just because it works in small batches just to complete 1 full regex list update (which will be programmed to run periodically every some months) not because I'm updating it over and over again all the time. - Klein Muçi (talk) 12:45, 7 March 2021 (UTC)
- Side quest: Can the regex list be made to be generated with the ASCII tab character separating regexes instead of plain space? I believe it would help me with code formatting in the final stage. - Klein Muçi (talk) 13:22, 7 March 2021 (UTC)
- If you want ...
- —Trappist the monk (talk) 13:46, 7 March 2021 (UTC)
- Seems extraordinarily odd to me that something that you can do manually cannot be done through the api – the api should be able to fetch the largest article so fetching the regexes for 50 codes shouldn't be a problem. Perhaps your first step is to find an api expert. I'm pretty sure that I've suggested this before: pywikibot is written using python; surely sq:Moduli:Smallem can be ported to python so that all it needs to fetch is the code list and it makes the regexes itself...
- —Trappist the monk (talk) 13:46, 7 March 2021 (UTC)
- Well, even if it did work with 50 codes (I tried it only 2 times and then went on to lower it not giving enough thought on why it wasn't working because I thought "anyway, Smallem doesn't get tired") it still would need 7 turns to finish all the codes and it would take only 1 edit conflict to make the final list miss all the regexes for 50 codes missed during that one iteration. But why not do all the codes in 1 turn then? You already know the answer to that question. (The Mediawiki limit.) So in one way or another I need to deal with the edit conflict problem. Making Smallem understand when the edit conflict happens and retry that iteration would take too much effort and I didn't want to follow that road given that running on edit conflicts should be relatively a very, very rare occasion for Smallem. Ironically, that's what's stopping it now. I can try to make it work with 50 codes but the problem persists if I don't understand correctly what's going on behind the curtain. I thought maybe you had encountered something similar with Monkbot or knew any kind of information in regard to phenomena like these but looks like Smallem will still be sailing alone on that problem. And yes, please, change it accordingly. Also, if you don't mind, tell me the correct name of that character. You're good with nomenclature. I only know it as "the tab character" or "the 5 spaces". :P - Klein Muçi (talk) 14:13, 7 March 2021 (UTC)
- Which is why smallem / pywikibot should be building the regexes string; no splitting the job into pieces and hoping that you got all of the peices.
- Tab key § Tab characters
- sq:Përdoruesi:Trappist the monk/Livadhi personal
- —Trappist the monk (talk) 14:42, 7 March 2021 (UTC)
- Well, even if it did work with 50 codes (I tried it only 2 times and then went on to lower it not giving enough thought on why it wasn't working because I thought "anyway, Smallem doesn't get tired") it still would need 7 turns to finish all the codes and it would take only 1 edit conflict to make the final list miss all the regexes for 50 codes missed during that one iteration. But why not do all the codes in 1 turn then? You already know the answer to that question. (The Mediawiki limit.) So in one way or another I need to deal with the edit conflict problem. Making Smallem understand when the edit conflict happens and retry that iteration would take too much effort and I didn't want to follow that road given that running on edit conflicts should be relatively a very, very rare occasion for Smallem. Ironically, that's what's stopping it now. I can try to make it work with 50 codes but the problem persists if I don't understand correctly what's going on behind the curtain. I thought maybe you had encountered something similar with Monkbot or knew any kind of information in regard to phenomena like these but looks like Smallem will still be sailing alone on that problem. And yes, please, change it accordingly. Also, if you don't mind, tell me the correct name of that character. You're good with nomenclature. I only know it as "the tab character" or "the 5 spaces". :P - Klein Muçi (talk) 14:13, 7 March 2021 (UTC)
- Side quest: Can the regex list be made to be generated with the ASCII tab character separating regexes instead of plain space? I believe it would help me with code formatting in the final stage. - Klein Muçi (talk) 13:22, 7 March 2021 (UTC)
- Yes, that's what I do. Every few months, I go and work with module Smallem and recreate the regexes. Now I want Smallem to go by itself to that module and recreate the regexes by itself, again, every few months. Now, when I do it manually, I feed around 50 codes at the module in one go, get the results and repeat until the list is exhausted. Smallem can't work with 50 codes in one go (one request) because it can't get that much of information in 1 request so, by trial and error, I found out that it must work with 5 codes at a time (request). So, considering we have around 330 codes, Smallem needs to send 66 POST requests (to add codes) and 66 GET requests (to get the regexes from those codes) in a row to complete the update, when the time comes (every 3 months, for example) [66*5=330] . This is done by creating a loop which, to put it simply, works like this: Gets the total list of codes and sends 1 request with 5 of them, deletes 5 codes from the list, save the regexes for the 5 codes sent, repeats until the list is exhausted. The problem is that every 5-6 requests, it encounters an edit conflict, for unknown reasons. No one other than Smallem is editing that page. What I described in details above. This doesn't cause the loop to stop which continues deleting those 5 codes from the list and going on with sending another request, so the results from those 5 codes, whatever they were, will be missing in the end. This is the problem I'm explaining. The big number of requests is just because it works in small batches just to complete 1 full regex list update (which will be programmed to run periodically every some months) not because I'm updating it over and over again all the time. - Klein Muçi (talk) 12:45, 7 March 2021 (UTC)
Thank you! You were right. It works even with 100 entries (150 entries reach the MediaWiki limit). Unfortunately, the problem persists. You need 4 iterations to finish the list and I can do only 1 or 2 iterations before it runs into an edit conflict and 100 entries are lost. :P I'll experiment more to try and solve it with good ol' trial and error. Thank you for your persistent help! :)) - Klein Muçi (talk) 15:19, 7 March 2021 (UTC)
- I was able to get past that problem with a very crude hack. Now I'm reaching the final steps of the update where I need to sort the total list and remove the duplicates. To do that, I have to have an unique delimiter on the list of regexes that separates each regex from each other so when I get the machine like form, I can rearrange them back into a sortable list and then remove the duplicates. Can you add a character after the comma (maybe replacing the whitespace, we're talking about the machine readable list) to use as a delimiter? Anything that is not used in the regex itself. I was thinking of adding a hash character but that's used for comments and I'm not really sure if it could be a good idea. - Klein Muçi (talk) 23:21, 10 March 2021 (UTC)
- Isn't that why you had me put in the tab character?
- —Trappist the monk (talk) 00:15, 11 March 2021 (UTC)
- Yes. The original thought wasn't precisely to use as a delimiter but more to have the tab character shown before every regex because in the final bash syntax, that's how those lines are arranged (cosmetic indentation). But I was disappointed when it wasn't rendered anywhere. Neither the wiki page itself, nor the bash results generated from it had any tab space in them. But please, since you reminded me, let me try something else before you do any change to it. My guess is I'll still have to put another symbol as a delimiter but first... (I'll tell you very soon.) - Klein Muçi (talk) 00:26, 11 March 2021 (UTC)
- Oops! It does show in the bash results even though it doesn't show on the wiki page. And it works like a charm as a delimiter. My bad my friend. :P :)) Bash scripting is a new world for me and working with it feels like re-inventing the wheel in every command I give (basically I need to search everything on Google). Putting everything together with the Wikimedia's API (again, something totally new for me) makes that new world feel like a minefield, albeit, still a nice adventure. At least you're helping me with Lua (even though, ironically, that's the only thing I know a bit compared to the other 2 concepts). Thank you for that. And sorry for the annoyance I may bring. - Klein Muçi (talk) 00:43, 11 March 2021 (UTC)
- Some more beauty subjects... Can we apply some formatting to the languages' codes' list like we did with the regex one? In the "human form" (not plain), given that entries are to be used in batches of 100 entries, can me format it in 4 groups of at least 100 entries each? I always spend quite some minutes counting them 1 by 1 manually until I'm able to select, copy and paste 100 codes exactly. - Klein Muçi (talk) 21:02, 11 March 2021 (UTC)
- Oops! It does show in the bash results even though it doesn't show on the wiki page. And it works like a charm as a delimiter. My bad my friend. :P :)) Bash scripting is a new world for me and working with it feels like re-inventing the wheel in every command I give (basically I need to search everything on Google). Putting everything together with the Wikimedia's API (again, something totally new for me) makes that new world feel like a minefield, albeit, still a nice adventure. At least you're helping me with Lua (even though, ironically, that's the only thing I know a bit compared to the other 2 concepts). Thank you for that. And sorry for the annoyance I may bring. - Klein Muçi (talk) 00:43, 11 March 2021 (UTC)
- Yes. The original thought wasn't precisely to use as a delimiter but more to have the tab character shown before every regex because in the final bash syntax, that's how those lines are arranged (cosmetic indentation). But I was disappointed when it wasn't rendered anywhere. Neither the wiki page itself, nor the bash results generated from it had any tab space in them. But please, since you reminded me, let me try something else before you do any change to it. My guess is I'll still have to put another symbol as a delimiter but first... (I'll tell you very soon.) - Klein Muçi (talk) 00:26, 11 March 2021 (UTC)
Yes, thank you! Now 1 technical question: The way Smallem autoupdates itself by first copying the codes from here then using them in batches in another page and getting the results back (and removing the duplicates in the end). Now assume that code list changes by MediaWiki. Would the change be reflected in the HTML source of that page I mentioned above automatically after a while? Or would it need to be edited somehow before reflecting the change? Because if it is the latter one, I should teach Smallem to do a dummy edit before starting the process. (A dummy edit would solve that problem, no?) - Klein Muçi (talk) 23:43, 11 March 2021 (UTC)
- A null edit should be all that is required to ensure that the code list is up to date. I don't know if non-mainspace pages are cached. I suspect that they are (at least for a while) so a null edit would make sense to ensure that the code list is up to date.
- —Trappist the monk (talk) 00:12, 12 March 2021 (UTC)
- Okay then. :) - Klein Muçi (talk) 00:22, 12 March 2021 (UTC)
- Can we add some kind of small mark (not newline) every 10 and 50 entries? The groups of 100 work very well when doing the update manually but when testing things, I still waste too much time counting to 50 one by one (and starting 3 times over when not sure if I counted that one code or not). Maybe the comma becomes bold after 50 entries? Do you think that's notable enough for the eyes? Something small that doesn't "pollute" the current format too much. - Klein Muçi (talk) 10:30, 12 March 2021 (UTC)
- What if, instead of a marker character,
|list=
takes a number and uses that number to create lines of codes that are that long? - —Trappist the monk (talk) 13:16, 12 March 2021 (UTC)
- So, I still get the full list but I'm able to specify how many entries does each batch/group of codes have? - Klein Muçi (talk) 14:41, 12 March 2021 (UTC)
- Yes. If the value assigned to
|list=
is not a number or the number is greater than the number of codes, defaults to lists of 100 codes. When the value assigned to|list=
is a number less than or equal to the number of language codes, creates lists of number codes. - —Trappist the monk (talk) 15:05, 12 March 2021 (UTC)
- Yes. If the value assigned to
- So, I still get the full list but I'm able to specify how many entries does each batch/group of codes have? - Klein Muçi (talk) 14:41, 12 March 2021 (UTC)
- What if, instead of a marker character,
- Can we add some kind of small mark (not newline) every 10 and 50 entries? The groups of 100 work very well when doing the update manually but when testing things, I still waste too much time counting to 50 one by one (and starting 3 times over when not sure if I counted that one code or not). Maybe the comma becomes bold after 50 entries? Do you think that's notable enough for the eyes? Something small that doesn't "pollute" the current format too much. - Klein Muçi (talk) 10:30, 12 March 2021 (UTC)
- Okay then. :) - Klein Muçi (talk) 00:22, 12 March 2021 (UTC)
Seems pretty good. - Klein Muçi (talk) 15:22, 12 March 2021 (UTC)
- sq:Përdoruesi:Trappist the monk/Livadhi personal
- —Trappist the monk (talk) 15:49, 12 March 2021 (UTC)
- Even more requests... Can the machine regex list show the total number of entries for that/those specific code/s rendered in the end of the list? The reason I want that is for a very specific problem. I was able to finish the whole autoupdate process yesterday but I have inconsistencies between the total number that I get manually (that the Lua logs give at the module) and the total number that Smallem gets automatically. My plan is to try each entry individually and see how much lines it should have and how much does Smallem get from it automatically. Maybe somewhere something is missing. I have no other idea on how to debug it. For now, I'm doing the counting process manually by copying everything on the Notepad+ but if I had the number shown below it would be 1 less step to be taken. Unfortunately, even though I believed it was working with batches of 100 entries, the results it was returning were empty. Turns out Smallem can't handle requests with more than 10 entries per request. If you surpass it, it starts returning empty results. When I completed the process with requests with 10 entries, the discrepancy was bigger. I tried comparing each group of 10 entries manually and automatically and it turned out there were some groups which were returning empty results again when the number of lines went more than 5000. So I tried lowering the number and the discrepancy got smaller and smaller until I got to 1 code per request at which time I got 1100 entries less than the total number I should be having, the smallest inconsistency I could get. Even then though, I tried comparing both lists (the automatic one and the manual one) for unique entries and my hope was that it would give us more or less 1100 unique entries (only the missing ones) but much to my dismay, it gave back 15000 unique entries so not only the final results are missing some lines but they're also not correct. And I'm sort of out of ideas what to try for debugging now. The only idea that I had was to compare the length of each code's list individually manually and automatically to see if I can find anything. It's a really tedious process but... I've got nothing else. At least removing 1 step would make it a bit easier. - Klein Muçi (talk) 10:59, 13 March 2021 (UTC)
- sq:Përdoruesi:Trappist the monk/Livadhi personal
- —Trappist the monk (talk) 12:25, 13 March 2021 (UTC)
- Hmm... My first instinct is to suggest to add the word "lines" after the number. Is there any particular reason you haven't added it? - Klein Muçi (talk) 12:30, 13 March 2021 (UTC)
- Because there is only one line. Each regex is concatenated onto the next separated by a tab character so the number reflects the number of regexes not lines.
- —Trappist the monk (talk) 12:38, 13 March 2021 (UTC)
- Oh! I was thinking now... Maybe we should have that even on the human form? Can't think of a particular reason now but it's an extra functionality so why not? And for you would it be 1 possible less request from me in the future. I mean, if you can think of any downsides, then tell me but if not... - Klein Muçi (talk) 12:46, 13 March 2021 (UTC)
- Hmm... My first instinct is to suggest to add the word "lines" after the number. Is there any particular reason you haven't added it? - Klein Muçi (talk) 12:30, 13 March 2021 (UTC)
- Even more requests... Can the machine regex list show the total number of entries for that/those specific code/s rendered in the end of the list? The reason I want that is for a very specific problem. I was able to finish the whole autoupdate process yesterday but I have inconsistencies between the total number that I get manually (that the Lua logs give at the module) and the total number that Smallem gets automatically. My plan is to try each entry individually and see how much lines it should have and how much does Smallem get from it automatically. Maybe somewhere something is missing. I have no other idea on how to debug it. For now, I'm doing the counting process manually by copying everything on the Notepad+ but if I had the number shown below it would be 1 less step to be taken. Unfortunately, even though I believed it was working with batches of 100 entries, the results it was returning were empty. Turns out Smallem can't handle requests with more than 10 entries per request. If you surpass it, it starts returning empty results. When I completed the process with requests with 10 entries, the discrepancy was bigger. I tried comparing each group of 10 entries manually and automatically and it turned out there were some groups which were returning empty results again when the number of lines went more than 5000. So I tried lowering the number and the discrepancy got smaller and smaller until I got to 1 code per request at which time I got 1100 entries less than the total number I should be having, the smallest inconsistency I could get. Even then though, I tried comparing both lists (the automatic one and the manual one) for unique entries and my hope was that it would give us more or less 1100 unique entries (only the missing ones) but much to my dismay, it gave back 15000 unique entries so not only the final results are missing some lines but they're also not correct. And I'm sort of out of ideas what to try for debugging now. The only idea that I had was to compare the length of each code's list individually manually and automatically to see if I can find anything. It's a really tedious process but... I've got nothing else. At least removing 1 step would make it a bit easier. - Klein Muçi (talk) 10:59, 13 March 2021 (UTC)
No lines? XD I'm mostly kidding now. Thank you! :) Sometimes I look back and think that I'd probably be adding regex lines by hand until now if you hadn't create that. I don't believe I'd be even close to finishing half of those by now. :P - Klein Muçi (talk) 14:04, 13 March 2021 (UTC)
- I believe there's a sort of shortcoming in the change we made. In human form it works as it should but in machine form, you get an error if you leave the
|list
parameter empty or if you don't put it at all. You must give it a random value for the list to be generated. And the value does nothing because the format is already fixed. Can it be made to behave like the parameter|plain
? - Klein Muçi (talk) 16:13, 14 March 2021 (UTC)- What that message was telling you is that
|lang=
is missing or empty. When|list=
has a value, sq:Moduli:Smallem renders the language list; when|list=
is empty or missing, Smallem renders the regex list according to the codes in|lang=
. I've tweaked it so that Smallem still emits an error message, but isn't so strident. - —Trappist the monk (talk) 17:23, 14 March 2021 (UTC)
- No, the error was good enough on its own but the problem is that there shouldn't be an error. So, basically it should work like this: All the codes are generated no matter if that parameter is there or not. Without bringing any errors. If that parameter is there, it changes only the formatting of how the codes are generated (the number of entries per group). If it's not, it defaults to 100 per group (per human form/plain=no) or all the codes in 1 line (per machine form/plain=yes). That's how I thought it would work with your explanation in the beginning but maybe I'm overlooking something on my reasoning. - Klein Muçi (talk) 19:36, 14 March 2021 (UTC)
- The default case for sq:Moduli:Smallem is to create regexes. We shortstop that to create the language codes list by setting
|list=
to some value. At sq:Përdoruesi:Trappist the monk/Livadhi personal there are two{{#invoke:Smallem|lang_lister|...}}
invokes. They both call the same functionlang_lister()
. In the first one,|list=
is not set solang_lister()
expects to find a list of language codes in|lang=
. That parameter is missing solang_lister()
cannot proceed except to emit an error message. In the second invoke,|list=
is missing but|lang=
has a list of language code solang_lister()
renders the regexes.|list=
is the parameter that decides what smallem will do. If you want 100-item lists from smallem, you must set|list=100
so that it knows that that is what you want. When you don't tell it to list codes, smallem will assume that you want list regexes. - —Trappist the monk (talk) 20:05, 14 March 2021 (UTC)
- The default case for sq:Moduli:Smallem is to create regexes. We shortstop that to create the language codes list by setting
- Also, I just noticed something totally unexpected now that we've add counting. This: {{#invoke:Smallem|lang_lister|plain=yes|lang=aa }} generates 857 results. This: {{#invoke:Smallem|lang_lister|plain=yes|lang=aa, }} generates 1246 results. Why is that? I'm afraid that's one of the factors that changes my expected total number of lines in the end. - Klein Muçi (talk) 19:46, 14 March 2021 (UTC)
- But... I'm confused... :/ It used to work without the parameter
|list=
. We only added that to be able to affect some more the formatting on the generated codes. Shouldn't it work again without it and just default to a specified default state? Or, asking in a more practical aspect, what am I supposed to write on|list=
when generating the list of codes with plain=yes? - Klein Muçi (talk) 20:13, 14 March 2021 (UTC)- Nope. To get a language-code list has always required
|list=
. See this diff. The old form was|list=yes
and it always shortstoppedlang_lister()
. The new form uses a number but if you give it something that is not a number (like|list=yes
) it defaults to 100-item lists. |plain=yes
selects the machine readable output forms. When|plain=yes
and|list=<anything>
you get the machine readable language-codes list. When|plain=yes
and|list=
(empty or omitted), you get the machine readable regexes list.- —Trappist the monk (talk) 21:11, 14 March 2021 (UTC)
- Ooh... I "should" put "yes"... Now it makes sense. What about the comma problem? What's happening there? And... Can we make the plain list of codes generated without commas? Now that I see that they do make a difference... - Klein Muçi (talk) 21:24, 14 March 2021 (UTC)
- I have added a line that strips trailing commas from the end of the
|lang=
code list before handing the list to the part oflang_lister()
that makes the regexes. sq:Përdoruesi:Trappist the monk/Livadhi personal - —Trappist the monk (talk) 22:24, 14 March 2021 (UTC)
- Sweet! - Klein Muçi (talk) 22:56, 14 March 2021 (UTC)
- I have added a line that strips trailing commas from the end of the
- Ooh... I "should" put "yes"... Now it makes sense. What about the comma problem? What's happening there? And... Can we make the plain list of codes generated without commas? Now that I see that they do make a difference... - Klein Muçi (talk) 21:24, 14 March 2021 (UTC)
- Nope. To get a language-code list has always required
- But... I'm confused... :/ It used to work without the parameter
- No, the error was good enough on its own but the problem is that there shouldn't be an error. So, basically it should work like this: All the codes are generated no matter if that parameter is there or not. Without bringing any errors. If that parameter is there, it changes only the formatting of how the codes are generated (the number of entries per group). If it's not, it defaults to 100 per group (per human form/plain=no) or all the codes in 1 line (per machine form/plain=yes). That's how I thought it would work with your explanation in the beginning but maybe I'm overlooking something on my reasoning. - Klein Muçi (talk) 19:36, 14 March 2021 (UTC)
- What that message was telling you is that
So, my autoupdate script works fine overall more or less. It still has some problems with accessing the Wikimedia API but I can't do much about those now, because I can't find anything to help me on the documentation. It requires a lot of trial-erroring to reverse engineer it and hopefully with time I'll be able to solve those too. (I've been talking with someone about the API problems but it seems like even that conversation it's coming to an end without solving all of the problems. Do take a look if you have time.) Meanwhile I had 1 problem I wanted to ask you about, maybe you can orientate me a bit on the right direction. I don't get consistent results. Most of the times I will be getting 41063 lines but the number can vary a lot (40603, 40778, 40842, 41413, 40735, etc). This is with the list of regexes + 9 lines of commands. Now the variation and the lack of consistency is part of the API problems. (Sometimes I also get errors.) But I'd like to be able to know more specifically about the variation between lines though. Do you have any ideas how I might understand more about what's going on behind the curtains in regard to this? What's changing between every run, what's getting left behind or ignored? The exact number should be 42597 + 9 command lines. Some common debugging tactics I can use. Don't worry if you can't help much though. As you can see above, I haven't been able to get help properly even on MediaWiki so... :P :) - Klein Muçi (talk) 12:15, 17 March 2021 (UTC)
- I suspect that I'm not going to be much help. So many things being done by so many different processes make finding the one or few that are causing the variations in the number of returned regexes really difficult. One would think that the difference between the expected and what you actually got would be the same given that the inputs are always the same. So I guess I would start there. Are the inputs to the auto update process chain always the same? If you force the inputs to always be the same, do you get consistent output? If you break the chain at the midpoint and insert known-good data at the beginning, is the midpoint output always correct?
- Aside from this, I don't think that I have any suggestions.
- —Trappist the monk (talk) 15:07, 17 March 2021 (UTC)
- Yes, that was my hope. I was trying to somehow make it always give the same result, even if that result was wrong. But I couldn't manage even that. What I did try though was only getting results for 1 code and comparing the script generated results by those gotten manually. It always had the same number of lines. I did try it for more than 30 codes. Then I got the list of results for the code aa and compared both lists (script/manually) together. I mashed them up and sorted to only leave the unique lines, meaning, if they would be the same, the result would be an empty list. What I did get though was a short list with some entries. There I noticed that the manually generated list had some entries with empty characters on them which the script one didn't have (this worried me a bit to be honest because I understood that that's a problem I might have been having all this time while generating those codes and basically there's no way to solve it) AND that the script couldn't deal with characters like single quotes, etc. For example N'go became Ngo in the script list. That's all I could understand but I don't know what to do with an information like that. I mean, should I deduce that all the missing entries are all because of problems with special characters like these? Leaving the number inconsistencies aside, I mean. Treating 41063 as being the only result. Just brainstorming now mostly. - Klein Muçi (talk) 21:15, 17 March 2021 (UTC)
manually generated list had some entries with empty characters
what do you mean by that? What does an[entry] with empty characters
look like? What should it look like?- sq:Moduli:Smallem doesn't do anything special for language names that contain single quote/apostrophe characters. Should it? In the
aa
regex list I found: Fe'Fe', Lamnso', Mi'kmaq, Mka'a, Nda'Nda', N’Ko (U+2019 – single comma quotation mark), and O'odham. Didn't find N'go. - —Trappist the monk (talk) 22:56, 17 March 2021 (UTC)
- Oh, not empty. Sorry. I meant to say invisible. Maybe it was N’Ko or maybe it was from another test I did with first 10 codes. I've done many tests trying to "crack the mystery" any way I can in these days. But anyway you got the general idea. Apparently my script just plainly removes those whenever it finds them. I don't know if that's coming from my bash code, the Wikipedia API or the curl command and I don't know its implications. Would the Smallem's regex still find N’Ko if it was written as Nko and apply the change? I guess not. Can Module Smallem help with that? I would answer if I knew. I don't know what's causing Smallem not to care about these characters so I don't know how you can twist the module to make that phenomenon not to happen again. :/ - Klein Muçi (talk) 23:11, 17 March 2021 (UTC)
- Just checked the full list. There's no N’go. It's N’Ko. Doesn't make any difference but just for clarification. :P - Klein Muçi (talk) 23:54, 17 March 2021 (UTC)
- Ok, so what does an
[entry] with
look like? What should it look like?emptyinvisible characters - If something removes single quote/apostrophe characters then a regex that is modified:
(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Mi'kmaq(\s*[\|\}])", r"\1mic\2")
→(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Mikmaq(\s*[\|\}])", r"\1mic\2")
- won't work as desired because the regex won't match and fix
|language=Mi'kmaq
→|language=mic
- But, that doesn't explain the variable regex line counts. Is it possible that something in the processing chain matches and consumes single quotes and everything between them? If you have this (from the
aa
regex generation):... (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Mi'kmaq(\s*[\|\}])", r"\1mic\2"), ... <twelve regexes> ... (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Mka'a(\s*[\|\}])", r"\1bqz\2") ...
- then, if something consumes single quotes and their content, you would get this:
... (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Mia(\s*[\|\}])", r"\1bqz\2") ...
- which is a valid regex (in form if not in function).
- —Trappist the monk (talk) 00:19, 18 March 2021 (UTC)
- An invisible character doesn't look like anything because it's invisible. :P The terminal percieves this as a code U+something or just shows a carriage return arrow sometimes. This only happens when you copy the regexes manually from the wikipage. Sometimes invisible characters will show up at some regex lines... apparently. I didn't know that. Only noticed now and that got me worried about Smallem in general because apparently they have always been there, modifying the regexes "silently". I really like your thinking about miss-consumption of lines but I don't believe that to be the case. The reason I say that is because as I said, I get the same number of lines when I do codes 1 by 1. There are no lines missing. To be practical, I'll run the script for
aa
, get the results manually, concatenate the lists and keep only the unique lines, which I'll show here. - Klein Muçi (talk) 00:38, 18 March 2021 (UTC)
- An invisible character doesn't look like anything because it's invisible. :P The terminal percieves this as a code U+something or just shows a carriage return arrow sometimes. This only happens when you copy the regexes manually from the wikipage. Sometimes invisible characters will show up at some regex lines... apparently. I didn't know that. Only noticed now and that got me worried about Smallem in general because apparently they have always been there, modifying the regexes "silently". I really like your thinking about miss-consumption of lines but I don't believe that to be the case. The reason I say that is because as I said, I get the same number of lines when I do codes 1 by 1. There are no lines missing. To be practical, I'll run the script for
- Ok, so what does an
- Yes, that was my hope. I was trying to somehow make it always give the same result, even if that result was wrong. But I couldn't manage even that. What I did try though was only getting results for 1 code and comparing the script generated results by those gotten manually. It always had the same number of lines. I did try it for more than 30 codes. Then I got the list of results for the code aa and compared both lists (script/manually) together. I mashed them up and sorted to only leave the unique lines, meaning, if they would be the same, the result would be an empty list. What I did get though was a short list with some entries. There I noticed that the manually generated list had some entries with empty characters on them which the script one didn't have (this worried me a bit to be honest because I understood that that's a problem I might have been having all this time while generating those codes and basically there's no way to solve it) AND that the script couldn't deal with characters like single quotes, etc. For example N'go became Ngo in the script list. That's all I could understand but I don't know what to do with an information like that. I mean, should I deduce that all the missing entries are all because of problems with special characters like these? Leaving the number inconsistencies aside, I mean. Treating 41063 as being the only result. Just brainstorming now mostly. - Klein Muçi (talk) 21:15, 17 March 2021 (UTC)
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)español\ \(formal\)(\s*[\|\}])", r"\1es-formal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)español\ \(formal\)(\s*[\|\}])", r"\1es-formal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)magyar\ \(formal\)(\s*[\|\}])", r"\1hu-formal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)magyar\ \(formal\)(\s*[\|\}])", r"\1hu-formal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Nederlands\ \(informeel\)(\s*[\|\}])", r"\1nl-informal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Nederlands\ \(informeel\)(\s*[\|\}])", r"\1nl-informal\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)N’Ko(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)NKo(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮ(\s*[\|\}])", r"\1ban-bali\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮᬶ(\s*[\|\}])", r"\1ban-bali\2"),
So, both methods showed the same number of lines. Script 858, manually 857 (script adds 1 empty line in the end of the results which is removed in the last concatenation). I then concatenated both lists, sorted them lexicographically and kept only the lines which weren't duplicated (if a line had a symmetrical friend, they both got deleted) which were the lines above. I was wrong with the U+code. It only shows the carriage return error which here, strangely enough, gets converted to that red dot (check the source code to see what I mean). So, as you can see, the lines above were the only ones who weren't fully symmetrical and that's about it in regard to the change. There are no totally new entries in a list which don't appear at all in the other list. I believe the red dot is coming from the script list judging by the N’Ko/Nko case in which the manual entri gets positioned in second place. - Klein Muçi (talk) 01:03, 18 March 2021 (UTC)
- The red dot is U+200E left-to-right mark. It is present in the MediaWiki data though I don't know why it is there. I know this because I can see it when I copy these magic word renderings to https://r12a.github.io/uniview/:
{{#language:es-formal|aa}}
→ Spanish (formal address){{#language:hu-formal|aa}}
→ Hungarian (formal address){{#language:nl-informal|aa}}
→ Dutch (informal address)
- Changing the rendered language to something other than
aa
doesn't change anything{{#language:es-formal|es}}
→ Spanish (formal address){{#language:es-formal}}
→ español (formal)
- I have added a snippet of code to sq:Moduli:Smallem that replaces U+200E LRM with an empty string. Doing that changes the total number of lines to 42596 (from 42597).
- No idea where you are loosing the U+2019 single comma quotation mark in
nqo
or loosing the U+1B36 Balinese vowel sign ulu inban-bali
. - —Trappist the monk (talk) 14:24, 18 March 2021 (UTC)
- There are a handful of other codes that include U+200E left-to-right mark:
{{#language:be-tarask}}
→ беларуская (тарашкевіца){{#language:be-x-old}}
→ беларуская (тарашкевіца){{#language:zh-cn}}
→ 中文(中国大陆){{#language:zh-tw}}
→ 中文(臺灣)
- —Trappist the monk (talk) 15:02, 18 March 2021 (UTC)
- Thank you! I'm curios, what happens that we get +1 line now with that change. I'll try the same test now and see what happens. And then try to see if there is any change on the grand total in the end. - Klein Muçi (talk) 20:39, 18 March 2021 (UTC)
- There are a handful of other codes that include U+200E left-to-right mark:
As expected, this is what I got now after redoing the same steps mentioned yesterday:
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)N’Ko(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)NKo(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮ(\s*[\|\}])", r"\1ban-bali\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮᬶ(\s*[\|\}])", r"\1ban-bali\2"), - Klein Muçi (talk) 20:53, 18 March 2021 (UTC)
- I tried making a full run a couple of times. Unfortunately, I never get the right amount but again, as expected, if I run it with 1 code instead of 10 codes per request I get "better" results in general. I don't know what makes the difference. You have suggested in the past to run it with 50 or 100 codes in a request but that just doesn't happen. If you raise the number past 20, you start getting empty responses.
- These are the results with 10 codes per request: 41063 40063 40778 40351 41063 40842 41413 40735 40543 40603 41063
- These are the results with 1 code per request: 41008 41413 41008 41015 41015 41420 41013 41421
- I think a good idea would be to try and compare lists by the script and manually for 10 codes, one by one, until the codes are over. And see what differences there are in each comparison. Maybe sometimes we'll see codes missing in the script generated list. Or anything strange (apart from the cases we've discussed above). What do you think? - Klein Muçi (talk) 23:57, 18 March 2021 (UTC)
- After that we can also try a comparison between a full script list and a full manual list and see if something changes when you run a full automatic run compared with when you do 35 requests manually with the script. I remember I tried doing this in the past but the results were too vast to check because of the U+200E character. Now that that problem is solved, maybe the list can be manageable. - Klein Muçi (talk) 01:46, 19 March 2021 (UTC)
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Gwich’in(\s*[\|\}])", r"\1gwi\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Gwichin(\s*[\|\}])", r"\1gwi\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)K’iche’(\s*[\|\}])", r"\1quc\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Kiche(\s*[\|\}])", r"\1quc\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)meta’(\s*[\|\}])", r"\1mgo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)meta(\s*[\|\}])", r"\1mgo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Meta’(\s*[\|\}])", r"\1mgo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Meta(\s*[\|\}])", r"\1mgo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)n’ko(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)nko(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)N’Ko(\s*[\|\}])", r"\1nqo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮ(\s*[\|\}])", r"\1ban-bali\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮᬶ(\s*[\|\}])", r"\1ban-bali\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ዕብራይስጥ(\s*[\|\}])", r"\1he\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ዕብራይስጥ(\s*[\|\}])", r"\1he\2"),
These are the comparisons of the first 10 codes. I thought there would be only some apostrophes missing but the final results surprised me. The Red Dot is back. I'll continue with the other comparisons and hopefully only post the unexpected results, if there are any. The length of the lists continues to be the same in both methods. - Klein Muçi (talk) 11:46, 19 March 2021 (UTC)
- This red dot is U+FEFF zero width no-break space. It only occurs in the Amharic rendering of
he
:{{#language:he|am}}
→ ዕብራይስጥ
- I have added a snippet of code to strip U+FEFF ZWNBSP when encountered.
- Here is a list of language names that contain U+2019 right single quotation mark:
- Ale’uhtesch
- Creole ta’ Haiti
- Cànan Hawai’i
- Du’ala
- D’zongqa
- Ge’ez
- Ghomala’
- Ghomálá’
- Gwich’in
- Ha’iihtesch
- Ikigaluwa cy’Igisweduwa
- Ingusch’sch
- Je’orjesch
- Ji’is-Ahl-Ättejohpesch
- Jöödsch-Pers’sch
- Kasach’sch
- Kige’ez
- Kon’kahnesch
- Koro’eeshiyaan
- K’iche’
- K’ische’
- Ma’ohresch
- Meta’
- Middelpers’sch
- Mi’kmaq
- N’Ko
- N’ko
- Ooldpers’sch
- Pers’sch
- Russ’sch
- Sorbjan ta’ Fuq
- Susbaint nach eil ’na chànan
- Swahili’r Congo
- Tiếng H’Mông
- Tiếng Meta’
- Tiếng N’Ko
- Tschech’sch
- Tschuwasch’sch
- Uj’juhresch
- Wittruss’sch
- Yup’ik
- alemán d’Austria
- amazighe de l’Atlas central
- cajun’i
- crioll d’Haití
- espagnol d’Amérique latine
- espagnol d’Espagne
- español d’América Llatina
- flandrezeg ar c’hornôg
- fris da l’ost
- ge’ez
- gheg d’Albania
- gwich’in
- gwich’inkielâ
- inglés d’Australia
- inglés d’Estaos Xuníos
- isi-Meta’
- isi-N’Ko
- ki’che’
- k’iche
- k’iche’
- meta’
- mi’kmaq
- noma’lum til
- n’Ko
- n’ko
- n’koera
- n’kó
- quechua dell’altopiano del Chimborazo
- same d’Inari
- sami d’Inari
- slavon d’église
- s’čchuanská iovčina
- vepsän kel’
- xitoy (an’anaviy)
- árabe d’Arxelia
- árabe d’Exiptu
- Ν’Κο
- в’етнамская
- гуіч’ін
- комі-перм’яцька
- нг’ембон
- пап’яменту
- т’яп
- مېتاچە’
- ग्विच’इन
- অ’চিটান
- এন’কো
- কোৱাছিঅ’
- গওইচ্’ইন
- জোলা-ফ’নি
- প’লিচ
- ফ’ন
- ল’ জাৰ্মান
- ল’ৱাৰ ছোৰ্বিয়ান
- ਗਵਿਚ’ਇਨ
- એન’કો
- ગ્વિચ’ઇન
- ଗୱିଚ’ଇନ୍
- ᎺᎳ’
- 曼德文字 (N’Ko)
- —Trappist the monk (talk) 12:52, 19 March 2021 (UTC)
- Hmm, at this point I should probably stop the general testing and see if I can fix the missing the right single quotation mark. I'll try copy-pasting a single entry from that list (a language name) into a wiki page, curl it, and see what really happens and how can I make it render like it should on bash. If I don't succeed, I'll go on with the general testing until the end so we at least can fix other errors along the way and look at the remaining ones in the end. - Klein Muçi (talk) 13:08, 19 March 2021 (UTC)
- These are the language names that use the simple keyboard apostrophe/single quote:
- Avañe'ẽ
- Fe'Fe'
- Gvich'in
- Lamnso'
- Mi'kmaq
- Mka'a
- Nda'Nda'
- O'odham
- —Trappist the monk (talk) 13:38, 19 March 2021 (UTC)
- I did some experiments with N'ko and I finally found out what was happening which will be hard to explain so I'll try to be as clear as possible. Turns out that the script list gets the name of the language correctly, without missing any marks. And so does the manual way. The copy-paste to the Notepad++ (the text editor that I use) works as it should without leaving anything behind. The problem comes when I copy from the Notepad++ to the terminal. As soon as I copy the text there, marks like these disappear. There is no problem with languages like Nda'Nda' or O'odham but you get the same error if you try any of the languages you've mentioned above them (and for languages in the bottom of that list, you get squares with question marks because the terminal can't render those languages at all - even though it still understands them strangely enough, as far as I've seen from some testing with the corrections in regard to languages like those). My plan for the moment will be to somehow make the terminal render correctly strings like these and maybe then do a final test between the total results from the script and the total results gotten manually and see what's changing. If I'm able to get past the terminal problem. - Klein Muçi (talk) 00:50, 20 March 2021 (UTC)
- Ok, so update: Turns out Windows' command line (cmd.exe) can't handle the characters we've been discussing so I switched it to git bash. I returned your changes on module Smallem and apparently git bash can "understand" the "red dot" characters we removed and doesn't show a change when compared to the manual list. I'm in a dilemma if it still would be better to remove them or not? What do you think? For the moment, just so we don't get confused, I'm leaving them removed, tell me later what should we do. I'm on a process of checking all the codes in batches of 10 entries. When I finish, I'll compare the total lists and report back here what's causing the change on the numbers. - Klein Muçi (talk) 02:56, 20 March 2021 (UTC)
- I did some experiments with N'ko and I finally found out what was happening which will be hard to explain so I'll try to be as clear as possible. Turns out that the script list gets the name of the language correctly, without missing any marks. And so does the manual way. The copy-paste to the Notepad++ (the text editor that I use) works as it should without leaving anything behind. The problem comes when I copy from the Notepad++ to the terminal. As soon as I copy the text there, marks like these disappear. There is no problem with languages like Nda'Nda' or O'odham but you get the same error if you try any of the languages you've mentioned above them (and for languages in the bottom of that list, you get squares with question marks because the terminal can't render those languages at all - even though it still understands them strangely enough, as far as I've seen from some testing with the corrections in regard to languages like those). My plan for the moment will be to somehow make the terminal render correctly strings like these and maybe then do a final test between the total results from the script and the total results gotten manually and see what's changing. If I'm able to get past the terminal problem. - Klein Muçi (talk) 00:50, 20 March 2021 (UTC)
- These are the language names that use the simple keyboard apostrophe/single quote:
- Hmm, at this point I should probably stop the general testing and see if I can fix the missing the right single quotation mark. I'll try copy-pasting a single entry from that list (a language name) into a wiki page, curl it, and see what really happens and how can I make it render like it should on bash. If I don't succeed, I'll go on with the general testing until the end so we at least can fix other errors along the way and look at the remaining ones in the end. - Klein Muçi (talk) 13:08, 19 March 2021 (UTC)
(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ইগ্<200c>বো(\s*[\|\}])", r"\1ig\2)),
Even though I'm still not finding any changes between lists, I found some lines appear like this (this was only one of many) when I was on these codes be-tarask, be-x-old, bg, bh, bi, bjn, bm, bn, bo, bpy
. Usually I get that symbol (<200c>) whenever there's a "strange" character. I used to get the same with what we've talked above until you removed those. What do you think is happening there? - Klein Muçi (talk) 03:07, 20 March 2021 (UTC)
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)گﻩ<200c>ﻟیﻙی\ ﺲﻛۆﺖﻠﻫ<200c>ﻧﺩی(\s*[\|\}])", r"\1gd\2"), Another line, don't know if it is the same character in both of the examples. This one was on codes
cho, chr, chy, ckb, co, cr, crh, cs, csb, cu
- U+200C zero width non-joiner is required by some languages to prevent adjacent unicode codepoints from combining. For
ig
in Bengali, the language name contains গ্ and বো (both of which are combinations of 2 and 3 codepoints). Without U+200C ZWNJ you get গ্বো but with it you get গ্বো. Your second example (gd
– from Central Kurdishckb
) has two U+200C ZWNJ (the flipflop between right-to-left Arabic script and left-to-right Latin script is confusing). U+200C ZWNJ is not a codepoint that should be removed or replaced. - Unless there is some way that you can replace the
<200c>
text with the U+200C ZWNJ codepoint in the regexes, the best that we can do is have sq:Moduli:Smallem skip language names that contain U+200C ZWNJ. I have tweaked Moduli:Smallem to skip. See sq:Përdoruesi:Trappist the monk/Livadhi personal which uses the same ten language codes that you listed above. Before the skip code was added, Moduli:Smallem returned 5250 regexes; after, 5245 (Bengali (bn
) and Bishnupriya (bpy
) both use Bengali script so for codesig
,kpe
,hmn
, andlv
they produce the same regex. - —Trappist the monk (talk) 14:01, 20 March 2021 (UTC)
- Hmm, so much to learn about languages. But... I believe even though the terminal doesn't show it, the replacement works out fine. I don't think we should remove those codes altogether. How can I test it if it works fine or not? What change should Smallem be able to make specifically? - Klein Muçi (talk) 14:49, 20 March 2021 (UTC)
- So not sure about what I asked above, nonetheless, I'm going on with the comparisons to see if I can find anymore "problematic" characters in other codes. And I did found these three (well, one) during the rendering of regexes for
gor, got, gsw, gu, gv, ha, hak, haw, he, hi
: -
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)<200f>אינטרלינגואה(\s*[\|\}])", r"\1ia\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)<200f>וולאפיק(\s*[\|\}])", r"\1vo\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)<200f>נורדית\ עתיקה(\s*[\|\}])", r"\1non\2"),
- Maybe time to make the module skip more codes? :P Even though, as I said, I'm still wondering if Smallem really doesn't know how to make these changes or not. The reason I say this is, pardon my ignorance, because there are some other cases when I don't get the
<number-something>
part but instead get some squares or just plain "space" (which is not space, per se, but more like, invisible characters) and I've noticed that in those cases, the bot works fine. I mean, I've done only 1 test some time ago. I must disclose though that in those cases, when you copy the line into a text editor, the squares and the spaces are converted to what the string really is, while the<number-something>
cases stay like that even when copied to somewhere else, so maybe there is a difference. Again, sorry for the lack of terminology. - Klein Muçi (talk) 23:04, 20 March 2021 (UTC)- And some more...
-
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ಟಷೆಲ್<200d>ಹಿಟ್(\s*[\|\}])", r"\1shi\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)<200b>កៀហ្ស៊ីស(\s*[\|\}])", r"\1ky\2"),
- These ones on codes
kg, ki, kj, kk, kl, km, kn, ko, koi, kr
- Klein Muçi (talk) 23:41, 20 March 2021 (UTC) - (edit conflict)
- U+200F right-to-left mark. The three that you found are (apparently) the only language names with U+200F rtl mark. This 'character' should probably be removed just as we remove U+200E left-to-right mark so I have added that to sq:Moduli:Smallem.
- I don't think that it is necessary to worry too much about square-with-question-mark characters; that is just an indication that whatever tool is displaying the codepoint doesn't have a font for that codepoint; the data behind the codepoint is likely not modified. But, we need to know about the
plain "space" (which is not space, per se, but more like, invisible characters)
. We need to know about these characters because they might prevent a regex match if they are in the MediaWiki list but not in|language=
as a human would write the parameter value. - Whatever is converting codepoints like U+200C zero width non-joiner from the native codepoint to the text string '<200c>' is a problem because that 'string' won't be appearing in a language name in a cs1|2 parameter value as a human would write it. You wrote:
when you copy the line into a text editor, the squares and the spaces are converted to what the string
. When I copy regexes that have U+200F rtl mark from Wikipedia's Moduli:Smallem rendering to Notepad++, the U+200F rtl mark is not converted to '<200f>'. - —Trappist the monk (talk) 00:44, 21 March 2021 (UTC)
- U+200B zero width space, U+200D zero width joiner. I'll look into those tomorrow.
- There are about thirty language names that use U+200D zero width joiner. These occur in
mr
,si
, andtcy
language sets. U+200D ZWJ is required to join certain codepoints together into a single character so we can do nothing with them. There are five U+200B ZWSP all from the Khmer language set. These are typographic indicators that can be used to indicate when a line break is permitted; I do not know if they are required or if humans writing these language names would include them. - —Trappist the monk (talk) 15:04, 21 March 2021 (UTC)
- There are about thirty language names that use U+200D zero width joiner. These occur in
- —Trappist the monk (talk) 00:44, 21 March 2021 (UTC)
- So not sure about what I asked above, nonetheless, I'm going on with the comparisons to see if I can find anymore "problematic" characters in other codes. And I did found these three (well, one) during the rendering of regexes for
- Hmm, so much to learn about languages. But... I believe even though the terminal doesn't show it, the replacement works out fine. I don't think we should remove those codes altogether. How can I test it if it works fine or not? What change should Smallem be able to make specifically? - Klein Muçi (talk) 14:49, 20 March 2021 (UTC)
- U+200C zero width non-joiner is required by some languages to prevent adjacent unicode codepoints from combining. For
Thank you for all the information! I have a clarification to make though: Regexes that have U+xxxX on them DO NOT get converted when you change the working space. The regexes that get converted are those with squares or "plain space". That's what I was saying and why I believe U+xxxX cases behave different from square/question mark AND plain space/invisible character cases. At this point we're suggesting the same thing apart from the plain space/invisible character cases, which I believe to behave the same as square/question mark cases, so in other words, the tool (Git Bash in my case) doesn't have a font to display them properly. But I don't know why some cases get squares/question marks and some get plain spaces/invisible characters. There are A LOT of both of these cases and, as it may be already expected now, they all happen in non-Latin characters. I'll try to paste some of them here from the K-codes mentioned above I was already working now to give some examples. - Klein Muçi (talk) 01:18, 21 March 2021 (UTC)
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)토켈라우제도어(\s*[\|\}])", r"\1tkl\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)토크\ 피신어(\s*[\|\}])", r"\1tpi\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)통가어(\s*[\|\}])", r"\1to\2"),
For me, all the Korean characters above (it is Korean, no?) are shown as empty squares in Git Bash. Just 1 example.
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)ᬩᬲᬩᬮᬶ(\s*[\|\}])", r"\1ban-bali\2"),
For me, the empty squares in the regex above (squares with question marks, if you see the source code) are shown as just space/void. These cases are rarer than the empty squares cases and after searching for more than 30 codes, I was able to refind only this case. But there are codes in which they appear more often.
So to make it more clear, there are 3 cases when the strings aren't rendered as they should:
- The U+xxxX cases (which stay like that everywhere)
- The empty square cases (which change to text strings in other workspaces, including here and on Notepad++)
- The empty space cases (which change to empty squares or squares with question marks if you see the source code in here and on other workspaces like Notepad++... I believe. Can't be too sure because I was only able to find 1 of those for the moment.) - Klein Muçi (talk) 01:41, 21 March 2021 (UTC)
- So where do the <200c> etc text strings come from?
- —Trappist the monk (talk) 15:04, 21 March 2021 (UTC)
I noticed that the 2nd case mentioned above only happens when using the manual method, so, in other words, when copying the results from the wikipage to the terminal manually. When you get the results there using the script, they render properly. This is referring only to the second case.- This is not true apparently. It appears to be random. I got the empty squares even on script generated results:
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Kʼabilan\ Scots\ Gaelic(\s*[\|\}])", r"\1gd\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Kʼicheʼ(\s*[\|\}])", r"\1quc\2"),
- The single quote got rendered as an empty square in both of these script generated results. Strangely enough, on the next result it gets rendered properly:
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)K’iche’(\s*[\|\}])", r"\1quc\2"),
- Hence, I believe it is random.
- This is not true apparently. It appears to be random. I got the empty squares even on script generated results:
- Side request: I was wondering... Should Module Smallem report an error if you put a langcode with a typo or if you have put the same code twice? For example, I want to put
xal, xh, xmf, yi, yo, yue, za, zea, zh, zh-classical
but instead, accidently, I putxal, xh, xmf, yi, yo, yue, za, zea, zh, zh-
or maybe evenxal, xh, xmf, yi, yo, yue, za, zea, zh, zh
. The way I think it could work is basically understand what codes are possible and if it finds alphabetical characters that are not creating a code or are creating the same code twice, it renders an error accordingly. - Klein Muçi (talk) 02:19, 21 March 2021 (UTC)- The characters 'ʼ' and '’' are not the same codepoints. The first one (from the group of three regexes) is U+02BC modifier letter apostrophe and the second is U+2019 right single quotation mark. I suspect that the U+02BC is probably the more semantically correct codepoint (the Kʼicheʼ language article here uses that codepoint). Nothing for us to do with these code points. Because they are different, sq:Moduli:Smallem will include both in the regex list.
- I've added a check for language codes in
|lang=
that end with a hyphen. See sq:Përdoruesi:Trappist the monk/Livadhi personal. - —Trappist the monk (talk) 15:04, 21 March 2021 (UTC)
- The empty space cases (which change to empty squares or squares with question marks if you see the source code in here) are not always true either. I just got a whole lot of these cases in codes
krc, ks, ksh, ku, kv, kw, ky, la, lad, lb
. 4 examples: -
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)קריאולית\ \(האיטי\)(\s*[\|\}])", r"\1ht\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)קריאולית\ \(סיישל\)(\s*[\|\}])", r"\1crs\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)קריאולית\ לואיזיאנית(\s*[\|\}])", r"\1lou\2"),
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)קריאולית\ מאוריציאנית(\s*[\|\}])", r"\1mfe\2"),
- Everything in Hebrew (it is Hebrew, no?) for me gets rendered as only empty space. I'm nearly sure I've seen Hebrew getting rendered properly before in my terminal so I'm starting to think this is random too. :/ - Klein Muçi (talk) 02:45, 21 March 2021 (UTC)
- In the above regexes, the Hebrew text is right-to-left so the escape character '\' precedes spaces and the parentheses on the right. There are no 'peculiar' code points in those language names.
- —Trappist the monk (talk) 15:04, 21 March 2021 (UTC)
- This has been the first time I've been dealing with Unicode characters at this level so only now I'm really starting to understand what's happening around with all those non-printing characters and marks. Yesterday I was reading a bit about Unicode and codepoints and I hope I'll have a better vocabulary when referring to these problems in the future. For that I thank you because it was only after your explanations that I started going further into understanding the computeric work done after scripts different from Latin ones. I have a naive question: Shouldn't we make Smallem skip those languages that use those 2 characters mentioned above like we did in other cases? What's changing in case? - Klein Muçi (talk) 15:32, 21 March 2021 (UTC)
- PS: The error is good. Shouldn't there be an error even for cases like this though:
xal, xh, xmf, yi, yo, yue, za, zea, zh, z
? Or is that not possible? - Klein Muçi (talk) 15:37, 21 March 2021 (UTC)- You mean: should sq:Moduli:Smallem skip U+200B zero width space and U+200D zero width joiner like it skips U+200C zero width non-joiner? Don't know. Perhaps, perhaps not. You didn't answer my question:
So where do the <200c> etc text strings come from?
If that text string is created by you actually editing the regex, then we should not bother to skip those language names. If those strings are created by a machine, then, yes, language names with U+200B ZWS and U+200D ZWJ should also be skipped. - Here is an interesting oddity:
{{#language:sq}}
→ shqip – 'to' language tag not specified so returns autonym{{#language:sq|sq}}
→ shqip – 'to' language tag specified as Albanian so returns Albanian language name in Albanian{{#language:sq|sq-}}
→ Albanian – malformed 'to' language tag not ignored as one would expect but instead, returns Albanian language name in English{{#language:sq|sq-L}}
→ Albanian – malformed 'to' language tag not ignored as one would expect but instead, returns Albanian language name in English{{#language:sq|s}}
→ shqip – malformed 'to' language tag is ignored as expected so returns Albanian autonym
- I've changed sq:Moduli:Smallem so that all codes in
|lang=
must be found in MediaWiki's English list of language codes. - —Trappist the monk (talk) 17:54, 21 March 2021 (UTC)
- You mean: should sq:Moduli:Smallem skip U+200B zero width space and U+200D zero width joiner like it skips U+200C zero width non-joiner? Don't know. Perhaps, perhaps not. You didn't answer my question:
- The empty space cases (which change to empty squares or squares with question marks if you see the source code in here) are not always true either. I just got a whole lot of these cases in codes
Hmm, I thought I had made that clear already... Those kind of text strings together with other oddities that I mentioned above (those 3 cases I explained) are all regexes that appear in my bash (shell). I use Git Bash (used to use cmd.exe) to SSH to ToolForge where I have an account. I've written a script (part of a bigger script aiming to deal with the whole update of the source code of Smallem - the list of regexes basically) that gets the result from this page using cURL and saves them in a file. When I see these results, the ones I've sent here appear like I've sent them. With the <200c> etc. text strings.
Interesting results the ones you showed. I'm curious if other languages may manifest problems like these as well. I don't know who exactly "looks after" these codes. Are they all dealt with from MediaWiki developers not related directly to the languages they deal with? Or are there global volunteers somehow helping in this subject? I believe we've already talked about this in the past. Either way, I expect small wikis' languages to suffer more from inaccurancies like this. - Klein Muçi (talk) 18:53, 21 March 2021 (UTC)
- Still isn't crystal-clear – too many words talking about something else. Regardless, I have tweaked sq:Moduli:Smallem so that it skips language names that have U+200B zero width space, U+200C zero width non-joiner, or U+200D zero width joiner codepoints.
- —Trappist the monk (talk) 13:29, 22 March 2021 (UTC)
- I'm really sorry. Let me rephrase: I've created a bash script that basically logs in Wikipedia's API through Smallem's account, gets the regex lines from that page I mentioned above and saves them in a Linux file in ToolForge servers. When this happens, some results on the saved file look normal, some look like what I've sent you above. That's it. If I manage to make the process work fine, Smallem will ultimately use this file to do its regex replacements it usually does, so that's basically an autoupdate for it. Is it more clear now? - Klein Muçi (talk) 13:39, 22 March 2021 (UTC)
- I finished comparing the lists (script/manual) of every code. There are no U+xxxX characters anymore anywhere and every list compared is totally the same. So basically the script method is as good as the manual one. But I'm yet to compare the total script list with the total manual list, which is where usually the errors happen, no idea why. I will do that tomorrow and report with the results. Meanwhile, some features related to the module:
-
- I believe module Smallem should report an error if we're using |list=<number> and |plain=yes. That's because it literally is an error. You can't group the codes if you have plain mode activated and it is confusing if you forget that detail because nothing changes.
- Is it possible somehow that the total number of regex lines gets rendered in the bottom of the language codes? (In both modes) That would be a gamechanger. I'm talking about the ultimate total.
- Would it be better if when module Smallem skipped a certain language, along the "skipped x language, from x.wiki" message was also shown the reason why it was skipped? This way we could easily remember the characters that are problematic now and possibly try to find ways to handle them without plainly skipping in the future.
- This reminds me about the first languages that we skipped because my command line couldn't read it. I switched back from cmd.exe to Git Bash lately and unlike cmd, Git Bash gets updated more frequently by its maintainers and maybe it could handle it now. Adding on that, maybe it would be wise to simplify the "skipping mechanism", have it standing somewhere aside on the module code so I can make changes to it easily for testing purposes. Maybe adding new languages to be skipped could be harder but removing them should be easier enough to be able to use it as a switch. Maybe you just remove the name from a certain "list" or comment out a certain line. I'm thinking something similar to Module:CS1/Configuration. Maybe it already is like that. I'm just brainstorming without much prior knowledge to the actual code arrangement in Module:Smallem. When it was only 1 language skipped, things looked a bit different. But now we're having many languages skipped and the number could potentially grow when trying to fix the problems with the total number. Couple that with the fact that ideally Git Bash can change for better in a near future and the whole thing takes more importance now. - Klein Muçi (talk) 04:05, 23 March 2021 (UTC)
- Is
|list=<number>
and|plain=yes
really an error?|list=
has to have some sort of an assigned value else sq:Moduli:Smallem attempts to make a regex list from the values assigned to|lang=
. - The only way to get the maximum-possible count of regexes that Moduli:Smallem thinks that it will produce is to run
lang_lister()
twice; once against the whole list of possible language codes (output suppressed – no error or skipped messages unless something is so wrong that it can't coninue) and the second time against the list of language codes in|lang=
. I can imagine Moduli:Smallem overrunning its allocated time if we try to do that. - I have thought about that now that the list of things-to-skip is growing.
- I'll think about that.
- —Trappist the monk (talk) 14:17, 23 March 2021 (UTC)
- Hmm but maybe a warning whatsoever? Call me stupid if you want but I set |list=10 in the past and I spent a couple of minutes not understanding why nothing had changed. Then I noticed |plain=yes and I remembered... In general the |list parameter confuses me because, at least in my mind, it "doubles" in function considering the values it can take. Those can be either yes or no OR numbers. I understand I can put anything instead of yes/no but just simplifying here. And you basically have 2 concepts merged in one parameter. At least in my mind. Coupling that with the |plain=yes situation makes it more confusing. Sad about the lack of possibility to show the total number but... What to do? - Klein Muçi (talk) 15:53, 23 March 2021 (UTC)
- I'm really sorry. Let me rephrase: I've created a bash script that basically logs in Wikipedia's API through Smallem's account, gets the regex lines from that page I mentioned above and saves them in a Linux file in ToolForge servers. When this happens, some results on the saved file look normal, some look like what I've sent you above. That's it. If I manage to make the process work fine, Smallem will ultimately use this file to do its regex replacements it usually does, so that's basically an autoupdate for it. Is it more clear now? - Klein Muçi (talk) 13:39, 22 March 2021 (UTC)
Hey man! I'm sorry that I kinda disappeared, especially when I was supposed to basically do the last step in testing, the reason why we've been doing all the aforementioned adjustments above. I made some changes to my internet at home and that required some days to take effect. I made Smallem make a full run of autoupdate. The total I get is 40635 lines. From that, remove 17 lines which are other regexes and command lines, and you get 40618 regex lines. I believe the number is still off the right total no? Even after making the module skip some "problematic" entries. I'm just asking now because I haven't checked the new total after we made the changes. - Klein Muçi (talk) 10:23, 30 March 2021 (UTC)
- sq:Moduli:Smallem says 42446.
- —Trappist the monk (talk) 12:50, 30 March 2021 (UTC)
- I got 40864 results now with 1 code per request. The first was with 10 codes per request. I'm gonna try creating the list manually and see what's missing. Is there any way I can show you the missing results here? I mean, we're talking about 2k results and I don't wanna overflow your page. Any other way? Maybe you can get an idea on why those codes are being left behind when you see what's actually being left behind. Because it is really strange. We basically tried together every code and there is literally no discrepancy between the script list and the manual one. Only when I make the script run all the codes automatically I start having problems. Which sort of seems bizarre to me. - Klein Muçi (talk) 21:28, 30 March 2021 (UTC)
- Apparently we're talking about exactly 406 lines. Still a lot to copy-paste them here. I'll try to find a webpage to host them temporarily and I'll post the link here when I can do that. Meanwhile I saw the new messages that you get when languages were skipped with Module Smallem. I like them a lot. - Klein Muçi (talk) 22:54, 30 March 2021 (UTC)
- These are the lines that are missing. (At least in this case. Unfortunately I still get different results every time.) Notice any pattern whatsoever in them? - Klein Muçi (talk) 23:00, 30 March 2021 (UTC)
- Zulu (
zu
) last code in the list of codes. Perhaps the results of the last code aren't being saved? - —Trappist the monk (talk) 23:34, 30 March 2021 (UTC)
- So all those lines are all coming from 1 single code? Pretty interesting! But Zulu has 859 lines. You're saying that only a part of those isn't being saved? I'm redoing the full run now (1 code per request) and I'll see what discrepancies I get this time. I so hope they're again from the Zulu language. That would mean I would have finally found a pattern. - Klein Muçi (talk) 00:02, 31 March 2021 (UTC)
- I can't say that they all come from code
zu
. Most of the language names begin with 'isi-' so I spotchecked several of them; all that I checked werezu
. I have looked at all of the language names at the top and bottom of the list and all of them arezu
except 'Patois' (jam
):- Patois; –
{{#language:jam|zu}}
→ Jamaican Creole English
- Patois; –
- Don't know where Patois comes from.
- It isn't surprising that Zulu by itself has 859 lines but the list you gave me has only 406. I suspect that there are a lot of codes that fall back to another language's name (probably English) so those would be duplicates that should be removed. That suggests that these 406 are not duplicated by any other language.
- —Trappist the monk (talk) 01:02, 31 March 2021 (UTC)
- I can't say that they all come from code
- So all those lines are all coming from 1 single code? Pretty interesting! But Zulu has 859 lines. You're saying that only a part of those isn't being saved? I'm redoing the full run now (1 code per request) and I'll see what discrepancies I get this time. I so hope they're again from the Zulu language. That would mean I would have finally found a pattern. - Klein Muçi (talk) 00:02, 31 March 2021 (UTC)
- Zulu (
- These are the lines that are missing. (At least in this case. Unfortunately I still get different results every time.) Notice any pattern whatsoever in them? - Klein Muçi (talk) 23:00, 30 March 2021 (UTC)
- Apparently we're talking about exactly 406 lines. Still a lot to copy-paste them here. I'll try to find a webpage to host them temporarily and I'll post the link here when I can do that. Meanwhile I saw the new messages that you get when languages were skipped with Module Smallem. I like them a lot. - Klein Muçi (talk) 22:54, 30 March 2021 (UTC)
- I got 40864 results now with 1 code per request. The first was with 10 codes per request. I'm gonna try creating the list manually and see what's missing. Is there any way I can show you the missing results here? I mean, we're talking about 2k results and I don't wanna overflow your page. Any other way? Maybe you can get an idea on why those codes are being left behind when you see what's actually being left behind. Because it is really strange. We basically tried together every code and there is literally no discrepancy between the script list and the manual one. Only when I make the script run all the codes automatically I start having problems. Which sort of seems bizarre to me. - Klein Muçi (talk) 21:28, 30 March 2021 (UTC)
Second list of discrepancies. Much to my amazement, the change between iterations wasn't that big. There were the same missing results except: (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Patois(\s*[\|\}])", r"\1jam\2"),
which in the second iteration wasn't missing. 405 results in total in the second run. I'll try it again because I believe the close numbers were just a coincidence. I'll post the results here in the same method when I have them. Looking forward to 5 or 10 tests. - Klein Muçi (talk) 00:56, 31 March 2021 (UTC)
- Hmm, missed that detail. But that's even better. I'm currently running a third test as I said but after that is completed, if the results again stay on the "Zulu realm", can we make module Smallem not generate the
zu
code temporarily? I can then do 2-3 test and see what happens. If we get no discrepancies in any of those tests, we would have successfully isolated the problem. My belief though is that the test that I'm running now won't stay on that realm. :/ - Klein Muçi (talk) 01:22, 31 March 2021 (UTC)- Completed test 3. Got exactly the same results as test2. This is strange. May it be that we have fixed the problem of results changing in every test by fixing the problems with those 3 nonprinting characters? I still believe this is just pure luck. I'll run 2 more tests tomorrow and see what happens. If it really doesn't change then we can go on with the removal of
zu
from Module:Smallem. But this is really strange and inspiring at the same time. - Klein Muçi (talk) 02:40, 31 March 2021 (UTC)- Completed test 4. Again exactly the same results. I'm genuinely surprised that we might have solved the inconsistency problem. After some hours I'll try the last test. - Klein Muçi (talk) 11:12, 31 March 2021 (UTC)
- Instead of dropping
zu
, perhaps it would be better to invert the language code sort so thatzu
is first andaa
is last. If the result is the same then that confirms your something-wrong-with-zu-handling hypothesis. If the result shows that all of the lost regexes are fromaa
then that suggests that somewhere the last groups of regexes isn't being correctly handled. - Easy to invert the sort; let me know when you're read for that.
- —Trappist the monk (talk) 11:22, 31 March 2021 (UTC)
- Test 5 complete. Again no difference whatsoever. Apparently we've solved the inconsistency problem. That was one of 2 of my main problems with the script so it does look unbelievable for me. You can go on with the code inversion now. - Klein Muçi (talk) 23:42, 31 March 2021 (UTC)
- Completed test 3. Got exactly the same results as test2. This is strange. May it be that we have fixed the problem of results changing in every test by fixing the problems with those 3 nonprinting characters? I still believe this is just pure luck. I'll run 2 more tests tomorrow and see what happens. If it really doesn't change then we can go on with the removal of
Test 1 with inversion complete. Things are starting to go into the unknown again. I got 41269 lines in total. Remove 17 "other" lines and you get 41252 lines of regexes. I'll soon post here the exact missing lines. - Klein Muçi (talk) 01:20, 1 April 2021 (UTC)
- Soo... I'm feeling extra confused by now. I concatenated the manual full list with the script full list and asked to get only the unique lines, which is what I've been doing all this time. And... I get only 1 unique line:
- (r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Qafár\ af(\s*[\|\}])", r"\1aa\2"),
- But that doesn't make much sense considering the total number I have and the total number I should be getting. Does that mean anything to you or is it a glitch and I should redo the whole procedure somehow? - Klein Muçi (talk) 02:05, 1 April 2021 (UTC)
- Test 2 complete. Again 41269 lines. At least we can be secure that the inconsistency is gone for good. I'll try sorting and comparing again. May it be that the manual list I have is outdated? Could it become outdated just for 2 days? - Klein Muçi (talk) 11:01, 1 April 2021 (UTC)
- And again, only that 1 line above gets chosen as unique. I'm confused. - Klein Muçi (talk) 11:26, 1 April 2021 (UTC)
- I think that what it means is that
aa
has only one language name written in Afar, its autonym. All other language codes fall back to some other language (likely English):{{#language:aa|aa}}
→ Qafár af{{#language:de|aa}}
→ German{{#language:en|aa}}
→ English{{#language:fr|aa}}
→ French{{#language:sq|aa}}
→ Albanian
- So, this means that there is nothing wrong with
zu
and nothing wrong withaa
. Youraa
results tend to lend credence to the notion that the last group of regexes isn't being included in the final list of regexes (it's the automated list that is coming up short, right?) - —Trappist the monk (talk) 15:12, 1 April 2021 (UTC)
- I think that what it means is that
- And again, only that 1 line above gets chosen as unique. I'm confused. - Klein Muçi (talk) 11:26, 1 April 2021 (UTC)
- Test 2 complete. Again 41269 lines. At least we can be secure that the inconsistency is gone for good. I'll try sorting and comparing again. May it be that the manual list I have is outdated? Could it become outdated just for 2 days? - Klein Muçi (talk) 11:01, 1 April 2021 (UTC)
Yes, that was my initial belief too, ever since you mentioned that all the problems arise with zu
, which was the last code. The reason for that being in the way the script works. Speaking in general lines, it gets the list of codes from a wiki page and then transforms that concatenated one-line list into an one-code-per-line list (using the commas as delimiters). Then it starts a loop that gets 1 line from that list (so 1 code), creates an API request with it, saves the results (the list of regexes generated) in a file and removes that 1 line (so 1 code) from the overall code list. Then it gets 1 line from that list (which is the next code in line, because the line before that just got deleted), creates an API request with it... So on until it runs out of codes. Being that I'm not good on scripting, I may have designed badly the loop and the results from the last API request fail to be saved, whatever that is. So, you arising at the same conclusion is a good sign because that would mean that I just need to fix the loop and one of my 2 problems would be solved. The problem that confuses me though is that the automated list, which is the one that is coming short, is not coming short of just 1 entry compared to the manual list. If we do a simple subtraction, the results missing are around 1000 lines, no? Or am I being misled in the logic that I'm using? Either way, in order for us to be totally sure that this really is the problem that we're having, I have to ask, how easy is to twist module Smallem to create the environment for this experiment: Make it only produce 2 codes. One is en
and the other one being a "popular" language that we know that it doesn't have any fallbacks. Maybe Italian (it
)? Then we can see what happens. If all the regexes from it
go missing, then we're sure that the problem is on the loop and I have to fix that. I don't know how easy/or hard is it for you to do that, though. If it is hard, maybe I can twist my script to work with only those 2 codes but I'm a bit reluctant to do further changes in it before being sure what the current problem is because of being afraid in introducing even more bugs. - Klein Muçi (talk) 15:40, 1 April 2021 (UTC)
- If I understand you, you want the code list returned from
{{#invoke:Smallem|lang_lister|list=yes |plain=yes}}
to beCodeListBegin:en, it:CodeListEnd
. In that case, sq:Përdoruesi:Trappist the monk/Livadhi personal. - —Trappist the monk (talk) 16:21, 1 April 2021 (UTC)
- Yes, make the code list generate only 2 hardcoded codes. En and another one that doesn't have fallbacks. I'm currently on the move and can't properly check what you've brought but I'll write as soon as I have my laptop back. - Klein Muçi (talk) 16:28, 1 April 2021 (UTC)
- Redid the test. (You had done exactly what I had asked.) I get 876 results (with 17 added "other" lines). Manually I get 1464 (without the 17 added "other" lines). I'll see what lines are missing now. - Klein Muçi (talk) 20:00, 1 April 2021 (UTC)
- The missing results. 605 results.
It
has Regex count: 859 though. Do you think fallbacks are to be blamed for the discrepancy? - Klein Muçi (talk) 20:07, 1 April 2021 (UTC)- When I compare the Italian total results with the 605 results above, these are the unique results: Here. Basically, these are the fallbacks and when you see the results, it makes sense to treat them as fallbacks. My hope was that
it
wouldn't have any fallbacks so the total number of regex lines from that code would also be the number of missing lines from the grand total (given that there would be only 2 codes and onlyen
would be used). And that would secure the fact that the problem is coming from the loop not saving the results from the last code. But maybe the experiment above is enough as proof nonetheless? - Klein Muçi (talk) 20:16, 1 April 2021 (UTC)- 876 (the total list) - 17 = 859 though, the exact total of
en
so I guess... - Klein Muçi (talk) 20:18, 1 April 2021 (UTC)
- 876 (the total list) - 17 = 859 though, the exact total of
- When I compare the Italian total results with the 605 results above, these are the unique results: Here. Basically, these are the fallbacks and when you see the results, it makes sense to treat them as fallbacks. My hope was that
- The missing results. 605 results.
- Redid the test. (You had done exactly what I had asked.) I get 876 results (with 17 added "other" lines). Manually I get 1464 (without the 17 added "other" lines). I'll see what lines are missing now. - Klein Muçi (talk) 20:00, 1 April 2021 (UTC)
- Yes, make the code list generate only 2 hardcoded codes. En and another one that doesn't have fallbacks. I'm currently on the move and can't properly check what you've brought but I'll write as soon as I have my laptop back. - Klein Muçi (talk) 16:28, 1 April 2021 (UTC)
I was able to fix the loop and now I get 1481 lines all the time, which is 1464 + 17, precisely how many lines there should be. I'm excited! Can you revert your change about the hardcoding to module:Smallem so I can do a full run with all the codes now to see what happens? - Klein Muçi (talk) 22:53, 1 April 2021 (UTC)
- done; ascending order sort restored.
- —Trappist the monk (talk) 23:00, 1 April 2021 (UTC)
- Much to my dismay, I get 40865 lines in total and not 42463. Judging by that number it does look like I haven't changed anything. But I did try it 3 times and it gave me the total number when only 2 codes were involved with the changes that I made to the script. :/ I'll try to see what specific lines are missing now that I fixed the problem that we had. - Klein Muçi (talk) 00:39, 2 April 2021 (UTC)
- Aaand, no. I get the
zu
regex lines missing again. :/ I'll retry switching the module to 2 codes/debug mode to be totally secure it works fine in that mode. :/ - Klein Muçi (talk) 01:03, 2 April 2021 (UTC)- No, it works fine like that. Tried it some more times now. Any suggestions before I'm forced to try the brute force way by adding the codes one by one in debug mode to see when it starts malfunctioning? - Klein Muçi (talk) 01:13, 2 April 2021 (UTC)
- So, I went on with the "brute force" method I mentioned above. I first added 10 codes in the debug mode, I got no discrepancies. Then I went on and added 10 more. Here things start behaving not as you'd expect them to. The manual list gives 4484 results. I get 4483.
(r"(\{\{\s*cit[aeio][^\}]*\|\s*language\s*=\s*)Qafár\ af(\s*[\|\}])", r"\1aa\2"),
This one goes missing. Any idea what might be happening? Do you think it's still the same problem that we're seeing? (The results from the last code are being ignored?) These were the codes used:aa, ab, ace, ady, af, ak, als, alt, am, an, ang, ar, arc, ary, arz, as, ast, atj, av, avk
Keep in mind that it worked perfectly fine withaa, ab, ace, ady, af, ak, als, alt, am, an
. - Klein Muçi (talk) 10:20, 2 April 2021 (UTC)
- So, I went on with the "brute force" method I mentioned above. I first added 10 codes in the debug mode, I got no discrepancies. Then I went on and added 10 more. Here things start behaving not as you'd expect them to. The manual list gives 4484 results. I get 4483.
- No, it works fine like that. Tried it some more times now. Any suggestions before I'm forced to try the brute force way by adding the codes one by one in debug mode to see when it starts malfunctioning? - Klein Muçi (talk) 01:13, 2 April 2021 (UTC)
- Aaand, no. I get the
- Much to my dismay, I get 40865 lines in total and not 42463. Judging by that number it does look like I haven't changed anything. But I did try it 3 times and it gave me the total number when only 2 codes were involved with the changes that I made to the script. :/ I'll try to see what specific lines are missing now that I fixed the problem that we had. - Klein Muçi (talk) 00:39, 2 April 2021 (UTC)
Meh, redid some more runs with the same codes. Sometimes I get that line missing, sometimes I don't. I'm disappointed that we got back the inconsistencies between runs. I know they are related to non-printing characters somehow confusing my script but I don't know how. Because the problems started happening as soon as the module started skipping results for the first time. Is there anything special with that single line? I believe whatever is happening in this case, is happening throughout the whole process and the missing lines get accumulated in the end, to give the 1k short total. There must be another character somehow (not necessarily in that line) that we also need to skip. I'm gonna take a close look at the whole list of results, see if I find anything strange. - Klein Muçi (talk) 10:40, 2 April 2021 (UTC)
- Nope, I couldn't find anything too strange. :/ At this point I'm out of suggestions. I don't know why I start getting inconsistencies with these specific codes and why I get the total to be 1k short in the end. I think I did fix the part at the script which was responsible for failing to save the last results but somehow... :/ Maybe I haven't? - Klein Muçi (talk) 11:00, 2 April 2021 (UTC)
- In that group of twenty language codes, there are 59 regexes that use non-ASCII characters in their names. In those 59, the only 'invisible' character used is U+200E left-to-right mark which sq:Moduli:Smallem removed. So the problem is not with invisible characters.
- I can mimick the results that you got by removing
aa
from the group and from the|lang=
parameter. The Qafár af regex is the only unique language name contributed byaa
(the autonym) all other language codes fallback to English which are contributed by the other languages in the group. Perhaps your 'fix' to the tail end disrupted the head end? - I have tweaked Moduli:Smallem so that
|lang=
gets the same language codes that are listed in the debuglang_codes
(line 104). What happens if you swapaa
withab
? - —Trappist the monk (talk) 14:08, 2 April 2021 (UTC)
- Hmm, I'm not sure if I do understand quite well your last paragraph. If I've understood correctly what you have changed, I'm not sure if my script can work properly with module:Smallem anymore. And what do you mean with swapping exactly? :/ - Klein Muçi (talk) 20:41, 2 April 2021 (UTC)
- Apparently it kept working. I removed
aa
from Module:Smallem (that's what you meant, no?) and tried a full run. I got no problems. 4500 results in total: 4483+17 "other" lines. What does this mean? I'm following blindly now. - Klein Muçi (talk) 21:06, 2 April 2021 (UTC)- Did 3 runs. I always got 4500 lines. No inconsistencies. - Klein Muçi (talk) 21:24, 2 April 2021 (UTC)
- I meant change line 103 to:
lang_codes = {'ab', 'aa', 'ace', 'ady', 'af', 'ak', 'als', 'alt', 'am', 'an', 'ang', 'ar', 'arc', 'ary', 'arz', 'as', 'ast', 'atj', 'av', 'avk'}; -- debug
- —Trappist the monk (talk) 21:29, 2 April 2021 (UTC)
- Apparently it kept working. I removed
- Hmm, I'm not sure if I do understand quite well your last paragraph. If I've understood correctly what you have changed, I'm not sure if my script can work properly with module:Smallem anymore. And what do you mean with swapping exactly? :/ - Klein Muçi (talk) 20:41, 2 April 2021 (UTC)
Okay then. I'll do that now. And what's the hypothesis we're testing with that? - Klein Muçi (talk) 21:35, 2 April 2021 (UTC)
Perhaps your 'fix' to the tail end disrupted the head end?
- —Trappist the monk (talk) 21:42, 2 April 2021 (UTC)
- I get 4501 lines. Consistently. So I've ruined the head? :P - Klein Muçi (talk) 22:00, 2 April 2021 (UTC)
- I don't know. But since I could mimick the problem by removing
aa
, perhaps you did... - —Trappist the monk (talk) 23:03, 2 April 2021 (UTC)
- I switched to
lang_codes = {'aa', 'ab', 'ace', 'ady', 'af', 'ak', 'als', 'alt', 'am', 'an', 'ang', 'ar', 'arc', 'ary', 'arz', 'as', 'ast', 'atj', 'av', 'avk'}; -- debug
. I still get 4501 results now. The correct number. Without doing any changes to the script. :/ Should we revert the debug mode now and try a full normal run? Even though, this doesn't make much sense. We didn't do any change basically, no? :P - Klein Muçi (talk) 23:22, 2 April 2021 (UTC)- I'm confused. Are you suggesting that something, somehow, fixed itself?
- —Trappist the monk (talk) 23:40, 2 April 2021 (UTC)
- I switched to
- I don't know. But since I could mimick the problem by removing
- I get 4501 lines. Consistently. So I've ruined the head? :P - Klein Muçi (talk) 22:00, 2 April 2021 (UTC)
Well... Basically, yes... I mean, my initial idea was to keep adding codes into the "old" debug mode gradually until I started having problems (which happened when I added the second group of 10 codes). Even then, the problems weren't consistent. They happened only half of the times. Then you changed module:Smallem to introduce the "new" debug mode which automatically takes care for the code and regex list. I tried a run with that with aa
missing. Then tried it with aa
in second place (after ab
) and then tried it with aa
in the beginning. All the tests brought the correct, expected results. Without doing any change to the script. Now I don't know what to do. :/ We either switch it to the normal form, or continue adding other codes in this mode and see what happens. The problem with the second option though is that you have changed module:Smallem to add the codes automatically to |lang=
. When more than 10 codes are used simultaneously, the script tends to malfunction in getting the generated regex lines and starts saving empty results. I don't know how it will behave if we add 10 more codes and basically are forced to do 30 codes simultaneously. - Klein Muçi (talk) 23:53, 2 April 2021 (UTC)
- And, yes. Tried adding 10 codes more (changing the codes added to
|lang=
from 20 to 30) and the end results after the script completed its run were with only those "other" 17 results, empty from any regex lines, as expected. :/ If we want to keep experimenting and add codes gradually by 10, we need to turn back to the old debug mode where codes weren't added automatically to|lang=
. I don't know what else to try now, really. Because everything seem to be working fine at the moment. :P In this mode with only those 20 codes that is. - Klein Muçi (talk) 00:32, 3 April 2021 (UTC) - I made sq:Moduli:Smallem automatically render the code and regex lists from the same set of code so that you don't have to edit the module and then edit whatever page it is that Smallem gets the code and regex lists for each test you wanted to do. You can disable the debug mode by simply commenting-out lines lines 103 & 104. You can fix the
|lang=
value to some subset oflang_codes
by writing a replacement for line 104:args.lang = 'ab, ab, ace';
- or whatever you want.
- Perhaps some time away is in order. Do something else for a while and then come back to Smallem...
- —Trappist the monk (talk) 00:39, 3 April 2021 (UTC)
- Yeah, I fully understand your intention and it does help a lot if you do it manually but when you put the script to work, it starts bringing out more problems than it should. And yes, soon I'll be forced to accept that suggestion. My last try is to switch back to the old debug mode and retry entering the codes gradually, see if I see any pattern or anything else of the sort "I can grab to", if you understand what I mean. If that fails as well, I'll give up. At least for now. To be honest, this part of the Smallem project has taken me way more time than I thought it would and I have been ready to abandon it a couple of times but only went forward after the help you gave me and the thought that I've already written a full script able to do 90% of the job correctly on it. Maybe I annoy you with some last messages as I run the last tests for these days (what I described above) but I'd wish to clarify not to take all my messages as requests. Sometimes, as you may have already seen, I'm really just brainstorming, and some other times, I just share the information here so if I need to ask for help in the future, we're both on the same page. Only answer whenever you can. I'm sorry for the continuous red +1 (or more) you might have because of me. Every once in a while I forget and I try using emails but you don't have that option. :P - Klein Muçi (talk) 01:06, 3 April 2021 (UTC)
- Hey there! I tried everything I could but I noticed that the results still weren't consistent all the time. As a last resort, I tried asking for help on Stack Overflow. Here, if you want to take a look. I don't think I'll really find help there (24 hours have passed and I still haven't got 1 single answer) but at least I tried everything I could anywhere. I wanted to ask a question out of personal curiosity, feel free not to answer it. What do you work/deal with in everyday life apart from Wikimedia? Programming? I really find it surprising you find the time and nerves to deal with my and other's from all around the world technical requests in details. :P I'm, of course, thankful for that but I was also curious. As I said though, feel free not to answer if you don't want. :) - Klein Muçi (talk) 09:36, 5 April 2021 (UTC)
- Yeah, I fully understand your intention and it does help a lot if you do it manually but when you put the script to work, it starts bringing out more problems than it should. And yes, soon I'll be forced to accept that suggestion. My last try is to switch back to the old debug mode and retry entering the codes gradually, see if I see any pattern or anything else of the sort "I can grab to", if you understand what I mean. If that fails as well, I'll give up. At least for now. To be honest, this part of the Smallem project has taken me way more time than I thought it would and I have been ready to abandon it a couple of times but only went forward after the help you gave me and the thought that I've already written a full script able to do 90% of the job correctly on it. Maybe I annoy you with some last messages as I run the last tests for these days (what I described above) but I'd wish to clarify not to take all my messages as requests. Sometimes, as you may have already seen, I'm really just brainstorming, and some other times, I just share the information here so if I need to ask for help in the future, we're both on the same page. Only answer whenever you can. I'm sorry for the continuous red +1 (or more) you might have because of me. Every once in a while I forget and I try using emails but you don't have that option. :P - Klein Muçi (talk) 01:06, 3 April 2021 (UTC)
Automated Url Archiver
Hello! I'm wondering if there is a bot that will read all the citations in a wikipedia page, recognize those ones without archive URLs and automatically archive the sites and then add the archive url and archive date. Does that exist?Brent Severnie (talk) 13:15, 3 April 2021 (UTC)
- I don't know. Best person to ask is likely to be Cyberpower678 who operates InternetArchiveBot.
- —Trappist the monk (talk) 13:45, 3 April 2021 (UTC)
Are you related to User: FeloniousMonk?
Just curious. 69.5.112.154 (talk) 20:35, 3 April 2021 (UTC)
I have noticed that you have some interest in this article I created. I am on a quest to make as many Good Articles as I can this year. If you think this is a possibility, would you be interested in copy editing the article and nominating it GAN. You would get credit for a Good Article as the nominator and I would get credit as the creator of the article when it gets promoted. I will be glad to solve most of the issues the reviewer brings up.--Doug Coldwell (talk) 14:34, 6 April 2021 (UTC)
- I don't have any interest in SS Jacona (1918) except to fix deficiencies when I see them. I don't think that I've looked at that article since I made the one edit that I did make 5-ish years ago. And, alas, I am wholly immune to enticements like GA credit.
- You might want to look at the article's referencing ... Business week, Electric Journal, Hearst Magazines, and Marine Engineering are not authors so do not belong in
|last=
. - —Trappist the monk (talk) 15:16, 6 April 2021 (UTC)
- Thanks for hints on authors. I'll look into that.--Doug Coldwell (talk) 12:04, 7 April 2021 (UTC)
developer-discouraged
Regarding: I don't know what to think about
developer discouraged
. Outside of the RFC, is there such a thing?
No, I literally just made it up. I'm not exactly wedded to the term, and I justed wanted people to understand what the exact status of these parameters were (to replace the old definition of deprecated but also describe something where support won't ever necessarily be removed).
I was exactly thinking it was going to be used for maintenance category names (figured it would be something like Category:CS1 maint: nonhyphenated parameter
), but that's not really for me to weigh in on. –MJL ‐Talk‐☖ 18:14, 7 April 2021 (UTC)
Semi-protected edit request on 10 April 2021
It is requested that an edit be made to the semi-protected redirect at User talk:Monkbot. (edit · history · last · links · protection log)
This template must be followed by a complete and specific description of the request, that is, specify what text should be removed and a verbatim copy of the text that should replace it. "Please change X" is not acceptable and will be rejected; the request must be of the form "please change X to Y".
The edit may be made by any autoconfirmed user. Remember to change the |
114.125.180.230 (talk) 05:06, 10 April 2021 (UTC)