View source for Migration to the new preprocessor

[[User:Tim Starling|Tim Starling]] (MediaWiki developer and Wikimedia system administrator) has rewritten the parser preprocessor to be more consistent and less flaky. This document represents a thorough but minimalistic live test and secondary debugging phase of the new preprocessor on Wikimedia, specifically the English-language Wikipedia, through random sampling of several thousand pages for differences in rendering.

To test a page under the new parser, you can use [[Special:ParserDiffTest]]. To test a page with its current text, use subpage syntax, for instance [[Special:ParserDiffTest/Main Page]] to test the main page. To test a page with arbitrary text, go to [[Special:ParserDiffTest]] and fill in the form. For more options, please see [[m:Migration_to_the_new_preprocessor/how_to_test]]

On January 25, 2008, en.wiki and test.wiki switched to the new preprocessor. On February 22, 2008, the new preprocessor went live on all Wikimedia wikis. '''Help is still needed to identify any bugs or issues with the transition.'''

To check which preprocessor is active, use <nowiki>{{#if:{{#if:x|{{{2}}}|}}|new pp|old pp}}</nowiki>, here producing "{{#if:{{#if:x|{{{2}}}|}}|new pp|old pp}}". Or, alternately, you can check the page source for "NewPP"; just be sure to purge the page first, so as to avoid caching.

__TOC__

==Expected differences==
These are differences in behavior that are expected. They are either bugs in the old parser that are now corrected, or "features" in the new parser. Changes have been marked "negotiable" by Tim Starling if restoration of the old behaviour may be feasible.

Many of these are differences in the way the old parser allowed you to get away with things that the new parser is more strict on. 

{| class="wikitable" style="width:100%; margin:auto; font-size:90%;"
! Test case
! Difference from old parser
! Explanation
! Examples/fixes
! Negotiable?
|-
| <code><nowiki><h2>Unlinked header</h2></nowiki></code>
| XML headers no longer make an edit link
| Headers created with <code><nowiki><h1><h2><h3>...</nowiki></code> etc no longer create an editable section. They also do not create a section divide when editing sections. They do however add a level to the TOC as well as an anchor for linking. This allows you to add uneditable sections to a template, and still have editable sections on the page, which isn't possible with <nowiki>__NOEDITSECTION__</nowiki>.
| [http://en.wikipedia.org/wiki/User_talk:Chri1753?timtest=newpp]
| Negotiable
|-
|<code><nowiki><noinclude>==Unlinked header==</noinclude></nowiki></code>
| Headers embedded inside noinclude/includeonly tags will now behave as XML headers (see above). 
| The new parser has stricter header requirements.
| <s>[http://en.wikipedia.org/w/index.php?title=-&curid=11861511&timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Wikipedia:Version_1.0_Editorial_Team/Smashing_Pumpkins_articles_by_quality&diff=184470274&oldid=181546082]
| Not really
|-
| <code><nowiki><!-- </includeonly> --></nowiki></code>
| The includeonly/noinclude tags can now be commented out.
| The new parser allows these tags to be disabled inside comments, a change that has to be taken into account in template design.
| <s>[http://en.wikipedia.org/wiki/Dmitri_Stratan?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:Asbox&diff=prev&oldid=184012807]
| no
|-
| ''template with editable sections''
| Transcluded edit section links have a <code>T-</code> prefix.
| This allows the editable sections on a page to be a bit more informative (T-(n-1) where n is the real section number). [[bugzilla:6563]]
| [http://en.wikipedia.org/wiki/Template:Infobox_Election?timtest=newpp Template:Infobox Election]
| no
|-
| <code><nowiki>{{template|{{#if:1|a=1}}}}</nowiki></code>
| Old parser: <code><nowiki>{{{a}}} == "1"</nowiki></code>
New parser: <code><nowiki>{{{1}}} == "a=1"</nowiki></code>
| The equals sign between ''parameter'' and ''value'' can no longer be generated (by transclusion, parameter, parserfunction, etc) as a delimiter in the template parameters, it is interpreted literally. This requires moving the equals sign outside the generator.
| <s>[http://en.wikipedia.org/wiki/Template:Sussex_County_Reserve_Premier_teamlist?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:Tnavbar-header&diff=prev&oldid=184016409] &bull; <s>[http://en.wikipedia.org/wiki/Nevada?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:United_States_topic&diff=prev&oldid=184185938] [http://en.wikipedia.org/w/index.php?title=Template:Nevada&diff=prev&oldid=184180951] &bull; <s>[http://en.wikipedia.org/wiki/Template:Wpd-catlist-up?timtest=newpp] [http://en.wikipedia.org/wiki/Template:Cms-catlist-up?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:Catlst/doc&diff=184634802&oldid=184625284]
| no
|-
| <code><poem><nowiki>{{#switch: {{#time:Y|blah}}
|Error: invalid time = Error
|{{#time:Y|blah}}
}}</nowiki></poem></code>
| Old parser would see the #time error (a '''class="error"''') as a #switch case.
| Interesting bug in the old parser inadvertently utilized to catch errors. The implementors thought they were catching the plain text error of #time, but they were actually creating a new self-deleting #switch case that returned nothing. If <code><nowiki>{{#time:Y|blah}}</nowiki></code> generated an error, it generated an equals sign as part of the error, and the #switch would then see the last parameter not as a default passthrough but as a match case, that would fail, returning no output. This could also be expressed as: <code><nowiki>{{#switch:1|{{#time:Y|Z}}}}</nowiki></code> ... A new [[Help:ParserFunctions#.23iferror:|parserfunction]] was created for parserfunction error handling (that is, those that return errors), eg: <code><nowiki>{{#iferror:{{#time:Y|Z}}|error}}</nowiki></code>
| <s>[http://en.wikipedia.org/w/index.php?title=Template:Citation&action=edit]</s>
| no
|-
| <code><nowiki>{{#if:1|{{template{{!}}parameter}} }}</nowiki></code>
| {{tl|!}} no longer works as template delimiter in Parserfunctions.
| This is related to [[bugzilla:5678]] and should not have worked, and was in fact never necessary. As seen in implementations of [[w:Template:Self]], this bug allowed parameters other than the first to pass template parameters.
| <s>[http://en.wikipedia.org/wiki/Nevada?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:US_Demographics&diff=prev&oldid=184179517] &bull; <s>[http://en.wikipedia.org/wiki/Shell_Lake_Township%2C_Minnesota?timtest=newpp]</s> [http://en.wikipedia.org/w/index.php?title=Template:Infobox_Settlement&diff=prev&oldid=184162381] &bull; [http://en.wikipedia.org/wiki/Image:PBB_GE_TCEA1_216241_s_at_fs.png?timtest=newpp]
| not with this syntax
|-
| <code><nowiki>{{#ifeq:1|1|[[Template:{{{1}}}|foo]]}}</nowiki></code>
| Renders correctly in new parser.
| [[bugzilla:5678]]
| ~
| no
|-
| <code><nowiki> {{#switch:{{{g|}}}|g=|{{{g}}}}}</nowiki></code>
| Renders correctly in new parser.
| [[bugzilla:5678]] ??
| <s>[http://en.wiktionary.org/wiki/vervaardigen?timtest=newpp]</s> [http://en.wiktionary.org/w/index.php?title=Template:infl&diff=prev&oldid=3573159]
| no
|-
| <code><nowiki>{{#if:{{{2}}}|{{{2}}}|{{PAGENAME}}}}</nowiki></code>
| Fails correctly in new parser.
| Bug in the old parser. Note that #if should never be used if there is no chance of the test case being null. Note also that this usage of a parserfunction is excessive when you can just use parameter defaults such as <code><nowiki>{{{2|{{PAGENAME}}}}}</nowiki></code> [[bugzilla:5678]]??
| [http://en.wikipedia.org/wiki/Template:Wikitravel?timtest=newpp] &bull; [http://en.wikipedia.org/wiki/Template:Talkarchive?timtest=newpp] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_hp&diff=184863653&oldid=156335215] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_in&diff=184863656&oldid=156335229] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_lb&diff=184863660&oldid=180167179] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_lb_ft&diff=184863662&oldid=171227336] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_mm&diff=184863677&oldid=156335441] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_mpg&diff=184863684&oldid=178241130] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_cc-cu_in&diff=prev&oldid=184862858] &bull; [http://en.wikipedia.org/w/index.php?title=Template:Auto_PS&diff=prev&oldid=184482425]
| no
|-
| Template:Loop/deloop -> <code><nowiki>{{Template:Loop/{{{1}}}}}</nowiki></code><br />called with: <code><nowiki>{{Loop/deloop|deloop}}</nowiki></code>
| Detects loop in new parser.
| Loop detection is more strict in the new parser. A certain number of loops are exempt on the same page, so that examples can be included such as in documentation in a noinclude section, but potential loops one or more levels deep as per example are no longer possible (even in the old parser they were somewhat buggy.)
| [http://en.wikipedia.org/wiki/User:Color_probe/Template/Archive1?timtest=newpp]
| Negotiable
|-
| <code><poem><nowiki>
x
<!-- 1 --><!-- 2 -->
y</nowiki></poem></code>
| Old parser: &lt;p>x y&lt;/p> <br>New parser: &lt;p>x&lt;/p>&lt;p>y&lt;/p>. 
| In the old parser, multiple comments on the same line, without any separation, cause the line to be "eaten". In the new parser, there must be only one comment alone on a line for line-eating to be triggered. Without line-eating, the preprocessor leaves two line breaks between the <code>x</code> and the <code>y</code>, causing a paragraph break in the main pass.
| [http://en.wikipedia.org/w/index.php?title=Wikipedia:Version_1.0_Editorial_Team/Smashing_Pumpkins_articles_by_quality&oldid=184470274&timtest=newpp]
| Negotiable
|-
| <code><nowiki><span title="{{#if:1|">x<span>}}">y</span></nowiki></code>
| Badly formed wikicode (eg a parserfunction inside an html tag) is cleaned up differently.
| Old parser would parse the html first (replaceVariables and escaping) and not parse the #if, showing a literal mess. New parser will expand the #if first and see <code><nowiki><span title="">x<span>">y</span></nowiki></code> cleansing (via removeHTMLtags) to <code><nowiki><span title="">x<span>"&gt;y</span></span></nowiki></code>.

A counter example would be something like: <code><nowiki><span id="{{#expr:Z}}">z</span></nowiki></code>. The old parser would run removeHTMLtags first and end up with <code><nowiki><span>z</span></nowiki></code> while the new one would expand the #expr first, generate <code><nowiki><strong class="error">Expression error: Unrecognised word "z"</strong></nowiki></code>, causing the external span to be escaped and producing <code><nowiki>&lt;span id="<strong class="error">Expression error: Unrecognised word "z"</strong>"&gt;z</nowiki></code> 
| [http://en.wikipedia.org/wiki/Satow%2C_Germany?timtest=newpp] vs [http://en.wikipedia.org/wiki/Satow%2C_Germany?timtest=oldpp]
| no
|-
| <code><nowiki>{{urlencode:<_&amp;lt;_>_&amp;gt;_}}</nowiki></code>
| <code><nowiki>{{urlencode}}</nowiki></code> and <code><nowiki>{{anchorencode}}</nowiki></code> escape '''<''' and '''&amp;lt;''' differently between parsers.
| The new parser can now tell the difference between '''< >''' and '''&amp;lt; &amp;gt;''' inside these encode functions. The old parser would escape them first and cause different results. 
 old: %26lt%3B_%26lt%3B_%26gt%3B_%23gt%3B
 new: %3C_%26lt%3B_%3E_%23gt%3B
| [http://en.wikipedia.org/wiki/Special:ParserDiffTest/Buffy_the_Vampire_Slayer_Season_Eight].
|
|-
| <poem>{{#if:1|
= Top level heading =
}}
</poem>
| Section edit link missing in new parser
| A wikitext-style level-1 heading (&lt;h1&gt;) is not marked with a section edit link if the heading appears in a place where the parser expects an equals sign separating a template parameter name from its value, as in <nowiki>{{foo | name </nowiki>'''='''<nowiki> value}}</nowiki>. 
| Use a [http://test.wikipedia.org/wiki/Template:Passthru?action=edit passthru template] with a named parameter.
| Negotiable
|-
| <code><nowiki><onlyinclude><span>text</span>
</onlyinclude></nowiki></code>
| Onlyinclude newline behavior changed.
| <code><nowiki><onlyinclude></nowiki></code> had to be completely rewritten, and the new behavior will now match that of <code><nowiki><includeonly>/<noinclude></nowiki></code> in regards to newlines.
| ?
| no
|-
| <code><nowiki>{{#expr:$1 + 1}}</nowiki></code> in template.
| $1 in transclusion to interface message.
| $1-style message variables working in templates that were transcluded to MediaWiki: namespace interface messages, was a quirk/bug of the old parser and will no longer work. 
| [http://en.wikipedia.org/w/index.php?title=MediaWiki:Longpagewarning&diff=186571143&oldid=148080862]
| no
|}

==Bugs in the new parser==
===Fixed bugs===
Bugs found in the new preprocessor during secondary debugging, that were fixed. 
:''Note: '''rev:''' links for the fixes might be handy''.
:''Note: this could be made a prettier table''.

{| class="wikitable" style="width:100%; margin:auto; font-size:90%;"
| [http://en.wikipedia.org/wiki/Portal:Geography?timtest=newpp Geography]
| <s>#160;</s>
|-
| [http://en.wikipedia.org/wiki/%C4%8Cajetina?timtest=newpp Cajetina]
| <s>Whitspace associated with comments in parameters</s>
|-
| [http://en.wikipedia.org/wiki/%C4%8Cajetina?timtest=newpp Cajetina]
| <s>Infobox issues: &lt;ref> oddness in Area and Population sections;</s> <s>Linking issues: UTC+2 in Time zone section</s> ''linking issue related to whitespace''
|-
| [http://en.wikipedia.org/wiki/Tyler_Knight?timtest=newpp Tyler]
| <s>&lt;ref> oddness (it adds two in the &lt;references/> section to the first item)</s>
|-
| [http://en.wikipedia.org/wiki/User:The_Ungovernable_Force/Wikipedian_kitchen?timtest=newpp#.C2.A0_Smurrayinchester Kitchen header oddness]
| <s><nowiki>=====</nowiki> alone on a line seems to parse as <nowiki><h2>=</h2></nowiki> in the old parser.</s>
|-
| [http://en.wikipedia.org/wiki/Template:X10/doc?timtest=newpp x10]
| <s>Sections immediately after a &lt;!-- --&gt; comment (one newline apart) are not editable sections.</s>
|-
| [http://en.wikipedia.org/wiki/User:Jnlin?timtest=newpp Jnlin] 
| <s>Gallery tags sometimes have extranuous <nowiki><br /></nowiki></s>
|-
| <code><nowiki>#REDIRECT [[Page#Section]]</nowiki></code> 
| <s>In the new parser, a link to a section redirect, causes all the section edit links on the page to get the #Section anchor. To duplicate: create <nowiki>[[Redirect]]</nowiki> page with content <code><nowiki>#REDIRECT [[Page#Section]]</nowiki></code>, create <nowiki>[[Page]]</nowiki> with several headers, and go to /wiki/Redirect, and click an edit link (in new parser).</s> fixed in <s>[[rev:29825]]</s>
|-
| example: [http://en.wikipedia.org/wiki/User_talk:Morwen/13?timtest=newpp#Wigan_Article_Merge Morwen/13]
| <s>Broken wikilink <nowiki>[[Foo]</nowiki>prevents all following headers from being edit sections.
<s>[http://en.wikipedia.org/wiki/Corchorus]</s> [http://en.wikipedia.org/w/index.php?title=Corchorus&diff=184461150&oldid=181015712]</s>
|-
| <code><nowiki>==foo==<!--bar--></nowiki></code>
| Previously an expected difference, now made the same in [[rev:30109]]
|-
| Behavior of <code><nowiki>{{msgnw:}}</nowiki></code>
| <s>There seems to be some parsing going on with msgnw that doesn't jive with the old parser, for example: [http://en.wikipedia.org/wiki/Special:ParserDiffTest?pdt_input=%7B%7Bmsgnw:Template:Edit%7D%7D msgnw:Edit], or [[Special:ParserDiffTest/Template talk:Ttc]]</s> [[rev:30106]]
|-
| Conflict between equals for section start and name/value separator
| First section in a parserfunction is uneditable in new parser. This now only occurs for level-1 headings, higher levels have been fixed.
<pre>
{{#if:1|
==== Uneditable ====
==== Editable ====
}}
</pre>
|-
| <code><nowiki><!-- ~~~~ --></nowiki></code>
| <s>Misbehavior of sigs in comments</s> fixed in [[rev:30142]].
|-
| [[w:PlayStation 3]]
| <s>SegFaults on save</s> fixed in [[rev:30144]].
|-
| Editable section inside malformed braces
| <s>Unclosed braces closed later can lead to broken section editing links (edit links shown where none should be).</s> [[rev:30370]]
<pre>== Section title ==

{{

== edit link broken ==

}}
</pre>
|}

===Pending bugs===
These are bugs found in the new parser that have not yet been fixed in trunk.

{| class="wikitable" style="width:100%; margin:auto;"
|-
|[[w:Special:ParserDiffTest/Newsome|Interesting]] bad input on a template ([http://en.wikipedia.org/w/index.php?title=Newsome&oldid=152073532 oldid if fixed]). An apparent start of heading "\n==" breaks proper identification of template syntax on the same line, even if there is no valid end of heading.
|<pre>
|postcode_district = HD4
|postcode_area= HD   

==Sport|dial_code= 01484
|os_grid_reference=     
}}</pre>
|-
|$-style parameters in interface messages don't seem to work inside Parserfunctions unless escaped one level.  (See e.g. the currently commented-out revert link on [[:en:MediaWiki:Movepage-moved]].)
|<pre>{{urlencode:$1}}</pre>
<pre>{{((}}urlencode:$1{{))}}</pre>
|-
|$1 in [[w:MediaWiki:Readonlytext]] does not appear to resolve, during intermittent database lag.
| ''confirmation?''
|-
|LabeledSectionTransclusion: Localized tags do not work with new parser, see [[Bugzilla:13024]]. On [[:s:de:Seite:Carl Gottfried Neumann - Die elektrischen Kräfte 120.jpg]] and [[:s:de:Seite:Carl Gottfried Neumann - Die elektrischen Kräfte 126.jpg]] the section are marked with <tt><nowiki><Abschnitt Anfang=§17 /> and <Abschnitt Ende=§17 /></nowiki></tt>
|Compare: [[:s:de:Die elektrischen Kräfte/Zusammenstellung:§17]] and http://de.wikisource.org/wiki/Die_elektrischen_Kr%C3%A4fte/Zusammenstellung:%C2%A717?timtest=oldpp
|}

===Possible bugs===
If you have a verified bug, add it to the previous section. '''Please add new ''possible'' bugs to [[Talk:{{PAGENAME}}#Possible bugs|this section of the talk page.]]''' 

:'''<span style="color:red;">Do not add them here. Nothing belongs in this section except this message. ''Verified, fixed, and expected'' bugs belong in the sections above, possible bugs belong on the talk page.</span>'''

==Related (TO DO)==
:'''See [[Talk:Migration to the new preprocessor]] for new-parser-related cleanup tasks.'''

* <s>add [[rev:29774|#iferror]] to [[m:Help:ParserFunctions]]</s> done
* [[m:Help:ParserFunctions#Caveats]] may need clarification.
* [[mw:Markup_spec#Parser_outline]] should be rewritten?