timeless Reporter
	Description • 23 years ago

reading spam in npm.builds. loaded this message and set the decoder to utf8

#3  0x28317299 in nsDebug::Error (
    aMessage=0x28384420 "Not a UTF-8 string. This code should only be used for converting from known UTF-8 strings.",
    aFile=0x28384500 "/home/timeless/mozilla/string/obsolete/nsString2.cpp", aLine=1500) at /home/timeless/mozilla/xpcom/glue/nsDebug.cpp:458
#4  0x2834742b in CalculateUTF8Length::write (this=0xbfbfe5d0, start=0x8b63420 "Uªº|nªB@lounge.mozilla.org", N=28)
    at /home/timeless/mozilla/string/obsolete/nsString2.cpp:1500
#5  0x283472d5 in nsCharSinkTraits<CalculateUTF8Length>::write (iter=@0xbfbfe5d0, s=0x8b63420 "Uªº|nªB@lounge.mozilla.org", n=28)
    at ../../dist/include/string/nsCharTraits.h:558
#6  0x2834718a in CalculateUTF8Length & copy_string<nsReadingIterator<char>, CalculateUTF8Length> (first=@0xbfbfe5ec, last=@0xbfbfe5d8,
    result=@0xbfbfe5d0) at ../../dist/include/string/nsAlgorithm.h:90
#7  0x283168f9 in NS_ConvertUTF8toUCS2::Init (this=0xbfbfe6a4, aCString=@0xbfbfe634)
    at /home/timeless/mozilla/string/obsolete/nsString2.cpp:1521
#8  0x2acc9d6c in NS_ConvertUTF8toUCS2::NS_ConvertUTF8toUCS2 (this=0xbfbfe6a4, aCString=0x8b63420 "Uªº|nªB@lounge.mozilla.org")
    at ../../../../dist/include/string/nsString2.h:577
#9  0x2acc5472 in nsMimeHtmlDisplayEmitter::WriteHTMLHeaders (this=0x8ab6500)
    at /home/timeless/mozilla/mailnews/mime/emitters/src/nsMimeHtmlEmitter.cpp:225
#10 0x2acc594e in nsMimeHtmlDisplayEmitter::EndHeader (this=0x8ab6500)
    at /home/timeless/mozilla/mailnews/mime/emitters/src/nsMimeHtmlEmitter.cpp:319
#11 0x2a37e41b in mimeEmitterEndHeader (opt=0x8b0d900) at /home/timeless/mozilla/mailnews/mime/src/mimemoz2.cpp:1836
#12 0x2a36d637 in MimeMessage_write_headers_html (obj=0x8edeb80) at /home/timeless/mozilla/mailnews/mime/src/mimemsg.cpp:756
#13 0x2a36ca34 in MimeMessage_close_headers (obj=0x8edeb80) at /home/timeless/mozilla/mailnews/mime/src/mimemsg.cpp:405
#14 0x2a36c710 in MimeMessage_parse_line (
    line=0x8f5f400 "\nref: secnews.netscape.com netscape.public.mozilla.builds:21529\ni=A5H=A5s=A7=DA=BE=A4=BE=A4.?=\n(\001", length=1,
    obj=0x8edeb80) at /home/timeless/mozilla/mailnews/mime/src/mimemsg.cpp:271
#15 0x2a379864 in convert_and_send_buffer (
    buf=0x8f5f400 "\nref: secnews.netscape.com netscape.public.mozilla.builds:21529\ni=A5H=A5s=A7=DA=BE=A4=BE=A4.?=\n(\001", length=1,
    convert_newlines_p=1, per_line_fn=0x2a36c2a8 <MimeMessage_parse_line(char *, int, MimeObject *)>, closure=0x8edeb80)
    at /home/timeless/mozilla/mailnews/mime/src/mimebuf.cpp:168
#16 0x2a379ab0 in mime_LineBuffer (
    net_buffer=0x8d79236 "\nThis is a multi-part message in MIME format.\n\n------=_NextPart_eW3eX0tzX8arX0kJ2\nContent-Type: multipart/alterna
tive;\n\tboundary=\"----=_NextPart_eW3eX0tzX8arX0kJ2AA\"\n\n\n------=_NextPart_eW3eX0tzX8arX0k"..., net_buffer_size=7626, bufferP=0x8edeba8,
    buffer_sizeP=0x8edebb0, buffer_fpP=0x8edebb8, convert_newlines_p=1,
    per_line_fn=0x2a36c2a8 <MimeMessage_parse_line(char *, int, MimeObject *)>, closure=0x8edeb80)
    at /home/timeless/mozilla/mailnews/mime/src/mimebuf.cpp:255
#17 0x2a370ff1 in MimeObject_parse_buffer (
    buffer=0x8d79000 "Path: secnews.netscape.com!gateway\nFrom: join_ideamore@hotmail.com\nNewsgroups: netscape.public.mozilla.builds\nSubject:
 =?big5?Q?=A7=DA=A5s=B1i=BE=A4=BE=A4,=B3=DF=C5w=A7=DA=A5i=A5H=A5s=A7=DA=BE=A4=BE=A"..., size=8192, obj=0x8edeb80)
    at /home/timeless/mozilla/mailnews/mime/src/mimeobj.cpp:255
#18 0x2a37be0f in mime_display_stream_write (stream=0x8b63480,
    buf=0x8d79000 "Path: secnews.netscape.com!gateway\nFrom: join_ideamore@hotmail.com\nNewsgroups: netscape.public.mozilla.builds\nSubject: =?
big5?Q?=A7=DA=A5s=B1i=BE=A4=BE=A4,=B3=DF=C5w=A7=DA=A5i=A5H=A5s=A7=DA=BE=A4=BE=A"..., size=8192)
    at /home/timeless/mozilla/mailnews/mime/src/mimemoz2.cpp:860
#19 0x2a388fdd in nsStreamConverter::OnDataAvailable (this=0x8bc5e80, request=0x8d91210, ctxt=0x0, aIStream=0x89c0ad0, sourceOffset=0,
    aLength=8192) at /home/timeless/mozilla/mailnews/mime/src/nsStreamConverter.cpp:902
#20 0x28d6945d in nsDocumentOpenInfo::OnDataAvailable (this=0x87aa120, request=0x8d91210, aCtxt=0x0, inStr=0x89c0ad0, sourceOffset=0,
    count=8192) at /home/timeless/mozilla/uriloader/base/nsURILoader.cpp:241

	ji
	Comment 1 • 23 years ago

I don't see on a Red Hat system. Since I don't have a FreeBSD system to test on,
I'd like to confirm the bug based on reporter's desription.

	Denis Antrushin
	Comment 2 • 23 years ago

This message has no charset info at all, so there is no way to avoid that
assertion - we're ending up with fallback assuption that this is utf8 message, which is wrong.
If you set charset to big5, message displays just fine.
It also displays fine (meaning no assertion) is you turn on charset
autodetection, so I vote for INVALID.
(timeless, no second look this time :-) )

	nhottanscp
	Comment 3 • 23 years ago

Denis, do you know the place where the invalid UTF-8 is generated?

	Denis Antrushin
	Comment 4 • 23 years ago

It's just not generated :-)
Mozilla asserts while preparing msg headers for frontend
(nsMimeHtmlDisplayEmitter::WriteHTMLHeaders) :

headerValues[numHeadersAdded] = 
nsCRT::strdup(NS_ConvertUTF8toUCS2(headerValue).get());
we assume here that headers are in utf8 and this is correct, because we convert
them to utf8 in MimeHeaders_write_all_headers() (src/mimehdrs.cpp, line 781):

MimeHeaders_convert_header_value(opt, &hdr_value);

This function just calls MIME_DecodeMimeHeader.
Please note, that we have assertion *only* when override charset is
incorrect.
But as soon as you set *correct* charset (Big5), message displays w/o
assertions.

	timeless Reporter
	Comment 5 • 23 years ago

sorry, i'm filing these bugs based on a perl concept of tainting, the assertion 
say "This code should only be used for converting from known UTF-8 strings." 
the stuff you're converting is *NOT* truly known to be utf8 so IMO you should 
use something to check to see if it's utf8 before you send it to that function 
(this would be detainting in perl).

	Denis Antrushin
	Comment 6 • 23 years ago

should != must ;-)
I think, there's some cases when assertion is acceptable.

	Martijn Wargers (dead)
	Comment 7 • 17 years ago

I'm seeing this assertion when copying pasting something from one application to Firefox in the url bar. But I guess I should file a new bug for that?

	Andrew Sutherland [:asuth] (he/him)
	Comment 8 • 15 years ago

I'm seeing this problem with spam messages in the thread pane.  It's a nuisance when running a debug build with XPCOM_DEBUG_BREAK=break; because the tree-view repaints as a response to most input, you either need to kill the app at that point or find a way to disable the break directive.

We should be scrubbing the input or something.  My stack looks like:
nsCharSinkTraits<CalculateUTF8Length>::write
copy_string<nsReadingIterator<char>, CalculateUTF8Length>
AppendUTF8toUTF16
CopyUTF8toUTF16
nsMimeConverter::DecodeMimeHeader
nsMsgDBView::FetchAuthor
nsMsgDBView::GetCellText

	David :Bienvenu
	Comment 10 • 15 years ago

I've had to run with this patch because I've run into the exact problem Andrew mentioned, which is that debug break is pretty much fatal with these junk messages.

	Andrew Sutherland [:asuth] (he/him)
	Comment 11 • 15 years ago

Maybe we should just land the quieting patch?  If it's redundant (aka when !DEBUG, the UTF8 checking isn't adding anything), then we could make it conditional on DEBUG.

I feel like debug spew aligns with the "broken windows" theory of cities.  I get so much debug spew on our DEBUG builds that I just tune it out, and only do anything when it passes the volume threshold where it sends the debug I care about off the screen faster than I can read it or scroll back to it.