Change 31699: Re: [PATCH] Adding more information to "Unrecognized character" error in toke.c

[Rafael Garcia-Suarez]

Change 31699 by ***@stcosmo on 2007/08/10 09:57:08

Subject: Re: [PATCH] Adding more information to "Unrecognized character" error in toke.c
From: Claes Jakobsson <***@surfar.nu>
Date: Sat, 4 Aug 2007 16:00:19 +0200
Message-Id: <57CEC660-0020-48DF-A72A-***@surfar.nu>

Affected files ...

... //depot/perl/pod/perldiag.pod#478 edit
... //depot/perl/t/base/lex.t#20 edit
... //depot/perl/toke.c#789 edit

Differences ...

==== //depot/perl/pod/perldiag.pod#478 (text) ====
Index: perl/pod/perldiag.pod
--- perl/pod/perldiag.pod#477~31592~ 2007-07-11 22:51:35.000000000 -0700
+++ perl/pod/perldiag.pod 2007-08-10 02:57:08.000000000 -0700
@@ -4410,11 +4410,11 @@
somehow, or insert an underbar into it. You might also declare it as a
subroutine.

-=item Unrecognized character %s
+=item Unrecognized character %s in column %d

(F) The Perl parser has no idea what to do with the specified character
-in your Perl script (or eval). Perhaps you tried to run a compressed
-script, a binary program, or a directory as a Perl program.
+in your Perl script (or eval) at the specified column. Perhaps you tried
+to run a compressed script, a binary program, or a directory as a Perl program.

=item Unrecognized escape \\%c in character class passed through in regex; marked by <-- HERE in m/%s/


==== //depot/perl/t/base/lex.t#20 (xtext) ====
Index: perl/t/base/lex.t
--- perl/t/base/lex.t#19~31601~ 2007-07-12 23:12:50.000000000 -0700
+++ perl/t/base/lex.t 2007-08-10 02:57:08.000000000 -0700
@@ -1,6 +1,6 @@
#!./perl

-print "1..55\n";
+print "1..56\n";

$x = 'x';

@@ -263,3 +263,7 @@

sub foo::::::bar { print "ok $test\n"; $test++ }
foo::::::bar;
+
+eval "\$x =\xE2foo";
+if ($@ =~ /Unrecognized character \\xE2 in column 5/) { print "ok $test\n"; } else { print "not ok $test\n"; }
+$test++;

==== //depot/perl/toke.c#789 (text) ====
Index: perl/toke.c
--- perl/toke.c#788~31587~ 2007-07-11 05:02:11.000000000 -0700
+++ perl/toke.c 2007-08-10 02:57:08.000000000 -0700
@@ -3561,7 +3561,8 @@
default:
if (isIDFIRST_lazy_if(s,UTF))
goto keylookup;
- Perl_croak(aTHX_ "Unrecognized character \\x%02X", *s & 255);
+ len = UTF ? Perl_utf8_length((U8 *) PL_linestart, (U8 *) s) : (STRLEN) (s - PL_linestart);
+ Perl_croak(aTHX_ "Unrecognized character \\x%02X in column %d", *s & 255, (int) len + 1);
case 4:
case 26:
goto fake_eof; /* emulate EOF on ^D or ^Z */
End of Patch.

[Jerry D. Hedden]

This patch is broken. I'm getting:

`sh cflags "optimize='-O3 -pipe -funit-at-a-time -mtune=pentium4m
-march=pentium4 -mfpmath=sse -mieee-fp -mmmx -msse -msse2'" toke.o`
toke.c
CCCMD = gcc -DPERL_CORE -c -DPERL_PATCHNUM=31699
-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -DNO_MATHOMS
-fno-strict-aliasing -pipe -std=c89 -O3 -pipe -funit-at-a-time
-mtune=pentium4m -march=pentium4 -mfpmath=sse -mieee-fp -mmmx -msse
-msse2 -Wall -ansi -W -Wextra -Wdeclaration-after-statement
-Wendif-labels
toke.c: In function `Perl_yylex':
toke.c:3564: warning: passing arg 1 of `Perl_utf8_length' from
incompatible pointer type
toke.c:3564: error: too few arguments to function `Perl_utf8_length'
make: *** [toke.o] Error 1
Make failed

Post by Rafael Garcia-Suarez
Subject: Re: [PATCH] Adding more information to "Unrecognized character" error in toke.c
Date: Sat, 4 Aug 2007 16:00:19 +0200
Affected files ...
... //depot/perl/pod/perldiag.pod#478 edit
... //depot/perl/t/base/lex.t#20 edit
... //depot/perl/toke.c#789 edit
Differences ...
==== //depot/perl/pod/perldiag.pod#478 (text) ====
Index: perl/pod/perldiag.pod
--- perl/pod/perldiag.pod#477~31592~ 2007-07-11 22:51:35.000000000 -0700
+++ perl/pod/perldiag.pod 2007-08-10 02:57:08.000000000 -0700
@@ -4410,11 +4410,11 @@
somehow, or insert an underbar into it. You might also declare it as a
subroutine.
-=item Unrecognized character %s
+=item Unrecognized character %s in column %d
(F) The Perl parser has no idea what to do with the specified character
-in your Perl script (or eval). Perhaps you tried to run a compressed
-script, a binary program, or a directory as a Perl program.
+in your Perl script (or eval) at the specified column. Perhaps you tried
+to run a compressed script, a binary program, or a directory as a Perl program.
=item Unrecognized escape \\%c in character class passed through in regex; marked by <-- HERE in m/%s/
==== //depot/perl/t/base/lex.t#20 (xtext) ====
Index: perl/t/base/lex.t
--- perl/t/base/lex.t#19~31601~ 2007-07-12 23:12:50.000000000 -0700
+++ perl/t/base/lex.t 2007-08-10 02:57:08.000000000 -0700
@@ -1,6 +1,6 @@
#!./perl
-print "1..55\n";
+print "1..56\n";
$x = 'x';
@@ -263,3 +263,7 @@
sub foo::::::bar { print "ok $test\n"; $test++ }
foo::::::bar;
+
+eval "\$x =\xE2foo";
+$test++;
==== //depot/perl/toke.c#789 (text) ====
Index: perl/toke.c
--- perl/toke.c#788~31587~ 2007-07-11 05:02:11.000000000 -0700
+++ perl/toke.c 2007-08-10 02:57:08.000000000 -0700
@@ -3561,7 +3561,8 @@
if (isIDFIRST_lazy_if(s,UTF))
goto keylookup;
- Perl_croak(aTHX_ "Unrecognized character \\x%02X", *s & 255);
+ len = UTF ? Perl_utf8_length((U8 *) PL_linestart, (U8 *) s) : (STRLEN) (s - PL_linestart);
+ Perl_croak(aTHX_ "Unrecognized character \\x%02X in column %d", *s & 255, (int) len + 1);
goto fake_eof; /* emulate EOF on ^D or ^Z */
End of Patch.

[Jerry D. Hedden]

Attached patch fixes error in change 31699.