Prevents matching partial words in lexer

[MihaMarkic]

Let's say I have these two rules in lexer:

PRINT: 'print';
STRING: [a-zA-Z0-9_]+ ;

With input
printxxx
lexer will create both PRINT and STRING tokens. Which is not what I want, I want a single STRING in that case instead - I don't want PRINT rule to match.

Now, I've tried something like the code below

fragment END: ~[0-9a-zA-Z]+;
PRINT: 'print' END;

which works except that it requires a character at the end.
So any idea? TIA

[kaby76]

With input printxxx lexer will create both PRINT and STRING tokens.

No, it doesn't. See below.

02/13-14:41:15 ~/issues/a4-4772
$ !trgen
trgen -t CSharp
CSharp  G.g4 success 0.0339229
Rendering template file from CSharp/MyParserInterpreter.cs to ./Generated-CSharp/MyParserInterpreter.cs
Rendering template file from CSharp/Other.csproj to ./Generated-CSharp/Other.csproj
Rendering template file from CSharp/st.build.ps1 to ./Generated-CSharp/st.build.ps1
Rendering template file from CSharp/st.build.sh to ./Generated-CSharp/st.build.sh
Rendering template file from CSharp/st.clean.ps1 to ./Generated-CSharp/st.clean.ps1
Rendering template file from CSharp/st.clean.sh to ./Generated-CSharp/st.clean.sh
Rendering template file from CSharp/st.Encodings.cs to ./Generated-CSharp/st.Encodings.cs
Rendering template file from CSharp/st.ErrorListener.cs to ./Generated-CSharp/st.ErrorListener.cs
Rendering template file from CSharp/st.makefile to ./Generated-CSharp/st.makefile
Rendering template file from CSharp/st.MyParser.cs to ./Generated-CSharp/st.MyParser.cs
Rendering template file from CSharp/st.perf.sh to ./Generated-CSharp/st.perf.sh
Rendering template file from CSharp/st.ProfilingCommonTokenStream.cs to ./Generated-CSharp/st.ProfilingCommonTokenStream.cs
Rendering template file from CSharp/st.run.ps1 to ./Generated-CSharp/st.run.ps1
Rendering template file from CSharp/st.run.sh to ./Generated-CSharp/st.run.sh
Rendering template file from CSharp/st.test-ambiguity.sh to ./Generated-CSharp/st.test-ambiguity.sh
Rendering template file from CSharp/st.test-cover.sh to ./Generated-CSharp/st.test-cover.sh
Rendering template file from CSharp/st.Test.cs to ./Generated-CSharp/st.Test.cs
Rendering template file from CSharp/st.test.ps1 to ./Generated-CSharp/st.test.ps1
Rendering template file from CSharp/st.test.sh to ./Generated-CSharp/st.test.sh
Rendering template file from CSharp/Test.csproj.st to ./Generated-CSharp/Test.csproj.st
Rendering template file from C:/msys64/home/Kenne/issues/a4-4772/G.g4 to ./Generated-CSharp/G.g4
02/13-14:41:26 ~/issues/a4-4772
$ cd Generated-CSharp/
02/13-14:41:29 ~/issues/a4-4772/Generated-CSharp
$ make
bash build.sh
warning(131): G.g4:2:8: greedy block ()* contains wildcard; the non-greedy syntax ()*? may be preferred
warning(131): G.g4:2:8: greedy block ()* contains wildcard; the non-greedy syntax ()*? may be preferred
Restore complete (0.6s)

Build succeeded in 1.0s
Restore complete (0.4s)
  Test succeeded (1.4s) → bin\Debug\net8.0\Test.dll

Build succeeded in 2.6s

Workload updates are available. Run `dotnet workload list` for more information.
02/13-14:41:37 ~/issues/a4-4772/Generated-CSharp
$ ./bin/Debug/net8.0/Test.exe -tokens -input 'printxxx'
[@0,0:7='printxxx',<2>,1:0]
[@1,8:7='<EOF>',<-1>,1:8]

CSharp 0 string0 success 0.0045668
Total Time: 0.0556779
02/13-14:42:27 ~/issues/a4-4772/Generated-CSharp
$ cat G.g4
grammar G;
start: .* EOF;
PRINT: 'print';
STRING: [a-zA-Z0-9_]+ ;
02/13-14:48:13 ~/issues/a4-4772/Generated-CSharp
$ grep 2 G.tokens
STRING=2

The lexer produces a single token for the input. This is because Antlr lexers follow two basic rules:

1. Select the rule that matches the longest string first.
2. If two or more lexer rules match the same string, return the first rule that matches.

In your grammar. PRINT occurs first. But STRING matches printxxx because the string length is 8. PRINT cannot match because the matching string length is 5.

It sounds like you have a build issue. Make sure to remove generated files, and rebuild completely from scratch. Or, there is something more to your grammar.

[MihaMarkic]

Hi, uh, thanks for looking and explanation. It makes sense now. I made a copy&paste error, it should be '.print', not 'print' and then it should match as I said when the input is '.printxxx'.

But thanks to you now I know what to do - introduce something like string prefixed with a dot.
Will try it tomorrow, but it should work

[MihaMarkic]

If anybody is curious, here is my solution, thanks to @kaby76

WS : [ \t]+ -> channel(HIDDEN) ; // skip spaces, tabs, newlines
EOL: '\r\n' | '\r' | '\n';

PRINT: '.print';
STRING: [a-zA-Z0-9_]+ ;
DOT_STRING: '.' [a-zA-Z0-9_]+ ;

Note: fragment could be used for both string parts.