Detecting closed and unclosed strings

[vieiro]

Hi all,

I want to detect both closed strings (such as 'hello') and unclosed strings (such as 'hello, note the missing '). These lines cannot span on multiple lines.

I am using a lexer such as this:

lexer grammar StringLexer;

WS : [ \t]+ -> skip ;
NL : ('\r'? '\n')+ ;

CLOSED_STRING: '\'' (~['\n])*? '\'' ;
UNCLOSED_STRING: '\'' (~['\n])*? '\n';

This lexer properly detects CLOSED_STRING tokens, but gets confused in UNCLOSED_STRING tokens.

Here's a parser:

parser grammar StringParser;
options { tokenVocab=StringLexer; }

program:
    (CLOSED_STRING | UNCLOSED_STRING | NL)*;

And a sample text input:

'123'
'abc
'xyz'

I tried this at http://lab.antlr.org/ and the parser seems to work properly, but the second token is not recognized:

Any ideas on how to make this working? Any way to improve the lexer rules?

[vieiro]

D'oh, it seems antlr4-parse does indeed detect the tokens, this may be a problem at lab.antlr4.org, then:

$ antlr4-parse StringParser.g4 StringLexer.g4 program -tokens -trace
'123'
'abc
'xyz'

[@0,0:4=''123'',<CLOSED_STRING>,1:0]
[@1,5:5='\n',<NL>,1:5]
[@2,6:10=''abc\n',<UNCLOSED_STRING>,2:0]
[@3,11:15=''xyz'',<CLOSED_STRING>,3:0]
[@4,16:17='\n\n',<NL>,3:5]
[@5,18:17='<EOF>',<EOF>,5:0]
enter   program, LT(1)='123'
consume [@0,0:4=''123'',<3>,1:0] rule program
consume [@1,5:5='\n',<2>,1:5] rule program
consume [@2,6:10=''abc\n',<4>,2:0] rule program
consume [@3,11:15=''xyz'',<3>,3:0] rule program
consume [@4,16:17='\n\n',<2>,3:5] rule program
exit    program, LT(1)=<EOF>