Mode toggling on not-so-simple tokens

[raffian]

What is the best method for switching lexer modes when the start and end tokens are not simple characters like < > in XML?

Background

I'm using ANTLR to parse C code containing embedded PlSql - the language is Oracle Pro*C. My goal is to parse the outer grammar as regular C and use modes for capturing embedded Pl/Sql as arbitrary text for parsing in a separate process.

##Test
For the sake of simplicity I'm using ExprParser/ExprLexer as the starting point just to get modes working.

lexer grammar ExprLexer;

AND    : 'and' ;
OR     : 'or' ;
NOT    : 'not' ; 
PLUS   : '+';
EQ     : '=' ;
COMMA  : ',' ;
SEMI   : ';' ; 
LPAREN : '(' ;
RPAREN : ')' ;
LCURLY : '{' ;
RCURLY : '}' ;
INT    : [0-9]+ ;
ID     : [a-zA-Z_][a-zA-Z_0-9]* ;  
WS     : [ \t\n\r]+ -> skip ;

parser grammar ExprParser;
options { tokenVocab=ExprLexer; } 

program
    : stat EOF
    | def EOF
    ;
stat: ID '=' expr SEMI
    | expr SEMI
    ;
    
def : ID '(' ID (',' ID)* ')' '{' stat* '}' ;

expr: ID
    | INT
    | func
    | 'not' expr
    | expr 'and' expr
    | expr 'or' expr
    | expr '+' expr
    ;
func : ID '(' expr (',' expr)* ')' ;

Results so far

Pro*C has various but distinct start and end tags for "inside content" - similar to XML, though not always single character tokens:

CASE 1

Simple scenario: Embedded block starts with EXEC SQL, followed by legit arbitrary SQL, ends with ;

main(x) {
    counter = 0;

    EXEC SQL select * from TABLE;
    calculate(counter);
    index = 1;

    EXEC 
         SQL select count(*) from dual ;
}

I got CASE 1 working with the following adjustments:

//ExprLexer
START_SQL  : 'EXEC' WS* 'SQL'  -> pushMode(SQL);
//other tokens here

mode SQL;
CLOSE_SQL : SEMI    -> popMode;
SqlWS     : WSql    -> skip; 
SqlText   : ~[;]+  ;

fragment  WSql : [ \t\f\r\n]+;

//ExprParser
stat: execSql
    | ID '=' expr SEMI
    | expr SEMI
    ;

execSql : START_SQL  SqlText  CLOSE_SQL;

[@19,79:86='EXEC SQL',<START_SQL>,6:8]
[@20,87:106=' select * from TABLE',<SqlText>,6:16]
[@21,107:107=';',<CLOSE_SQL>,6:36]

Case 2

Embedded block starts with EXEC SQL EXECUTE, followed by anonymous PL/Sql (not just Sql), ends with END-EXEC;

main(x) {
    counter = 0;
    double balance = 0;
    
    EXEC SQL EXECUTE
      DECLARE 
        old_bal NUMBER(9,2);
      BEGIN
        IF :balance >  THEN  
           UPDATE accts SET bal = bal + :amt WHERE acctId = accnum;      
      END;
    END-EXEC;
    
    calculate(counter);
}

//ExprLexer
START_SQLEXEC : 'EXEC' WS* 'SQL' WS* 'EXECUTE' -> pushMode(SQLEXEC);
//more tokens here

mode SQLEXEC;
CLOSE_SQLEXEC   : 'END-EXEC;'  -> popMode;
SqlExecWS       :  WSql        -> skip; 
//SqlExecText   : ~['END-EXEC;']+  ;
//SqlExecText   : .  ;
//SqlExecText   : .  ->  more;
//SqlExecText   :  .*?  ;
SqlExecText     : ~(EOF)*  ;

fragment  WSql : [ \t\f\r\n]+;

Problems &Questions

Mode SQLEXEC triggers on EXEC SQL EXECUTE, but SqlExecText consumes the end tag END-EXEC; before popping back to default mode - this is not a surprise since unlike the previous scenario I can't exclude the end tag from the SqlExecText rule - this is where I'm stuck.

Questions

• Why doesn't lexer rule CLOSE_SQLEXEC trigger popMode() given its priority over other rules in mode SQLEXEC?

Potential Options

• Custom TokenSource for manipulating/combining tokens before they hit TokenStream , but I don't know enough about it yet to determine if it's worth pursuing.

[@11,43:58='EXEC SQL EXECUTE',<START_SQLEXEC>,4:8]
[@12,59:127='\n  BEGIN select * from TABLE; END;\n  END-EXEC;\n }\n\n',<SqlExecText>,4:24]
[@13,128:127='<EOF>',<EOF>,9:0]

[kaby76]

Case 2 doesn't work because:

1. The lexer will always try to match the rule with the longest text.
2. If two rules match the input string with the same length, the first one "wins".

SqlExecText matches the longest text.

[raffian]

Hi kaby,

There has to be a way to solve this - surely I'm not the first soul facing this predicament.

I tried using SqlExecText : (options{greedy=false} : .)* ; to suppress the rules' greediness, but it seems that option is not supported in ANTLR4.

[kaby76]

Perhaps you can do something like this.

lexer grammar ExprLexer;

tokens {
	EXEC_SQL_STMT,
	EXEC_SQL_EXECUTE_STMT
}

AND    : 'and' ;
OR     : 'or' ;
NOT    : 'not' ; 
PLUS   : '+';
EQ     : '=' ;
COMMA  : ',' ;
SEMI   : ';' ; 
LPAREN : '(' ;
RPAREN : ')' ;
LCURLY : '{' ;
RCURLY : '}' ;
INT    : [0-9]+ ;
ID     : [a-zA-Z_][a-zA-Z_0-9]* ;  
WS     : [ \t\n\r]+ -> channel(HIDDEN) ;
EXEC_SQL_START: 'EXEC' [ \t\n\r]+ 'SQL' -> pushMode(EXEC_SQL), more;
EXEC_SQL_EXECUTE_START: 'EXEC' [ \t\n\r]+ 'SQL' [ \t\n\r]+ 'EXECUTE' -> pushMode(EXEC_SQL_EXECUTE), more;

mode EXEC_SQL;
EXEC_SQL_CLOSE: SEMI -> popMode, type(EXEC_SQL_STMT);
EXEC_SQL_TEXT: . -> more;

mode EXEC_SQL_EXECUTE;
EXEC_SQL_EXECUTE_CLOSE: 'END-EXEC' [ \t\n\r]* SEMI -> popMode, type(EXEC_SQL_EXECUTE_STMT);
EXEC_SQL_EXECUTE_TEXT: . -> more;

parser grammar ExprParser;

options { tokenVocab=ExprLexer; } 

program:
    (stat | def) EOF
    ;

stat:
    execSql
    | ID? ID '=' expr SEMI
    | expr SEMI
    ;
    
execSql:
    EXEC_SQL_STMT
    | EXEC_SQL_EXECUTE_STMT
    ;

def:
    ID '(' ID (',' ID)* ')' '{' stat* '}' ;

expr:
    ID
    | INT
    | func
    | 'not' expr
    | expr 'and' expr
    | expr 'or' expr
    | expr '+' expr
    ;

func:
    ID '(' expr (',' expr)* ')'
    ;

For Antlr4, change the Antlr3 syntax (options{greedy=false} : .)* to .*?.

[raffian]

Thanks, kaby- this is working but I have some questions. I apologize if some of these seem trivial - parts of your solution run contradictory to my understanding of some ANTLR concepts.

Question 1

What factors motivate explicit definition of tokens in this manner?

tokens {
	EXEC_SQL_STMT,
	EXEC_SQL_EXECUTE_STMT
}

I looked to XMLLexer.g4 for better understanding but tokens{} isn't used there.
I found this in Lexer Rules, and my takeaway from that example is tokens{STRING} allows for typing different flavors of strings (DOUBLE or SINGLE quoted) as just token type STRING. I suppose this simplifies things in the lexer and parser. And as I'm typing this it just hit me 💡 as to why you have tokens{} in ExprLexer, lol, you're using more to suppress the lexer from emitting tokens while accumulating all parts of EXEC SQL block, and upon consuming close tag ; we then tell the lexer to emit a token as type EXEC_SQL_STMT with everything accumulated, so three intermediate tokens become one token for the parser, is this accurate?

Question 2

Why is it necessary to handle whitespace explicitly using [ \t\n\r]+ throughout the lexer? Fragments are global, so why not just define it once and reuse wherever needed in the lexer?

Question 3

Lexer rule EXEC_SQL_START appears in the lexer before EXEC_SQL_EXECUTE_START , but the latter is a longer token. Since the lexer tries to match as much text per token as possible, shouldn't EXEC_SQL_EXECUTE_START appear before EXEC_SQL_START for higher priority handling?

EXEC_SQL_START: 'EXEC' [ \t\n\r]+ 'SQL' -> pushMode(EXEC_SQL), more;
EXEC_SQL_EXECUTE_START: 'EXEC' [ \t\n\r]+ 'SQL' [ \t\n\r]+ 'EXECUTE' -> pushMode(EXEC_SQL_EXECUTE), more;

Thanks again,
Best
-R

[kaby76]

The tokens { ... } is used to declare a token type, which is just an integer. Usually, you don't need to declare one because the left-hand side symbol of a rule is the "token type". However, sometimes--as with lexer rules that use the more operator--the left-hand side symbol does not declare a token type because no token is returned with more. So, the rule EXEC_SQL_EXECUTE_TEXT: . -> more; does not return a token type, but appends the text. Then, when rule EXEC_SQL_EXECUTE_CLOSE: 'END-EXEC' [ \t\n\r]* SEMI -> popMode, type(EXEC_SQL_EXECUTE_STMT); matches, I just want to return EXEC_SQL_EXECUTE_STMT. That returns a token type EXEC_SQL_EXECUTE_STMT. We could have just written the rule EXEC_SQL_EXECUTE_STMT: 'END-EXEC' [ \t\n\r]* SEMI -> popMode;. The tokens {...} syntax becomes vital when the code is implemented in a base class using target-specific code. See the Python3 lexer grammar, and the base class code that returns a token.

You could write a rule fragment FOOBAR : [ \t\n\r]+; and use that in place of all applied occurrences of the expression in the grammar. I just prefer to see the expression rather than a symbol. The same thing applies for string literals. But, it makes no difference.

As for your third question, the order of rules almost always should not matter. Antlr always matches the longest string.

For input "EXEC SQL EXECUTE DECLARE old_bal NUMBER(9,2); BEGIN IF :balance > THEN UPDATE accts SET bal = bal + :amt WHERE acctId = accnum; END; END-EXEC;", EXEC_SQL_START and EXEC_SQL_EXECUTE_START both EXEC_SQL_START and EXEC_SQL_EXECUTE_START match. For EXEC_SQL_START, the matched input is "EXEC SQL". For EXEC_SQL_EXECUTE_START, the matched input is "EXEC SQL EXECUTE". The longest match always wins though.

[raffian]

This was immensely helpful - thank you, kaby!