[protobuf2] Extensions not parsed fully in Protobuf2

[dsyer]

The current grammar for Protobuf2 has an "extensions" keyword, but it only supports a range qualifier, whereas .proto files allow verification and declaration arguments as well. Nearly all the examples from the Protobuf docs fail to parse.

[kaby76]

When I go to the link you provide (https://protobuf.dev/programming-guides/proto2/#extensions), the very first extensions example (https://protobuf.dev/programming-guides/proto2/#ext-example) is this:

// file kittens/video_ext.proto

import "kittens/video.proto";
import "media/user_content.proto";

package kittens;

// This extension allows kitten videos in a media.UserContent message.
extend media.UserContent {
  // Video is a message imported from kittens/video.proto
  repeated Video kitten_videos = 126;
}

Let's assume that the comment is not part of the example, but it should not matter. Does this parse with the protobuf2 grammar? No.

09/26-04:51:27 ~/issues/g4-current/protobuf/protobuf2/Generated-CSharp
$ ./bin/Debug/net8.0/Test.exe ../examples/video_ext.proto
line 1:0 mismatched input 'import' expecting 'syntax'
CSharp 0 ../examples/video_ext.proto fail 0.036639
Total Time: 0.0907517
09/26-04:51:41 ~/issues/g4-current/protobuf/protobuf2/Generated-CSharp
$

This fails because the syntax specifies a manadatory syntax nonterminal. It is not because extensions are not parsed.

grammars-v4/protobuf/protobuf2/Protobuf2.g4

Lines 16 to 18 in 6b51773

	proto
	: syntax (importStatement | packageStatement | optionStatement | topLevelDef | emptyStatement_)* EOF
	;

As a matter of fact, the Protobuf version 2 Specification says very, very clearly that syntax is optional, not mandatory: https://protobuf.dev/reference/protobuf/proto3-spec/#proto_file.

[dsyer]

Yeah, but that's not really relevant for this issue is it? I can raise another ticket for the syntax tag if you lilke (but I think the Google docs are wrong - how can you tell the difference between proto2 and proto3 otherwise?).

[kaby76]

Yeah, but that's not really relevant for this issue is it? I can raise another ticket for the syntax tag if you lilke (but I think the Google docs are wrong - how can you tell the difference between proto2 and proto3 otherwise?).

It's not relevant to whatever issue you are raising about extensions.

But, you said: Nearly all the examples [from the Protobuf docs](https://protobuf.dev/programming-guides/proto2/#extensions) fail to parse. Which example fails to parse? Give me the code.

The first extensions example does not have a syntax declaration, which is exactly why it fails to parse. When I add syntax="proto2";, it parses.

The second example, user_content.proto, corrected to work with the current invalid mandatory syntax nonterminal, fails.

syntax="proto2";

// adjusted file media/user_content.proto

package media;

// A container message to hold stuff that a user has created.
message UserContent {
  // Set verification to `DECLARATION` to enforce extension declarations for all
  // extensions in this range.
  extensions 100 to 199 [verification = DECLARATION];
}

Output from parse:

09/26-06:07:59 ~/issues/g4-current/protobuf/protobuf2/Generated-CSharp
$ ./bin/Debug/net8.0/Test.exe ../examples/user_content.proto
line 11:24 mismatched input '[' expecting ';'
CSharp 0 ../examples/user_content.proto fail 0.0291935
Total Time: 0.0838021
09/26-06:12:38 ~/issues/g4-current/protobuf/protobuf2/Generated-CSharp
$

Sorry if I sound irritating. But, we need to have concrete examples that fail, not go fishing around.

Please include this example, the adjusted user_content.proto file, included in your PR #4631.

how can you tell the difference between proto2 and proto3 otherwise?).

You can't. On face value, you can't say whether a package was written for proto2 or proto3. If there is no syntax="proto2"; or syntax="proto3"; statement, there is nothing in a file to indicate whether the file is Protobuf2 or Protobuf3--or both, or Protobuf 2023, which we don't have. You have to use an oracle, such as the protobuf tool version 2 or version 3, or a parser for protobuf2 or protobuf3. I had to do this when I added many examples to the protobuf3 grammar, weeding out very carefully those examples that were not correct.

[dsyer]

I already included an example with the PR. Isn't that part of this conversation a needless duplicate of the comments there?

I take your point about the syntax, but that's not on me is it, and it's nothing to do with this issue? Do you want me to open a new issue or not?

[kaby76]

I already included an example with the PR. Isn't that part of this conversation a needless duplicate of the comments there?

If it's there, great. I will review it.

I take your point about the syntax, but that's not on me is it, and it's nothing to do with this issue? Do you want me to open a new issue or not?

Thanks, but I will open a GitHub Issue for the problem.