Performance regression between 1.5.2 and 1.5.3?

[gpetrou]

Run the ParlotBenchmarks in 1.5.2 and 1.5.3. For 1.5.3, revert to .NET 9.0 to get comparable results.
Is the difference in memory usage expected?

1.5.2:

Method	Mean	Error	StdDev	Gen0	Gen1	Allocated
CreateCompiledSmallParser	9,732.196 ns	18,334.2461 ns	1,004.9619 ns	0.7935	0.7324	10040 B
CreateCompiledExpressionParser	8,045.589 ns	1,110.8016 ns	60.8868 ns	0.3662	0.3052	5031 B
CursorCtor	3.306 ns	1.1112 ns	0.0609 ns	0.0038	-	48 B
ScannerCtor	8.766 ns	6.0965 ns	0.3342 ns	0.0064	-	80 B
ParseContextCtor	4.515 ns	1.7046 ns	0.0934 ns	0.0057	-	72 B
CursorMatchHello	21.616 ns	38.8473 ns	2.1294 ns	0.0121	-	152 B
CursorMatchGoodbye	21.640 ns	12.0084 ns	0.6582 ns	0.0121	-	152 B
CursorMatchNone	21.615 ns	0.7638 ns	0.0419 ns	0.0121	-	152 B
DecodeStringWithoutEscapes	2.314 ns	1.4556 ns	0.0798 ns	-	-	-
DecodeStringWithEscapes	71.621 ns	13.7036 ns	0.7511 ns	0.0088	-	112 B
ExpressionRawBig	1,107.523 ns	167.3927 ns	9.1754 ns	0.0954	-	1200 B
ExpressionCompiledBig	2,756.820 ns	1,471.9709 ns	80.6837 ns	0.2441	-	3104 B
ExpressionFluentBig	2,678.503 ns	184.2264 ns	10.0981 ns	0.2441	-	3104 B
ExpressionRawSmall	223.309 ns	28.9398 ns	1.5863 ns	0.0241	-	304 B
ExpressionCompiledSmall	502.223 ns	240.2079 ns	13.1666 ns	0.0563	-	712 B
ExpressionFluentSmall	545.004 ns	326.0303 ns	17.8708 ns	0.0563	-	712 B
BigJson	82,732.829 ns	12,053.5906 ns	660.6980 ns	7.4463	1.3428	93960 B
BigJsonCompiled	77,133.736 ns	15,387.3568 ns	843.4330 ns	7.4463	1.3428	93960 B
DeepJson	42,634.896 ns	13,605.7714 ns	745.7782 ns	7.9956	0.9766	100680 B
DeepJsonCompiled	40,209.605 ns	17,009.3289 ns	932.3387 ns	7.9956	0.9155	100680 B
LongJson	60,510.852 ns	19,134.0010 ns	1,048.7992 ns	9.6436	2.3193	121176 B
LongJsonCompiled	60,270.664 ns	31,256.9127 ns	1,713.2969 ns	9.6436	2.3804	121176 B
WideJson	39,029.478 ns	2,018.7316 ns	110.6535 ns	3.2959	0.3052	41528 B
WideJsonCompiled	34,107.861 ns	11,941.1174 ns	654.5329 ns	3.2959	0.3052	41528 B
Lookup	22.359 ns	5.1020 ns	0.2797 ns	0.0121	-	152 B
SkipWhiteSpace_0	17.211 ns	7.1977 ns	0.3945 ns	0.0121	-	152 B
SkipWhiteSpace_1	23.612 ns	16.4202 ns	0.9000 ns	0.0121	-	152 B
SkipWhiteSpace_10	37.222 ns	31.5314 ns	1.7283 ns	0.0121	-	152 B

1.5.3:

Method	Mean	Error	StdDev	Gen0	Gen1	Allocated
CreateCompiledSmallParser	15,214.805 ns	9,013.274 ns	494.0479 ns	0.7935	0.7324	10112 B
CreateCompiledExpressionParser	14,189.396 ns	2,743.311 ns	150.3701 ns	0.3662	0.3052	5031 B
CursorCtor	8.927 ns	3.239 ns	0.1775 ns	0.0038	-	48 B
ScannerCtor	18.119 ns	4.408 ns	0.2416 ns	0.0063	-	80 B
ParseContextCtor	26.400 ns	73.642 ns	4.0366 ns	0.0127	-	160 B
CursorMatchHello	62.078 ns	89.554 ns	4.9088 ns	0.0191	-	240 B
CursorMatchGoodbye	58.814 ns	75.853 ns	4.1578 ns	0.0191	-	240 B
CursorMatchNone	60.401 ns	46.619 ns	2.5553 ns	0.0191	-	240 B
DecodeStringWithoutEscapes	4.252 ns	5.998 ns	0.3288 ns	-	-	-
DecodeStringWithEscapes	111.267 ns	27.922 ns	1.5305 ns	0.0088	-	112 B
ExpressionRawBig	1,841.215 ns	1,291.945 ns	70.8158 ns	0.0954	-	1200 B
ExpressionCompiledBig	4,652.133 ns	1,934.792 ns	106.0525 ns	0.2594	-	3328 B
ExpressionFluentBig	5,301.556 ns	316.142 ns	17.3288 ns	0.2823	-	3576 B
ExpressionRawSmall	357.182 ns	170.294 ns	9.3344 ns	0.0238	-	304 B
ExpressionCompiledSmall	788.800 ns	251.577 ns	13.7898 ns	0.0629	-	800 B
ExpressionFluentSmall	1,067.554 ns	885.050 ns	48.5126 ns	0.0744	-	936 B
BigJson	152,754.281 ns	63,199.198 ns	3,464.1613 ns	7.3242	1.2207	94432 B
BigJsonCompiled	168,060.799 ns	121,956.966 ns	6,684.8728 ns	7.3242	1.2207	94432 B
DeepJson	83,565.283 ns	72,078.499 ns	3,950.8658 ns	9.7656	0.7324	123000 B
DeepJsonCompiled	91,653.959 ns	80,464.506 ns	4,410.5311 ns	9.7656	0.7324	123000 B
LongJson	121,018.880 ns	49,945.524 ns	2,737.6827 ns	9.5215	2.4414	121400 B
LongJsonCompiled	130,657.194 ns	83,835.858 ns	4,595.3263 ns	9.5215	2.4414	121400 B
WideJson	67,403.206 ns	25,326.938 ns	1,388.2549 ns	3.2959	0.3662	41752 B
WideJsonCompiled	67,312.520 ns	28,847.255 ns	1,581.2154 ns	3.2959	0.3662	41752 B
Lookup	53.929 ns	5.116 ns	0.2804 ns	0.0191	-	240 B
SkipWhiteSpace_0	45.348 ns	15.817 ns	0.8670 ns	0.0191	-	240 B
SkipWhiteSpace_1	52.963 ns	30.248 ns	1.6580 ns	0.0191	-	240 B
SkipWhiteSpace_10	74.230 ns	35.977 ns	1.9720 ns	0.0191	-	240 B

[sebastienros]

What was your benchmarking protocol exactly? Run from source, nuget packages, did you set the tfm explicitely to net9.0 for both runs? What scenario do you have in mind for the allocations differences?

[sebastienros]

This is what I think the regression (1.5.3):

• Fix stack overflow in Deferred parsers with infinite recursion by @Copilot in Fix stack overflow in Deferred parsers with infinite recursion #270

Which I fixed in 1.5.6:

• Optimize loop detection code by @sebastienros in Optimize loop detection code #289

You may want to create a Parser by setting disableLoopDetection = false when creating a ParseContext once your grammar is correctly tested. I know this has some perf implication but it's a balance between default behavior and perf ...

[gpetrou]

What was your benchmarking protocol exactly? Run from source, nuget packages, did you set the tfm explicitely to net9.0 for both runs? What scenario do you have in mind for the allocations differences?

I just cloned this repository, checked out each tag, built Release (1.5.3 with net9.0), ran the specific benchmarks from command line.
I observed some small performance changes in my own code, so I thought about trying the benchmarks of this repository first to see where the difference comes from.
Indeed, #270 seems to have caused some changes.
For example, SkipWhiteSpace_0 allocates 168 B in the commit before that.
#271 has caused the rest.
For example, SkipWhiteSpace_0 allocates 152 B in the commit before that.
I will check the latest release and disableLoopDetection.
I guess we can close this issue.

[sebastienros]

Not closing until we confirm the latest is fine. The skip whitespace for instance should not be impacted by that...

[gpetrou]

If I run the tests with 1.5.6 release, I get the same results.
If I change the value of disableLoopDetection, I see less memory usage, but still a bit more than 1.5.2, I guess due to #271.