Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/release-notes/.FSharp.Compiler.Service/11.0.200.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
* Preserve SRTP extension scopes across source files and competing witnesses through imported inline calls. Compile generic extension forwarding, including quotations, in unoptimized builds. Preserve applicable built-in operator priority in imported inline quotations. Do not reuse SRTP specializations across different selected witnesses. ([Issue #20683](https://github.com/dotnet/fsharp/issues/20683), [Issue #20684](https://github.com/dotnet/fsharp/issues/20684), [Issue #20685](https://github.com/dotnet/fsharp/issues/20685), [PR #20717](https://github.com/dotnet/fsharp/pull/20717))
* Transparent compiler: with `enablePartialTypeChecking`, a project that is only referenced by another project is type checked from its signatures. Implementation files that have a signature file are no longer parsed or type checked to produce the referenced assembly. `ParseAndCheckProject` still checks all files. ([PR #20703](https://github.com/dotnet/fsharp/pull/20703))
* With `enablePartialTypeChecking`, a project that is only referenced by another project no longer type checks its implementation files that have a signature file. Producing the referenced assembly only needs the signatures, but finalizing the project also collected the diagnostics of the skipped files, which type checked all of them and kept their results in memory. `ParseAndCheckProject` still reports the diagnostics of these files. ([PR #20699](https://github.com/dotnet/fsharp/pull/20699))
* Bound the copies of `when` guards in compiled pattern matches. A match whose decision tree would repeat its guards more than 32 times is compiled in groups, each falling through to the rest of the match once. This halves the IL of `LexFilter.hwTokenFetch` (110 KB to 48 KB), so the JIT optimizes it again, and parsing is about 27% faster where FCS is JIT-compiled. ([Issue #20632](https://github.com/dotnet/fsharp/issues/20632), [PR #20718](https://github.com/dotnet/fsharp/pull/20718))

### Changed

Expand Down
148 changes: 111 additions & 37 deletions src/Compiler/Checking/PatternMatchCompilation.fs
Original file line number Diff line number Diff line change
Expand Up @@ -1080,11 +1080,51 @@ let mkThrowUsingEDICapture (infoReader: InfoReader) tcVal m resultTy exnExpr =
| _ ->
mkThrow m resultTy exnExpr

exception private GuardCopyBudgetExceeded

/// Whether the input is read through an immutable byref. A struct union input read this way is tested
/// in place, not through a copy.
let private isCopyableInputExpr origInputExpr =
match origInputExpr with
| Expr.Op (TOp.LValueOp (LByrefGet, v), [], [], _) when not v.IsMutable -> true
| _ -> false

/// Whether the pattern tests an array element or a mutable record field, which a 'when' guard can write.
let rec private testsMutableData inpPat =
match inpPat with
| TPat_array _ -> true
| TPat_recd (tcref, _, subPats, _) ->
(tcref.TrueInstanceFieldsAsList, subPats)
||> List.exists2 (fun field subPat ->
match subPat with
| TPat_wild _ -> false
| _ -> field.IsMutable || testsMutableData subPat)
| TPat_query (_, subPat, _)
| TPat_as (subPat, _, _) -> testsMutableData subPat
Comment on lines +1102 to +1103

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cut can still change which rule a match picks when a total active pattern reads data that a guard writes, because this arm only checks the patterns on the active pattern's result. Here is the array case from Guards that write the tested data keep the rule the whole match would pick, with the array read through an active pattern:

let (|Elems|) (a: int[]) = a.[0], a.[1]

let array (arr: int[]) col =
    match arr with
    | Elems (0, 0) when (arr.[0] <- 7; arr.[1] <- 7; false) -> 0
    | Elems (_, 0) when col <= 0 -> 10
    // ... the same rule for 1 to 6
    | Elems (_, 7) when col <= 7 -> 17
    | Elems (1, _) -> 1
    // ... the same rule for 2 to 6
    | Elems (7, _) -> 7
    | _ -> -1

With this PR's build, array [| 0; 0 |] 100 returns 7 and calls Elems twice, with --optimize- and with --optimize+. The merge-base build returns -1 and calls Elems once, as SDK 10.0.402 does. In a variant where the eight col <= c guards call a non-inlined le c col instead, the merge-base build's IL has 72 calls to le. One call per guard would give 8, so that is 64 extra copies, well over this PR's budget of 32. The patterns matched against the result of Elems contain no array, mutable field or byref, so canRetestInput allows the cut, and the rest of the match calls Elems again, on [| 7; 7 |].

The obvious fix, TPat_query _ -> true, would also turn the cut off for hwTokenFetch, because its match uses the partial TokenRExprParen (LexFilter.fs line 1702). An Elems variant in guardWritesTestedDataSource would cover whichever fix you choose.

| TPat_disjs (subPats, _)
| TPat_conjs (subPats, _)
| TPat_tuple (_, subPats, _, _)
| TPat_exnconstr (_, subPats, _)
| TPat_unioncase (_, _, subPats, _) -> List.exists testsMutableData subPats
| TPat_isinst (_, _, subPatOpt, _) -> Option.exists testsMutableData subPatOpt
| TPat_const _
| TPat_wild _
| TPat_null _
| TPat_error _ -> false

[<Struct; NoEquality; NoComparison>]
type private CompiledMatch =
{ Tree: DecisionTree
Targets: DecisionTreeTarget list
Joins: Bindings
GuardCopies: int }

let private CompilePatternBasic
(g: TcGlobals) denv amap tcVal infoReader mExpr mMatch
warnOnUnused
warnOnIncomplete
(joinPromotion: JoinPromotion)
(guardCopyBudget: int voption)
actionOnFailure
(origInputVal, origInputValTypars, _origInputExprOpt: Expr option)
(clauses: MatchClause list)
Expand Down Expand Up @@ -1270,6 +1310,9 @@ let private CompilePatternBasic
let idx = matchBuilder.AddTarget(TTarget(targetParams, mkApps g ((joinE, joinThunkTy), [], args, mMatch), None))
TDSuccess(caps |> List.map (exprForVal mMatch), idx)

let guardEmitted = HashSet<ClauseNumber>()
let mutable guardCopies = 0

let rec InvestigateFrontiers refuted frontiers =
Cancellable.CheckAndThrow()
stackGuard.Guard(fun () -> InvestigateFrontiersImpl refuted frontiers)
Expand Down Expand Up @@ -1324,6 +1367,11 @@ let private CompilePatternBasic
match GetWhenGuardOfClause i refuted with
| Some whenExpr ->
let m = whenExpr.Range
if not (guardEmitted.Add i) then
guardCopies <- guardCopies + 1
match guardCopyBudget with
| ValueSome budget when guardCopies > budget -> raise GuardCopyBudgetExceeded
| _ -> ()
let whenExprWithBindings = mkLetsFromBindings m (mkInvisibleBinds vs2 es2) whenExpr
let failureTree = investigateMemoized (RefutedWhenClause :: refuted) rest
mkBoolSwitch m whenExprWithBindings successTree failureTree
Expand Down Expand Up @@ -1388,11 +1436,6 @@ let private CompilePatternBasic
else
None)

and IsCopyableInputExpr origInputExpr =
match origInputExpr with
| Expr.Op (TOp.LValueOp (LByrefGet, v), [], [], _) when not v.IsMutable -> true
| _ -> false

and ChoosePreBinder simulSetOfEdgeDiscrims subexpr =
match simulSetOfEdgeDiscrims with
// Very simple 'isinst' tests: put the result of 'isinst' in a local variable
Expand Down Expand Up @@ -1424,7 +1467,7 @@ let private CompilePatternBasic
let argExpr = GetSubExprOfInput subexpr
let argExpr =
match argExpr, _origInputExprOpt with
| Expr.Val (v1, _, _), Some origInputExpr when valEq origInputVal v1.Deref && IsCopyableInputExpr origInputExpr -> origInputExpr
| Expr.Val (v1, _, _), Some origInputExpr when valEq origInputVal v1.Deref && isCopyableInputExpr origInputExpr -> origInputExpr
| _ -> argExpr
let vOpt, addrExp, _readonly, _writeonly = mkExprAddrOfExprAux g true false NeverMutates argExpr None mMatch
match vOpt with
Expand Down Expand Up @@ -1823,7 +1866,7 @@ let private CompilePatternBasic
if warnOnUnused then
ReportUnusedTargets clauses dtree

dtree, matchBuilder.CloseTargets(), List.ofSeq joinBindings
{ Tree = dtree; Targets = matchBuilder.CloseTargets(); Joins = List.ofSeq joinBindings; GuardCopies = guardCopies }

// Three pattern constructs can cause significant code expansion in various combinations
// - Partial active patterns
Expand Down Expand Up @@ -1887,41 +1930,72 @@ let isProblematicClause (clause: MatchClause) =
let ips = investigationPoints clause.Pattern
ips.Length > 0 && Span.exists id (ips.AsSpan (0, ips.Length - 1))

let rec CompilePattern g denv amap tcVal infoReader mExpr mMatch warnOnUnused actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) (clausesL: MatchClause list) inputTy resultTy =
match clausesL with
| _ when List.exists isProblematicClause clausesL ->

// First make sure we generate at least some of the obvious incomplete match warnings.
let warnOnUnused = false // we can't turn this on since we're pretending all partials fail in order to control the complexity of this.
let warnOnIncomplete = true
let clausesPretendAllPartialFail = clausesL |> List.collect (fun (MatchClause(p, whenOpt, tg, m)) -> [MatchClause(erasePartialPatterns p, whenOpt, tg, m)])
let _ = CompilePatternBasic g denv amap tcVal infoReader mExpr mMatch warnOnUnused warnOnIncomplete JoinPromotion.Disabled actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) clausesPretendAllPartialFail inputTy resultTy
let warnOnIncomplete = false
/// Extra copies of 'when' guards a decision tree may hold. A match over this budget is cut into
/// groups, each compiled with the rest of the match as a single fall-through target.
let private guardCopyBudget = 32

// Partial and when clauses cause major code explosion if treated naively
// Hence treat any pattern matches with any partial clauses clause-by-clause
let rec atMostOneProblematicClauseAtATime clauses =
match List.takeUntil isProblematicClause clauses with
| l, [] ->
CompilePatternBasic g denv amap tcVal infoReader mExpr mMatch warnOnUnused warnOnIncomplete JoinPromotion.Enabled actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) l inputTy resultTy
| l, h :: t ->
// Add the problematic clause.
doGroupWithAtMostOneProblematic (l @ [h]) t
let CompilePattern g denv amap tcVal infoReader mExpr mMatch warnOnUnused actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) (clausesL: MatchClause list) inputTy resultTy =
let compile warnOnUnused warnOnIncomplete joinPromotion budget clauses =
CompilePatternBasic g denv amap tcVal infoReader mExpr mMatch warnOnUnused warnOnIncomplete joinPromotion budget actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) clauses inputTy resultTy

and doGroupWithAtMostOneProblematic group rest =
// Compile the remaining clauses.
let decisionTree, targets, joins = atMostOneProblematicClauseAtATime rest
let restPlaceholder = MatchClause(TPat_wild mMatch, None, TTarget([], mkDefault (mMatch, resultTy), None), mMatch)

// Make the expression that represents the remaining cases of the pattern match.
let expr = mkLetsBind mMatch joins (mkAndSimplifyMatch DebugPointAtBinding.NoneAtInvisible mExpr mMatch resultTy decisionTree targets)
let fitsBudget clauses =
try
compile false false JoinPromotion.Enabled (ValueSome guardCopyBudget) (clauses @ [ restPlaceholder ]) |> ignore
true
with GuardCopyBudgetExceeded ->
false

// Make the clause that represents the remaining cases of the pattern match
let clauseForRestOfMatch = MatchClause(TPat_wild mMatch, None, TTarget(List.empty, expr, None), mMatch)
// The longest prefix of the group that fits the budget, at least one clause.
let cutToBudget (group: MatchClause list) rest =
let rec search lo hi =
if lo >= hi then
lo
else
let mid = (lo + hi + 1) / 2
if fitsBudget (List.take mid group) then search mid hi else search lo (mid - 1)

CompilePatternBasic g denv amap tcVal infoReader mExpr mMatch warnOnUnused warnOnIncomplete JoinPromotion.Enabled actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) (group @ [clauseForRestOfMatch]) inputTy resultTy
if fitsBudget group then
group, rest
else
let k = search 1 (group.Length - 1)
List.take k group, List.skip k group @ rest

// Partial patterns and 'when' clauses cause major code explosion if treated naively, as every
// path that falls through a group would get its own copy of the clauses after it. So each group
// ends at a problematic clause or at the budget, and the rest of the match is compiled once.
// The groups report no warnings: the whole-match compile already did.
let rec compileGroups cutLargeGroups clauses =
let group, rest =
match List.takeUntil isProblematicClause clauses with
| l, [] -> l, []
| l, h :: t -> l @ [ h ], t

let group, rest = if cutLargeGroups then cutToBudget group rest else group, rest

atMostOneProblematicClauseAtATime clausesL
match rest with
| [] -> compile false false JoinPromotion.Enabled ValueNone group
| _ ->
let rest = compileGroups cutLargeGroups rest
let expr = mkLetsBind mMatch rest.Joins (mkAndSimplifyMatch DebugPointAtBinding.NoneAtInvisible mExpr mMatch resultTy rest.Tree rest.Targets)
let clauseForRestOfMatch = MatchClause(TPat_wild mMatch, None, TTarget(List.empty, expr, None), mMatch)
compile false false JoinPromotion.Enabled ValueNone (group @ [ clauseForRestOfMatch ])

// The rest of a cut match tests the input again, and sees what a failed guard wrote to it, where
// the whole tree would not. So only a match whose tested data no guard can write is cut by the budget.
let canRetestInput () =
not (Option.exists isCopyableInputExpr origInputExprOpt)
&& not (clausesL |> List.exists (fun clause -> testsMutableData clause.Pattern))

let result =
if List.exists isProblematicClause clausesL then
// Partial patterns are treated as failing here, so unused clauses can't be reported.
let clausesPretendAllPartialFail = clausesL |> List.map (fun (MatchClause(p, whenOpt, tg, m)) -> MatchClause(erasePartialPatterns p, whenOpt, tg, m))
let whole = compile false true JoinPromotion.Disabled ValueNone clausesPretendAllPartialFail
compileGroups (whole.GuardCopies > guardCopyBudget && canRetestInput ()) clausesL
else
let whole = compile warnOnUnused true JoinPromotion.Disabled ValueNone clausesL
if whole.GuardCopies > guardCopyBudget && canRetestInput () then compileGroups true clausesL else whole

| _ ->
CompilePatternBasic g denv amap tcVal infoReader mExpr mMatch warnOnUnused true JoinPromotion.Disabled actionOnFailure (origInputVal, origInputValTypars, origInputExprOpt) clausesL inputTy resultTy
result.Tree, result.Targets, result.Joins
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,108 @@ let f (buffer: byref<int>) x =
| _ -> 0
"""
|> compiles

// https://github.com/dotnet/fsharp/issues/20632
// Guarded rules on the second column between rules on the first column, as in LexFilter's hwTokenFetch.
// Every branch of a switch on the first column would get its own copy of the guarded rules.
let private interleavedGuardsSource =
let ctxs, toks = 20, 12

let rules =
[ yield "T0", "C0 p :: _", "col < p"
for k in 0 .. ctxs - 1 -> "_", $"C%d{k} p :: _", $"col <= p + %d{k}"
for k in 1 .. toks - 1 -> $"T%d{k}", "_ :: _", (if k % 3 = 0 then $"col %% %d{k} = 0" else "true") ]

let clauses =
rules
|> List.mapi (fun i (tok, stack, guard) -> $" | %s{tok}, %s{stack} when %s{guard} -> %d{i}")
|> String.concat "\n"

let reference =
rules
|> List.mapi (fun i (tok, stack, guard) ->
let tokTest = if tok = "_" then "true" else $"tok = %s{tok}"
$" elif (%s{tokTest}) && (match stack with %s{stack} -> %s{guard} | _ -> false) then %d{i}")
|> String.concat "\n"

let cases prefix n field =
[ for k in 0 .. n - 1 -> $" | %s{prefix}%d{k}%s{field}" ] |> String.concat "\n"

$"""module Test
open Microsoft.FSharp.Reflection
type Ctx =
{cases "C" ctxs " of int"}
type Tok =
{cases "T" toks ""}
let f (tok: Tok) (stack: Ctx list) (col: int) =
match tok, stack with
{clauses}
| _ -> -1
let reference (tok: Tok) (stack: Ctx list) (col: int) =
if false then -2
{reference}
else -1
[<EntryPoint>]
let main _ =
let tokens = FSharpType.GetUnionCases typeof<Tok> |> Array.map (fun c -> FSharpValue.MakeUnion(c, [||]) :?> Tok)
let ctx k (p: int) = FSharpValue.MakeUnion((FSharpType.GetUnionCases typeof<Ctx>).[k], [| box p |]) :?> Ctx
let stacks = [ yield []; for k in 0 .. {ctxs - 1} do for p in 0 .. 3 -> [ ctx k p ] ]
let mismatches = Seq.length (seq {{ for tok in tokens do for stack in stacks do for col in 0 .. 25 do if f tok stack col <> reference tok stack col then yield () }})
let ilSize = typeof<Ctx>.DeclaringType.GetMethod("f").GetMethodBody().GetILAsByteArray().Length
printfn "mismatches=%%d small=%%b" mismatches (ilSize < 4000)
0
"""

[<Fact>]
let ``Issue 20632 - guarded rules between first-column rules are emitted once`` () =
interleavedGuardsSource
|> FSharp
|> withOptimize
|> compileExeAndRun
|> shouldSucceed
|> withStdOutContains "mismatches=0 small=true"

// https://github.com/dotnet/fsharp/pull/20718#discussion_r4219278089
// A failed guard that writes the data the match tests must not change which later rule matches.
let private guardWritesTestedDataSource =
let matchOn header write (pat: string -> string -> string) =
[ yield header
yield $""" | %s{pat "0" "0"} when (%s{write}; false) -> 0"""
for c in 0..7 -> $""" | %s{pat "_" (string c)} when col <= %d{c} -> 1%d{c}"""
for t in 1..7 -> $""" | %s{pat (string t) "_"} -> %d{t}"""
yield " | _ -> -1" ]
|> String.concat "\n"

let pair =
matchOn "let pair (value: byref<Pair>) col =\n match value with" "value <- Pair (7, 7)" (fun t c -> $"Pair (%s{t}, %s{c})")

let array =
matchOn "let array (arr: int[]) col =\n match arr with" "arr.[0] <- 7; arr.[1] <- 7" (fun t c -> $"[| %s{t}; %s{c} |]")

let record =
matchOn "let record (r: R) col =\n match r with" "r.T <- 7; r.C <- 7" (fun t c -> $"{{ T = %s{t}; C = %s{c} }}")

$"""module Test
[<Struct>]
type Pair = Pair of token: int * context: int
type R = {{ mutable T: int; mutable C: int }}
{pair}
{array}
{record}
[<EntryPoint>]
let main _ =
let mutable input = Pair (0, 0)
printfn "pair=%%d array=%%d record=%%d" (pair &input 100) (array [| 0; 0 |] 100) (record {{ T = 0; C = 0 }} 100)
0
"""

[<Theory>]
[<InlineData(false)>]
[<InlineData(true)>]
let ``Guards that write the tested data keep the rule the whole match would pick`` optimize =
guardWritesTestedDataSource
|> FSharp
|> (if optimize then withOptimize else withNoOptimize)
|> compileExeAndRun
|> shouldSucceed
|> withStdOutContains "pair=-1 array=-1 record=-1"
Loading