Skip to content

Make the editor's token cache safe for concurrent use - #20113

Open
xperiandri wants to merge 15 commits into
dotnet:mainfrom
xperiandri:copilot/fix-source-text-data-thread-safety
Open

xperiandri wants to merge 15 commits into
dotnet:mainfrom
xperiandri:copilot/fix-source-text-data-thread-safety

Conversation

@xperiandri

@xperiandri xperiandri commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #20112

SourceTextData, the editor's per-document token cache, is read and written by classification and symbol lookup on different threads, and growing its ResizeArray could race. Reads now index a plain array without a lock; writes and growth take a lock and publish a grown array only once it is filled, so a reader always sees a whole array.

  • Resuming a cache scan starts from the entry it has just validated instead of reading a neighbouring slot that was never checked, which could misclassify or drop tokens when scans of different text interleaved.
  • Adds FSharpLineTokenizer.ScanTokenValue, an allocation-free counterpart of ScanToken, used on the classification hot path; ScanToken is unchanged.

Validation stays per line, as on main: after an edit that keeps every later line's position and text, a scan can still resume from a stale lex state until the edited line is rescanned.

@xperiandri

Copy link
Copy Markdown
Contributor Author

Fixes #20112

@xperiandri xperiandri changed the title Make SourceTextData thread-safe with ConcurrentDictionary Make SourceTextData thread-safe with ConcurrentDictionary Aug 2, 2026
@github-actions github-actions Bot added the AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files label Aug 2, 2026
@T-Gro T-Gro added the AI-reviewed PR reviewed by AI review council label Aug 3, 2026
@T-Gro
T-Gro self-requested a review August 3, 2026 19:22
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from 8bee41a to bb0043a Compare August 17, 2026 01:28
@github-actions

github-actions Bot commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Release notes checked


✅ Found changes and release notes in following paths:

Change path Release notes path Description
`src/Compiler` docs/release-notes/.FSharp.Compiler.Service/11.0.200.md
`vsintegration/src` docs/release-notes/.VisualStudio/18.vNext.md

@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from bb0043a to 84f9fe8 Compare August 17, 2026 01:48
@github-actions

This comment has been minimized.

@github-actions github-actions Bot added ⚠️ Affects-Bootstrap Tooling check: PR touches compiler bootstrap chain ⚠️ Affects-Build-Infra Tooling check: PR touches build infrastructure labels Aug 17, 2026
@xperiandri

Copy link
Copy Markdown
Contributor Author

Pushed another round of improvements building on the thread-safety fix:

FSharpLineTokenizer.ScanToken allocation-free voption conversion (commit 7b724e470)

  • Changed the public signature from ScanToken: lexState -> FSharpTokenInfo option * FSharpTokenizerLexState to ScanToken: lexState -> struct (FSharpTokenInfo voption * FSharpTokenizerLexState) in ServiceLexing.fsi/ServiceLexing.fs.
  • This removes a per-token option heap allocation on the hot tokenization path (every single token scanned during classification, brace matching, and IntelliSense previously boxed a Some/None).
  • Updated every in-tree caller to match:
    • FSharpChecker.TokenizeLine (service.fs) — rewritten with a small scanNext () helper feeding the existing [| while ... do yield ... |] loop, avoiding the earlier fst/snd mutable-tuple indirection while staying close to the original shape.
    • editor Tokenizer.fs — pattern-matches on voption directly.
    • deprecated FSharp.LanguageService Colorize.fs/Intellisense.fs — ScanTokenWithDetails/ScanTokenAndProvideInfoAboutIt and the recursive token-scanning helpers now match on ValueSome/ValueNone.
  • Incidental: normalized trailing whitespace in Intellisense.fs that formatting picked up along the way (no behavior change).
  • Added/updated TokenizerTests.fs coverage for the new ScanToken shape.
  • Release notes entries added under ### Improved and ### Changed in docs/release-notes/.FSharp.Compiler.Service/11.0.100.md.

All affected projects (FSharp.Compiler.Service, FSharp.LanguageService) build cleanly with these changes.

xperiandri added a commit to xperiandri/fsharp that referenced this pull request Aug 17, 2026
@xperiandri
xperiandri requested a review from T-Gro August 17, 2026 02:10
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Aug 17, 2026
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from 4ff5e30 to c20c864 Compare August 17, 2026 02:11
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Aug 17, 2026
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from c20c864 to 287c161 Compare August 17, 2026 11:33

@T-Gro T-Gro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One note on public API compatibility.

Comment thread src/Compiler/Service/ServiceLexing.fsi Resolved
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Aug 20, 2026
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from 287c161 to 4b3e2af Compare August 20, 2026 14:39
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Aug 21, 2026
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from 4b3e2af to b5b7fcb Compare August 21, 2026 14:37
@xperiandri
xperiandri requested a review from T-Gro August 21, 2026 14:38
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Sep 3, 2026

@T-Gro T-Gro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diff is still way too big - apply the NoBloat instructions and CodeCompaction skill.

Especially the massive code comments would benefit not just with drastic reduction, but ideally with well named functions/arguments/bindings/types making the intention clear.

If a well named piece of code just tells what the comment says - delete the comment.

  • Excluding 2 public API documentation lines, this diff still adds 26 implementation and test comment lines and 2131 comment chars.
  • Comments bypass the type system and become stale maintenance state. Keep only durable public contracts; express implementation intent through names, functions, active patterns, discriminated unions, and types.

xperiandri added a commit to xperiandri/fsharp that referenced this pull request Oct 19, 2026
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Oct 19, 2026
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Oct 23, 2026
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Oct 23, 2026
@xperiandri
xperiandri requested a review from T-Gro October 29, 2026 19:49
@xperiandri

Copy link
Copy Markdown
Contributor Author

@T-Gro Trimmed in 8e9b2c2 (and 3720e88, a one-character fix in the test file), on top of a merge of main; no history rewrite.

  • Comments: 26 added implementation and test comment lines → 0. Only the two public API doc lines in ServiceLexing.fsi remain. What the comments said is now in names (writeLock, snapshot, and in the tests readLines, documentIdWithCacheOfOneLine, expectNoDivergence) and an assertion message.
  • Diff: 11 files, +353/−117 → 8 files, +244/−49. Removed what the race fix does not need: the IsValid(textLine, lineContents) overload, List.filter in tokensUnderCursor, the hoisted forbiddenSymbolNameChars, the re-indented processToken/scanAndColorNextToken, and the ScanTokenValue callers in FSharpChecker.TokenizeLine and the deprecated FSharp.LanguageService colorizer. The editor tokenizer keeps ScanTokenValue.
  • Tests: the three concurrency tests share helpers instead of repeating the loop; same scenarios. Against main's Tokenizer.fs the growth and two-versions tests fail in 6/6 runs (the overlapping-reads test only now and then, as before); all three pass here in 6/6 runs.
  • Release notes: one entry per file; 18.vNext.md had three copies.

xperiandri added a commit to xperiandri/fsharp that referenced this pull request Oct 29, 2026

@T-Gro T-Gro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 🕵️ Re-reviewed all recent changes as of 3b2d39d; LGTM

Comment thread src/Compiler/Service/ServiceLexing.fsi Resolved
xperiandri added a commit to xperiandri/fsharp that referenced this pull request Nov 5, 2026
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from 7c35740 to cb4826f Compare November 5, 2026 17:19
Copilot AI and others added 14 commits November 5, 2026 20:07
Co-authored-by: xperiandri <2365592+xperiandri@users.noreply.github.com>
- Reuse already-materialized line contents in SourceTextData cache
  lookups instead of re-stringifying the line via IsValid
- processToken now takes the token as a parameter instead of reading
  a mutable voption repeatedly; scanAndColorNextToken returns whether
  a token was scanned, removing the tokenInfoOption mutable
- tokensUnderCursor uses List.filter instead of a list comprehension
- Hoist forbiddenSymbolNameChars to a module-level private array so
  it is not recreated on every isValidNameForSymbol call
…ct return

Change ScanToken's signature from 'FSharpTokenInfo option * FSharpTokenizerLexState' to 'struct (FSharpTokenInfo voption * FSharpTokenizerLexState)', eliminating a per-token option allocation on the hot tokenization path. Update all in-tree callers: FSharpChecker.TokenizeLine, the editor Tokenizer, and the deprecated FSharp.LanguageService colorizer/scanner (Colorize.fs, Intellisense.fs). Also clean up incidental trailing whitespace in Intellisense.fs picked up by formatting.
… caught

Address review comment: add a concurrent test against one shared
SourceTextData, covering overlapping reads and invalidation.

The test alternates classifySpans/getSymbolAtPosition calls across many
threads on one document, interleaving two text versions of it (one with
an unclosed block comment) so cache invalidation is exercised alongside
concurrent reads.

It reliably reproduced a real bug in getFromRefreshedTokenCache: after
walking back to find the nearest valid cache entry at scanStartLine, the
resume lex state was read from scanStartLine - 1 - one entry lower than
the one just validated, and never checked itself. Under concurrent scans
of different text on the same document, that neighboring entry can be
stale, so classification would carry over a wrong lex state (typically
"still inside a comment"), turning code after that point into one giant
comment span, or occasionally throwing on a missing entry (silently
swallowed by Assert.Exception, dropping the classification for that
call). Fixed by reading LexStateAtStartOfLine off scanStartLine itself,
the entry the preceding loop already confirmed valid.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…free path

Address review comment: changing ScanToken to return
struct (voption * lexState) broke source and binary compatibility for
every existing caller of this public member.

ScanToken now returns FSharpTokenInfo option * FSharpTokenizerLexState
again, unchanged from before this PR. The allocation-free behavior is
kept as a new member, ScanTokenValue, returning
struct (FSharpTokenInfo voption * FSharpTokenizerLexState); ScanToken
is a thin wrapper over it. The hot-path callers that motivated the
original change (FSharpChecker.TokenizeLine, the editor Tokenizer, and
the deprecated FSharp.LanguageService colorizer) now call
ScanTokenValue directly, so they keep the allocation-free win.

TokenizerTests.fs and docs/fcs/tokenizer.fsx go back to matching on
Some/None against the restored ScanToken, exercising it (and
ScanTokenValue transitively) unchanged from before this PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
11.0.100 has shipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ConcurrentDictionary lookup costs every read of the cache, although
only extending the backing array races. Reads now index a plain array;
writes and growth take a lock, and a grown array is published only once
it holds every entry of the one it replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A concurrent scan of other text can clear the slot between the validity
check and the second read, and .Value then throws. The concurrency test
caught it once the reads stopped taking a lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each round warms a document's cache with a one-line text, so the cache
starts with a single slot, then races scans of the full document on
growing it, half of them reading from the first line through the copied
entries. Against main's unsynchronized ResizeArray the test fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review asked for a smaller diff and names instead of comments:

- Drop the allocation tweaks unrelated to the race (IsValid overload,
  List.filter in tokensUnderCursor, hoisted forbidden characters) and the
  cold-path ScanTokenValue callers (FSharpChecker.TokenizeLine, the
  deprecated FSharp.LanguageService colorizer). The editor tokenizer keeps
  ScanTokenValue and its original shape.
- Remove the implementation comments in Tokenizer.fs; writeLock, the
  volatile array and snapshot say what they did.
- Fold the three concurrency tests onto shared helpers named for what they
  check and replace their comments with names and an assertion message.
- Keep one release note entry per file.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The previous commit rewrote the literal as "C:\test.fs", which F# reads
with a tab in it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@xperiandri
xperiandri force-pushed the copilot/fix-source-text-data-thread-safety branch from cb4826f to 426f8c8 Compare November 5, 2026 19:07
Comment thread src/Compiler/Service/ServiceLexing.fsi Resolved
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@xperiandri
xperiandri requested a review from T-Gro November 5, 2026 21:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

⚠️ Affects-Bootstrap Tooling check: PR touches compiler bootstrap chain ⚠️ Affects-Build-Infra Tooling check: PR touches build infrastructure ⚠️ Affects-Design-Time Tooling check: PR touches type providers or dependency manager AI-reviewed PR reviewed by AI review council AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files ⚠️ Scope-Review-Needed Tooling check: PR scope exceeds title/description

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

VS F# editor: race in Tokenizer SourceTextData cache can corrupt per-line token data under concurrent access

4 participants