Remove unnecessary consistency checks for VarNamedVector #1092

mhauru · 2025-10-27T10:10:29Z

The VarNamedVector inner constructor has some checks to make sure that the data being fed in is consistent. This seems like a sensible idea. However, especially one of the checks is a bit expensive, and they were being called all the time. This PR turns them off when they are unnecessary, most notably when calling unflatten.

github-actions · 2025-10-27T10:12:39Z

Benchmark Report for Commit `427581b`

Computer Information

Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            6.5 │             1.7 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          741.0 │            44.8 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          355.8 │            67.2 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │          784.6 │            36.6 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6786.2 │            25.6 │
│           Smorgasbord │   201 │ forwarddiff │      typed_vector │   true │          813.2 │            40.3 │
│           Smorgasbord │   201 │ forwarddiff │    untyped_vector │   true │          734.8 │            40.3 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │          905.4 │            45.6 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          718.9 │             5.9 │
│           Smorgasbord │   201 │      enzyme │             typed │   true │          894.0 │             3.9 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         3893.3 │             6.6 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1029.4 │             8.7 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        43060.8 │             6.1 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         8577.8 │            10.0 │
│               Dynamic │    10 │    mooncake │             typed │   true │          118.4 │            12.1 │
│              Submodel │     1 │    mooncake │             typed │   true │            9.0 │             6.3 │
│                   LDA │    12 │ reversediff │             typed │   true │          987.4 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

codecov · 2025-10-27T10:21:55Z

Codecov Report

❌ Patch coverage is 82.05128% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.17%. Comparing base (90b591b) to head (427581b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/varnamedvector.jl	77.50%	18 Missing ⚠️
src/varinfo.jl	93.33%	2 Missing ⚠️
src/debug_utils.jl	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1092      +/-   ##
==========================================
- Coverage   81.43%   81.17%   -0.26%     
==========================================
  Files          40       40              
  Lines        3749     3793      +44     
==========================================
+ Hits         3053     3079      +26     
- Misses        696      714      +18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mhauru · 2025-10-27T10:22:35Z

Some benchmarks of the Smorgasbord model of VNV vs Metadata.

EDIT: The untyped_vector benchmarks here are wrong. See below for fixed benchmarks.

On main:

┌─────────────┬─────┬─────────────┬────────────────┬────────┬────────────────┬─────────────────┐
│       Model │ Dim │  AD Backend │        VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├─────────────┼─────┼─────────────┼────────────────┼────────┼────────────────┼─────────────────┤
│ Smorgasbord │ 201 │ forwarddiff │        untyped │  false │         3946.7 │             3.5 │
│ Smorgasbord │ 201 │ forwarddiff │ untyped_vector │  false │          581.5 │            32.3 │
│ Smorgasbord │ 201 │ forwarddiff │          typed │  false │          352.4 │            46.1 │
│ Smorgasbord │ 201 │ forwarddiff │   typed_vector │  false │          580.7 │            32.6 │
│ Smorgasbord │ 201 │ reversediff │        untyped │   true │          270.9 │            78.3 │
│ Smorgasbord │ 201 │ reversediff │ untyped_vector │   true │          511.8 │            41.7 │
│ Smorgasbord │ 201 │ reversediff │          typed │   true │          297.1 │            71.6 │
│ Smorgasbord │ 201 │ reversediff │   typed_vector │   true │          513.6 │            41.6 │
└─────────────┴─────┴─────────────┴────────────────┴────────┴────────────────┴─────────────────┘

On this PR:

┌─────────────┬─────┬─────────────┬────────────────┬────────┬────────────────┬─────────────────┐
│       Model │ Dim │  AD Backend │        VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├─────────────┼─────┼─────────────┼────────────────┼────────┼────────────────┼─────────────────┤
│ Smorgasbord │ 201 │ forwarddiff │        untyped │  false │         3806.3 │             3.6 │
│ Smorgasbord │ 201 │ forwarddiff │ untyped_vector │  false │          378.8 │            42.7 │
│ Smorgasbord │ 201 │ forwarddiff │          typed │  false │          346.5 │            43.3 │
│ Smorgasbord │ 201 │ forwarddiff │   typed_vector │  false │          381.4 │            42.3 │
│ Smorgasbord │ 201 │ reversediff │        untyped │   true │          263.0 │            79.2 │
│ Smorgasbord │ 201 │ reversediff │ untyped_vector │   true │          310.8 │            66.4 │
│ Smorgasbord │ 201 │ reversediff │          typed │   true │          290.8 │            71.1 │
│ Smorgasbord │ 201 │ reversediff │   typed_vector │   true │          311.7 │            66.3 │
└─────────────┴─────┴─────────────┴────────────────┴────────┴────────────────┴─────────────────┘

mhauru · 2025-10-27T10:52:16Z

Test failures are the same as on main.

benchmarks/src/DynamicPPLBenchmarks.jl

src/varnamedvector.jl

Co-authored-by: Penelope Yong <[email protected]>

github-actions · 2025-10-27T12:35:23Z

DynamicPPL.jl documentation for PR #1092 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR1092/

mhauru · 2025-10-27T12:59:05Z

Fixed benchmarks:

On main:

┌─────────────┬─────┬─────────────┬────────────────┬────────┬────────────────┬─────────────────┐
│       Model │ Dim │  AD Backend │        VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├─────────────┼─────┼─────────────┼────────────────┼────────┼────────────────┼─────────────────┤
│ Smorgasbord │ 201 │ forwarddiff │        untyped │  false │        10970.4 │             3.6 │
│ Smorgasbord │ 201 │ forwarddiff │ untyped_vector │  false │       240018.2 │            16.3 │
│ Smorgasbord │ 201 │ forwarddiff │          typed │  false │          919.3 │            47.1 │
│ Smorgasbord │ 201 │ forwarddiff │   typed_vector │  false │         1502.9 │            32.9 │
│ Smorgasbord │ 201 │ reversediff │        untyped │   true │          743.1 │            75.3 │
│ Smorgasbord │ 201 │ reversediff │ untyped_vector │   true │       234413.8 │             1.2 │
│ Smorgasbord │ 201 │ reversediff │          typed │   true │          800.3 │            70.0 │
│ Smorgasbord │ 201 │ reversediff │   typed_vector │   true │         1374.2 │            41.4 │
└─────────────┴─────┴─────────────┴────────────────┴────────┴────────────────┴─────────────────┘

On this PR:

┌─────────────┬─────┬─────────────┬────────────────┬────────┬────────────────┬─────────────────┐
│       Model │ Dim │  AD Backend │        VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├─────────────┼─────┼─────────────┼────────────────┼────────┼────────────────┼─────────────────┤
│ Smorgasbord │ 201 │ forwarddiff │        untyped │  false │        11705.7 │             3.8 │
│ Smorgasbord │ 201 │ forwarddiff │ untyped_vector │  false │        32367.4 │            12.0 │
│ Smorgasbord │ 201 │ forwarddiff │          typed │  false │         1068.9 │            44.9 │
│ Smorgasbord │ 201 │ forwarddiff │   typed_vector │  false │         1105.7 │            41.2 │
│ Smorgasbord │ 201 │ reversediff │        untyped │   true │          819.4 │            75.4 │
│ Smorgasbord │ 201 │ reversediff │ untyped_vector │   true │        26486.9 │             2.9 │
│ Smorgasbord │ 201 │ reversediff │          typed │   true │          874.6 │            70.2 │
│ Smorgasbord │ 201 │ reversediff │   typed_vector │   true │          945.5 │            65.4 │
└─────────────┴─────┴─────────────┴────────────────┴────────┴────────────────┴─────────────────┘

You may wonder why the whole scale of these numbers as jumped up, compared to the earlier benchmarks. I wonder that too. Maybe my laptop started throttling or something. :/

penelopeysm

I'm happy! Could you do the usual version stuff? Feel free to merge after that.

penelopeysm · 2025-10-27T14:32:51Z

(Feel free to claim v0.38.3, I'll unrevert my other PR later.)

mhauru · 2025-10-27T15:24:43Z

I'm working on more, very similar performance improvements to VNV. I would fold them into the same release.

…-speed

* Change VNV to use Dict rather than OrderedDict * Change concretisation from map(identity, x) to a comprehension * Improve tighten_types!! and loosen_types!! * Fix use of set_transformed!! * Fix push!! for VarInfos * Change the default element types in VNV to be Union{} * In untyped_vector_varinfo, don't rely on Metadata * Code style * Run formatter * In VNV, use typejoin rather than promote_type * Bump patch version to 0.38.4

Remove unnecessary consistency checks for VarNamedVector

1e1be7b

github-actions bot assigned mhauru Oct 27, 2025

mhauru marked this pull request as ready for review October 27, 2025 10:51

mhauru requested a review from penelopeysm October 27, 2025 10:52

penelopeysm requested changes Oct 27, 2025

View reviewed changes

benchmarks/src/DynamicPPLBenchmarks.jl Outdated Show resolved Hide resolved

src/varnamedvector.jl Outdated Show resolved Hide resolved

Fix benchmark setting

585f4eb

Co-authored-by: Penelope Yong <[email protected]>

mhauru added 2 commits October 27, 2025 12:35

Fix typo

e0b46e0

Merge branch 'main' into mhauru/varnamedvector-speed

84f4f9e

Add two benchmarks

9d4d2e9

mhauru requested a review from penelopeysm October 27, 2025 13:01

penelopeysm approved these changes Oct 27, 2025

View reviewed changes

mhauru and others added 3 commits October 29, 2025 14:30

Merge remote-tracking branch 'origin/main' into mhauru/varnamedvector…

1fa3109

…-speed

Merge branch 'main' into mhauru/varnamedvector-speed

427581b

mhauru added this pull request to the merge queue Oct 29, 2025

Merged via the queue into main with commit 80cf12d Oct 29, 2025
18 of 19 checks passed

mhauru deleted the mhauru/varnamedvector-speed branch October 29, 2025 17:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove unnecessary consistency checks for VarNamedVector #1092

Remove unnecessary consistency checks for VarNamedVector #1092

Uh oh!

mhauru commented Oct 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 27, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 27, 2025 •

edited

Loading

Uh oh!

mhauru commented Oct 27, 2025 •

edited

Loading

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 27, 2025

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

penelopeysm left a comment

Uh oh!

penelopeysm commented Oct 27, 2025 •

edited

Loading

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Remove unnecessary consistency checks for VarNamedVector #1092

Remove unnecessary consistency checks for VarNamedVector #1092

Uh oh!

Conversation

mhauru commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Report for Commit 427581b

Computer Information

Benchmark Results

Uh oh!

codecov bot commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mhauru commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 27, 2025

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

penelopeysm left a comment

Choose a reason for hiding this comment

Uh oh!

penelopeysm commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhauru commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhauru commented Oct 27, 2025 •

edited

Loading

github-actions bot commented Oct 27, 2025 •

edited

Loading

Benchmark Report for Commit `427581b`

codecov bot commented Oct 27, 2025 •

edited

Loading

mhauru commented Oct 27, 2025 •

edited

Loading

penelopeysm commented Oct 27, 2025 •

edited

Loading