Optimize `permutations` by implementing it via `multiset_permutations` #186

FedericoStra · 2025-05-06T13:01:25Z

Code changes

As noted in #151, multiset_permutations is much faster than permutations, so we can exploit it to optimize the latter (this was suggested in a comment here).

The code does the equivalent of

permutations(a, t::Integer=length(a)) =
    Iterators.map(
        indices -> [a[i] for i in indices],
        multiset_permutations(eachindex(a), t))

but we construct the iterator manually so that we can define eltype or it (otherwise, with Iterators.map, type inference would deduce Any).

This closes #151; it possibly closes #185 too.

Simple benchmark

Benchmark code

using BenchmarkTools
using Combinatorics

count_permutations(a) = count(Returns(true), permutations(a))
count_permutations(a, t) = count(Returns(true), permutations(a, t))

# compile
count_permutations(1:3)
for t in 0:3
    count_permutations(1:3, t)
end

for n in [3, 5, 7, 8, 9]
    println("\nn = $(n)\n")
    a = collect(1:n)
    display(@benchmark count_permutations($a))
end

println("\nn = 10, t = 6\n")
display(@benchmark count_permutations(1:10, 6))

Before

n = 3

BenchmarkTools.Trial: 10000 samples with 20 evaluations per sample.
 Range (min … max):  977.800 ns … 108.978 μs  ┊ GC (min … max): 0.00% … 97.93%
 Time  (median):       1.003 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):     1.160 μs ±   1.842 μs  ┊ GC (mean ± σ):  2.62% ±  1.68%

  █▇▃▅▃                                   ▃▁                    ▁
  █████▆▃▆▆▆▆▅▇▆▆▆▅▃▅▅▃▃▅▄▅▄▄▃▃▅▅▆▅▆▆▇▆▇▆▆██▇▇▇▇▇▆▇▇▆▆▇▄▅▆▅▅▄▅▆ █
  978 ns        Histogram: log(frequency) by time       2.47 μs <

 Memory estimate: 624 bytes, allocs estimate: 16.

n = 5

BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample.
 Range (min … max):  109.018 μs …  2.817 ms  ┊ GC (min … max): 0.00% … 94.72%
 Time  (median):     118.300 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   121.273 μs ± 36.502 μs  ┊ GC (mean ± σ):  0.40% ±  1.33%

       █  ▂
  ▁▁▇▅▂█▅▃█▅▃▇▆▃▃▄▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  109 μs          Histogram: frequency by time          173 μs <

 Memory estimate: 11.41 KiB, allocs estimate: 244.

n = 7

BenchmarkTools.Trial: 155 samples with 1 evaluation per sample.
 Range (min … max):  27.639 ms … 41.011 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     31.259 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   32.352 ms ±  3.509 ms  ┊ GC (mean ± σ):  0.06% ± 0.47%

     ▃▆▆▁ ██▆▆▁▃ ▁ ▁ ▁    ▃ ▁  ▁            ▁       ▁
  ▄▄▇████▇██████▇█▇█▇█▆▄▁▇█▄█▆▆█▇▆▁▆▁▁▇▇▄▄▄▁█▄▄▄▄▆▁▁█▆▄▁▁▄▁▇▄ ▄
  27.6 ms         Histogram: frequency by time        40.7 ms <

 Memory estimate: 551.42 KiB, allocs estimate: 10084.

n = 8

BenchmarkTools.Trial: 9 samples with 1 evaluation per sample.
 Range (min … max):  591.173 ms … 673.411 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     604.497 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   609.657 ms ±  24.487 ms  ┊ GC (mean ± σ):  0.03% ± 0.08%

  █   ██ █ ████                                               █
  █▁▁▁██▁█▁████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  591 ms           Histogram: frequency by time          673 ms <

 Memory estimate: 4.92 MiB, allocs estimate: 80644.

n = 9

BenchmarkTools.Trial: 1 sample with 1 evaluation per sample.
 Single result which took 14.473 s (0.01% GC) to evaluate,
 with a memory estimate of 44.30 MiB, over 725764 allocations.

n = 10, t = 6

BenchmarkTools.Trial: 121 samples with 1 evaluation per sample.
 Range (min … max):  39.323 ms … 45.142 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     41.299 ms              ┊ GC (median):    3.36%
 Time  (mean ± σ):   41.409 ms ±  1.035 ms  ┊ GC (mean ± σ):  2.86% ± 1.18%

                    █       ▄
  ▄▃▃▁▁▁▁▆▆▆▆▅▄▇▄▇▄▆█▅▆▇▆▄▄▆█▄▆▅▄▄▇▃▃▇▁▃▁▃▃▁▁▁▁▁▁▃▃▁▁▁▁▁▁▁▁▁▃ ▃
  39.3 ms         Histogram: frequency by time          45 ms <

 Memory estimate: 16.15 MiB, allocs estimate: 302403.

After

n = 3

BenchmarkTools.Trial: 10000 samples with 10 evaluations per sample.
 Range (min … max):  1.071 μs … 217.489 μs  ┊ GC (min … max): 0.00% … 97.94%
 Time  (median):     1.126 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   1.271 μs ±   4.475 μs  ┊ GC (mean ± σ):  9.03% ±  2.59%

    █▆
  ▂▇███▄▃▂▂▂▂▂▂▂▂▂▁▁▁▂▂▂▂▂▂▂▂▂▂▂▁▂▁▂▂▂▂▂▁▂▂▂▂▂▂▂▁▁▁▂▂▂▂▂▂▂▁▂▂ ▂
  1.07 μs         Histogram: frequency by time        2.13 μs <

 Memory estimate: 2.56 KiB, allocs estimate: 63.

n = 5

BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample.
 Range (min … max):  11.350 μs …  2.772 ms  ┊ GC (min … max): 0.00% … 98.82%
 Time  (median):     12.043 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   13.577 μs ± 56.440 μs  ┊ GC (mean ± σ):  9.17% ±  2.21%

    █▇
  ▃███▆▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▂▂ ▂
  11.4 μs         Histogram: frequency by time          25 μs <

 Memory estimate: 35.22 KiB, allocs estimate: 755.

n = 7

BenchmarkTools.Trial: 8577 samples with 1 evaluation per sample.
 Range (min … max):  461.444 μs …   3.088 ms  ┊ GC (min … max):  0.00% … 73.05%
 Time  (median):     480.591 μs               ┊ GC (median):     0.00%
 Time  (mean ± σ):   580.974 μs ± 262.912 μs  ┊ GC (mean ± σ):  15.87% ± 19.68%

  ██▅▄▃▂                                           ▂▃▃▃▂▁▁▁     ▂
  ███████▇▇▇▆▆▆▆▄▄▃▅▅▁▄▁▁▁▃▄▄▄▄▁▁▃▄▄▁▄▃▁▁▄▁▁▃▁▁▃▁▁█████████▇▇▆▆ █
  461 μs        Histogram: log(frequency) by time       1.38 ms <

 Memory estimate: 1.62 MiB, allocs estimate: 30283.

n = 8

BenchmarkTools.Trial: 1046 samples with 1 evaluation per sample.
 Range (min … max):  3.837 ms …  10.550 ms  ┊ GC (min … max):  0.00% … 17.33%
 Time  (median):     4.818 ms               ┊ GC (median):    19.09%
 Time  (mean ± σ):   4.778 ms ± 488.635 μs  ┊ GC (mean ± σ):  16.78% ±  7.33%

  ▃▄▁▂▂                      ▄█▇▅▅▆▅▄▂▁▁
  █████▇█▇▄▅▄▅▄▅▄▄▅▁▄▁▄▁▄▁▁▁▁████████████▇█▇▇▆▇▇▅▆▆▇▆▄▆▆▆▆▅▆▅ █
  3.84 ms      Histogram: log(frequency) by time      5.79 ms <

 Memory estimate: 14.77 MiB, allocs estimate: 241967.

n = 9

BenchmarkTools.Trial: 114 samples with 1 evaluation per sample.
 Range (min … max):  41.082 ms …  47.521 ms  ┊ GC (min … max):  8.59% … 16.64%
 Time  (median):     43.978 ms               ┊ GC (median):    16.26%
 Time  (mean ± σ):   44.119 ms ± 926.061 μs  ┊ GC (mean ± σ):  16.19% ±  1.33%

                      ▃▃  ▂ ▅▃▃█▅▃  ▅  ▆
  ▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇▇██▇▇█▅██████▅▄█▄▅█▇▅▅▁▄▄▅█▁▁▁▁▄▁▁▁▄▁▁▁▄▄ ▄
  41.1 ms         Histogram: frequency by time           47 ms <

 Memory estimate: 132.89 MiB, allocs estimate: 2177332.

n = 10, t = 6

BenchmarkTools.Trial: 308 samples with 1 evaluation per sample.
 Range (min … max):  13.125 ms …  19.379 ms  ┊ GC (min … max):  0.00% … 17.51%
 Time  (median):     16.235 ms               ┊ GC (median):    18.53%
 Time  (mean ± σ):   16.227 ms ± 654.551 μs  ┊ GC (mean ± σ):  18.60% ±  2.95%

                                ▂██ ▂▂▂▂ ▅█▄▁▃▂▁  ▂  ▁
  ▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▁▁▁▃▃▆▅▅▇███▇████████████▅▆█▃▇█▃▃▁▁▃▁▃▃ ▄
  13.1 ms         Histogram: frequency by time         17.9 ms <

 Memory estimate: 53.07 MiB, allocs estimate: 907255.

…ations` As it currently stands, `multiset_permutations` is more efficient than `permutations`; see JuliaMath#151. We can exploit it to optimize `permutations`.

codecov · 2025-05-06T13:02:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.17%. Comparing base (e005ee0) to head (3f6ee61).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #186      +/-   ##
==========================================
- Coverage   97.22%   97.17%   -0.06%     
==========================================
  Files           8        8              
  Lines         829      813      -16     
==========================================
- Hits          806      790      -16     
  Misses         23       23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

oameye

This looks great. No comments

FedericoStra · 2025-05-28T16:48:24Z

Of course I don't wish to take any credit at all for this silly PR. All the merit goes to whoever coded the fast nextpermutation(m, t, state) function which is used by Base.iterate(p::MultiSetPermutations, s=p.ref) (I believe they are mostly @randy3k and @natemcintosh).

Without this fix there are issues with `OffsetArray`: `eachindex` returns an `OffsetArrays.IdOffsetRange`, which causes troubles when passed to `multiset_permutations` because `f = [sum(c == x for c in a)::Int for x in m]` becomes an `OffsetArray` instead of a Vector`.

FedericoStra · 2025-05-28T17:36:37Z

I merged master because there was a conflict (caused my myself in #184 😅).

I then had to apply a small fix (collect the indices on line 28) because otherwise it failed a test with OffsetArray (check out the merge commit to see what I'm referring to).

natemcintosh

I'm glad to see this! Speedups are always great.

perf: optimize permutations by implementing it via `multiset_permut…

bc3931f

…ations` As it currently stands, `multiset_permutations` is more efficient than `permutations`; see JuliaMath#151. We can exploit it to optimize `permutations`.

FedericoStra added 2 commits May 6, 2025 15:10

fix: use explicit NamedTuple notation tu support Julia 1.0

397a684

fix: don't use isnothing to support Julia 1.0

995a046

FedericoStra force-pushed the permutations-trick branch from 9ebd8fa to 995a046 Compare May 6, 2025 13:10

This was referenced May 6, 2025

permutations is much slower than it needs to be #185

Open

Optimize permutations with low effort (no change in algorithm) #184

Merged

oameye approved these changes May 28, 2025

View reviewed changes

FedericoStra added 2 commits May 28, 2025 19:02

Merge branch 'master' into permutations-trick

41aed76

natemcintosh approved these changes May 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize `permutations` by implementing it via `multiset_permutations` #186

Optimize `permutations` by implementing it via `multiset_permutations` #186

Uh oh!

FedericoStra commented May 6, 2025 •

edited

Loading

Uh oh!

codecov bot commented May 6, 2025 •

edited

Loading

Uh oh!

oameye left a comment

Uh oh!

FedericoStra commented May 28, 2025

Uh oh!

FedericoStra commented May 28, 2025 •

edited

Loading

Uh oh!

natemcintosh left a comment

Uh oh!

Uh oh!

Optimize permutations by implementing it via multiset_permutations #186

Are you sure you want to change the base?

Optimize permutations by implementing it via multiset_permutations #186

Uh oh!

Conversation

FedericoStra commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code changes

Simple benchmark

Uh oh!

codecov bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

oameye left a comment

Choose a reason for hiding this comment

Uh oh!

FedericoStra commented May 28, 2025

Uh oh!

FedericoStra commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

natemcintosh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Optimize `permutations` by implementing it via `multiset_permutations` #186

Optimize `permutations` by implementing it via `multiset_permutations` #186

FedericoStra commented May 6, 2025 •

edited

Loading

codecov bot commented May 6, 2025 •

edited

Loading

FedericoStra commented May 28, 2025 •

edited

Loading