Using TypedTables limited column number #140

Activity

1. gbaraldi commented on May 13, 2026

   

   gbaraldi

   on May 13, 2026

   Member

   More actions

   You are hitting an annoying thing where Tuples are type stable up to 32. Then they become type unstable
2. gbaraldi

   added

   trimmingRelated to --trim mode and verifier errorsRelated to --trim mode and verifier errors

   upstreamRequires fix in Julia base or external packageRequires fix in Julia base or external package

   on May 13, 2026
3. Thomas008 commented on Jun 22, 2026

   

   Thomas008

   on Jun 22, 2026

   Author

   More actions

   Is there work to increase the type-stability of tuples?
   Is there even a constant in the julia compiler to set the maximal number of elements in a tuple such that it is type-stable?
4. Thomas008 commented on Jun 22, 2026

   

   Thomas008

   on Jun 22, 2026

   Author

   More actions

   There are further limitations when using TypedTables:

   1. Filtering rows: causes compiler errors, if the number of columns is greater than 5:

   using TypedTables
   
     function (@main)(args::Vector{String})::Cint
       t = Table(c1 = [1], c2 = [1], c3 = [1], c4 = [1], c5 = [1], c6 = [1])
       t1 = filter(r -> r.c1 == 1, t)
       return 0
     end
   

   Verifier error #1: unresolved call from statement (f::Function)(Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 1)::Vector{String})::Any
   Stacktrace:
    [1] map(f::Function, t::Tuple{Vector{String}, Vararg{Vector{Int64}, 5}})
      @ Base tuple.jl:357 [inlined]
    [2] map(::Function, ::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}})
      @ Base namedtuple.jl:263
    [3] getindex(t::Table{@NamedTuple{c1::String, c2::Int64, c3::Int64, c4::Int64, c5::Int64, c6::Int64}, 1, @NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}}, inds::Vector{Int64})
      @ TypedTables C:\Users\Thoma\.julia\packages\TypedTables\ItVth\src\Table.jl:318 [inlined]
    [4] filter(f::var"#main##0#main##1"{String}, a::Table{@NamedTuple{c1::String, c2::Int64, c3::Int64, c4::Int64, c5::Int64, c6::Int64}, 1, @NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}})
      @ Base array.jl:3011
    [5] main(args::Vector{String})
      @ Main C:\jul\JulWork\parajul\jct\test_jl5.jl:1556
    [6] _main(argc::Int32, argv::Ptr{Ptr{Int8}})
      @ Main C:\Julia-1.13.0-beta1\share\julia\juliac\juliac-buildscript.jl:35
   
     Verifier error #2: unresolved call from statement Base.map(f::Function, Core.tuple(Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 2)::Vector{Int64}, Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 3)::Vector{Int64}, Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 4)::Vector{Int64}, Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 5)::Vector{Int64}, Core.getfield(nt::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}, 6)::Vector{Int64})::NTuple{5, Vector{Int64}})::NTuple{5, Any}
     Stacktrace:
    [1] map(f::Function, t::Tuple{Vector{String}, Vararg{Vector{Int64}, 5}})
      @ Base tuple.jl:357 [inlined]
    [2] map(::Function, ::@NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}})
      @ Base namedtuple.jl:263
    [3] getindex(t::Table{@NamedTuple{c1::String, c2::Int64, c3::Int64, c4::Int64, c5::Int64, c6::Int64}, 1, @NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}}, inds::Vector{Int64})
      @ TypedTables C:\Users\Thoma\.julia\packages\TypedTables\ItVth\src\Table.jl:318 [inlined]
    [4] filter(f::var"#main##0#main##1"{String}, a::Table{@NamedTuple{c1::String, c2::Int64, c3::Int64, c4::Int64, c5::Int64, c6::Int64}, 1, @NamedTuple{c1::Vector{String}, c2::Vector{Int64}, c3::Vector{Int64}, c4::Vector{Int64}, c5::Vector{Int64}, c6::Vector{Int64}}})
      @ Base array.jl:3011
    [5] main(args::Vector{String})
      @ Main C:\jul\JulWork\parajul\jct\test_jl5.jl:1556
    [6] _main(argc::Int32, argv::Ptr{Ptr{Int8}})
      @ Main C:\Julia-1.13.0-beta1\share\julia\juliac\juliac-buildscript.jl:35
   

   2. Setting cells in the tables: causes compile errors, if the number of different column types is greater than 2:

   using TypedTables
   
   function (@main)(args::Vector{String})::Cint
     t = Table(c1 = ["1"], c2 = [1.2], c3 = [1])
     d = getfield(t, :data)
     r = 1
     for s in 1:3
       d[s][r] = getfield(d, s)[r]
     end
     return 0
   end
   

   It causes the compile errors

   Verifier error #1: unresolved call from statement Base.convert((Core._svec_ref)((Core._compute_sparams)(setindex!(A::Array{T}, x, i::Int64) where T @ Base array.jl:1017::Method, Base.setindex!, Base.getfield(%new()::@NamedTuple{c1::Vector{String}, c2::Vector{Float64}, c3::Vector{Int64}}, φ ()::Int64)::Union{Vector{Float64}, Vector{Int64}, Vector{String}}, φ ()::Union{Float64, Int64, String}, 1)::Core.SimpleVector, 1)::Any, φ ()::Union{Float64, Int64, String})::Any
   Stacktrace:
    [1] setindex!(A::Union{Vector{Float64}, Vector{Int64}, Vector{String}}, x::Union{Float64, Int64, String}, i::Int64)
      @ Base array.jl:1019 [inlined]
    [2] main(args::Vector{String})
      @ Main C:\jul\JulWork\parajul\jct\test_jl5.jl:1573
    [3] _main(argc::Int32, argv::Ptr{Ptr{Int8}})
      @ Main C:\Julia-1.13.0-beta1\share\julia\juliac\juliac-buildscript.jl:35
   
   Verifier error #2: unresolved call from statement Base.convert((Core._svec_ref)((Core._compute_sparams)(setindex!(A::Array{T}, x, i::Int64) where T @ Base array.jl:1017::Method, Base.setindex!, Base.getfield(%new()::@NamedTuple{c1::Vector{String}, c2::Vector{Float64}, c3::Vector{Int64}}, φ ()::Int64)::Union{Vector{Float64}, Vector{Int64}, Vector{String}}, φ ()::Union{Float64, Int64, String}, 1)::Core.SimpleVector, 1)::Any, φ ()::Union{Float64, Int64, String})::Any
   Stacktrace:
    [1] setindex!(A::Union{Vector{Float64}, Vector{Int64}, Vector{String}}, x::Union{Float64, Int64, String}, i::Int64)
      @ Base array.jl:1019 [inlined]
    [2] main(args::Vector{String})
      @ Main C:\jul\JulWork\parajul\jct\test_jl5.jl:1573
    [3] _main(argc::Int32, argv::Ptr{Ptr{Int8}})
      @ Main C:\Julia-1.13.0-beta1\share\julia\juliac\juliac-buildscript.jl:35
   
5. ameligrana commented on Aug 1, 2026

   

   ameligrana

   on Aug 1, 2026

   Last edited by ameligrana

   More actions

   I investigated a similar issue where I had problems with Core._apply_iterate with >31 types. In practice though in my case it is fixable:

   • by not using Vararg (since it is not statically expanded with >31 elements) and instead use a single Tuple
   • by fixing some type instabilities due to Base functions (e.g. map) giving up at >31 elements by default. So I think the problem is more in TypedTables than in JuliaC.

   Concretely I used

   Base.@assume_effects :foldable function _world_type_params(comp_types::Tuple)
       raw_types = map(arg -> arg isa Type ? arg : arg.first, comp_types)
       types = map(_unwrap_relation_type, raw_types)
       storages = map(arg -> arg isa Type ? Storage{Vector} : arg.second, comp_types)
       relation_types = map(_unwrap_relation_type, filter(_declares_relation, raw_types))
       return (Tuple{types...}, Tuple{storages...}, Tuple{relation_types...})
   end

   which doesn't cause problems with JuliaC since it uses a single tuple and it assumes foldability. Maybe similar bugs apply to TypedTables. I think these are indeed workarounds.
6. Thomas008 commented on Aug 4, 2026

   

   Thomas008

   on Aug 4, 2026

   Last edited by Thomas008

   Author

   More actions

   Here is a solution to the example of filtering rows of a Table. It is of Claude.
   The general solution pattern is: using a column-wise instead of row-wise processing. More specific, filter() runs on the generic rows-/ tuple-based Table mechanisms that cannot be resolved statically anymore.
   So one can apply a mask that is calculated only on the used columns and is applied on them.

   using TypedTables
    
   macro apply_mask(functionname::Symbol, columnnames_expression)
     columnnames = eval(columnnames_expression)
     column_values = [:(getproperty(table, $(QuoteNode(c)))[mask]) for c in columnnames]
     return quote
       @inline function $(esc(functionname))(table, mask)
         return Table(NamedTuple{$columnnames}(($(column_values...),)))
       end
     end
   end
    
   @apply_mask filter_with_mask (:c1, :c2, :c3, :c4, :c5, :c6, :c7, :c8)
    
   function (@main)(args::Vector{String})::Cint
     t = Table(c1 = [1,2], c2 = [1,2], c3 = [1,2], c4 = [1,2], c5 = [1,2], c6 = [1,2], c7 = [1,2], c8 = [1,2])
     mask = getproperty(t, :c1) .== 1
     t1 = filter_with_mask(t, mask)
     return 0
   end