embeds_many not supported

[scottmessinger]

Hi! I really appreciate the library, but we're running into an issue where we can't used embedded documents. It seems those schemas aren't used in this library's tests and/or are commented out, so this might be something that's not supported or in development. Here's what we're seeing:

Here's our schema:

      create table standards_usage (
        standards Nested(
          standard_id FixedString(32),
        ) default []
       -- snip --
     );

Here's our model:

    embeds_many(:standards, Spikey.StandardsUsage.Standard)

and the embedded model look like this:

  defmodule Standard do
    use Ecto.Schema

    @primary_key false
    embedded_schema do
      field :standard_id, Ch, type: "FixedString(32)"
      # ...snip....
    end
end

When we insert, we get:

[error] ** (ArgumentError) unknown type: `:any`
    (ecto_ch 0.3.1) lib/ecto/adapters/clickhouse/schema.ex:333: Ecto.Adapters.ClickHouse.Schema.ch_type_hint/1
    (ecto_ch 0.3.1) lib/ecto/adapters/clickhouse/schema.ex:328: Ecto.Adapters.ClickHouse.Schema.ch_type_hint/1
    (ecto_ch 0.3.1) lib/ecto/adapters/clickhouse/schema.ex:304: Ecto.Adapters.ClickHouse.Schema.remap_type/4
    (elixir 1.17.3) lib/enum.ex:1703: Enum."-map/2-lists^map/1-1-"/2
    (elixir 1.17.3) lib/enum.ex:1703: Enum."-map/2-lists^map/1-1-"/2
    (ecto_ch 0.3.1) lib/ecto/adapters/clickhouse/schema.ex:44: Ecto.Adapters.ClickHouse.Schema.insert_rows/5
    (ecto_ch 0.3.1) lib/ecto/adapters/clickhouse/schema.ex:33: Ecto.Adapters.ClickHouse.Schema.insert_all/8
    (ecto 3.11.2) lib/ecto/repo/schema.ex:59: Ecto.Repo.Schema.do_insert_all/7
    (spikey 5.4.513) lib/spikey/clickhouse_repo.ex:37: Spikey.ClickhouseRepo.safe_insert_all/3
    (elixir 1.17.3) src/elixir.erl:386: :elixir.eval_external_handler/3
    (stdlib 4.3.1.6) erl_eval.erl:748: :erl_eval.do_apply/7
    (elixir 1.17.3) src/elixir.erl:364: :elixir.eval_forms/4
    (elixir 1.17.3) lib/module/parallel_checker.ex:112: Module.ParallelChecker.verify/1
    (iex 1.17.3) lib/iex/evaluator.ex:332: IEx.Evaluator.eval_and_inspect/3
    (iex 1.17.3) lib/iex/evaluator.ex:306: IEx.Evaluator.eval_and_inspect_parsed/3
    (iex 1.17.3) lib/iex/evaluator.ex:295: IEx.Evaluator.parse_eval_inspect/4
    (iex 1.17.3) lib/iex/evaluator.ex:187: IEx.Evaluator.loop/1
    (iex 1.17.3) lib/iex/evaluator.ex:32: IEx.Evaluator.init/5
    (stdlib 4.3.1.6) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

I go to lib/ecto/adapters/clickhouse/schema.ex:333 and log the arguments:

  defp remap_type(other, original, schema, field) do
    IO.inspect(other, label: "other")
    IO.inspect(original, label: "original")
    IO.inspect(schema, label: "schema")
    IO.inspect(field, label: "field")
    ch_type = ch_type_hint(original)
    
    # ...snip...

I get

other: {:map, :any}
original: {:parameterized, Ecto.Embedded,
 %Ecto.Embedded{
   cardinality: :many,
   field: :standards,
   owner: Spikey.StandardsUsage,
   related: Spikey.StandardsUsage.Standard,
   on_cast: nil,
   on_replace: :raise,
   unique: true,
   ordered: true
 }}
schema: Spikey.StandardsUsage
field: :standards

It appears that the embeds_many is being converted to {:map, :any}. How can we add support for inserted embedded documents in this adapter?

Thanks in advance for any insight you have here!

[ruslandoga]

👋 @scottmessinger

We probably need to be able to support something like this but it might get tricky due to possible ambiguity (is it a JSON, is it a Nested table, or something else entirely). And you are right, we kind of skipped it during development since this wasn't useful for us. Note however, that Plausible is using nested tables for metadata fields in events and sessions tables and they are represented as just arrays. Seems to work there. I wonder if it could be used as a workaround while we are figuring out the embedded schemas?

schema "standards_usage" do
  # ...
  field :"standards.standard_id", {:array, Ch}, type: "FixedString(32)"
  # ...
end

[scottmessinger]

@ruslandoga Thank you! I'm still running into an issue when I try to create the struct:

        %Spikey.StandardsUsage{
          standards: standards,
        }

I get the compile message:

== Compilation error in file lib/spikey/models/analytics/standards_usage.ex ==
** (KeyError) key :standards not found
    (spikey 5.4.513) expanding struct: Spikey.StandardsUsage.__struct__/1

When I try to use a changset to cast (similar to how you do it here), it doesn't seem to capture that this is an array:

        |> cast(
          %{
            standards: card_stack.standards
          },
          [:"standards.standard_id"]
        )
        |> apply_changes()

produces

 %{
    "standards.jurisdiction_id": nil,
    "standards.jurisdiction_title": nil,
    "standards.standard_id": nil,
    "standards.standard_set_id": nil,
    "standards.standard_set_title": nil,
}

it should look like:

%{
   standards: [
      jurisdiction_id: "asfasd",
     # ...snip...
   ]
}

In the examples from Plausible, I'm not seeing where you all set entry_meta or meta.

Any ideas how to get around that part? How do I populate the field?

[ruslandoga]

Maybe something like this:

ids = ["11111111111111111111111111111111", "11111111111111111111111111111111",
 "11111111111111111111111111111111", "11111111111111111111111111111111",
 "11111111111111111111111111111111"]

%Spikey.StandardsUsage{
  "standards.standard_id": ids
}

And for casting you would need to use the "full" column name, like standards.standard_id. AFAIK the Nested tables are sort-of flattened out in ClickHouse into a bunch of "namespaced" array columns, so they can be treated as such.

|> cast(
  %{
    # it can probably be made prettier, but this should work too
    "standards.standard_id": Enum.map(card_stack.standards, & &1.standard_id)
  },
  [:"standards.standard_id"]
)
|> apply_changes()

[scottmessinger]

@ruslandoga that worked! You're a life saver. Thank you!