Help with deep (recursive) merging

[axyz]

Hi, I'm trying to evaluate nickel for a use case that involves deep merging.

Imagine a set of variables use to theme a website or an app with a structure along those lines:

{
  # base here defined all the available variables and their default values
  base = { 
    color.text.primary | doc "primary color"
                       | default = "red",
    color.text.secondary | doc "secondary color"
                         | default = "red",
    color.background.primary | doc "background primary color"
                             | default = "white"
  },

  # There are also a set of default themes defined by listing what variables are overridden in the base (with the same nesting structure)
  themes | default = {
    dark | default = {
      color.text.secondary = "green",
      color.background.primary = "black"
    }
  }
}

given those base definitions, then I'd like to be able to define a set of differend "designs", each one extending the base and the set of themes, overriding only what is needed. e.g.

let root = import "root.ncl" in

{
  default = root.base & { themes = root.themes },
  design_a = root.base & {
    color.text.primary = "blue",
    themes = root.themes & {
      dark = {
        color.text.secondary = "pink"
      }
    }
  }
}

I've been struggling to get the result I need, If you try this code you'll notice that the "dark" theme within design_a now only contains the pink override, but not the black primary background. I also got different results by using dot separated keys as well as nesting dictionaries (probably related on where the "default" is added. Is this only possible by using a flat structure and marking each single key as "default"?)

Am I in the right direction? What is the idiomatic way to achieve this kind of "inheritance"?

Additionally I'd like to eventually explore some validation aspects:

• all designs should always override keys that already exist in the base, and never add new keys
• the nested structure of "themes" should be exactly the same as the one of the "base" and themes should also only be able to override keys that already exist in the "top level"
• also nice to have validation on the value of each key, but this is likely unrelated to the current issues

I am curious to hear if you have suggestion or even completely different approaches.

[jneem]

I think your approach is basically fine, but you've misunderstood the behavior of default: since dark is marked as default within themes, when you merge themes with { dark = blah }, the blah "wins" and completely overwrites the previous settings in themes.dark.

I'm sorry that the previous sentence isn't very clear; I'm having trouble figuring out a better way to explain it. But here's an example:

{ a | default = { b = 1, c = 2 } } & { a = { b = 2 } }
# => { a = { b = 2 } }

because the default value of a is overridden by the higher-priority value { b = 2 }. If you want to do recursive merging instead of overriding, you need the two as to have the same priority instead of having one as the lower default priority.

In other words, you should probably have the default annotations at the leaves:

{ a = { b | default = 1, c | default = 2 } } & { a = { b = 2 } }
# => { a = { b = 2, c | default = 2 } }

[axyz]

yeah, if I move the default to every single leaf it seems to work correctly. Is there a shortcut to mark the entire record as default? Or is it intentional to force it to be explicit on every field?

[axyz]

Any suggestion for the validation part?

Additionally I'd like to eventually explore some validation aspects:

• all designs should always override keys that already exist in the base, and never add new keys
• the nested structure of "themes" should be exactly the same as the one of the "base" and themes should also only be able to override keys that already exist in the "top level"
• also nice to have validation on the value of each key, but this is likely unrelated to the current issues

I am curious to hear if you have suggestion or even completely different approaches.

[yannham]

yeah, if I move the default to every single leaf it seems to work correctly. Is there a shortcut to mark the entire record as default? Or is it intentional to force it to be explicit on every field?

There was in an old version of Nickel, but it has been disabled because it wasn't clear how to implement it since a deeper rework of the metadata system. It has been on hold for now; not necessarily buried, but as it's not been re-discussed or mentioned again since then, other priorities took over.

[yannham]

Regarding your other questions, it sound like you should make base a contract instead of a record, and apply it with | instead of merging it. The differences are subtle - applying a record contract is in many ways very similar to merging two records - but basically contract application is stricter, in some sense. In particular, it can do what you want around "not allowing additional keys".

Here is how I would do it: basically, you say that your designs have a common shape that is constrained. I would write a contract for that, called Design. base and dark would just be instances of Design, which is the single source of truth on what is allowed or not:

{
  # the general shape of a design
  Design = {
    color
      | {
        text
          | {
            primary | doc "primary color",
            secondary | doc "secondary color",
          },

        background
          | {
            primary | doc "background primary color",
          }
      }
  },

  # base here defined all the available variables and their default values
  base | Design
    = {
      color.text.primary | default = "red",
      color.text.secondary | default = "red",
      color.background.primary | default = "white"
    },

  # There are also a set of default themes defined by listing what variables are overridden in the base (with the same nesting structure)
  themes = {
    dark | Design = {
      color.text.secondary = "green",
      color.background.primary = "black"
    }
  }
}

This will fail because dark doesn't define color.text.primary. If you don't want to define everything, you can set fields to optional in your original contract:

            primary | doc "primary color" | optional,

[yannham]

If you want base to always be used as a base default, there is another possibility: make use of numeral priority. In Nickel, the standard priorities we use are default (lowest), normal (when you don't specify any priority), and force (highest). However, when that is not enough, you can use | priority xxx where xxx is a number literal. In particular you can use negative priorities, that takes precedence over a default, but will be overridden by a normal priority (not specifying any priority is equivalent to using | priority 0). So, to be used with care, but another approach could be the following, to combine base and the Design contract into one thing that 1. check that no additional fields are there 2. provide defaults, and then provide "second-level" defaults using negative priorities:

{
  # the general shape of a design
  Design = {
    color
      | {
        text
          | {
            primary | doc "primary color" | default = "red",
            secondary | doc "secondary color" | default =" red",
          },

        background
          | {
            primary | doc "background primary color" | default = "white",
          }
      }
  },

  # There are also a set of default themes defined by listing what variables are overridden in the base (with the same nesting structure)
  themes = {
    dark | Design = {
      color.text.secondary | priority -1 = "green",
      color.background.primary | priority -1 = "black",
    }
  }
}

Now yo don't need any optional anymore, because Design always provide default values. But you can provide "higher-precedence" default values as in dark. If you merge dark with a normal record (without any default or | priority -n), then the normal values will take precedence.

[jneem]

Or is it intentional to force it to be explicit on every field?

I think there was some discussion about having some metadata that could propagate recursively, but I can't find it now. In any case, you currently need to put it on every field.

About the validation, I'd probably define something like

Theme = { 
    color.text.primary 
     | doc "primary color"
     | String,
    color.text.secondary 
     | doc "secondary color"
     | String,
     # etc
}

and then apply this contract to each theme. If the names of the themes are also fixed, you could also do

Themes = {
  base | Theme,
  dark | Theme,
  light | Theme, # etc
}

and then you can validate your themes with

{
  design_a
    | Themes
    = root & {
    base.color.text.primary = "blue",
    dark.color.text.primary = "white",
  }
}

[axyz]

thanks, will try that out

[yannham]

Beware, @jneem: using field paths in a contract definition can have surprising result. Your contract desugars to something like:

Theme = { 
  color = {
    text = ...
}

The = there means that the contract will allow for extra fields, because what you want is really |. Cf the warning in the section Giving values to fields of the manual. So it's a bit annoying, but if you want all levels of your contract to not allow additional fields, you need |, and thus you need to spell out everything explicitly instead of using foo.bar.baz = , beause the foo.bar.baz will always desugar to foo = {bar = {baz =, and not foo | {bar | {baz | ....