feature: column level exceptions

[dennisUhrskov]

As a developer I would like the ability to define which columns to exclude on a table level.

Use case:
We have a data warehouse that serves many purposes, but some columns are very specific and adds ambiguity to the model. So would be neat to be able to exclude columns that is not adding value.

The alternative is to duplicate the tables.

[Bl3f]

Hello @dennisUhrskov interesting, what would be the best interface for this? Like a * pattern like in the include/exclude so it?

By curiosity how many distincts columns does it represent?

[dennisUhrskov]

Yes, that would make perfect sense in the nao_config, something like below. I think it would make your tool easier to deploy for teams with existing data warehouses (which would be most teams), that way they can just use their existing setup instead of having to build new models for nao. In my use case it is because I include some KPI tables that do some heavy computations that I want to feed downstream, like daily running 30-day download counts with +500m downloads pr day, etc.

For the time being (I am still testing out nao) I have only stumbled upon one column, that is named downloads, with a grain column called daily, the nao agent (ofc) thought this was daily download counts, but it is really 30-day rolling windows of downloads on a daily basis. But could also just be system columns like last_update, version etc. no need to have these.

For now I 'fixed' it, by adding to the various context files and rules, which does work, but would be safer to just be able to exclude.

One could build new models just for this use case, but would be a waste, so I think it would make your tool more accessible that way.

I hope it makes sense :)

Could be something like:

databases:

• type: clickhouse
  name: nexus-clickhouse
  include:
  • "nexus_demand.*"
  • "nexus_supply.*"
    exclude_columns:
  • ".peerdb" # internal replication metadata
  • "*.sign" # ReplacingMergeTree internals
  • "*.version" # ReplacingMergeTree internals
    accessors:
  • columns
  • description