Auto Import Database Schema

[oxzi]

To get rid of docker-icingadb and its additional entry point, the schema import functionality has been implemented directly in Icinga DB. Using the new --database-auto-import command line argument will result in an automatic schema import if no schema is found.

The implementation is split between the already existing CheckSchema function and the introduced ImportSchema function.

The CheckSchema function is now able to distinguish between the absence of a schema and an incorrect schema version. Both situations return a separate error type.

As before, CheckSchema is called in the main function. If the error type now implies the absence of a schema (ErrSchemaNotExists) and the --database-auto-import flag is set, the auto-import is started.

The schema import itself is performed in the new ImportSchema function, which loads the schema from a given file and inserts it within a transaction, allowing to rollback in case of an error.

MySQL Run

$ docker run --name mariadb-icingadb -e MARIADB_ROOT_PASSWORD=supersicher -p 3306:3306 -d mariadb:latest

$ docker exec -it mariadb-icingadb mariadb --password=supersicher
MariaDB [(none)]> create database icingadb;
Query OK, 1 row affected (0.001 sec)
MariaDB [(none)]> ^D

$ # Start w/o a schema
$ ./icingadb --config config.yml
2025-03-21T13:55:49.505+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:55:49.505+0100    INFO    icingadb        Connecting to database at 'mysql://root@localhost:3306/icingadb'
2025-03-21T13:55:49.508+0100    FATAL   icingadb        The database schema is missing

$ # Start w/o a schema, but allow imports
$ ./icingadb --config config.yml --database-auto-import
2025-03-21T13:56:02.930+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:56:02.930+0100    INFO    icingadb        Connecting to database at 'mysql://root@localhost:3306/icingadb'
2025-03-21T13:56:02.933+0100    INFO    icingadb        Starting database schema auto import
2025-03-21T13:56:03.256+0100    INFO    icingadb        The database schema was successfully imported
[ . . . ]

$ # Simulate an outdated schema version
$ docker exec -it mariadb-icingadb mariadb --password=supersicher icingadb
MariaDB [icingadb]> update icingadb_schema set version = 1;
Query OK, 1 row affected (0.008 sec)
Rows matched: 1  Changed: 1  Warnings: 0
MariaDB [icingadb]> ^D
$ ./icingadb --config config.yml
2025-03-21T13:57:24.745+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:57:24.745+0100    INFO    icingadb        Connecting to database at 'mysql://root@localhost:3306/icingadb'
2025-03-21T13:57:24.748+0100    FATAL   icingadb        unexpected database schema version: v1 (expected v6), please make sure you have applied all database migrations after upgrading Icinga DB

$ # Reset schema version
$ docker exec -it mariadb-icingadb mariadb --password=supersicher icingadb
MariaDB [icingadb]> update icingadb_schema set version = 6;
Query OK, 1 row affected (0.008 sec)
Rows matched: 1  Changed: 1  Warnings: 0
MariaDB [icingadb]> ^D

$ # Start w/ correct schema
./icingadb --config config.yml
2025-03-21T13:58:01.618+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:58:01.618+0100    INFO    icingadb        Connecting to database at 'mysql://root@localhost:3306/icingadb'
[ . . . ]

$ # Retry without a database
$ docker stop mariadb-icingadb
$ docker container rm mariadb-icingadb
$ ./icingadb --config config.yml
2025-03-21T13:58:57.484+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:58:57.484+0100    INFO    icingadb        Connecting to database at 'mysql://root@localhost:3306/icingadb'
2025-03-21T13:58:57.486+0100    WARN    database        Can't connect to database. Retrying     {"error": "dial tcp [::1]:3306: connect: connection refused"}
[ . . . ]

PostgreSQL Run

$ docker run --name postgres-icingadb -e POSTGRES_PASSWORD=supersicher -p 5432:5432 -d postgres:latest

$ docker exec -it postgres-icingadb psql -U postgres
postgres=# create database icingadb;
CREATE DATABASE
postgres=# ^D

$ # Start w/o a schema
$ ./icingadb --config config.yml
2025-03-21T13:46:28.030+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:46:28.031+0100    INFO    icingadb        Connecting to database at 'pgsql://postgres@localhost:5432/icingadb'
2025-03-21T13:46:28.046+0100    FATAL   icingadb        The database schema is missing

$ # Start w/o a schema, but allow imports
$ ./icingadb --config config.yml --database-auto-import
2025-03-21T13:47:30.096+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:47:30.096+0100    INFO    icingadb        Connecting to database at 'pgsql://postgres@localhost:5432/icingadb'
2025-03-21T13:47:30.109+0100    INFO    icingadb        Starting database schema auto import
2025-03-21T13:47:30.164+0100    INFO    icingadb        The database schema was successfully imported
[ . . . ]

$ # Simulate an outdated schema version
$ docker exec -it postgres-icingadb psql -U postgres
icingadb=# update icingadb_schema set version = 1;
UPDATE 1
icingadb=# ^D
$ ./icingadb --config config.yml
2025-03-21T13:48:55.762+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:48:55.762+0100    INFO    icingadb        Connecting to database at 'pgsql://postgres@localhost:5432/icingadb'
2025-03-21T13:48:55.778+0100    FATAL   icingadb        unexpected database schema version: v1 (expected v4), please make sure you have applied all database migrations after upgrading Icinga DB

$ # Reset schema version
$ docker exec -it postgres-icingadb psql -U postgres
icingadb=# update icingadb_schema set version = 4;
UPDATE 1
icingadb=# ^D

$ # Start w/ correct schema
./icingadb --config config.yml
2025-03-21T13:50:32.951+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:50:32.951+0100    INFO    icingadb        Connecting to database at 'pgsql://postgres@localhost:5432/icingadb'
[ . . . ]

$ # Retry without a database
$ docker stop postgres-icingadb
$ docker container rm postgres-icingadb
$ ./icingadb --config config.yml
2025-03-21T13:52:10.914+0100    INFO    icingadb        Starting Icinga DB daemon (1.2.1-g4d6c74e-dirty)
2025-03-21T13:52:10.914+0100    INFO    icingadb        Connecting to database at 'pgsql://postgres@localhost:5432/icingadb'
2025-03-21T13:52:10.915+0100    WARN    database        Can't connect to database. Retrying     {"error": "dial tcp [::1]:5432: connect: connection refused"}
[ . . . ]

Fixes #896.

[lippserd]

In addition to the changes requested I want to propose/discuss two additional changes:

1. Embed schema files to reduce code related to flag and file handling. Also, this ensures functionality across all contexts without needing to specify the correct schema path.

2. Introduce something like ErrNoSchema and ErrSchemaMismatch, and modify CheckSchema() accordingly. Build error handling and schema import around these changes. This would enhance code flow and readability, as we now call ImportSchema() followed by CheckSchema().

[julianbrost]

• Embed schema files to reduce code related to flag and file handling. Also, this ensures functionality across all contexts without needing to specify the correct schema path.

I though about that as well while loosely following this PR. How would this work for package installs? Don't install as file? Both embed and install as a file? Introduce some build flags to control embedding?

Like when I was thinking about this, I wasn't sure if this would even simplify things in the big picture (maybe unless the schema is shipped twice, once for manual use in a file, once embedded for automated use).

[lippserd]

• Embed schema files to reduce code related to flag and file handling. Also, this ensures functionality across all contexts without needing to specify the correct schema path.

I though about that as well while loosely following this PR. How would this work for package installs? Don't install as file? Both embed and install as a file? Introduce some build flags to control embedding?

Like when I was thinking about this, I wasn't sure if this would even simplify things in the big picture (maybe unless the schema is shipped twice, once for manual use in a file, once embedded for automated use).

I would ship the schema twice and not change anything in the documentation and package installation, as this currently only serves the purpose of moving our container file to this repo. As you said, the embedded file is for automated use and the file on disk is for package/manual use.

[oxzi]

In addition to the changes requested I want to propose/discuss two additional changes:

1. Embed schema files to reduce code related to flag and file handling. Also, this ensures functionality across all contexts without needing to specify the correct schema path.

At the moment, I am against embedding the schema files because the container still needs the schema update files. So embedding the schema itself while still having to ship the schema updates is an inconsistency, also for the user.

If at some point we do have automatic schema updates, then I would argue in favor of embedding everything in the binary, but until we get there, I would leave it the way it is.

2. Introduce something like `ErrNoSchema` and `ErrSchemaMismatch`, and modify `CheckSchema()` accordingly. Build error handling and schema import around these changes. This would enhance code flow and readability, as we now call `ImportSchema()` followed by `CheckSchema()`.

I liked this idea and changed the PR in its favor. The initial check logic (schema exists, schema has expected version) went in CheckSchema, while ImportSchema now only does the import based on the CheckSchema's error.

So I have updated the commit message and the initial post within this PR thread. I have also included two runs with hopefully every possible situations for MySQL/MariaDB and PostgreSQL in the OP.

Furthermore, gofmt was now unhappy about ./cmd/icingadb-migrate/main.go, which was fixed as well.

[lippserd]

At the moment, I am against embedding the schema files because the container still needs the schema update files. So embedding the schema itself while still having to ship the schema updates is an inconsistency, also for the user.

Our container doesn't ship schema upgrade files, and I don't think it should unless we have support for automatic schema migrations (which would then be embedded as well anyway). I think you were thinking of the schema upgrade situation in container setups, but this is something where we leave our users alone at the moment as there is no documentation, which of course is not good. But that should rather be a place in the documentation where the schema upgrade files can be downloaded, otherwise you would have to document the path in the container and then users would have to copy the scripts out of the container.

[oxzi]

At the moment, I am against embedding the schema files because the container still needs the schema update files. So embedding the schema itself while still having to ship the schema updates is an inconsistency, also for the user.

Our container doesn't ship schema upgrade files, and I don't think it should unless we have support for automatic schema migrations (which would then be embedded as well anyway).

You are right. I assumed that our container image would contain the schema upgrades as well, but in its current form it only contains a zipped version of the schema.

However, for the moment I would still keep it as it is. When eventually adding automatic schema updates, I would argue for embedding the schema file and schema upgrades into the binary, adding Docker (or container) documentation and removing all SQL files from our package builds as well.