Alembic paths and environment organization

[willsthompson]

We added a database bootstrapping step to our project that can run migrations automatically during deployment, but to get both that and the CLI working, we restorted to several workarounds, and I'm wondering if I might have missed a better way to do it.

When the app's Docker container runs, it executes some setup scripts (via docker-entrypoint.sh), including database_bootstrap.py, which runs migrations using the alembic.command module. Here is a slimmed down directory listing:

.
├── alembic.ini
├── app
│   ├── Dockerfile
│   ├── __init__.py
│   ├── api
│   ├── database_bootstrap.py   
│   ├── database_model.py
│   ├── database_types.py
│   ├── docker-entrypoint.sh
│   └── alembic
│       ├── README.md
│       ├── __init__.py
│       ├── env.py
│       ├── script.py.mako
│       └── versions
├── docker-compose.yml
├── services
├── setup.py
└── test

To use the CLI without the -c option (which our developers were tripping over), we moved alembic.ini up to the project root and set script_location = %(here)s/app/alembic. And in order for the CLI to resolve imports from our app module, we had to explcitly add the project root to sys.path in the mako template and in env.py. Now the CLI just works, as long as its run from the project root.

The path hacking seems unavoidable for a lot of Alembic use cases, but I don't like having alembic.ini outside of the application module directory. We've tried several permutations of project organizations, and this was the only one that worked for our setup.

If you have any suggestions on how to better organize a project like this, I'd appreciate it!

[zzzeek]

We added a database bootstrapping step to our project that can run migrations automatically during deployment, but to get both that and the CLI working, we restorted to several workarounds, and I'm wondering if I might have missed a better way to do it.

When the app's Docker container runs, it executes some setup scripts (via docker-entrypoint.sh), including database_bootstrap.py, which runs migrations using the alembic.command module. Here is a slimmed down directory listing:

.
├── alembic.ini
├── app
│   ├── Dockerfile
│   ├── __init__.py
│   ├── api
│   ├── database_bootstrap.py   
│   ├── database_model.py
│   ├── database_types.py
│   ├── docker-entrypoint.sh
│   └── alembic
│       ├── README.md
│       ├── __init__.py
│       ├── env.py
│       ├── script.py.mako
│       └── versions
├── docker-compose.yml
├── services
├── setup.py
└── test

To use the CLI without the -c option (which our developers were tripping over),

There's an ALEMBIC_CONFIG environment variable. I would set that in your scripts so that there is no need to use -c.

we moved alembic.ini up to the project root and set script_location = %(here)s/app/alembic. And in order for the CLI to resolve imports from our app module, we had to explcitly add the project root to sys.path in the mako template and in env.py. Now the CLI just works, as long as its run from the project root.

that also sounds overly complicated, you can set PYTHONPATH on the outside to refer to wherever you want. Also even if you are setting sys.path inside of env.py, there is certainly no reason for that to be in the script.py.mako template, env.py is sufficient.

The path hacking seems unavoidable for a lot of Alembic use cases, but I don't like having alembic.ini outside of the application module directory. We've tried several permutations of project organizations, and this was the only one that worked for our setup.

I would keep alembic.ini inside the app directory. i'm not really following the problem with that because when you say "which our developers were tripping over" I don't know what the overall environment is, are the devs bashing into the docker container directly or something like that? Overall, just set ALEMBIC_CONFIG to where the file is, the devs when they bash in should have a .bashrc that sets basic environmental details so that things like this are set up for devs, this is normal stuff for environments.

If you have any suggestions on how to better organize a project like this, I'd appreciate it!

[zzzeek]

you can also set sys.path directly in the alembic.ini file too:

# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
# (new in 1.5.5)
prepend_sys_path = .

that's in the sample alembic.ini

[willsthompson]

Ok I see, we missed this.

[willsthompson]

Thank you for taking the time to reply in detail. I think the key thing I failed to understand was that using %(here)s could save a lot of trouble dealing with relative path resolution happening in the CLI and our bootstrapper code. I didn't think to try that in the project organization that came before this one.

Using prepend_sys_path we were able to eliminate all the explicit path hacking, and using %(here)s/alembic we were able to move alembic.ini back into the module directory.

And this is an obsolete problem now, but when we only added the project root to sys.path in env.py (and not in the mako template and migrations), alembic current worked, but alembic history did not. The latter is what prompted us to update the migration template. But prepend_sys_path solves both.

Thanks again.

[zzzeek]

oh i see because your template files would import things. there's another config option that will force env.py to be used for all commands which is "revision_environment=True". looks like the docstring is not clear on this but that flag should cause all commands that use those files to also set up env.py.

[willsthompson]

Right, we're using some custom column types generated with sqlalchemy-utils. Is the idea behind revision_environment=True to avoid duplicating setup (assuming there's a lot of it) across env.py and migration files?

[zzzeek]

yes it's for when the migration files have things that need to be set up code-wise from env.py when any kind of command is being run. originally env.py only came in during migrations and autogenerate.