-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Missing interface connections after upgrade from v2.8.6 to v2.10.5 #5984
-
Hello, I am trying to solve an issue with missing data. Any help / suggestion / insight is appreciated. After an upgrade, I am missing the data under "Devices -> Interface Connections" and "Devices -> Console Connections". To be clear, I am using netbox-docker, and the upgrade was from netbox-docker 0.24.1 (commit e9a9d9b7) + Netbox v2.8.6 to netbox-docker 1.0.1 (commit 4dd7a51c) + Netbox v2.10.5. The upgrade procedure was this:
Note: if I run the postgres service by itself, I don't get errors during the backup restoration. If I start all services with a "docker-compose up -d", and then restore the backup, I get errors during that process. Either way, to end result is the same: I have all the data, except for interface/console connections. I did notice that netbox-docker v0.27.0 required running 'docker-compose run --rm netbox ./manage.py trace_paths', so I also attempted that as a step "3.5" (in other words, after restoring the database, but before starting the whole thing again). When I try that, I get an exception "django.db.utils.ProgrammingError: relation "virtualization_vminterface" already exists". Right now if I try to access the REST API with a GET /api/dcim/interface-connections/, I get the following error: {"error": "Got AttributeError when attempting to get a value for field Finally, looking at an interface that previously had a connection, I can still see the connection data in there, in a new field "cable_peer". However, the old "connected_endpoint" is null: GET /api/dcim/interfaces/?id=14315 (shortened, to highlight the connection data)
If there is any other information that might be of use, please let me know. Thank you in advance. |
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 2 comments · 7 replies
-
You said:
But you didn't show those errors. My guess is that there was some error when netbox-docker applied the data migrations on startup, and so they weren't completed properly. Can you restore postgres from the backup again, restart the container, and capture all the logs from container startup? |
Beta Was this translation helpful? Give feedback.
All reactions
-
My apologies, @candlerb. I wasn't sure if pasting the entire thing here would be ok. Here it is:
For comparison, this is what happens if I bring up just the postgres service, then restore the database:
I am guessing the netbox container runs and does something to the database before I get around to restoring it. I just tested something which solved the initial issue: 1) run just the postgres service, 2) restore the database, 3) run the 'manage.py trace_paths', 4) restart everything. When I do that, the trace_paths command works and I can see the interface/console connections. I am still hoping I can recover from the state the DB is in, though. |
Beta Was this translation helpful? Give feedback.
All reactions
-
You also requested the container logs; here it is (for the version where we bring the postgres container up by itself):
I tried the postgres logs for the case in which we run everything, then restore the DB, but it was massive amounts of logs - I can see data from the database dumped into the logs. |
Beta Was this translation helpful? Give feedback.
All reactions
-
I managed to get the logs for postgres for the case of 1) running everything with "docker-compose up -d", 2) restoring the backup. As it turns out, there is a difference depending on whether you actually wait until netbox is 100% done launching, or not; the massive logs I saw were likely errors due to trying to restore the backup while netbox was still launching and running migrations. Here are the logs:
|
Beta Was this translation helpful? Give feedback.
All reactions
-
That all looks very broken. Are you sure you're restoring a pg_dump backup taken from netbox 2.8 before you started the 2.10 container?? When you start the container, the migrations are applied automatically. https://github.com/netbox-community/netbox-docker/blob/release/docker/docker-entrypoint.sh#L18 |
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes, I am sure. Please note that, since my original post, I have managed to get the very same database backup successfully to Netbox 2.10, with interface/console connections showing up. The steps were:
The only difference between what I have in production (broken, without connections) and the successful upgrade is step 2. As far as I can tell: if I only start postgres and restore the DB, and then run 'manage.py trace_paths', it works. However, if I simply start everything in step 2, then Netbox runs before I can restore the database, and everything goes wrong. I am hoping there is some way to get my production database back to a working state. |
Beta Was this translation helpful? Give feedback.
All reactions
-
Just to be clear - I am sure this is a "backup of netbox 2.8 before I started the 2.10 container" because I took the backup, then cloned a new netbox-docker folder, and restored the database to that one instead. I still have the original Netbox 2.8 with its own netbox-docker folder (and docker volumes) where they were. Thank you for the help, @candlerb. |
Beta Was this translation helpful? Give feedback.
All reactions
-
It sounds like you've found a solution then. Do the same sequence of steps in your production environment - you'll only have to do this once to get the database into the right state, and subsequent container restarts should be fine. I'm sure Netbox used to carry a prominent warning saying that it was not advised to use the docker container in production, although I can't find it right now. I do note that Netbox-docker has its own discussion group, so maybe that would be a better place to ask for docker-related problems. |
Beta Was this translation helpful? Give feedback.
All reactions
-
Thank you for the help and the link, @candlerb. I was concerned that these steps would not work on the production database (because it had already been migrated/broken, and it had a few days of data generated on v2.10 after the failed upgrade), but I was able to upgrade it successfully. Just in case someone finds this, the solution was: stopping everything (docker-compose down), starting only postgres (docker-compose run -d postgres), running the trace_paths (docker-compose run --rm netbox ./manage.py trace_paths), and bringing everything back up (docker-compose up -d). |
Beta Was this translation helpful? Give feedback.
That all looks very broken. Are you sure you're restoring a pg_dump backup taken from netbox 2.8 before you started the 2.10 container??
When you start the container, the migrations are applied automatically. https://github.com/netbox-community/netbox-docker/blob/release/docker/docker-entrypoint.sh#L18