-
Notifications
You must be signed in to change notification settings - Fork 18
aenigma upgrades on Ubuntu 18.04 are temporarily broken. Here's how to fix them.
It's about a delicate situation regarding all existing aenigma users currently running any version of aenigma on Ubuntu 18.04, except those users who have a beta version of v0.7.1 after [and including] v0.7.1.beta.2. If you are in the latter situation, write me on the chatroom and I'll let you know what to do. Anyone else, follow these instructions.
Daily server functionality is unaffected. The issue affects apt upgrades / dist-upgrades and aenigma re-installs/upgrades: if you perform an APT upgrade or re-run the aenigma installer in any form without following this custom procedure you will render your instance temporarily unusable, with no loss of data however.
Ubuntu recently release openssl v1.1.1, which breaks TLS in ejabberd 18.12.1, the version currently installed by anyone using aenigma until today.
The decision was made to switch to a deb-package ejabberd install rather than one using the unofficial jabber.at APT repository. This will also ensure new ejabberd will always make their way faster onto your system, provided you re-run the aenigma installer to upgrade, something you should be doing every so often anyway. This was probably the best way to do things anyway the entire time, so it's good that we're doing this now rather than later anyway.
We were very lucky that this issue came after the introduction of v0.7.0, which exports the built-in ejabberd database "mnesia" to PostgreSQL, therefore allowing us to purge APT-based ejabberd and install DEB-based ejabberd without any loss of data. We could have fixed this even if it had happened earlier, but this is a much better time for it to happen.
Nothing too bad actually, ejabberd will simply fail at any TLS session it starts, and both clients and other servers will see it as offline. No data will be lost, but with a little extra effort you can migrate your way seamlessly beyond this issue.
Theoretically, anyone already on v0.7.0 already has a purgeable ejabberd install as the database has already been moved to psql, but they still need to inhibit apt upgrades from their systems, so this is a unified guide for everyone expect those running a version more recent than [or equal to] v0.7.1-beta.2.
I will guide you into a custom, two-step upgrade process to circumnavigate this situation with no downtime at all on your systems [hopefully].
v0.7.1 switches to nginx-habdled xmpp uploads, so you will lose all previously uploaded files. There is a way to save them and even to migrate them to your v0.7.1 instance, but it's outside the scope of this guide. If you absolutely need to keep them, write me on the chatroom and we'll do this together on a shared ssh session. This would have happened anyway when upgrading to v0.7.1 regardless of the openssl situation.
Read through the background section if you'd like and then follow the simple guide below to seamlessly bypass this catch-22 situation.
On 12/jun/2019, shortly after the v0.7.0 stable release, and on the verge of the v0.7.1 stable release, but just before it was ready, this happened:
VERY IMPORTANT: Hi guys and girls using aenigma! If you're running on Ubuntu 18, absolutely do *NOT* run any kind of apt upgrade from now until further notice. A new version of openssl [v1.1.1] was released today on the Ubuntu repo and this will break TLS in ejabberd for you until we update it. Right now we use the jabber.at repo for ejabberd which however hasn't released updates in some months, so this is probably the ideal time to switch to a direct-building provisioning method which I will start working on tomorrow. This will be a pretty invasive change but it will be handled - hopefully - seamlessly by aenigma. In the mean time simply refrain from doing anything on your Ubuntu 18 aenigma servers, no apt upgrades, no aenigma upgrades/reinstalls [as those will trigger an apt upgrade automatically].
After some days of work, on 17/jun/2019:
Hi all! It seems like we have a functioning ejabberd installer from DEB package, now testing out migration from old repo-based ejabberd to deb-based ejabberd, which will also upgrade the version from 18.12.1 to 19.05.
aenigma v0.7.1, when ready, will bring deb-based ejabberd v19.05 and nginx-handled xmpp uploads. To upgrade to v0.7.1 it is *absolutely essential* to first upgrade to v0.7.0, as - out of a streak of pure luck - we introduced postgresql backend in v0.7.0, separating the database data from the ejabberd installation. This means we can entirely purge ejabberd from the system without any data loss, which is essential for the migration.
However, if you perform any kind of re-installation, aenigma will trigger an apt upgrade and your system will install openssl v1.1.1, therefore breaking ejabberd until you upgrade to aenigma v0.7.1.
Here's how.
Again, this affects anyone running any version of aenigma on Ubuntu 18.04 from v0.0.1 until v0.7.1-beta.2 excluded. Anyone running more recent versions write me on the chatroom.
Therefore [TL;DR] when you decide to upgrade your aenigma instance, follow these instructions, and write me on xmpp:[email protected] before you do anything if you want so we can verify together that you're doing everything perfectly.
This is a little tricky, but hopefully as we get towards v1.0 we'll never have any of these issues again:
cd
if [ -d ~/aenigma ]; then rm -r ~/aenigma; fi
git clone https://github.com/openspace42/aenigma
bash aenigma/setup -d "v0.3.14-beta.2" -p "v0.7.1-beta.2" ### read following instructions immediately after issuing this command before proceeding with the setup.
When prompted:
- skip the introduction
- press to continue if you are warned about an incomplete installation
- run one last backup before continuing when prompted
After you see the aenigma logo splash screen, when you're asked whether this installation is for a single node or a cluster, exit the installer with "^-c" [control + c].
cd
nano +4863 ~/aenigma/functions
In the text editor that opens, comment out the line you're automatically taken to [4863]:
This:
dna-apt_full_upgrade -e
must become this:
# dna-apt_full_upgrade -e
Now press ^-x [control + x] to save and answer yes at the filename prompt.
We can now proceed with the full upgrade to v0.7.1-beta.2, which introduces nginx-handled xmpp uploads already, but still retains the old APT-repo-based ejabberd v18.12.1. However, thanks to the changes you've just made following the steps above, your system will not upgrade any system package and therefore it'll stay on openssl v1.1.0g, therefore not breaking TLS on ejabberd. After all of this is done, I'll show you how to upgrade to aenigma v0.7.1 stable in a second phase.
cd
bash aenigma/setup -dl -pl ### Note the "-dl -pl" arguments, these are essential.
You can follow through with the installer up until the end. If you have any issues just write me on the chatroom.
Wait for aenigma v0.7.1 stable to be released, when that is you will be able to upgrade to it with a simple aenigma-upgrade with no arguments of any kind, and that will actually purge ejabberd v18.12.1, now that, after following all of the above steps, your database will have been migrated to PostgreSQL and your system won't be upgrading any apt package since apt upgrades will be temporarily disabled in the v0.7.1 installer.
Thanks for being an aenigma user, and for your time reading this guide.
Nz