Unable to deploy a custom image on a dedicated server

[ClementCastel]

Hello,
The new BringYourOwnLinux feature seems very interesting so I wanted to test it.
However, I followed the instructions in the README, but it always crashes at the "Configuring Boot" stage.

Here are the different tests I made:

• Debian 12, without script make_image_bootable.sh

• Debian 12, with script make_image_bootable.sh (the one supplied)

• Debian 11, with and without script (since the script's comments state that it is intended for Debian 11, I preferred to test both)

• Ubuntu 22.04 server cloudimg, with and without script.

My aim is to get the feature working with a custom Ubuntu 22.04 image.

Here is the hardware configuration of the dedicated server:

• Advance STOR-1 Gen 2

Please let me know if the problem is on my side or yours. If you need any further information, please do not hesitate to contact me.

[lolo6oT]

same problem

[n0xena]

So I'm not sure if any or both of you managed to solve your issues (if so: please report back) or if you did what I did and just took one of the default installs to get the server up and running.
Anyway, I tinkered with it and here's how I got it working:

1. bring your own image
   the more simpler option

• set up a new VM in qemu with a qcow2 drive of at least 3GB
• boot up archinstall
• create a 50MB ESP, rest for the root partition
  don't worry - we will use ESP only for grub which only requires roughly 20MB
• pacstrap: base linux linux-firmware vim sudo grub efibootmgr openssh
• add your ip config to systemd-networkd and a public key to authorized_keys
• mount the ESP at /efi
• grub-install --target=x86_64-efi --efi-direcotry=/efi --boot-directory=/efi --no-nvram --removable
• grub-mkconfig -o /efi/grub/grub.cfg
• enable systemd-networkd, systemd-resolved and sshd
• shutdown the VM

use the resulting qcow2 as the image to provide - should go fine and the server should boot into arch

2. bring your own linux
   a bit more complicated due to OVH not provide how the partition works - but here you go:
   when using the manager interface just use the default partitions /boot and / - they will get mounted as such
   additional the install environment creates a ESP and mounts it to /boot/efi
   looks like this:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1     0B  0 disk 
sdb           8:16   1     0B  0 disk 
sdc           8:32   1     0B  0 disk 
sdd           8:48   1     0B  0 disk 
sde           8:64   1     0B  0 disk 
sdf           8:80   1     0B  0 disk 
sdg           8:96   1     0B  0 disk 
sdh           8:112  1     0B  0 disk 
nbd0         43:0    0     0B  0 disk 
nbd1         43:32   0     0B  0 disk 
nbd2         43:64   0     0B  0 disk 
nbd3         43:96   0     0B  0 disk 
nbd4         43:128  0     0B  0 disk 
nbd5         43:160  0     0B  0 disk 
nbd6         43:192  0     0B  0 disk 
nbd7         43:224  0     0B  0 disk 
nvme0n1     259:0    0 476.9G  0 disk 
|-nvme0n1p1 259:6    0   511M  0 part /boot/efi
|-nvme0n1p2 259:7    0     1G  0 part /boot
|-nvme0n1p3 259:10   0 475.4G  0 part /
`-nvme0n1p4 259:11   0     2M  0 part 
nvme1n1     259:1    0 476.9G  0 disk 
|-nvme1n1p2 259:2    0   100G  0 part 
`-nvme1n1p3 259:3    0 375.9G  0 part 
nbd8         43:256  0     0B  0 disk 
nbd9         43:288  0     0B  0 disk 
nbd10        43:320  0     0B  0 disk 
nbd11        43:352  0     0B  0 disk 
nbd12        43:384  0     0B  0 disk 
nbd13        43:416  0     0B  0 disk 
nbd14        43:448  0     0B  0 disk 
nbd15        43:480  0     0B  0 disk

/dev/nvme0n1p3 on / type ext4 (rw,relatime)
/dev/nvme0n1p2 on /boot type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=16145696k,nr_inodes=4036424,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,size=32612656k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,noexec,size=102400k,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3261264k,mode=700)

The difference now becomes:

• only create one partition as required by the docs
  don't add an additional ESP - the install environment will do this on its own
• create the required script /root/.ovh/make_image_bootable.sh, make it executable by chmod +x

#!/bin/bash
grub-install --target=x86_64-efi --efi-directory=/boot/efi --no-nvram --removable
grub-mkconfig -o /boot/grub/grub.cfg

• you don't have to worry about copy the kernel to /boot or run mkinitcpio - the install environment somehow does this on its own correctly
• the setup should complete successfully and boot into arch
  it may required to use the KVM to keep the install system from restart over again

overall the current documentation lacks a lot of required information and a proper step-by-step guide - another option is just to use one of the default templates like debian 12 to get the server bootstrapped - login via ssh and add an entry to boot an install yourself
one could also use grub-install without --no-nvram but with --bootloader-id= - but this could fail as the install environment doesn't provide efibootmgr (or at least it failed in my tests)
in the end I used both BYOI and BYOL for a small arch stub to use it to boot an opensuse install from it

side note to ovh staff: I'm aware that your job is to keep an eye on your servers - but if you give us the option to do such installs but not the proper documentation - please train your stuff TO NOT INTERFERE while WE as the customer are actively monitoring the setup via KVM konsole
I had to start over twice because your staff killed the install before I could analyze what went wrong - so either please update the doc or don't mess with the system while the customer is already on it

If you have a system with two or more drives: Be aware that the install environment will kill ANY ESP it finds on ALL drives. Restoring an already installed system on another drive can be a bit tricky as re-creating a new ESP also changes its serial so you have to check your FSTAB and maybe INITRD

[deric]

@ClementCastel Unfortunately the output of make_image_bootable.sh can't be checked via IPMI. However after unsuccessful install you can reboot into rescue image and chroot into partially installed system.

For default partitions (boot and root partitions both in RAID1), e.g.:

mkdir -p /mnt
mount /dev/md3 /mnt
mkdir -p /mnt/{boot,proc,sys,dev/pts}
mount /dev/md2 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev
mount --bind /dev/pts /mnt/dev/pts
mount --bind /run /mnt/run
chroot /mnt /bin/bash

then you can run directly /root/.ovh/make_image_bootable.sh and check the output.

There might be several issues:

• DNS not working, try adding echo "nameserver 1.1.1.1" > /etc/resolv.conf at the begining of make_image_bootable.sh
• missing mdadm package (the script assumes it's always installed)

Here's packer config to build Debian 12 image

[ashmonger]

@ClementCastel @deric @lolo6oT

Hello folks
I see your problems with the images, I can provide some answers

• /root/.ovh/make_image_bootable.sh is mandatory. The BYOLinux workflow works exactly like the basic install workflow we use everyday for ovh-made images. We are aware of the problem to not have the output of the script if it fails, we try to find a way to do that, but it's not really easy.
• DO NOT provide a qcow2 with more than one partition in it. You can customize the partitioning in the UI. I know this step is a bit strange and complicated, but I have some colleagues working on it to provide something more simple.
• about mdadm, in the example, it's installed in the pre-install-script run within packer

FYI the OVH DNS Servers
cdns.ovh.net has address 213.186.33.99
cdns.ovh.net has IPv6 address 2001:41d0:3:163::1

To @n0xena, sorry, I thought the documentation (README.md) was clear enough.

[p10tyr]

You used to be able to install Debian with ZFS from OVH.
I have tried all of these steps and image things.. getting stuck on the make_image_bootable.sh file. I dont know why.

Because I changed the partitions to ZFS I dont know how to mount them in recovery to see why its failing. I tried adding the nameservers and also the pre-install script stuff for mdadm.

Still failed.. dont know if the supplied debian 12 with grub is not compatible with zfs? but the pre init script seems to suggest it should be if you install all the zfs stuff.

After 4 days. I gave up.. and just gonna use proxmox

[sbraz]

You used to be able to install Debian with ZFS from OVH.

Hi @p10tyr, we do support ZFS for Debian 12. What issue did you have with our standard installer?

The only case we can't support is servers with less than 8 GiB of memory as the ZFS build might fail on low-end systems (because of not enough free memory or because one of our hard timeouts was reached). But when this happens, we do show an error message so I assume that's not the issue you're facing.

Still failed.. dont know if the supplied debian 12 with grub is not compatible with zfs? but the pre init script seems to suggest it should be if you install all the zfs stuff.

This should work indeed:

bringyourownlinux/example_build/files/make_image_bootable.sh

Lines 36 to 41 in 4c1e12b

	if lsblk -lno FSTYPE | grep -qxiF zfs_member; then
	apt-get -y install --no-install-recommends linux-headers-amd64 zfs-dkms zfs-initramfs zfs-zed
	# Make sure zpools are imported at boot, this is not required when / is ZFS because the initramfs
	# imports the pool. However, it is necessary if e.g. /home is ZFS and / is ext4.
	systemctl enable zfs-import-scan.service
	fi

Did you make any changes to the example image? If so, can you show us your repo?

Also, can you give me the server on which you tried? I'll check our logs.

I dont know how to mount them in recovery to see why its failing

You can use commands such as zpool import -R /mnt -a to mount the pool in /mnt and check what's happening there.

[n0xena]

@ashmonger
Sorry for the late reply, I'm currently involved into https://github.com/archzfs/archzfs which takes quite some effort along my regular job - I just not checked back here.

To me, the documentation currently lacks these points:

• When using the web interface the user is supposed to only select a /root partition - because a) the preset /boot parition can't be removed and b) the install environment adds an ESP mounted to /boot/efi on it's own. This has to be explained way more in-depth.
• The install environment deploys the kernel and an initrd to /boot hence this doesn't has to be done in the script. From what I was able to figure out this seems to get done via kernel-install from systemd. This information is completely missing. Also it has to be checked if this implies that the prepared image also has to use systemd or if other init systems like openRC are compatible.
• How to configure the network? As mentioned I gave up and used the provided debian12 preset to at least get the server initialized and get the proper network config from it. I wasn't able to find the required information neither here nor in the web interface. There has to be some explanation how to properly configure at least IPv4 to get access to the system via ssh - or how to configure the network afterwards via IPMI-console.

I'm sure there're various reasons for the current workflow - I don'T question this. But in order for a customer to properly prepare a BYOL image it has to be explained at least as far as to end up with an image that is compatible with the workflow.
Add information to the web interface that an ESP will be created and mounted to /boot/efi so the customer doesn'T has to tinker with it. Add information that the kernel and initramfs get installed automatic via kernel-install so that's why the customer has to only deal with setting up the bootloader. Add information for how to properly setup the network - both for IPv4 and IPv6.
It's nice to see OVH granting its customers such freedom and direct access to an IPMI console. But in its current state one has to tinker with it to get it working properly.

[deric]

How to configure the network?

setting e.g. in /etc/network/interfaces

auto eno1
iface eno1 inet dhcp

is usually enough. However the "predictable" interface naming is a bit unpredictable - you don't know the interface name util the server boots installed kernel (naming depends on motherboard, number of slots...).

You have several options:

1. Disable predictable interfaces net.ifnames=0 biosdevname=0 and stick with eth0 as primary interface - though with multiple network cards you might get into problems
2. Run dhclient eno1 after boot
3. Install cloud-init which on the first run tries to obtain IPv4 and IPv6 from DHCP and stores config e.g. into /etc/network/interfaces.d/50-cloud-init. On Debian simply run apt-get install cloud-init e.g. in make_image_bootable.sh script. Configuration is stored in /etc/cloud/cloud.cfg

[n0xena]

@deric
You missed my point - my question clearly doesn't go into the direction "I'm a noob to linux - how do I get internet?" - my question goes: "What exact config do I have to use for X (systemd-networkd/-resolved, NetworkManager, wicked,netplan, ...)?".
Although both IPv4 and IPv6 are given in the webinterface it's hard to figure out how to correctly set them up with correct routes and dns. Fun fact: the ipv6 /64 block assigned to my server has a route given that's outside of it? Also: Unless I'm wrong with IPv6 an IP ending in 0 always designates the network and is not a valid host IP - hence for some magic reason the xxxx::/64 (which expands to xxxx:0000:0000:0000:0000) worked with the debian preset *confused
Also, while we're already at it: The webinterface hints one can chose as many IPv6 freely from the assigned block and even set different PTR. But what's missing: To set a PTR first a matching A/AAAA record has to be set in the domain. Yes, in hindsight this makes sense - but's it's nowhere explained. It just gives you a useless "generic error" instead of a more useful "no matching A/AAAA record".
You mention DHCP - although not tested I doubt I would get the assigned public IP on eth0 - if any DHCPOFFER at all. I likely only get the PXE on eth1. Oh, and trying to use the rescue system via the IPMI kvm console is also useless - one HAS TO connect via ssh.

TLDR: To keep this on topic instead of drifting away into a rant: Yes, for power users or those you want something different than the provided presets BYOI/BYOL is a very cool idea - but its documentation lacks so many important information and required way more detailed sub-parts meaningful split into an order one can follow while preparing an image. So, to re-iterate how this stuff could gain massively from some simple imporvements of its docs:

1.) Add a meaningful error message in the IP config to explain: "To set a PTR a matching A/AAAA is required".
2.) Add the ESP with its mountpoint /boot/efi the install environment creates anyway. The current display with just /boot and /root is misleading at best.
3.) Explain proper networkconfig using the actual values shown in the webinterface along with routing and dns - systemd, NetworkManager and netplan copy'n'paste configs should be no problem and give enough hints how to do it correctly if using something different.
4.) Explain the fact that the kernel and its initramfs gets installed to /boot and how this works - is it just kernel-install from systemd? If so: Is the provided image supposed to be systemd compatible?
5.) Either automated or put a big red box around it: Disable automatic intervention on fault during manual setup! As explained: Staff killed my TWICE while I was actively troubleshooting - it should be a dead giveaway when the server is in setup mode and an active kvm session is running that it's likely either the customer or some other supporter already on it.

Don'T get me wrong - this isn't meant offensive - but yet again it clearly shows: The docs were written by the features dev: They know how all this stuff works exactly and assume so does a new customer just by pure magic. Get someone unrelated to it and have them check if what's given works or if there're defecites as right now.
This feature has quite some potential - but requires a lot of rework of its docs.

[deric]

@n0xena Sorry, I didn't mea to offend you.

You mention DHCP - although not tested I doubt I would get the assigned public IP on eth0 - if any DHCPOFFER at all

I just wanted to help. It would much easier if you would actually try to run those commands.

[p10tyr]

You used to be able to install Debian with ZFS from OVH.

Hi @p10tyr, we do support ZFS for Debian 12. What issue did you have with our standard installer?

The only case we can't support is servers with less than 8 GiB of memory as the ZFS build might fail on low-end systems (because of not enough free memory or because one of our hard timeouts was reached). But when this happens, we do show an error message so I assume that's not the issue you're facing.

Still failed.. dont know if the supplied debian 12 with grub is not compatible with zfs? but the pre init script seems to suggest it should be if you install all the zfs stuff.

This should work indeed:

bringyourownlinux/example_build/files/make_image_bootable.sh

Lines 36 to 41 in 4c1e12b

	if lsblk -lno FSTYPE | grep -qxiF zfs_member; then
	apt-get -y install --no-install-recommends linux-headers-amd64 zfs-dkms zfs-initramfs zfs-zed
	# Make sure zpools are imported at boot, this is not required when / is ZFS because the initramfs
	# imports the pool. However, it is necessary if e.g. /home is ZFS and / is ext4.
	systemctl enable zfs-import-scan.service
	fi

Did you make any changes to the example image? If so, can you show us your repo?

Also, can you give me the server on which you tried? I'll check our logs.

I dont know how to mount them in recovery to see why its failing

You can use commands such as zpool import -R /mnt -a to mount the pool in /mnt and check what's happening there.

I managed to get an Epyc 7351 with 128GB RAM. So it cant be an issue with the min requirements?
Steps I used for my other instance with ZFS but not available here


No ZFS?

Server (with proxmox now)
E105B06-1383647

I used the supplied script as standard and that failed with ZFS partitions.
Then I tried to add the suggested lines in to the boot and the init script

######################################################
# This is an example script that works for debian 11 #
######################################################

echo "nameserver 1.1.1.1" > /etc/resolv.conf

set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

apt-get updateapt-get -y install --no-install-recommends mdadm lvm2 patch btrfs-progs amd64-microcode intel-microcode

# We will install these in make_image_bootable.sh and only when ZFS is used
# linux-headers-amd64 needs to be manually specified because dkms only recommends it
apt-get -y install --no-install-recommends --download-only linux-headers-amd64 zfs-dkms zfs-initramfs zfs-zed

# This script will run inside the newly installed system, no need to call chroot

The rest of file is original

The image is here
https://drive.usercontent.google.com/download?id=1YBmKkzpFas_NFK1SKp94-h78a3xSiKjT&export=download&confirm=t&uuid=715f71ea-92e1-44d5-b121-150dbee564ab

--EDIT
I have just noticed the OVH template is Debian 11.. not 12 ? I usually use Ubuntu so I didn't notice my other instance is Debian 12 .. Why don't I have Debian 12 on the select list?

Proxmox is also 7 instead of 8?

[sbraz]

@p10tyr Your server has a Tyan S8026 board which has issues with recent kernels in some case (we're working on it). This is why we don't offer Debian 12 or Proxmox 8 on it.

If I have enough time, I will check your image and logs from your server to see what happened during the BYOLinux installation.

@n0xena We will reply more in depth later to address all the points you raised. Thanks for archzfs by the way, I use it and it's really useful :)
To address the networking configuration question:

• Yes, the gateway is outside of the subnet, most network configuration generators support this, (e.g. via ip route's onlink option).
• For IPv4, you can simply use DHCP which is what we do when we install Linux distributions.
• For IPv6, you may check the network configuration while booted into rescue-customer and copy this.
• Feel free to check the contents of the config-2 partition after e.g. a Debian 12/Ubuntu 24.04 installation to see what we put in there and take inspiration from this to build your cloud-init config.