100-fold increase in regular incremental backup size using blksnap over veeamsnap

[bickford]

Distribution

Ubuntu 22.04.5 LTS

Architecture

x86_64 intel

Kernel version

Linux 6.8.0-52-generic #53~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jan 15 19:18:46 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Blksnap version

Version: 6.3.0.73 deb from apt-get

Bug description

Hi, I'm not sure if this is a bug per-se. I'm checking if it's normal for the Veeam incremental files with blksnap to be 100 times larger than the Veeam incremental files created with the older veeamsnap. Mine have ballooned from ~14MB daily to ~1.2GB daily.

I updated Veeam Backup Agent for Linux a couple of weeks ago and found that the new version required the use of blksnap. So I installed that which replaced veeamsnap, I also noticed that blksnap supports Secure Boot now - so I enabled this in my BIOS and went through the process of authorizing the MOK. I re-ran the backup config to reconfigure and run a test backup. Everything worked great, so far so good.

However I've checked on the backups now and noticed that the incremental backup files (.VIB) which were previously ~14MB each day are now ~1.2GB, this is not a server that has a lot of data change daily so the increase is unexpected and I can't account for it other than the veeamsnap to blksnap change. Screenshots of the NAS file sytem showing the backups, attached.

veeamsnap:

The transition from veeamsnap to blksnap:

Steps to reproduce

Perhaps hard to reproduce.

• Configure entire machine backup in Veeam to shared folder (SMB), monthly full backup, daily incrementals.
• Run a veeamsnap full backup, run some incremental backups (preferrably over a few days, so some data has changed)
• Install blksnap (uninstalling veamsnap automatically), run a full backup, run some incremental backups (pref over several days)
• Compare incremental backup sizes between the two systems

Expected behavior

I'd expect the incremental backup sizes to be similar betwen both backup mechanisms as they're both disk-level, a 100-fold increase is absolutely massive.

Additional information

I could not find any other threads discussing this increase so I thought it may be unique to my situation, or something I've misconfigured.
Normal? Any suggestions? Is there a configuration option I can change for better compression or something?

Thanks for your help.

[SergeiShtepa]

Hi!
Thank you for feedback!

I can assume that the differences are caused by differences in the operation of the change tracker (CBT).
We will try to confirm this in our labs and find out the cause of the problem.

I hope that we can make the next version of blksnap better.

[SergeiShtepa]

Hi @bickford .

Unfortunately, we were unable to confirm this problem.
Maybe we're not taking something into account.
The only thing I can suggest is to contact Veeam customer support.
They will request all logs and complete information about the system.

Thank you.

[aj-potc]

Hi @SergeiShtepa ,

I'm chasing a similar issue and have just opened a ticket with Veeam support about it.

In my case, I recently put a Web server into production that uses Apache to serve up static content (images). Before going into production, daily backups were extremely small. But immediately after making the server live, daily incremental backups of my data volume spiked in size massively. As an example, on a recent day, the volume that holds the static content is showing a backup size of 200+ GB, despite only about 2 GB of new images added to the server that day.

I've done a bit of troubleshooting and found that Apache is updating the access time (atime) metadata for files it serves. On my server, that's a lot of files and directly corresponds to how busy the server is.

Is it reasonable that blksnap could be detecting these atime updates as block changes, thus greatly increasing the number of blocks in the snapshot?

Thank you very much for any thoughts!

[Fantu]

If you don't really use access time of files you can set relatime as workaround so that update access time only on files changes.

[SergeiShtepa]

Hi.
@aj-potc, I think you are facing another problem unrelated to the one described by @bickford.

The kernel module tracks activity on the block layer. The nature of the data doesn't matter to him, and the file system doesn't matter.
But the change tracker works with blocks. An entire block is marked as changed if at least one sector has changed in it. Therefore, highly fragmented data can increase the amount of incremental backup data.

Changing the file access time causes the file metadata to change. That is, one or four kilobytes of file system metadata are updated each time an image is read. I think that the images are read in random order, which means that the metadata changes throughout the disk.

Try mount with "-o noatime".
This will reduce the number of writes to disk. It will increase system performance.

Of course, I can't guarantee that this is the problem. This is just a guess. To identify the cause of the problem, you will need to monitor and analyze the changed data. See: blktrace.

@bickford - we are still interested in your detailed logs.

[aj-potc]

Hi @Fantu and @SergeiShtepa,

Thanks for your replies, and especially for confirming my suspicion that the modification of the access time could be the issue.

I've currently got the volume mounted with relatime, but will try remounting with noatime and see what effect it has after a full 24-hour cycle.

Hopefully this will be successful, as I can find no other reason so much data would be changing. Since you feel this issue is not related to the one described by @bickford, I won't add anything else.

Thanks again!

[bickford]

@bickford - we are still interested in your detailed logs.

Hi again Sergei,

I've logged a case with Veeam just now. I initially thought as a Free version user I didn't have access to log a case with Veeam - but I checked again now and see that's incorrect and that it's 'best efforts' basis.

In the interim I've updated the server to Ubuntu 24.04.2 LTS to see if that improves things - no change.
Also changed my BIOS settings back to Secure Boot: disabled - no change.

At the moment I am trying to see if I can find anything on the system that would be making sudden increases in size - eg programs having errors dumping excess logs. I noticed Nomachine was creating some significant logs so I've uninstalled it to see if it improves the issue. Will update back in a day or two to let you know if I see reductions in backup size or with feedback from Veeam support.

Thanks again

[aj-potc]

Hi @bickford,

To search for changes on disk, I successfully used the following commands:

For modified files:
find /data -type f -mtime -1

And what really helped me nail down the problem was looking for files with a changed access time:
find /data -type f -atime -1 -exec ls -lh --time=atime {} +

Those may be useful for you as well.

[Flole998]

Is there any chance the old behavior from veeamsnap could be implemented in blksnap? I assume there is no way to inform blksnap from a filesystem driver level that a changed sector only includes metadata changes? I'm thinking about passing down a flag that indicates a metadata only change, but that'll be a major refactoring of all drivers I suppose.

[bickford]

Just an update from my end, nothing I've tried (above) has improved the size of backups after moving to blksnap. They are still about normal size for monthly, but 100-fold for incrementals. So my storage for 2 months of backups for this machine has gone from a steady ~44GB to a steady ~150GB.

Unfortunately the case I opened with Veeam support was closed with a note that they were too low on staff to take a look into it: "Unfortunately, due to low support staff availability we were unable to process your request. This support case has been archived automatically" (ticket #07623287 ).

I will live with it since 150GB is not a huge imposition on my NAS. I am kinda surprised that more people are not affected as my setup is pretty vanilla - Ubuntu Desktop v24 LTS, Docker running a few microservices (Navidrome, Nextcloud for 2 users syncing very little data [contacts & tasks], Nginx Proxy Manager, Portainer, Watchtower). The hardware is not exotic either, just a Thinkcentre SFF PC with a SATA drive (formatted in ext4), backing up to SMB NAS over cabled LAN.

Thanks again for the help & suggestions all.

Current screenshot:

[SergeiShtepa]

Hi.

I spoke with QA about this case — they were unable to reproduce your scenario.
Possibly because they didn't have Docker with micro services in their setup.

Thank you.

[Flole998]

@bickford If this is an issue for you and you need an immediate fix: I have ported veeamsnap to the 6.8 kernel, it's available at https://github.com/Flole998/veeamsnap and I am running it on Ubuntu 24.04 as I am also having issues with blksnap.