Retention policy seems not working

[http403]

I have configured resticprofile and run for a bit of time, and I found out that the retention policy didn't execute. At least doesn't remove snapshots according to the policy after backup. I tried doing backup manually and use forget with dry run afterward. There are snapshots need to be removed, proofing the retention policy is working in the way I expected. Before calling it a bug, I would like to make sure my configuration is correct, and my understanding of retention is right.

base config

#:schema https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json

version = "1"

includes = ["profile.d/*.toml"]

{{ define "discord_webhook_url" }}""{{ end }}

{{ define "hook_discord" }}
method = "POST"
url = {{ template "discord_webhook_url" . }}
body-template = "templates/discord-webhook.json"
{{ end }}

{{ define "hook_discord_fail" }}
method = "POST"
url = {{ template "discord_webhook_url" . }}
body-template = "templates/discord-webhook-fail.json"
{{ end }}

{{ define "hook_header_json" }}
name = "Content-Type"
value = "application/json"
{{ end }}

[global]
restic-stale-lock-age = "45m"

[base]
lock = "{{ .TempDir }}/resticprofile-{{ .Profile.Name }}.lock"

[base.backup]
check-after = true
schedule = "*-*-* *:00:00"
schedule-permission = "user"

{{ if .Env.DEBUG }}
[base.backup.send-after]
{{ template "hook_discord" . }}

[[base.backup.send-after.headers]]
{{ template "hook_header_json" . }}
{{ end }}

[base.backup.send-after-fail]
{{ template "hook_discord_fail" . }}

[[base.backup.send-after-fail.headers]]
{{ template "hook_header_json" . }}

[base.prune]
schedule = "*-*-01 14:30:00"
schedule-permission = "user"

{{ if .Env.DEBUG }}
[base.prune.send-after]
{{ template "hook_discord" . }}

[[base.prune.send-after.headers]]
{{ template "hook_header_json" . }}
{{ end }}

[base.prune.send-after-fail]
{{ template "hook_discord_fail" . }}

[[base.prune.send-after-fail.headers]]
{{ template "hook_header_json" . }}

{{ if .Env.DEBUG }}
[base.check.send-after]
{{ template "hook_discord" . }}

[[base.check.send-after.headers]]
{{ template "hook_header_json" . }}
{{ end }}

[base.check.send-after-fail]
{{ template "hook_discord_fail" . }}

[[base.check.send-after-fail.headers]]
{{ template "hook_header_json" . }}

[base.retention]
after-backup = true

profile `personal` config

#:schema https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json

version = "1"

[personal]
inherit = "base"
repository = ""
password-file = "profile.d/personal.passwd"

[personal.env]
AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""

[personal.backup]
# verbose = true
use-fs-snapshot = true
exclude-file = "profile.d/personal.exclude"
exclude-caches = true
source = [ 
    '{{ .Env.USERPROFILE | replace "\\" "\\\\" }}',
]

[personal.forget]
group-by = "host,paths"
keep-hourly = 8
keep-daily = 7
keep-weekly = 4
keep-monthly = 12
keep-yearly = 5

[personal.retention]
group-by = "host,paths"
keep-hourly = 8
keep-daily = 7
keep-weekly = 4
keep-monthly = 12
keep-yearly = 5

[jkellerer]

Hi @http403 , can't find any obvious mistakes in your config. Could you share the non-confidential parts of resticprofile --dry-run personal.backup and some info on snapshots and which you'd expect to get removed?

[http403]

@jkellerer Of course. Those logs are too long so I uploaded to a gist.

As you can see, the forget command is being ran right after backup. However, when I manually run forget, it shows it accumulated 72 snapshots need to be removed. They shouldn't be there and should be all gone according to the policy when the backup command is ran.

[jkellerer]

Sorry for the late reply.

I think the problem is related to "%USERPROFILE%". Some time ago we started to process environment variables in paths and arguments since the may or may not be resolved in the shell. For example they are resolved if they stand on its own (e.g. for backup) but not when specified as "--path=%USERPROFILE%", so we decided to resolve them prior to creating the restic commandline.

That said, we use the built-in functionality of go os.Expand to expand variables and this only supports $VARIABLE and ${VARIABLE} but not %VARIABLE% nor !VARIABLE!.

As a workaround you could use ${USERPROFILE} instead of %USERPROFILE% and backup as well as forget should work fine as resticprofile will then resolve it prior to creating the commandlines.

[jkellerer]

Just reviewed the config again.. looks like you had replaced paths with %USERPROFILE% just as placeholder and it isn't really used. Am I correct? So the actual issue may be elsewhere.

[http403]

Yes, %USERPROFILE% is just a placeholder. Resticprofile resolve the actual path correctly.

[KreativeKrise]

For me the retention policy also seems not to work.
I'm running resticprofile v0.29.0.

Here is my dry-run log. It seems that the retention command was not executed.

/resticprofile # resticprofile --name "home_stefan_onedrive" --dry-run
2024/12/02 13:58:05 using configuration file: profiles.yaml
2024/12/02 13:58:05 profile 'home_stefan_onedrive': initializing repository (if not existing)
2024/12/02 13:58:05 dry-run: /usr/bin/restic init --password-file=/volume1/homes/stefan/.config/restic/password_onedrive.txt --repo=rclone:onedrive_onedrvbackup2:restic/
2024/12/02 13:58:05 profile 'home_stefan_onedrive': starting 'snapshots'
2024/12/02 13:58:05 dry-run: /usr/bin/restic snapshots --password-file=/volume1/homes/stefan/.config/restic/password_onedrive.txt --repo=rclone:onedrive_onedrvbackup2:restic/
2024/12/02 13:58:05 profile 'home_stefan_onedrive': finished 'snapshots'
2024/12/02 13:58:05 dry-run: chown 1024:100 /root/.config/rclone/rclone.conf && chmod a+rwx /root/.config/rclone/rclone.conf 

profile.yaml

global:
  initialize: true
  scheduler: crond
  verbose: true

default:
  force-inactive-lock: true
  
  backup:
    check-after: true
    exclude-caches: true
    schedule-lock-wait: 15m
    schedule-permission: system
    
  check:
    schedule-lock-wait: 15m
    schedule-permission: system
    
  retention:
    before-backup: false
    after-backup: true
    prune: true

home_stefan_onedrive:
  inherit: default
  
  password-file: "/volume1/homes/stefan/.config/restic/password_onedrive.txt"
  repository: "rclone:onedrive_onedrvbackup2:restic/"
  
  run-finally:
    - "chown 1024:100 /root/.config/rclone/rclone.conf && chmod a+rwx /root/.config/rclone/rclone.conf"
  
  backup:
    source:
      - "/volume1/homes/stefan/"
    iexclude-file: "/volume1/homes/stefan/.config/restic/exclude.txt"
    schedule: "03:00"
  
  check:    
    read-data-subset: "{{ len (printf "a%*s" .Now.Weekday "") }}/7"
    
  retention:
    keep-daily: 28

[creativeprojects]

You didn't specify a command to run (like backup) so the default command is snapshots.

As a matter of fact, the log shows the execution of the snapshots command.

The retention only runs before or after the backup command.

Try with

resticprofile --name "home_stefan_onedrive" --dry-run backup

[KreativeKrise]

You're absolutely right. I just forgot the backup command. Sorry for the interruption!