Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement multithreading for database checks with pgbackrest_auto #18

Merged
merged 6 commits into from
Nov 30, 2023
Merged

Implement multithreading for database checks with pgbackrest_auto #18

merged 6 commits into from
Nov 30, 2023

Conversation

vitabaks
Copy link
Owner

@vitabaks vitabaks commented Nov 29, 2023 •
edited
Loading

  • add 'process-max' option
  • add multithreading to "dummy-dump" chek
  • add multithreading to "amcheck" chek

Additionally:

  • Fix compatibility of "checksums" checks with Postgres 11 and older

@vitabaks vitabaks self-assigned this Nov 29, 2023
@vitabaks vitabaks added the enhancement New feature or request label Nov 29, 2023
@vitabaks
Copy link
Owner Author

vitabaks commented Nov 30, 2023 •
edited
Loading

Test

pgbackrest_auto --config=/etc/pgbackrest/pgbackrest.conf --from=apdb-cluster --to=/var/lib/postgresql/rst/apdb-cluster --checkdb --clear --process-max=4

2023-11-30 14:00:20 INFO: [STEP 1]: Starting
2023-11-30 14:00:20 INFO: Starting. Restore Type: Full PostgreSQL Restore FROM Stanza: apdb-cluster --> TO Directory: /var/lib/postgresql/rst/apdb-cluster
2023-11-30 14:00:20 INFO: Starting. Restore Settings: immediate
2023-11-30 14:00:20 INFO: Starting. Run settings: Log: /var/log/pgbackrest/pgbackrest_auto_apdb-cluster.log
2023-11-30 14:00:20 INFO: Starting. Run settings: Lock run: /tmp/pgbackrest_auto_apdb-cluster.lock
2023-11-30 14:00:20 INFO: Starting. PostgreSQL version: 10
2023-11-30 14:00:20 INFO: Starting. PostgreSQL data directory: /var/lib/postgresql/rst/apdb-cluster
2023-11-30 14:00:20 INFO: Starting. PostgreSQL Database Validation: yes
2023-11-30 14:00:20 INFO: Starting. Clear Data Directory after restore: yes
2023-11-30 14:00:20 WARN: Restoring to /var/lib/postgresql/rst/apdb-cluster Waiting 30 seconds. The directory will be overwritten. If mistake, press ^C
2023-11-30 14:00:50 INFO: [STEP 2]: Stopping PostgreSQL
2023-11-30 14:00:50 INFO: attempt: 1/3600
2023-11-30 14:00:50 INFO: PostgreSQL check status
2023-11-30 14:00:50 INFO: [STEP 3]: Restoring from backup
2023-11-30 14:00:50 INFO: See detailed log in the file /var/log/pgbackrest/apdb-cluster-restore.log
2023-11-30 14:00:50 INFO: Restore from backup started. Type: Full PostgreSQL Restore
pgbackrest --config=/etc/pgbackrest/pgbackrest.conf --stanza=apdb-cluster --pg1-path=/var/lib/postgresql/rst/apdb-cluster  --type=immediate --delta restore --process-max=4 --log-level-console=error --log-level-file=detail --recovery-option=recovery_target_action=promote --tablespace-map-all=/var/lib/postgresql/rst/apdb-cluster_remapped_tablespaces
WARN: configuration file contains invalid option 'include-path'
2023-11-30 14:01:32 INFO: Restore from backup done
2023-11-30 14:01:32 INFO: [STEP 4]: PostgreSQL Starting for recovery
2023-11-30 14:01:32 INFO: Starting PostgreSQL on port 5432
2023-11-30 14:01:33 INFO: attempt: 1/3600
2023-11-30 14:01:33 INFO: PostgreSQL instance 5432 started and accepting connections
2023-11-30 14:01:33 INFO: [STEP 5]: PostgreSQL Recovery Checking
2023-11-30 14:01:33 INFO: Checking if restoring from archive is done
2023-11-30 14:01:33 INFO: Replayed: 2023-11-29 22:01:55.196855+00
2023-11-30 14:01:33 INFO: Restoring from archive is done
2023-11-30 14:01:33 INFO: [STEP 6]: Verify data checksums
2023-11-30 14:01:34 INFO: PostgreSQL stop
2023-11-30 14:01:34 INFO: PostgreSQL instance stopped
2023-11-30 14:01:34 INFO: pg_checksums: starting data checksums validation
2023-11-30 14:02:29 INFO: pg_checksums: data checksums validation - Successful
2023-11-30 14:02:29 INFO: [STEP 7]: Verify that data can be read out
2023-11-30 14:02:29 INFO: Starting PostgreSQL on port 5432
2023-11-30 14:02:30 INFO: Start data validation for database postgres
2023-11-30 14:02:30 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d postgres -F d -j 4 -f /var/lib/postgresql/rst/apdb-cluster/dump/postgres
2023-11-30 14:02:31 INFO: Data validation in the database postgres - Successful
2023-11-30 14:02:31 INFO: Start data validation for database applatform
2023-11-30 14:02:31 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d applatform -F d -j 4 -f /var/lib/postgresql/rst/apdb-cluster/dump/applatform
2023-11-30 14:04:18 INFO: Data validation in the database applatform - Successful
2023-11-30 14:04:18 INFO: [STEP 8]: Verify indexes
2023-11-30 14:04:20 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database postgres
2023-11-30 14:04:21 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database applatform
2023-11-30 14:09:01 INFO: [STEP 9]: Stopping PostgreSQL and Clear Data Directory
2023-11-30 14:09:01 INFO: PostgreSQL stop
2023-11-30 14:09:01 INFO: PostgreSQL instance stopped
2023-11-30 14:09:01 INFO: attempt: 1/3600
2023-11-30 14:09:01 INFO: PostgreSQL check status
2023-11-30 14:09:01 INFO: PostgreSQL instance not running
2023-11-30 14:09:05 INFO: Finish

passed

@vitabaks
Copy link
Owner Author

Test 2


postgres@smh-backuper:~$ pgbackrest_auto --config=/etc/pgbackrest/pgbackrest.conf --from=shdb-cluster --to=/var/lib/postgresql/rst/shdb-cluster --checkdb --clear --process-max4

2023-11-30 14:10:15 INFO: [STEP 1]: Starting
2023-11-30 14:10:15 INFO: Starting. Restore Type: Full PostgreSQL Restore FROM Stanza: shdb-cluster --> TO Directory: /var/lib/postgresql/rst/shdb-cluster
2023-11-30 14:10:15 INFO: Starting. Restore Settings: immediate
2023-11-30 14:10:15 INFO: Starting. Run settings: Log: /var/log/pgbackrest/pgbackrest_auto_shdb-cluster.log
2023-11-30 14:10:15 INFO: Starting. Run settings: Lock run: /tmp/pgbackrest_auto_shdb-cluster.lock
2023-11-30 14:10:15 INFO: Starting. PostgreSQL version: 14
2023-11-30 14:10:15 INFO: Starting. PostgreSQL data directory: /var/lib/postgresql/rst/shdb-cluster
2023-11-30 14:10:15 INFO: Starting. PostgreSQL Database Validation: yes
2023-11-30 14:10:15 INFO: Starting. Clear Data Directory after restore: yes
2023-11-30 14:10:15 WARN: Restoring to /var/lib/postgresql/rst/shdb-cluster Waiting 30 seconds. The directory will be overwritten. If mistake, press ^C
2023-11-30 14:10:45 INFO: [STEP 2]: Stopping PostgreSQL
2023-11-30 14:10:45 INFO: attempt: 1/3600
2023-11-30 14:10:45 INFO: PostgreSQL check status
2023-11-30 14:10:45 INFO: [STEP 3]: Restoring from backup
2023-11-30 14:10:45 INFO: See detailed log in the file /var/log/pgbackrest/shdb-cluster-restore.log
2023-11-30 14:10:45 INFO: Restore from backup started. Type: Full PostgreSQL Restore
pgbackrest --config=/etc/pgbackrest/pgbackrest.conf --stanza=shdb-cluster --pg1-path=/var/lib/postgresql/rst/shdb-cluster  --type=immediate --delta restore --process-max=1 --log-level-console=error --log-level-file=detail --recovery-option=recovery_target_action=promote --tablespace-map-all=/var/lib/postgresql/rst/shdb-cluster_remapped_tablespaces
WARN: configuration file contains invalid option 'include-path'
WARN: --delta or --force specified but unable to find 'PG_VERSION' or 'backup.manifest' in '/var/lib/postgresql/rst/shdb-cluster' to confirm that this is a valid $PGDATA directory. --delta and --force have been disabled and if any files exist in the destination directories the restore will be aborted.
2023-11-30 14:11:29 INFO: Restore from backup done
2023-11-30 14:11:29 INFO: [STEP 4]: PostgreSQL Starting for recovery
2023-11-30 14:11:29 INFO: Starting PostgreSQL on port 5432
2023-11-30 14:11:30 INFO: attempt: 1/3600
2023-11-30 14:11:30 INFO: PostgreSQL instance 5432 started and accepting connections
2023-11-30 14:11:30 INFO: [STEP 5]: PostgreSQL Recovery Checking
2023-11-30 14:11:30 INFO: Checking if restoring from archive is done
2023-11-30 14:11:30 INFO: Replayed: 2023-11-30 00:00:25.639085+00
2023-11-30 14:11:30 INFO: Restoring from archive is done
2023-11-30 14:11:30 INFO: [STEP 6]: Verify data checksums
2023-11-30 14:11:30 INFO: PostgreSQL stop
2023-11-30 14:11:30 INFO: PostgreSQL instance stopped
2023-11-30 14:11:30 INFO: pg_checksums: starting data checksums validation
2023-11-30 14:11:32 INFO: pg_checksums: data checksums validation - Successful
2023-11-30 14:11:32 INFO: [STEP 7]: Verify that data can be read out
2023-11-30 14:11:32 INFO: Starting PostgreSQL on port 5432
2023-11-30 14:11:33 INFO: Start data validation for database voice-assistant
2023-11-30 14:11:33 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d voice-assistant >> /dev/null
2023-11-30 14:11:33 INFO: Data validation in the database voice-assistant - Successful
2023-11-30 14:11:33 INFO: Start data validation for database lambda-pechkin
2023-11-30 14:11:33 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d lambda-pechkin >> /dev/null
2023-11-30 14:11:42 INFO: Data validation in the database lambda-pechkin - Successful
2023-11-30 14:11:42 INFO: Start data validation for database errmap
2023-11-30 14:11:42 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d errmap >> /dev/null
2023-11-30 14:11:43 INFO: Data validation in the database errmap - Successful
2023-11-30 14:11:43 INFO: Start data validation for database pechkin
2023-11-30 14:11:43 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d pechkin >> /dev/null
2023-11-30 14:11:44 INFO: Data validation in the database pechkin - Successful
2023-11-30 14:11:44 INFO: Start data validation for database smarthome
2023-11-30 14:11:44 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d smarthome >> /dev/null
2023-11-30 14:11:53 INFO: Data validation in the database smarthome - Successful
2023-11-30 14:11:53 INFO: Start data validation for database monitor
2023-11-30 14:11:53 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d monitor >> /dev/null
2023-11-30 14:11:53 INFO: Data validation in the database monitor - Successful
2023-11-30 14:11:53 INFO: Start data validation for database smarthomealice_old
2023-11-30 14:11:53 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d smarthomealice_old >> /dev/null
2023-11-30 14:11:53 INFO: Data validation in the database smarthomealice_old - Successful
2023-11-30 14:11:53 INFO: Start data validation for database postgres
2023-11-30 14:11:53 INFO:   starting pg_dump -p 5432 -h 127.0.0.1 -d postgres >> /dev/null
2023-11-30 14:11:53 INFO: Data validation in the database postgres - Successful
2023-11-30 14:11:53 INFO: [STEP 8]: Verify indexes
2023-11-30 14:11:54 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database voice-assistant
  0/271 relations (0%),    0/1029 pages (0%)
271/271 relations (100%), 1029/1029 pages (100%)
2023-11-30 14:11:54 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database lambda-pechkin
  0/283 relations (0%),     0/85400 pages (0%)
  2/283 relations (0%), 84094/85400 pages (98%)
  3/283 relations (1%), 84200/85400 pages (98%)
283/283 relations (100%), 85400/85400 pages (100%)
2023-11-30 14:11:59 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database errmap
  0/324 relations (0%),    0/1093 pages (0%)
324/324 relations (100%), 1093/1093 pages (100%)
2023-11-30 14:11:59 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database pechkin
  0/280 relations (0%),     0/17916 pages (0%)
  3/280 relations (1%), 16257/17916 pages (90%)
  5/280 relations (1%), 16930/17916 pages (94%)
280/280 relations (100%), 17916/17916 pages (100%)
2023-11-30 14:12:01 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database smarthome
  0/292 relations (0%),      0/162927 pages (0%)
  3/292 relations (1%), 106869/162927 pages (65%)
  4/292 relations (1%), 126256/162927 pages (77%)
  5/292 relations (1%), 140142/162927 pages (86%)
  6/292 relations (2%), 149115/162927 pages (91%)
  7/292 relations (2%), 158085/162927 pages (97%)
  8/292 relations (2%), 160939/162927 pages (98%)
  9/292 relations (3%), 161748/162927 pages (99%)
292/292 relations (100%), 162927/162927 pages (100%)
2023-11-30 14:12:24 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database monitor
  0/264 relations (0%),    0/6346 pages (0%)
  5/264 relations (1%), 5663/6346 pages (89%)
264/264 relations (100%), 6346/6346 pages (100%)
2023-11-30 14:12:25 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database smarthomealice_old
  0/265 relations (0%),    0/1002 pages (0%)
265/265 relations (100%), 1002/1002 pages (100%)
2023-11-30 14:12:25 INFO: amcheck: verify the logical consistency of the structure of indexes and heap relations in the database postgres
  0/259 relations (0%),   0/954 pages (0%)
259/259 relations (100%), 954/954 pages (100%)
2023-11-30 14:12:25 INFO: [STEP 9]: Stopping PostgreSQL and Clear Data Directory
2023-11-30 14:12:25 INFO: PostgreSQL stop
2023-11-30 14:12:26 INFO: PostgreSQL instance stopped
2023-11-30 14:12:26 INFO: attempt: 1/3600
2023-11-30 14:12:26 INFO: PostgreSQL check status
2023-11-30 14:12:26 INFO: PostgreSQL instance not running
2023-11-30 14:12:27 INFO: Finish

passed

@vitabaks vitabaks merged commit f0b9307 into master Nov 30, 2023
1 check passed
@vitabaks vitabaks deleted the process-max branch November 30, 2023 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@vitabaks vitabaks

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vitabaks