Migrate EZID Stage RDS to graviton

[jsjiang]

Created IAS ticket for migrating EZID Stage RDS to graviton: https://github.com/cdlib/cdlsys/issues/968

Tasks includes:

• refresh the stage database from production database - IAS
• restore passwords - EZID
• perform functional tests - EZID
• perform performance tests - EZID
• migrate stage RDS to graviton - IAS
• post-migration functional tests - EZID
• post-migration performance tests - EZID
• report and resolve issues if needed - IAS/EZID
• recreate OpenSearch indexes

[jsjiang]

Tasks to perform after refreshing the stage database from production database:

• reset the password for the database users
  • ezidro
  • ezidrw
  • eziddba
• reset the password for the ezid app admin account
• the apitest account password: no changes as the prd and stg shares the same password
• recreate OpenSearch indexes

How to change MySQL user password

[jsjiang]

Tests with prd datasets on current database instance type:

• EZID functional tests - Done
• EZID UI tests - Done
• Performance and get statistics (a week)
  • load tests with Locust - Done
  • background jobs - Need to prepare large test datasets for the background jobs to process. Possible data sources:
    • datasets produced by the load tests with Locust
    • the link checker job - it is disable on stage. The first step which is preparing datasets for link checker requires a lot of database resources and can be used for load test. The url checking function should be avoided as we do not want to add unnecessary load to the target sites.
      • started link checker on stg: 12/15, 2:30pm
      • stopped: 6:15pm (3 hours 45min)
      • code: a copy of ezid in /ezid/tmp/ezid
      • command: screen python manage.py proc-link-checker
    • recreating OpenSearch indexes
      • started on stg: 12/15, 3:15pm
      • code: a copy of ezid in /ezid/tmp/ezis
      • command: screen python -u manage.py opensearch-update > ../logs/opensearch-reindex.log 2>&1
      • stop in 1 or 2 days to save at least half of the index work for post-migration testing use
      • terminated due to low disk space issue at 10:45pm (ran for 7 hours)

Issues:

• The ezid-stg01 instance reported out-of-disk alert at 7:30am 12/16
• The openserach-update process was terminated (at 10:45pm) probably due to low disk issue
  • last a few identifiers processed:
    • ark:/28722/k2086fd0d: ID=2422093
    • ark:/28722/k2086fd5t: ID=2422098
    • ark:/28722/k2086ff0v: ID=2422103
  • processed 2.3m identifiers (out of (total 36.6m) in 7 hours ;
• Large log files were found in the /ezid/tmp/logs directory

Log files

[root@uc3-ezidui-stg01 tmp]# cd logs/
[root@uc3-ezidui-stg01 logs]# du -sh * | grep G
9.1G    ecs_json.log
4.0G    request.log
4.1G    trace.log

Log entries for processed identifiers:

tail -1000 request.log | grep ezid-app-search-stg-alias
... ...
2025-12-15 20:45:27,548    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086f902"}}
2025-12-15 20:45:27,599    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086f95g"}}
2025-12-15 20:45:27,654    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fb0h"}}
2025-12-15 20:45:27,714    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fb5x"}}
2025-12-15 20:45:27,771    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fc0z"}}
2025-12-15 20:45:27,827    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fc5c"}}
2025-12-15 20:45:27,886    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fd0d"}}
2025-12-15 20:45:27,941    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086fd5t"}}
2025-12-15 20:45:27,997    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2086ff0v"}}

Possible solutions:

• request more disk space when running full reindexing
• run smaller batches by limiting run time, by ID or by timestamp
• trim log files

[jsjiang]

Note:

• Stopped EZID services on stage
  • triggered Nagios alerts
  • need to learn how to schedule downtime in escalope https://escalope.cdlib.org/
• TO-DO: schedule downtime in escalope for Stg & prod RDS migration

How to pause alerts in escalope:

• logon to https://escalope.cdlib.org/
• click on Blocks in the left-hand menu bar, then create a new block specifying a time period and a host pattern

Hosts: uc3-ezidui-stg*

Services: (any)

Start: 2025-12-16 16:55

End: 2026-01-05 09:00

Active
Delete
Created by: jjiang at 2025-12-16 16:57

Comment: Testing RDS migration

[jsjiang]

Database insights on current platform:

[jsjiang]

EZID Stage RDS Graviton migration

• stopped ezid services at 8:55am
• RDS was migrated on 12/17, at 9am
• restarted ezid services 9:15am
• UI and functional tests: passed

[jsjiang]

Performance tests:

• monitor RDS performance in Grafana
• run load tests using Locust
• start the link checker background job and have it up and running for 3 hours
• run the OpenSearch index job for about 7 hours (starting ID=2422103)

[jsjiang]

Compare load tests:

Platform	Date	Request per second	50%ile Response time (ms)	95%ile Response time (ms)
Intel	2025-12-15	2.07	4,600	13,000
Graviton	2025-12-17	1.54	3,000	21,000
Graviton	2025-12-18	2.09	5,000	13,000

Note:

• 2025-12-17 test results showed slightly degraded performance on Graviton
• to-do: perform another set of tests
  • 2025-12-18 test results were close to pre-migration results

Intel platform:

Graviton 12/17

Graviton 12/18

[jsjiang]

Link checker

• code: a copy of ezid in /ezid/tmp/ezid
  • modified the settings/settings.py file to have the ecs_json.log and trace.log disabled. Only allowed the request.log
• started link checker on stg: 12/17, 1:00pm
• stopped: 5pm (run for 4 hours)
• command:

screen python manage.py proc-link-checker

1486747.pts-1.uc3-ezidui-stg01	(Detached)

[jsjiang]

Recreate OpenSearch index:

• code: a copy of ezid in /ezid/tmp/ezid
  • modified the settings/settings.py file to have the ecs_json.log and trace.log disabled. Only allowed the request.log
• started: 12/17, 1:20pm
• stop around 8:30pm (run for 7 hours)
• command:

screen python manage.py opensearch-update --starting_id 2422103

[detached from 1487937.pts-4.uc3-ezidui-stg01]

Issues with the request.log

• only logged 432,232 indexed log entries;
• the identifiers' IDs are not ordered in ascending order
  • the first 5 identifiers' ID are in the 326xxxx range and the last 5 are in the 160xxxx range

ezid@uc3-ezidui-stg01:11:19:41:~/tmp/logs$  grep ezid-app-search-stg-alias request.log | grep DEBUG | wc -l
432232
ezid@uc3-ezidui-stg01:11:20:25:~/tmp/logs$  grep ezid-app-search-stg-alias request.log | grep DEBUG | head -5
2025-12-17 13:19:12,607    DEBUG https://search-os-uc3-logging-stg-suwz42vownvyte6ivqazeifz44.us-west-2.es.amazonaws.com:443 "HEAD /ezid-app-search-stg-alias HTTP/1.1" 200 0
2025-12-17 13:19:12,804    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2v98344g"}}
2025-12-17 13:19:12,885    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2v98349w"}}
2025-12-17 13:19:12,940    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2v98354x"}}
2025-12-17 13:19:13,011    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/28722/k2v98359b"}}

ezid@uc3-ezidui-stg01:11:20:55:~/tmp/logs$  grep ezid-app-search-stg-alias request.log | grep DEBUG | tail -5
2025-12-17 20:58:49,916    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "doi:10.18739/A2V94J"}}
2025-12-17 20:58:49,975    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/13030/qt7494c1dn"}}
2025-12-17 20:58:50,037    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/13030/qt207330cx"}}
2025-12-17 20:58:50,106    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/13030/qt1bh705mk"}}
2025-12-17 20:58:50,172    DEBUG > {"index": {"_index": "ezid-app-search-stg-alias", "_id": "ark:/13030/qt68p819kd"}}

TO-DO: rerun the OpenSearch Index job to find out

• how identifiers are picked up for processing
• how may identifiers can be process in an hour
• how to track job progress by monitoring lob files such as
  • request.log
  • screen output send to a log file

[jsjiang]

Database insights on current & graviton:

• activities were created by the link checker and OpenSearch indexing
• no noticeable pattern changes

Top SQLs Image 1:

Image 2:

Image 3:

Top waits image 1:

image 2: