Once bootstrapping failed, core fails to start

[Nuc1eoN]

When the initial bootstrapping process fails for some reason (eg #94), it is impossible to launch the core afterwards (even though always shutting down gracefully with stop.sh).

There is the following error message upon launch:

Logs:

2022-09-24 06:56:53 INFO  Controller:366 - Starting up...
2022-09-24 06:56:54 INFO  Controller:262 - Build timestamp: 20220919162926
2022-09-24 06:56:54 INFO  Controller:274 - Build version: qortal-3.6.0-b99b1f5
2022-09-24 06:56:54 INFO  Settings:431 - Using settings file: settings.json
2022-09-24 06:56:54 INFO  BlockChain:278 - Using default, resources-based blockchain config
2022-09-24 06:56:54 INFO  Controller:262 - Build timestamp: 20220919162926
2022-09-24 06:56:54 INFO  Controller:274 - Build version: qortal-3.6.0-b99b1f5
2022-09-24 06:56:54 INFO  Controller:387 - Starting NTP
2022-09-24 06:56:54 INFO  Controller:394 - Starting repository
2022-09-24 06:56:54 INFO  HSQLDBDatabaseUpdates:834 - Altering AT table in repository - this might take a while... (approx. 20 seconds on high-spec)
2022-09-24 06:56:54 ERROR Controller:410 - Unable to start repository
org.qortal.repository.DataException: Repository initialization error
	at org.qortal.repository.hsqldb.HSQLDBRepositoryFactory.<init>(HSQLDBRepositoryFactory.java:71) ~[qortal.jar:3.6.0]
	at org.qortal.controller.Controller.main(Controller.java:396) [qortal.jar:3.6.0]
Caused by: java.sql.SQLSyntaxErrorException: object name already exists in statement [ALTER TABLE ATs ADD sleep_until_message_timestamp BIGINT]
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBDatabaseUpdates.databaseUpdating(HSQLDBDatabaseUpdates.java:835) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBDatabaseUpdates.updateDatabase(HSQLDBDatabaseUpdates.java:34) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBRepositoryFactory.<init>(HSQLDBRepositoryFactory.java:69) ~[qortal.jar:3.6.0]
	... 1 more
Caused by: org.hsqldb.HsqlException: object name already exists
	at org.hsqldb.error.Error.error(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.error.Error.error(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.TableWorks.checkAddColumn(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.TableWorks.addColumn(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.StatementSchema.getResult(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.StatementSchema.execute(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.Session.executeDirectStatement(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.Session.execute(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source) ~[qortal.jar:3.6.0]
	at org.hsqldb.jdbc.JDBCStatement.execute(Unknown Source) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBDatabaseUpdates.databaseUpdating(HSQLDBDatabaseUpdates.java:835) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBDatabaseUpdates.updateDatabase(HSQLDBDatabaseUpdates.java:34) ~[qortal.jar:3.6.0]
	at org.qortal.repository.hsqldb.HSQLDBRepositoryFactory.<init>(HSQLDBRepositoryFactory.java:69) ~[qortal.jar:3.6.0]
	... 1 more

The workaround is to delete the db folder and relaunch.

Expected results:
Core should not fail to start with such a cryptic error. It should recognize the issue and "self heal" (or shut down gracefully in the first place).

[crowetic]

this will depend on WHEN the bootstrap process fails... accommodating for EVERY potential bootstrap failure in any potential stage, I'm not sure is entirely possible, but we'll let someone with more in-depth knowledge of this come in...

from the log it seems like you tried to start it manually AFTER a failed bootstrap, and we have no idea at which point during the bootstrap it failed. Accommodation for something like this (bootstrap failure, then complete restart) would be quite difficult I would imagine, as the core wouldn't even KNOW that you recently attempted a bootstrap in a prior start attempt. It would have to be done during the SAME attempt as the failure I would think... so I don't see this being something that would be possible to 'resolve'.

But again, we'll see what the others have to say.