qortal core freezes after being unable to delete transaction

[lifeboy]

The run.log shows this:

2021-02-09 16:31:49 ERROR Controller:814 - Repository issue while deleting expired unconfirmed transactions
org.qortal.repository.DataException: Unable to delete transaction from repository
at org.qortal.repository.hsqldb.transaction.HSQLDBTransactionRepository.delete(HSQLDBTransactionRepository.java:1275) ~[qortal.jar:1.4.1]
at org.qortal.controller.Controller.deleteExpiredTransactions(Controller.java:808) [qortal.jar:1.4.1]
at org.qortal.controller.Controller.run(Controller.java:569) [qortal.jar:1.4.1]
Caused by: java.sql.BatchUpdateException: transaction rollback: serialization failure
at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source) ~[qortal.jar:1.4.1]
at org.qortal.repository.hsqldb.HSQLDBRepository.executeCheckedBatchUpdate(HSQLDBRepository.java:657) ~[qortal.jar:1.4.1]
at org.qortal.repository.hsqldb.HSQLDBRepository.executeCheckedUpdate(HSQLDBRepository.java:633) ~[qortal.jar:1.4.1]
at org.qortal.repository.hsqldb.HSQLDBRepository.delete(HSQLDBRepository.java:747) ~[qortal.jar:1.4.1]
at org.qortal.repository.hsqldb.transaction.HSQLDBTransactionRepository.delete(HSQLDBTransactionRepository.java:1273) ~[qortal.jar:1.4.1]
... 2 more

There are no more entries in log.txt.1

[lifeboy]

Also, I see this when I start core.

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

Is that true and relevant?

[lifeboy]

Adding some setting to settings.json seems to have fixed the issue, but can we create a set of defautl values and comment the other options out, as is standard with config file? I could do that, except I don't know what the options are.

If there is a set of parameters listed somewhere, I get to work on it.

[archived-2]

Thanks - having them commented out by default sounds like a good approach. Or perhaps including an example settings file with them commented out, which keeps the actual settings.json clean.

Here's an example settings.json file I put together back in September 2020 (so it may be out of date):

{
    "bindAddress": "::",
    "uiPort": 12388,
    "uiLocalServers": [
        "localhost", "127.0.0.1", "172.24.1.1", "qor.tal"
    ],
    "uiRemoteServers": [
        "node1.qortal.org", "node2.qortal.org", "node3.qortal.org", "node4.qortal.org", "node5.qortal.org",
        "node6.qortal.org", "node7.qortal.org", "node8.qortal.org", "node9.qortal.org", "node10.qortal.org"
    ],
    "apiEnabled": true,
    "apiPort": 12391,
    "apiWhitelist": [
        "::1",
        "127.0.0.1"
    ],
    "apiRestricted": false,
    "apiLoggingEnabled": false,
    "apiDocumentationEnabled": false,
    "sslKeystorePathname": null,
    "sslKeystorePassword": null,
    "wipeUnconfirmedOnStart": false,
    "maxUnconfirmedPerAccount": 25,
    "maxTransactionTimestampFuture": 86400000,
    "autoUpdateEnabled": true,
    "repositoryBackupInterval": 0,
    "showBackupNotification": false,
    "isTestNet": false,
    "listenPort": 12392,
    "minBlockchainPeers": 5,
    "minOutboundPeers": 16,
    "maxPeers": 32,
    "maxNetworkThreadPoolSize": 20,
    "networkPoWComputePoolSize": 2,
    "slowQueryThreshold": null,
    "repositoryPath": "db",
    "autoUpdateRepos": [
        "https://github.com/Qortal/qortal/raw/%s/qortal.update",
        "https://[email protected]/Qortal/qortal/%s/qortal.update"
    ],
    "ntpServers": ["pool.ntp.org",
        "0.pool.ntp.org",
        "1.pool.ntp.org",
        "2.pool.ntp.org",
        "3.pool.ntp.org",
        "cn.pool.ntp.org",
        "0.cn.pool.ntp.org",
        "1.cn.pool.ntp.org",
        "2.cn.pool.ntp.org",
        "3.cn.pool.ntp.org"
    ],
    "testNtpOffset": null
}

You can see the default settings at the top of Settings.java, so it may be worth checking that all the keys and values are up to date. Also, I also have no idea if there are any errors in that JSON values that will cause problems - it was just put together quickly to give people an idea of what's available.

[lifeboy]

The file I got with the Ubuntu / Debian .deb contained only

{
}

Probably technically a bug then to be fixed in the next release.

[archived-2]

Yes all releases ship with an empty settings.json file right now - which is intended. From my point of view it would be nice to have an example file though, as otherwise it's not obvious what settings are available.

[catbref]

Caused by: java.sql.BatchUpdateException: transaction rollback: serialization failure

This exception is thrown by HSQLDB when different threads try to update/delete the same row.

In this case, one thread is trying to delete a transaction because it has 'expired' but it's not clear what the other thread is trying to do. From memory I think it might be possible that another thread has started an SQL 'transaction', read that Qortal 'Transaction' but not yet performed COMMIT/ROLLBACK and so held a form of row-lock, causing the exception.

The thread that is deleting expired transactions has captured this exception and not expired any transactions (this time).
There doesn't seem to be any easy way to find out what the other thread was doing via HSQLDB itself.

So maybe the best way forward here is to:

1. detect this specific situation in deleteExpiredTransactions and only log at INFO
2. improve the HSQLDB-interfacing code to detect "serialization failure"s and report other thread/SQL session states