cli: temp dir should be cleared before disk full warning on startup

[1lann]

Currently, if CockroachDB is run with the --temp-dir being on the same disk as --store and an intensive operation is ran that spills to disk more than the available remaining disk capacity, CockroachDB will not automatically recover by clearing out the temporary directory on next start even if doing so would be sufficient to clear out the out of disk state. It would be ideal if it could as to prevent us from needing to manually delete the temporary files.

To Reproduce

1. Run CockroachDB without a specified --temp-dir so it defaults to the --store location, and a --max-disk-temp-storage more than the available remaining disk capacity.
2. Run an intensive query that spills to disk more than the remaining disk capacity.
3. CockroachDB crashes with exit code 10 (as expected when it runs out of disk).
4. On restart, CockroachDB fails fast with the error message that the disk is out of space, even though if it were to clear the --temp-dir it would have enough space to start.

Expected behavior

1. Run CockroachDB without a specified --temp-dir so it defaults to the --store location, and a --max-disk-temp-storage more than the available remaining disk capacity.
2. Run an intensive query that spills to disk more than the remaining disk capacity.
3. CockroachDB crashes with exit code 10 (as expected when it runs out of disk).
4. On restart, CockroachDB clears the --temp-dir and then checks if the disk is out of space, in which it finds it isn't, and starts normally.

Environment:

• CockroachDB version: 22.1.4
• Server OS: Docker image on amd64 Linux
• Client app: N/A

Jira issue: CRDB-18064

[agsti]

Hey @nicktrav

I'm taking a look at this

So far:

• I've found exitIfDiskFull which is what triggers early exit
• Also found fs.CleanupTempDirs which should be used to delete old temp folders i believe

Therefore:
Should I just create a method that gets called before exitIfDiskFull that clears temp paths under storage path?