chore: [ANDROSDK-2173] Handle FileResource status when posting files #2503
Conversation
…e status - Add FileResourceStorageStatusVerifier for batch verification of uploaded files - Add FileResourceVerificationResult data class to hold verification results - Add FileResourceUploadResult data class to track upload operations - Implement exponential backoff retry mechanism for verification - Support configurable batch size, max attempts, and delay parameters
…urce upload flows - Refactor DataValueFileResourcePostCall to use batch verification - Refactor OldTrackerImporterFileResourcesPostCall with helper functions - Refactor TrackerImporterFileResourcesPostCall to use batch verification - Update FileResourcePostCall with uploadFileResourceWithoutUpdate method - Extract helper functions to reduce complexity and improve readability - Change Triple to Pair where third parameter was redundant - Add @Suppress annotations for TooManyFunctions and TooGenericExceptionCaught
… verification - Add FileResourceStorageStatusVerifierShould test suite - Test batch verification with multiple files - Test verification with all files stored - Test verification with mixed statuses (stored, pending, failed) - Test verification timeout handling - Test empty batch handling - Test network error handling - Test single file verification - Use fake CoroutineAPICallExecutor to avoid Mockito matcher issues - Update tests in TrackerImporterFileResourcesPostCallShould.kt
|
|
||
| // If there are still pending files and we haven't reached max attempts, wait before next poll | ||
| if (pendingUids.isNotEmpty() && attempt < maxAttempts) { | ||
| delay((delayMs * (2.0.pow(attempt) - 1)).toLong()) |
There was a problem hiding this comment.
I think there's a critical bug in the exponential backoff calculation:
delay((delayMs * (2.0.pow(attempt) - 1)).toLong())
This formula grows exponentially without bounds:
- Attempt 10: ~17 minutes
- Attempt 20: ~12 days
- Attempt 30: ~34 years
With DEFAULT_MAX_ATTEMPTS = 30, this will cause the app to freeze indefinitely when files fail verification.
Maybe we could do a Linear backoff delay(delayMs * attempt)or cap the exponential backoff with a maxDelay like this:
val maxDelay = 10_000L
delay(minOf(delayMs * (2.0.pow(attempt - 1)).toLong(), maxDelay))
| * Verifies a single batch of file resources. | ||
| */ | ||
| private suspend fun verifyBatch(uids: List<String>): Map<String, FileResourceVerificationResult> { | ||
| return uids.associateWith { uid -> |
There was a problem hiding this comment.
I think the verifyBatch processes files sequentially rather than in parallel. With each verification taking 1-2 seconds, a batch of 10 files takes 10-20 seconds instead of a couple of seconds.
Should we try something like this?
private suspend fun verifyBatch(uids: List<String>): Map<String, FileResourceVerificationResult> {
return coroutineScope {
uids.map { uid ->
async { uid to verifyFileResource(uid) }
}.awaitAll().toMap()
}
}
There was a problem hiding this comment.
I fact, it is possible to request multiple file resources on a single query. Let me change it
change api request to fetch several fileResources at once
|
This PR implements batch verification for file resource storage status to prevent issues with files being wrongly assigned when posted without waiting for the STORED status.
Key changes:
Related task: ANDROSDK-2173