forked from cline/cline
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Add global state storage size logging with threshold filter #3810
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
KJ7LNW
wants to merge
1
commit into
RooCodeInc:main
Choose a base branch
from
KJ7LNW:storage-size-threshold
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+91
−3
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
import { getExtensionContext } from "../extension" | ||
|
||
/** | ||
* Formats a number of bytes into a human-readable string with units (Bytes, KB, MB, GB, etc.). | ||
* | ||
* @param bytes The number of bytes. | ||
* @param decimals The number of decimal places to include (default is 2). | ||
* @returns A string representing the formatted bytes. | ||
*/ | ||
export function formatBytes(bytes: number, decimals = 2): string { | ||
if (bytes === 0) { | ||
return "0 Bytes" | ||
} | ||
const k = 1024 | ||
const dm = decimals < 0 ? 0 : decimals | ||
const sizes = ["Bytes", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"] | ||
const i = Math.floor(Math.log(bytes) / Math.log(k)) | ||
return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + " " + sizes[i] | ||
} | ||
|
||
/** | ||
* Logs the estimated sizes of items in the VS Code global state to the console. | ||
* Only logs items that exceed the specified minimum size threshold. | ||
* | ||
* @param minSizeBytes The minimum size in bytes to log (default: 10KB) | ||
*/ | ||
export function logGlobalStorageSize(minSizeBytes: number = 10 * 1024): void { | ||
const context = getExtensionContext() | ||
try { | ||
console.log(`[Roo Code] Global State Storage Estimates (items > ${formatBytes(minSizeBytes)}):`) | ||
const globalStateKeys = context.globalState.keys() | ||
const stateSizes: { key: string; size: number }[] = [] | ||
let totalSize = 0 | ||
let itemsSkipped = 0 | ||
|
||
for (const key of globalStateKeys) { | ||
const value = context.globalState.get(key) | ||
try { | ||
const valueString = JSON.stringify(value) | ||
const size = valueString.length | ||
totalSize += size | ||
|
||
if (size >= minSizeBytes) { | ||
stateSizes.push({ key, size }) | ||
} else { | ||
itemsSkipped++ | ||
} | ||
} catch (e) { | ||
// Handle cases where value might not be stringifiable | ||
stateSizes.push({ key, size: -1 }) // Indicate an error or unmeasurable size | ||
console.log(` - ${key}: (Error calculating size)`) | ||
} | ||
} | ||
|
||
stateSizes.sort((a, b) => b.size - a.size) | ||
|
||
stateSizes.forEach((item) => { | ||
if (item.size === -1) { | ||
// Already logged error | ||
} else if (item.size === undefined) { | ||
console.log(` - ${item.key}: (undefined value)`) | ||
} else { | ||
console.log(` - ${item.key}: ${formatBytes(item.size)}`) | ||
} | ||
}) | ||
|
||
console.log(` Total size of all items: ${formatBytes(totalSize)}`) | ||
console.log(` Items below threshold (${itemsSkipped}): not shown`) | ||
console.log("---") | ||
} catch (e: any) { | ||
console.log(`Error displaying global state sizes: ${e.message}`) | ||
} | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the loop where items are logged, there's a check for
item.size === undefined
. Ifcontext.globalState.get(key)
actually returnsundefined
,JSON.stringify(undefined)
produces the string"undefined"
, sosize
would be9
. Could we clarify how explicitlyundefined
values retrieved fromglobalState
should be handled or reported here? For instance, if a key exists but its value isundefined
, should that be logged differently than an error in size calculation?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting point.
That code is correct, however, as it should measure
len(json).toString()
because the size-estimate assumption is that the global state backend storage size is stored as JSON text, so that is the size of the element. (Even if vscode's global state does not store on disk as JSON, it still correctly provides the magnitude of key sizes and the sort order would still be correct as well, and that is all we care about here.)In any event, the result of a size being only 9 is rather moot: the entire point of this is to provide inspection of large global state sizes, because when the global state gets too big (ie, megabytes) it causes problems, hence PR #3785 and possibly related issues about memory crashes.
This pull request is a result of inspecting global state storage for #3785 to generally see where the problems lie: I thought it could be useful in general for logging Roo startup, to keep an eye on things, and also give users a way to report information like this if troubleshooting is necessary (ie, paste your console logs into this ticket)---however I did not think it should be part of #3785 since that is getting big and I want to keep it focused.