Add global state storage size logging with threshold filter

[KJ7LNW]

Context

This PR adds a utility function to log global state storage sizes with a configurable threshold filter (default 10KB) to help users identify large items.

Implementation

• Created a new utility file src/utils/storageUtils.ts with:
  • formatBytes function to convert byte sizes to human-readable format
  • logGlobalStorageSize function with threshold parameter (default 10KB)
• Updated extension.ts to use the new utility functions
• Added comprehensive tests for the formatBytes function
• Changed logging to use console.log instead of outputChannel

How to Test

• Run the extension and check the console output
• Verify that only storage items larger than 10KB are shown
• Verify that total size and number of skipped items are displayed

Fixes #3809

---

Important

Adds utility to log global state storage sizes with a threshold filter and updates logging behavior in the extension.

• Behavior:
  • Adds logGlobalStorageSize in storageUtils.ts to log global state items exceeding a size threshold (default 10KB).
  • Updates extension.ts to use logGlobalStorageSize during activation.
  • Changes logging from outputChannel to console.log in extension.ts.
• Utilities:
  • Adds formatBytes in storageUtils.ts to convert byte sizes to human-readable format.
• Testing:
  • Adds tests for formatBytes in storageUtils.test.ts.

^{This description was created by for 2f7ab2d. You can customize this summary. It will automatically update as commits are pushed.}

[KJ7LNW]

@cte @mrubens @hannesrudolph what does knip test?

It looks like the intention is to prevent files that are not used, but this new file is definitely used (imported by extension.ts) and I cannot figure out how to fix this error:

Run npm run knip
  npm run knip
  shell: /usr/bin/bash -e {0}
  env:
    NODE_VERSION: 20.18.1

> [email protected] knip
> knip --include files

src/utils/storageUtils.ts  
Error: Process completed with exit code 1.

[webpro]

First of all, apologies for my creepy unsolicited entrance. Creator of Knip here. Sometimes I see projects using Knip and this caught my eye. Feel free to ignore me! But I noticed that this is a monorepo and the root "." workspace could be better configured like so:

{
	"$schema": "https://unpkg.com/knip@latest/schema.json",
	"workspaces": {
		".": {
			"entry": ["src/extension.ts", "src/workers/countTokens.ts", "src/**/__tests__/**/*{b,B}enchmark.ts"],
			"project": ["src/**/*.ts"]
		},
		"webview-ui": {
			"entry": ["src/index.tsx"],
			"project": ["src/**/*.{ts,tsx}"]
		}
	},
	"ignore": ["**/__mocks__/**"]
}

This way, there's no need to ignore as much! In the next release of Knip you'll be able to remove that "**/__mocks__/**" from ignore as well (and they'll automatically be added as entry files by the Jest and Vitest plugins).

Have a great day ☀️

[KJ7LNW]

First of all, apologies for my creepy unsolicited entrance. Creator of Knip here. Sometimes I see projects using Knip and this caught my eye. Feel free to ignore me! But I noticed that this is a monorepo and the root "." workspace could be better configured like so:

{
	"$schema": "https://unpkg.com/knip@latest/schema.json",
	"workspaces": {
		".": {
			"entry": ["src/extension.ts", "src/workers/countTokens.ts", "src/**/__tests__/**/*{b,B}enchmark.ts"],
			"project": ["src/**/*.ts"]
		},
		"webview-ui": {
			"entry": ["src/index.tsx"],
			"project": ["src/**/*.{ts,tsx}"]
		}
	},
	"ignore": ["**/__mocks__/**"]
}

This way, there's no need to ignore as much! In the next release of Knip you'll be able to remove that "**/__mocks__/**" from ignore as well (and they'll automatically be added as entry files by the Jest and Vitest plugins).

Have a great day ☀️

@webpro, Thank you for dropping in!

Do you understand what causes the following? As far as I can tell, src/utils/storageUtils.ts is being used by this PR but for some reason is it is getting triggered:

Run npm run knip
  npm run knip
  shell: /usr/bin/bash -e {0}
  env:
    NODE_VERSION: 20.18.1

> [email protected] knip
> knip --include files

src/utils/storageUtils.ts  
Error: Process completed with exit code 1.

[webpro]

In a monorepo, the top-level entry and project options are not effective, because these should be in the root workspace config at workspaces["."]. Use --debug and look for "Included workspace configs" to see the config Knip uses.

Knip could def need some more config validation, etc. but we're not there yet.

Btw the ignore option in the config I posted can be removed entirely after installing [email protected] (or later).

[KJ7LNW]

In a monorepo, the top-level entry and project options are not effective, because these should be in the root workspace config at workspaces["."]. Use --debug and look for "Included workspace configs" to see the config Knip uses.

Knip could def need some more config validation, etc. but we're not there yet.

Btw the ignore option in the config I posted can be removed entirely after installing [email protected] (or later).

thank you I am going to reference this in a new issue

[daniel-lxs]

Hey @KJ7LNW This is great to help us debug potential issues that users might run into with the global state while we fix the implementation, just a nit on the size calculation for the undefined values in the global state, everything else lgtm

[daniel-lxs]

In the loop where items are logged, there's a check for item.size === undefined. If context.globalState.get(key) actually returns undefined, JSON.stringify(undefined) produces the string "undefined", so size would be 9. Could we clarify how explicitly undefined values retrieved from globalState should be handled or reported here? For instance, if a key exists but its value is undefined, should that be logged differently than an error in size calculation?

[KJ7LNW]

Interesting point.

That code is correct, however, as it should measure len(json).toString() because the size-estimate assumption is that the global state backend storage size is stored as JSON text, so that is the size of the element. (Even if vscode's global state does not store on disk as JSON, it still correctly provides the magnitude of key sizes and the sort order would still be correct as well, and that is all we care about here.)

In any event, the result of a size being only 9 is rather moot: the entire point of this is to provide inspection of large global state sizes, because when the global state gets too big (ie, megabytes) it causes problems, hence PR #3785 and possibly related issues about memory crashes.

This pull request is a result of inspecting global state storage for #3785 to generally see where the problems lie: I thought it could be useful in general for logging Roo startup, to keep an eye on things, and also give users a way to report information like this if troubleshooting is necessary (ie, paste your console logs into this ticket)---however I did not think it should be part of #3785 since that is getting big and I want to keep it focused.

[KJ7LNW]

blocked by #3884

[KJ7LNW]

this is a good idea but I do not see it getting off the ground. someone can reopen if they want the feature