Skip to content

Optimize DB usage & size #31

Open
Open
@MHHukiewitz

Description

@MHHukiewitz

There are multiple reasons why the indexing framework is using too much disk space right now:

  • Data is being stored as serialized UTF-8 chars of its JSON representation. This could be replaced by ProtoBuf serialized data.
  • The EntityRequestResponseDAL stores all responses indefinitely. Freeing up space there would bring the biggest benefits but is not trivial as expected. Multiple indexers can request the same events, so we need to only delete these EntityRequestResponses if all Requests have been processed and satisfied. This might be solved by introducing a new job that periodically checks the DB for these finished Requests and deletes their Responses. Solved here: Clean up responses in indexer instances #34

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestperformanceImpacts the runtime performance

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions