File read log interval

[xufeixianggithub]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Currently, the cpu is higher when using files to read logs, because there is a practical scenario that sometimes you need to restart the application, or just access the Vector, the file is read hundreds of meters, then the log will be read quickly, and the cpu will be high. Of course, I know that some solutions connect to middleware such as Mq first. But it comes at a premium.

Attempted Solutions

No response

Proposal

No response

References

No response

Version

No response

[jszwedko]

Hi @xufeixianggithub ,

I see the issue you are describing, but I don't think we'd solve it by adding configuration to the file source. Instead, I think a more holistic way to solve this problem is for the throttle transform to support applying back-pressure. This is being tracked by #13651

You could also consider configuring the sink to apply back-pressure by limiting the concurrency or batch sizes.

I'll close this issue, but let me know if you disagree with my assessment!