Vector Job Management

[ProBrian]

Hi, I'm a fresh player of Vector. After some docs reading and experiments on local machine, I do have some question about the Vector's job management(I know there's no "job" concept in Vector, here I use "job" to present a complete pipeline for specific source and sink).
In my case, I'm using Vector as a batch processing tool. e.g. I have a log file(or a directory of log files) and I want to collect these logs and sink them to a kafka instance. It's easy to configure a file source and a kafka sink in toml file to finish that job. But my question is: is there a concept of "batch processing" in Vector? since Vector has collected all logs from the source, it would still be running and waiting for new appending file contents. If my source files are immutable(no new file contents will be added), how could I know if Vector has finished the processing of all logs? Can I rely on the Graphql API's processedEventsTotal field in sinks : metrics to judge that if all logs processed? Or there are better API for checking this?
Furthermore, In my usage, when I know all logs are processed, I want to stop the Vector instance and start it again if there are new processing job(seems not a good practice?), or remove the finished job's config segment and then reload so that the finished processing job can be "removed". I see that Vector supports hot-reloading when watched config changes so I can do "job add/remove" by changing config files, but seems that's not a graceful way. Do we support changing config by Graphql API? If so, that'll be more convenient for me since I want to use Vector as a micro service and other service can operate it by APIs.
Please forgive my ignorance and correct me if there are misunderstanding of Vector features. Thanks.

[spencergilbert]

Hey, welcome to Vector!

We don't really support batch processing today but it has come up and we have issues open to explore the idea.

Similarly the graphql API is going to be a read only interface into the running instance. I know write capabilities have been discussed in the past but nothing has been implemented at this time.

[ProBrian]

Thank you @spencergilbert , good to know that there's a plan for batch processing, could you give me the issue id so that I can follow the progress?
One more question is about the processing status. As I mentioned in the first question, If I rely on the Graphql API's processedEventsTotal field in sinks : metrics to get the events Vector processed, I can fetch this data if Vector is running normally, but if the Vector instance restarts abnormally, the processedEventsTotal field will be the number of events it processed after restart, but not all events count. Is there any other way I can rely to get the total events Vector processed for a specific source from the time the source is created?
I noticed that there's a checkpoint json file that records the checkpoint of source file, is there any API that I can use to get the checkpoint information of a source?

[spencergilbert]

#4271

No, I believe any metric will be relative to the instance's lifetime.

We currently don't have any public API for the checkpoints, I'm not sure I've seen that requested or discussed before either.

[ProBrian]

Got it, Thank you @spencergilbert