Two and more sources

[kirill-ratkin]

Hi,
Could you advice, how to configure 'vector' to process several sources?
I have two (or maybe three) files on a disk. There are csv and json. They are quite large.
And I'd like to get data from them, mix data got from all and sink to database.

It's quite easy to do it in 'awk' or 'python' but I'd like to have more performance and 'vector' looks very very exciting. But I can't understand how to configure several inputs .. :(

[jszwedko]

Hi @kirill-ratkin !

You can provide multiple paths to the file source like:

[sources.my_source_id]
type = "file"
include = [ "/tmp/foo.log", "/tmp/bar.log" ]

Does this help answer your question?

[kirill-ratkin]

Hi, Yes but ... not fully :).
In fact I have several sources, for example some table in database and csv file.
I have to read table line by line and replace (and add) some columns based on data in csv.
In other words table and csv have logical foreign key between each other.

Here I say table and csv file just to define some source objects.
In real life one of it, for example 'table' will be stream from Kafka (probably).

This is one case.

Second one is two or more streams. And I need to join columns into one target stream or (which is more possible) to big table in Clickhouse database. In this case I don't have correlations between sources but I need just to make union of these sets.

[jszwedko]

Hey!

I see, I think the CSV enrichment feature of Vector could help here? You can see a demo here: https://github.com/vectordotdev/vector-demos/tree/main/csv-enrichment