Vector Unit Testing - Cant send or parse array/json elements directly

[satscreate]

Config yaml:

sources:
  stdin:
    type: stdin
    decoding:
      codec: json
      
transforms:
  filter-empty:
    type: filter
    inputs:
      - stdin
    condition: >-
      .message != ""
  
  parse-common-stuff:
    type: remap
    inputs:
      - filter-empty
    drop_on_error: true
    source: |-
      .name = string!(.actors[0].name)
      .src = string!(.source)
      .eId = string!(.id)

  route-logs:
    type: route
    inputs:
      - parse-common-stuff
    route:
      result-log: '.result != null'
      empty-result-log: '.result == null'
  
  parse-result-log:
    type: remap
    inputs:
      - route-logs.result-log
    drop_on_error: true
    source: |-
      .message = .result.message
      .status = string!(.result.status)
      .host = string!(.host)
      .session = string!(.resources[0].session)
  
  parse-empty-result-log:
    type: remap
    inputs:
      - route-logs.empty-result-log
    drop_on_error: true
    source: |-
      if is_array(.resources) {
        if length!(.resources) >= 1 {
          web = get!(value: .resources[0], path: ["session"])
          .session = if web == null { null } else { string!(web) }
        }
      }
      . = {
        "session" : .session
      }

sinks:
  stdout:
    type: console
    inputs:
      - stdin
    target: stdout
    encoding:
      codec: ndjson

Test file:

tests:
  - name: filter-empty-logs
    inputs:
      - insert_at: filter-empty
        type: log
        log_fields:
          host: 0.0
          message: ""
    no_outputs_from:
      - filter-empty

  - name: validate-result-log-block
    inputs:
      - insert_at: filter-empty
        type: log
        log_fields:
          host: 0.0
          message: "{\"source\": \"test\", \"id\": \"testid\", \"recorded\": \"date\", \"actors\": [{\"type\": \"user\", \"name\": \"somename\", \"id\": \"someid\"}], \"resources\": [{\"id\": null, \"type\": null, \"name\": null, \"session\": \"test\"}], \"result\": {\"status\": \"test\", \"message\": \"somemessage\"}}"
    outputs:
      - extract_from: parse-result-log
        conditions:
          - type: vrl
            source: |-
              assert!(.host == "0.0")
              assert!(.session == "test")
   

Command:
vector -v test --config-yaml test.yaml test2-test.yaml

Error:

WARN transform{component_kind="transform" component_id=parse-common-stuff component_type=remap component_name=parse-common-stuff}: vector::internal_events::remap: Mapping failed with event; discarding event. error="function call error for \"string\" at (8:32): expected \"string\", got \"null\"" internal_log_rate_secs=30
INFO vector::topology::builder: Healthcheck: Passed.
test validate-result-log-block ... �[31mfailed�[0m

failures:

test validate-result-log-block:

checks for transform "parse-result-log" failed: no events received. Topology may be disconnected or transform is missing inputs.

But when i run the vector file alone with below message input, it does parse it. but how can i test the use case by just sending message json alone.

1. i think message json not being parsed as an input with the type of json.
2. or how can we send array under log_fields:

[jszwedko]

Hi @satscreate !

The issue here is that you are passing in a raw string where the actual topology will parse the string as JSON due to setting decoding.codec = "json" on the stdin source.

As you note, there is a bug prohibiting arrays to be used as fields that is being tracked by #9386

You can, however, flatten out the object keys like this:

  - name: validate-result-log-block
    inputs:
      - insert_at: filter-empty
        type: log
        log_fields:
          host: "0.0"
          source: test
          id: testid
          recorded: date
          message: some message
          actors[0].type: user
          actors[0].name: somename
          actors[0].id: someid
          resources[0].id: ""
          resources[0].type: ""
          resources[0].name: ""
          resources[0].session: test
          result.status: test
          result.message: somemessage
    outputs:
      - extract_from: parse-result-log
        conditions:
          - type: vrl
            source: |-
              assert!(.host == "0.0")
              assert!(.session == "test")

This will result in the JSON you had (I made some slight tweaks to the fields to get the tests passing: adding a message field and making host a string).

[satscreate]

Hi @satscreate !

The issue here is that you are passing in a raw string where the actual topology will parse the string as JSON due to setting decoding.codec = "json" on the stdin source.

As you note, there is a bug prohibiting arrays to be used as fields that is being tracked by #9386

You can, however, flatten out the object keys like this:

  - name: validate-result-log-block
    inputs:
      - insert_at: filter-empty
        type: log
        log_fields:
          host: "0.0"
          source: test
          id: testid
          recorded: date
          message: some message
          actors[0].type: user
          actors[0].name: somename
          actors[0].id: someid
          resources[0].id: ""
          resources[0].type: ""
          resources[0].name: ""
          resources[0].session: test
          result.status: test
          result.message: somemessage
    outputs:
      - extract_from: parse-result-log
        conditions:
          - type: vrl
            source: |-
              assert!(.host == "0.0")
              assert!(.session == "test")

This will result in the JSON you had (I made some slight tweaks to the fields to get the tests passing: adding a message field and making host a string).

Yes @jszwedko , i think its good to flatten out the objects and really test the core. But surprisingly not many objects tree isn't to be validated, imagine if i have a an objects of 1000+ in a tree, its harder in this case. Hoping for quick feature release anyway...
Another request is to add support for both text, json, array, syslog formats as well., Thanks

[jszwedko]

https://github.com/tomnomnom/gron looks to do a translation to flattened JSON that could be useful.

Agreed though, we should support nested object fields.