[BUG] Parsed field data missing when source field not included

[dai-chen]

What is the bug?

When using the PPL parse command, if the source field that was parsed is not included in the subsequent fields command, the parsed field data is missing from the final result.

This behavior appears to be by design in the code: ProjectOperator.java#L76.

How can one reproduce the bug?

POST parse_command_test/_doc
{
  "@timestamp": "2025-01-04T04:00:00Z",
  "line": {
    "@message": "Request from AWS for client ID 123 is OVER_LIMIT"
  }
}

POST _plugins/_ppl
{
  "query": """
    search source=parse_command_test
    | parse line.@message 'Request from (?<service>.+) for client ID (?<clientId>.+) is OVER_LIMIT'
    | fields @timestamp, service, clientId
  """
}
{
  "schema": [
    {
      "name": "@timestamp",
      "type": "timestamp"
    },
    {
      "name": "service",
      "type": "string"
    },
    {
      "name": "clientId",
      "type": "string"
    }
  ],
  "datarows": [
    [
      "2025-01-04 04:00:00"
    ]
  ],
  "total": 1,
  "size": 1
}

What is the expected behavior?

• If intentional: Provide clear documentation to explain why parsed field data depends on the inclusion of the original field in the fields command.
• Otherwise: Modify the behavior to ensure parsed fields remain available even if the original field is excluded.

Do you have any screenshots?

N/A

Do you have any additional context?

N/A

[RyanL1997]

Adding the action item for document this for now.

[penghuo]

The results should be null if no data.

  "datarows": [
    [
      "2025-01-04 04:00:00",
      null,
      null
    ]
  ]

[dai-chen]

Tasks:

• Update PPL parse command doc with limitation #3274
• Fix missing value in data rows bug above
• [TBD] Change parse command behavior to remove the limitation