-
Notifications
You must be signed in to change notification settings - Fork 526
[amazon_bedrock_agentcore] add gateway_application_logs data stream #15991
#16587
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gpop63
wants to merge
9
commits into
elastic:main
Choose a base branch
from
gpop63:gateway-logs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,228
−3
Open
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
da1fde9
add gateway data stream
gpop63 885b3fc
Merge remote-tracking branch 'upstream/main' into gateway-logs
gpop63 f8e6a67
bump package version
gpop63 f202b6f
update readme
gpop63 0787d8d
fix pr id
gpop63 a064952
Merge remote-tracking branch 'upstream/main' into gateway-logs
gpop63 6181cc7
rename body field
gpop63 52aa394
improve pipeline
gpop63 16e5c5b
Merge remote-tracking branch 'upstream/main' into gateway-logs
gpop63 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...stream/gateway_application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore-gateway.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"resource_arn":"arn:aws:bedrock-agentcore:us-west-2:845123678901:gateway/ordermanager-gw-abc12def34","event_timestamp":1768432156789,"body":{"isError":false,"log":"Processing request for tool DocumentRetrieval___fetch_order_details from target XYZPROD001","id":"7"},"account_id":"845123678901","request_id":"a1b2c3d4-5678-9012-ef34-567890abcdef","trace_id":"8e45f912abcd3456ef7890123456abcd","span_id":"9f8e7d6c5b4a3210","resource":{"attributes":{"service.name":"ordermanager-gw-abc12def34","cloud.resource_id":"arn:aws:bedrock-agentcore:us-west-2:845123678901:gateway/ordermanager-gw-abc12def34","cloud.platform":"aws_bedrock_agentcore"}},"attributes":{"aws.request.id":"a1b2c3d4-5678-9012-ef34-567890abcdef","aws.account.id":"845123678901","aws.resource.type":"AWS::BedrockAgentCore::Gateway"},"timeUnixNano":1768432156789123456,"severityNumber":9,"severityText":"INFO","traceId":"8e45f912abcd3456ef7890123456abcd","spanId":"9f8e7d6c5b4a3210"} |
53 changes: 53 additions & 0 deletions
53
..._application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore-gateway.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2026-01-14T23:09:16.789Z", | ||
| "aws": { | ||
| "bedrock_agentcore": { | ||
| "body": { | ||
| "id": "7", | ||
| "isError": false, | ||
| "log": "Processing request for tool DocumentRetrieval___fetch_order_details from target XYZPROD001" | ||
| }, | ||
| "operation_name": "invoke_gateway", | ||
| "provider_name": "aws_bedrock_agentcore", | ||
| "request_id": "a1b2c3d4-5678-9012-ef34-567890abcdef", | ||
| "resource_arn": "arn:aws:bedrock-agentcore:us-west-2:845123678901:gateway/ordermanager-gw-abc12def34", | ||
| "severity_number": 9 | ||
| } | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "845123678901" | ||
| }, | ||
| "provider": "aws", | ||
| "service": { | ||
| "name": "bedrock-agentcore" | ||
| } | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "event": { | ||
| "original": "{\"resource_arn\":\"arn:aws:bedrock-agentcore:us-west-2:845123678901:gateway/ordermanager-gw-abc12def34\",\"event_timestamp\":1768432156789,\"body\":{\"isError\":false,\"log\":\"Processing request for tool DocumentRetrieval___fetch_order_details from target XYZPROD001\",\"id\":\"7\"},\"account_id\":\"845123678901\",\"request_id\":\"a1b2c3d4-5678-9012-ef34-567890abcdef\",\"trace_id\":\"8e45f912abcd3456ef7890123456abcd\",\"span_id\":\"9f8e7d6c5b4a3210\",\"resource\":{\"attributes\":{\"service.name\":\"ordermanager-gw-abc12def34\",\"cloud.resource_id\":\"arn:aws:bedrock-agentcore:us-west-2:845123678901:gateway/ordermanager-gw-abc12def34\",\"cloud.platform\":\"aws_bedrock_agentcore\"}},\"attributes\":{\"aws.request.id\":\"a1b2c3d4-5678-9012-ef34-567890abcdef\",\"aws.account.id\":\"845123678901\",\"aws.resource.type\":\"AWS::BedrockAgentCore::Gateway\"},\"timeUnixNano\":1768432156789123456,\"severityNumber\":9,\"severityText\":\"INFO\",\"traceId\":\"8e45f912abcd3456ef7890123456abcd\",\"spanId\":\"9f8e7d6c5b4a3210\"}", | ||
| "outcome": "success" | ||
| }, | ||
| "log": { | ||
| "level": "INFO" | ||
| }, | ||
| "service": { | ||
| "name": "ordermanager-gw-abc12def34" | ||
| }, | ||
| "span": { | ||
| "id": "9f8e7d6c5b4a3210" | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event", | ||
| "preserve_duplicate_custom_fields" | ||
| ], | ||
| "trace": { | ||
| "id": "8e45f912abcd3456ef7890123456abcd" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
5 changes: 5 additions & 0 deletions
5
..._agentcore/data_stream/gateway_application_logs/_dev/test/pipeline/test-common-config.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| fields: | ||
| tags: | ||
| - preserve_original_event | ||
| - preserve_duplicate_custom_fields |
106 changes: 106 additions & 0 deletions
106
...edrock_agentcore/data_stream/gateway_application_logs/agent/stream/aws-cloudwatch.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| {{#unless log_group_name}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_arn }} | ||
| log_group_arn: {{ log_group_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name}} | ||
| {{#if log_group_name_prefix }} | ||
| log_group_name_prefix: {{ log_group_name_prefix }} | ||
| {{/if}} | ||
| {{#if include_linked_accounts_with_prefix }} | ||
| include_linked_accounts_for_prefix_mode: {{ include_linked_accounts_with_prefix }} | ||
| {{/if}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_name }} | ||
| log_group_name: {{ log_group_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| region_name: {{ region_name }} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_stream_prefix}} | ||
| {{#if log_streams }} | ||
| log_streams: {{ log_streams }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_streams}} | ||
| {{#if log_stream_prefix }} | ||
| log_stream_prefix: {{ log_stream_prefix }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#if start_position }} | ||
| start_position: {{ start_position }} | ||
| {{/if}} | ||
|
|
||
| {{#if scan_frequency }} | ||
| scan_frequency: {{ scan_frequency }} | ||
| {{/if}} | ||
|
|
||
| {{#if api_sleep }} | ||
| api_sleep: {{ api_sleep }} | ||
| {{/if}} | ||
|
|
||
| {{#if latency }} | ||
| latency: {{ latency }} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
171 changes: 171 additions & 0 deletions
171
...es/aws_bedrock_agentcore/data_stream/gateway_application_logs/agent/stream/aws-s3.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| {{! The aws-s3 input can be configured to read from an SQS queue or an S3 bucket. }} | ||
|
|
||
| {{! start SQS queue }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if queue_url }} | ||
| queue_url: {{ queue_url }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{! end SQS queue }} | ||
|
|
||
| {{#unless queue_url}}{{! start S3 bucket polling }} | ||
|
|
||
| {{! | ||
| When using an S3 bucket, you can specify only one of the following options: | ||
| - An AWS bucket ARN | ||
| - A non-AWS bucket name | ||
| }} | ||
|
|
||
| {{! shared S3 bucket polling options }} | ||
| {{#if bucket_list_prefix }} | ||
| bucket_list_prefix: {{ bucket_list_prefix }} | ||
| {{/if}} | ||
|
|
||
| {{#if bucket_list_interval }} | ||
| bucket_list_interval: {{ bucket_list_interval }} | ||
| {{/if}} | ||
|
|
||
| {{#if start_timestamp}} | ||
| start_timestamp: {{start_timestamp}} | ||
| {{/if}} | ||
|
|
||
| {{#if ignore_older}} | ||
| ignore_older: {{ignore_older}} | ||
| {{/if}} | ||
|
|
||
| {{! AWS S3 bucket ARN options }} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if bucket_arn }} | ||
| bucket_arn: {{ bucket_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! non-AWS S3 bucket ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless access_point_arn}} | ||
| {{#if non_aws_bucket_name }} | ||
| non_aws_bucket_name: {{ non_aws_bucket_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! AWS S3 Access Point ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#if access_point_arn }} | ||
| access_point_arn: {{ access_point_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{/unless}}{{! end S3 bucket polling }} | ||
|
|
||
| {{! allows number of workers to be configured for SQS queue and S3 buckets}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
|
|
||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if content_type }} | ||
| content_type: {{ content_type }} | ||
| {{/if}} | ||
| {{#if encoding }} | ||
| encoding: {{ encoding }} | ||
| {{/if}} | ||
| {{#if expand_event_list_from_field }} | ||
| expand_event_list_from_field: {{ expand_event_list_from_field }} | ||
| {{/if}} | ||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if fips_enabled }} | ||
| fips_enabled: {{ fips_enabled }} | ||
| {{/if}} | ||
| {{#if include_s3_metadata }} | ||
| include_s3_metadata: {{ include_s3_metadata }} | ||
| {{/if}} | ||
| {{#if max_bytes }} | ||
| max_bytes: {{ max_bytes }} | ||
| {{/if}} | ||
| {{#if max_number_of_messages }} | ||
| max_number_of_messages: {{ max_number_of_messages }} | ||
| {{/if}} | ||
| {{#if path_style }} | ||
| path_style: {{ path_style }} | ||
| {{/if}} | ||
| {{#if provider }} | ||
| provider: {{ provider }} | ||
| {{/if}} | ||
| {{#if sqs.max_receive_count }} | ||
| sqs.max_receive_count: {{ sqs.max_receive_count }} | ||
| {{/if}} | ||
| {{#if sqs.wait_time }} | ||
| sqs.wait_time: {{ sqs.wait_time }} | ||
| {{/if}} | ||
|
|
||
| {{#if file_selectors}} | ||
| file_selectors: | ||
| {{file_selectors}} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if visibility_timeout}} | ||
| visibility_timeout: {{visibility_timeout}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if endpoint}} | ||
| endpoint: {{endpoint}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if fips_enabled}} | ||
| fips_enabled: {{fips_enabled}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we extract the key parts of this and store them in ECS fields?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
body.logfield does not always follow a consistent format. In some cases, we may be able to extract values such as the tool name and target but these fields will not be present in all gateway logs.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the gateway, the tool operation name, target, and outcome of the tool operation (including target invocation results) are important observability information. When present, let's capture it and store them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, you may store the log data entirely in the
messsagefield