-
Notifications
You must be signed in to change notification settings - Fork 525
[amazon_bedrock_agentcore] add runtime_application_logs data stream
#15991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 22 commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
4cdbe25
Amazon Bedrock AgentCore new integration package
agithomas 21ddd0e
Updated PR link, removed unused configs
agithomas 9182cf0
Merge branch 'main' into issue-obs-integration-670
agithomas 4d06423
Updated dashboard conditions
agithomas c9faea5
Updated the readme with the name of the agentcore components
agithomas a953aa9
add logs data stream
gpop63 c873bed
update dashboard
gpop63 bbe0c10
address comments
gpop63 2fc9ee4
merge
gpop63 39996cd
update
gpop63 0a43cbb
remove sample logo
gpop63 3088a7a
fix policy
gpop63 0ebfadc
add observability category
gpop63 53ee3b9
address comments
gpop63 e7bb9a9
fix pipeline indenting
gpop63 05b69ff
improve field descriptions
gpop63 5c2712d
fix sample event
gpop63 67dce02
update sample event
gpop63 920216e
add aws-s3 input
gpop63 391937f
improve policy templates descriptions
gpop63 a404df2
fix pr id
gpop63 cb1d145
update dashboard img
gpop63 2b39bcf
update dashboard
gpop63 5e57de4
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 390ec23
update pipeline
gpop63 f543a44
update sample event
gpop63 2dbc61a
add tags for debugging
gpop63 a9d5e76
add agent and endpoint names
gpop63 63efdfb
update fields mappings and pipeline
gpop63 de0f5d5
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 0db7ce5
move gen_ai fields
gpop63 6a595d7
fix tags
gpop63 87d4dc2
update sample event
gpop63 32f4e6f
Merge branch 'main' into runtime-logs
gpop63 9ced8b2
flatten response payload
gpop63 93f046b
update dashboard
gpop63 4c5e568
update dashboard image
gpop63 7523956
handle request and response of multiple types
gpop63 ea71ee6
update fields mappings and pipeline
gpop63 74f3dff
update readme
gpop63 bd57644
update pipeline
gpop63 3404575
update dashboard
gpop63 0482b86
revert dashboard
gpop63 c6f6f66
revert dashboard img
gpop63 50be0c2
fix dashboard img
gpop63 8cda6c4
remove comment
gpop63 4457bda
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 8af7e15
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 8b95ed2
update version
gpop63 5d096bd
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 1b59aa0
update version
gpop63 17ec2bb
update readme
gpop63 7780b2f
Merge remote-tracking branch 'upstream/main' into runtime-logs
gpop63 2972896
fix changelog
gpop63 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...re/data_stream/runtime_application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"resource_arn": "arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0", "event_timestamp": 1762144239907, "account_id": "627286350134", "request_id": "136b5865-b303-43a7-b737-3fed1ba17341", "session_id": "090ab333-40ff-49cd-ade1-1527b3a8ede7", "span_id": "3f2b2909093d27f3", "trace_id": "69082fe91a0ed1872de8e57378fe229b", "service_name": "AgentCoreCodeRuntime", "operation": "InvokeAgentRuntime", "request_payload":{"prompt": "What is this agent about, this is claudserver?"}} |
56 changes: 56 additions & 0 deletions
56
.../runtime_application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2025-11-03T04:30:39.907Z", | ||
| "aws": { | ||
| "bedrock_agentcore": { | ||
| "account_id": "627286350134", | ||
| "request_id": "136b5865-b303-43a7-b737-3fed1ba17341", | ||
| "resource_arn": "arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0" | ||
| } | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "627286350134" | ||
| }, | ||
| "provider": "aws", | ||
| "service": { | ||
| "name": "bedrock-agentcore" | ||
| } | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "event": { | ||
| "original": "{\"resource_arn\": \"arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0\", \"event_timestamp\": 1762144239907, \"account_id\": \"627286350134\", \"request_id\": \"136b5865-b303-43a7-b737-3fed1ba17341\", \"session_id\": \"090ab333-40ff-49cd-ade1-1527b3a8ede7\", \"span_id\": \"3f2b2909093d27f3\", \"trace_id\": \"69082fe91a0ed1872de8e57378fe229b\", \"service_name\": \"AgentCoreCodeRuntime\", \"operation\": \"InvokeAgentRuntime\", \"request_payload\":{\"prompt\": \"What is this agent about, this is claudserver?\"}}", | ||
| "outcome": "success" | ||
| }, | ||
| "gen_ai": { | ||
| "conversation": { | ||
| "id": "090ab333-40ff-49cd-ade1-1527b3a8ede7" | ||
| }, | ||
| "operation": { | ||
| "name": "InvokeAgentRuntime" | ||
| }, | ||
| "prompt": "What is this agent about, this is claudserver?", | ||
| "provider": { | ||
| "name": "aws" | ||
| }, | ||
| "system": "aws_bedrock_agentcore" | ||
| }, | ||
| "service": { | ||
| "name": "AgentCoreCodeRuntime" | ||
| }, | ||
| "span": { | ||
| "id": "3f2b2909093d27f3" | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event", | ||
| "preserve_duplicate_custom_fields" | ||
| ], | ||
| "trace": { | ||
| "id": "69082fe91a0ed1872de8e57378fe229b" | ||
| } | ||
| } | ||
| ] | ||
| } |
5 changes: 5 additions & 0 deletions
5
..._agentcore/data_stream/runtime_application_logs/_dev/test/pipeline/test-common-config.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| fields: | ||
| tags: | ||
| - preserve_original_event | ||
| - preserve_duplicate_custom_fields |
106 changes: 106 additions & 0 deletions
106
...edrock_agentcore/data_stream/runtime_application_logs/agent/stream/aws-cloudwatch.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| {{#unless log_group_name}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_arn }} | ||
| log_group_arn: {{ log_group_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name}} | ||
| {{#if log_group_name_prefix }} | ||
| log_group_name_prefix: {{ log_group_name_prefix }} | ||
| {{/if}} | ||
| {{#if include_linked_accounts_with_prefix }} | ||
| include_linked_accounts_for_prefix_mode: {{ include_linked_accounts_with_prefix }} | ||
| {{/if}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_name }} | ||
| log_group_name: {{ log_group_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| region_name: {{ region_name }} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_stream_prefix}} | ||
| {{#if log_streams }} | ||
| log_streams: {{ log_streams }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_streams}} | ||
| {{#if log_stream_prefix }} | ||
| log_stream_prefix: {{ log_stream_prefix }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#if start_position }} | ||
| start_position: {{ start_position }} | ||
| {{/if}} | ||
|
|
||
| {{#if scan_frequency }} | ||
| scan_frequency: {{ scan_frequency }} | ||
| {{/if}} | ||
|
|
||
| {{#if api_sleep }} | ||
| api_sleep: {{ api_sleep }} | ||
| {{/if}} | ||
|
|
||
| {{#if latency }} | ||
| latency: {{ latency }} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
171 changes: 171 additions & 0 deletions
171
...es/aws_bedrock_agentcore/data_stream/runtime_application_logs/agent/stream/aws-s3.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| {{! The aws-s3 input can be configured to read from an SQS queue or an S3 bucket. }} | ||
|
|
||
| {{! start SQS queue }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if queue_url }} | ||
| queue_url: {{ queue_url }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{! end SQS queue }} | ||
|
|
||
| {{#unless queue_url}}{{! start S3 bucket polling }} | ||
|
|
||
| {{! | ||
| When using an S3 bucket, you can specify only one of the following options: | ||
| - An AWS bucket ARN | ||
| - A non-AWS bucket name | ||
| }} | ||
|
|
||
| {{! shared S3 bucket polling options }} | ||
| {{#if bucket_list_prefix }} | ||
| bucket_list_prefix: {{ bucket_list_prefix }} | ||
| {{/if}} | ||
|
|
||
| {{#if bucket_list_interval }} | ||
| bucket_list_interval: {{ bucket_list_interval }} | ||
| {{/if}} | ||
|
|
||
| {{#if start_timestamp}} | ||
| start_timestamp: {{start_timestamp}} | ||
| {{/if}} | ||
|
|
||
| {{#if ignore_older}} | ||
| ignore_older: {{ignore_older}} | ||
| {{/if}} | ||
|
|
||
| {{! AWS S3 bucket ARN options }} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if bucket_arn }} | ||
| bucket_arn: {{ bucket_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! non-AWS S3 bucket ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless access_point_arn}} | ||
| {{#if non_aws_bucket_name }} | ||
| non_aws_bucket_name: {{ non_aws_bucket_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! AWS S3 Access Point ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#if access_point_arn }} | ||
| access_point_arn: {{ access_point_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{/unless}}{{! end S3 bucket polling }} | ||
|
|
||
| {{! allows number of workers to be configured for SQS queue and S3 buckets}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
|
|
||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if content_type }} | ||
| content_type: {{ content_type }} | ||
| {{/if}} | ||
| {{#if encoding }} | ||
| encoding: {{ encoding }} | ||
| {{/if}} | ||
| {{#if expand_event_list_from_field }} | ||
| expand_event_list_from_field: {{ expand_event_list_from_field }} | ||
| {{/if}} | ||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if fips_enabled }} | ||
| fips_enabled: {{ fips_enabled }} | ||
| {{/if}} | ||
| {{#if include_s3_metadata }} | ||
| include_s3_metadata: {{ include_s3_metadata }} | ||
| {{/if}} | ||
| {{#if max_bytes }} | ||
| max_bytes: {{ max_bytes }} | ||
| {{/if}} | ||
| {{#if max_number_of_messages }} | ||
| max_number_of_messages: {{ max_number_of_messages }} | ||
| {{/if}} | ||
| {{#if path_style }} | ||
| path_style: {{ path_style }} | ||
| {{/if}} | ||
| {{#if provider }} | ||
| provider: {{ provider }} | ||
| {{/if}} | ||
| {{#if sqs.max_receive_count }} | ||
| sqs.max_receive_count: {{ sqs.max_receive_count }} | ||
| {{/if}} | ||
| {{#if sqs.wait_time }} | ||
| sqs.wait_time: {{ sqs.wait_time }} | ||
| {{/if}} | ||
|
|
||
| {{#if file_selectors}} | ||
| file_selectors: | ||
| {{file_selectors}} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if visibility_timeout}} | ||
| visibility_timeout: {{visibility_timeout}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if endpoint}} | ||
| endpoint: {{endpoint}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if fips_enabled}} | ||
| fips_enabled: {{fips_enabled}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think, it will be good to mention the steps needed to fetch the log_group_arn, needed for configuring this dataset?
As it is the same steps for all the log-based datasets, maybe a common section (may be under
What do I need to use this integration?) should suffice.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added a small section to explain how to get the log group arn 👍