feat: add observability (APPLICATION_LOGS + X-Ray + Identity) to AgentCore Gateway

[EashanKaushik]

Description

Add observability delivery to the AgentCore Gateway using the existing RuntimeObservability construct. The gateway has a different log group pattern than runtimes and does NOT have USAGE_LOGS.

Context

PRs #80 and #86 deployed observability for the coding assistant and MCP server runtimes respectively. The gateway is the last resource without observability configured.

Key difference from runtimes:

• Gateway log group: /aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{GatewayID}
• Runtime log group: /aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/{runtimeId}
• Gateway does NOT have USAGE_LOGS (only runtimes do)

What to do

1. Generalize RuntimeObservability construct

Add a resourceType prop to lib/constructs/observability/runtime-observability.ts:

interface RuntimeObservabilityProps {
  runtimeArn: string;
  runtimeId: string;
  resourceType?: "runtime" | "gateway";  // default: "runtime"
  logRetentionDays?: number;
  enableIdentityLogs?: boolean;
}

When resourceType = "gateway":

• APPLICATION_LOGS log group: /aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{runtimeId} (use gateway instead of runtime)
• Skip USAGE_LOGS entirely (don't create LogGroup, DeliverySource, DeliveryDestination, Delivery for usage)
• X-Ray TRACES: same pattern as runtime (custom resource Lambda)
• Identity: same pattern (shared /aws/vendedlogs/bedrock-agentcore/workload-identity-directory/APPLICATION_LOGS/default)

2. Add IAM to gateway role

In lib/constructs/gateway/mcp-gateway.ts, add to the gateway role:

gatewayRole.addToPolicy(new iam.PolicyStatement({
  actions: ["bedrock-agentcore:AllowVendedLogDeliveryForResource"],
  resources: ["*"],
}));
gatewayRole.addToPolicy(new iam.PolicyStatement({
  actions: ["cloudwatch:PutMetricData"],
  resources: ["*"],
  conditions: { StringEquals: { "cloudwatch:namespace": "bedrock-agentcore" } },
}));

3. Instantiate in gateway stack

In lib/nested/gateway-stack.ts:

import { RuntimeObservability } from "../constructs/observability/runtime-observability";

new RuntimeObservability(this, "Observability", {
  runtimeArn: this.gateway.gatewayArn,
  runtimeId: this.gateway.gatewayId,
  resourceType: "gateway",
  logRetentionDays: 30,
});

Gateway deliveries (3 total)

Tab	Log Type	Destination	Log Group
Gateway	APPLICATION_LOGS	CWL	/aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{gatewayId}
Gateway	TRACES	X-Ray	(custom resource)
Identity	APPLICATION_LOGS	CWL	/aws/vendedlogs/bedrock-agentcore/workload-identity-directory/APPLICATION_LOGS/default (shared)

Acceptance Criteria

• RuntimeObservability accepts resourceType?: "runtime" | "gateway" prop (default: "runtime")
• When resourceType = "gateway": log group uses /gateway/APPLICATION_LOGS/ prefix, no USAGE_LOGS created
• Gateway role has AllowVendedLogDeliveryForResource + cloudwatch:PutMetricData
• RuntimeObservability instantiated in gateway stack with resourceType: "gateway"
• Existing runtime observability (assistant + MCP servers) unchanged (default resourceType: "runtime")
• npx cdk synth --quiet passes
• All delivery source/destination names ≤60 chars
• Post-deploy: Gateway shows APPLICATION_LOGS + TRACES in AgentCore console

Files to modify

lib/constructs/observability/runtime-observability.ts   MODIFY — add resourceType prop, conditional USAGE_LOGS
lib/constructs/gateway/mcp-gateway.ts                   MODIFY — add IAM for observability
lib/nested/gateway-stack.ts                             MODIFY — instantiate RuntimeObservability

Constraints

• The gatewayId is already exposed by McpGateway construct (this.gateway.gatewayId)
• The gatewayArn is already exposed (this.gateway.gatewayArn)
• Shared Identity log group is idempotent (custom resource handles it)
• Do NOT break existing runtime/MCP server observability

[sdlc-ai-developer]

Exploration Report

Add observability (APPLICATION_LOGS, X-Ray TRACES, and Identity logs) to the AgentCore Gateway by extending the existing RuntimeObservability construct. The gateway differs from runtimes in log group paths and lacks USAGE_LOGS.

Relevant Files

Files to Modify

lib/constructs/observability/runtime-observability.ts (244 lines)

• Core observability construct that creates CloudWatch Logs Delivery for runtimes
• Currently hardcoded to runtime-specific log group paths and always creates USAGE_LOGS
• Must be generalized to support both runtime and gateway resource types
• Key sections:
  • Lines 10-32: RuntimeObservabilityProps interface
  • Lines 50-99: APPLICATION_LOGS delivery (CWL)
  • Lines 100-141: USAGE_LOGS delivery (CWL) — must become conditional
  • Lines 142-199: X-Ray TRACES delivery (custom resource)
  • Lines 200-241: Identity APPLICATION_LOGS delivery (shared log group)

lib/constructs/gateway/mcp-gateway.ts (180 lines)

• Defines the McpGateway construct that creates the gateway resource
• Exposes: gatewayArn, gatewayId, gatewayUrl, gatewayRole
• Currently missing IAM permissions for observability
• Must add to gatewayRole (lines 26-37):
  • bedrock-agentcore:AllowVendedLogDeliveryForResource on *
  • cloudwatch:PutMetricData on * with namespace condition

lib/nested/gateway-stack.ts (82 lines)

• Stack that instantiates the gateway
• Must import and instantiate RuntimeObservability with resourceType: "gateway"
• Gateway properties already exposed via this.gateway.gatewayArn and this.gateway.gatewayId

Supporting Files (Read-Only)

lib/constructs/observability/xray-delivery-lambda/index.py (197 lines)

• Custom resource handler for X-Ray traces delivery
• Handles both runtime and identity TRACES via CloudWatch Logs Delivery API
• Pattern: creates delivery source/destination/delivery for X-Ray
• Idempotent cleanup in Delete handler
• No changes needed — already supports parameterized runtime ARNs

lib/nested/assistant-stack.ts (instantiation example)

• Lines 84-88: Shows how RuntimeObservability is instantiated for the assistant runtime

new RuntimeObservability(this, "Observability", {
  runtimeArn: this.assistant.runtimeArn,
  runtimeId: this.assistant.runtimeId,
  logRetentionDays: 30,
});

test/config.test.ts (218 lines)

• Jest snapshot tests for config loading and policy name construction
• Test command: npm test
• No tests currently exist for RuntimeObservability construct

Patterns and Conventions

CDK Construct Patterns

1. L1 constructs for CloudWatch Logs Delivery: Uses cdk.CfnResource with type AWS::Logs::LogGroup, AWS::Logs::DeliverySource, AWS::Logs::DeliveryDestination, AWS::Logs::Delivery because these resources don't have L2 constructs yet.

2. Custom resource pattern for X-Ray: X-Ray delivery destination is not supported by CloudFormation, so uses Lambda-backed custom resource (cr.Provider with onEventHandler).

3. Naming convention for delivery resources: Uses shortId (first 49 chars of runtimeId) to ensure delivery source/destination names stay under 60-character limit:

   • {shortId}-app-src, {shortId}-app-dst (application logs)
   • {shortId}-usg-src, {shortId}-usg-dst (usage logs)
   • {shortId}-xray-src, {shortId}-xray-dst (runtime traces)
   • {shortId}-id-src, {shortId}-id-dst (identity application)
   • {shortId}-idxr-src, {shortId}-idxr-dst (identity traces)

4. Log group paths:

   • Runtime APPLICATION_LOGS: /aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/{runtimeId}
   • Runtime USAGE_LOGS: /aws/vendedlogs/bedrock-agentcore/runtime/USAGE_LOGS/{runtimeId}
   • Gateway APPLICATION_LOGS: /aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{gatewayId} (different!)
   • Identity: /aws/vendedlogs/bedrock-agentcore/workload-identity-directory/APPLICATION_LOGS/default (shared)

5. IAM permissions for observability:

   • Runtime role needs: bedrock-agentcore:AllowVendedLogDeliveryForResource, cloudwatch:PutMetricData
   • X-Ray Lambda handler needs: logs delivery CRUD operations, xray:PutResourcePolicy, xray:ListResourcePolicies
   • Pattern: assistant-stack.ts doesn't add these to assistant execution role (observability construct is self-contained)

6. cdk-nag suppressions: Uses NagSuppressions.addResourceSuppressions() with justification strings. Example from runtime-observability.ts:

NagSuppressions.addResourceSuppressions(
  xrayDeliveryHandler.role!,
  [{ id: "AwsSolutions-IAM5", reason: "CloudWatch Logs Delivery API requires wildcard resources..." }],
  true  // applyToChildren
);

Testing Patterns

• Jest for TypeScript unit tests (test/config.test.ts)
• Test command: npm test
• CDK synthesis validation: npx cdk synth --quiet (must pass before commit)
• No snapshot tests for CDK constructs yet (only config/util functions)
• Python tests at project-management/shared/tests/ (38 tests, pytest)

TypeScript Style

• Strict mode, NodeNext module resolution
• No implicit any
• Optional properties marked with ?
• Default values via ?? operator (e.g., props.logRetentionDays ?? 30)
• Public properties exposed via public readonly fields
• Interface naming: <ConstructName>Props

Entry Points and Integration

Implementation Flow

1. Generalize RuntimeObservability construct:

   • Add resourceType?: "runtime" | "gateway" to props (default: "runtime")
   • Use resourceType to compute log group prefix: runtime or gateway
   • Skip USAGE_LOGS creation entirely when resourceType === "gateway" (don't create LogGroup, DeliverySource, DeliveryDestination, Delivery)
   • X-Ray TRACES and Identity deliveries work the same for both types

2. Update McpGateway construct:

   • Add IAM statements to this.gatewayRole:
     • bedrock-agentcore:AllowVendedLogDeliveryForResource on *
     • cloudwatch:PutMetricData on * with StringEquals condition on cloudwatch:namespace === "bedrock-agentcore"

3. Instantiate in GatewayStack:

   • Import RuntimeObservability from ../constructs/observability/runtime-observability
   • After gateway creation, add:

new RuntimeObservability(this, "Observability", {
  runtimeArn: this.gateway.gatewayArn,
  runtimeId: this.gateway.gatewayId,
  resourceType: "gateway",
  logRetentionDays: 30,
});

Deliveries Created (Gateway)

Tab	Log Type	Destination	Log Group
Gateway	APPLICATION_LOGS	CloudWatch Logs	/aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/{gatewayId}
Gateway	TRACES	X-Ray	(custom resource)
Identity	APPLICATION_LOGS	CloudWatch Logs	/aws/vendedlogs/bedrock-agentcore/workload-identity-directory/APPLICATION_LOGS/default

No USAGE_LOGS for gateway.

Testing Strategy

• Manual: npx cdk synth --quiet to validate TypeScript compilation and cdk-nag
• Post-deploy verification: Check AgentCore console for gateway observability tab showing APPLICATION_LOGS + TRACES
• Regression check: Assistant and MCP server runtimes must continue working (default resourceType: "runtime")

Specification Gaps and Clarifications

Resolved from Code

1. Gateway ARN/ID exposure: Confirmed that McpGateway already exposes both gatewayArn and gatewayId as public properties.

2. Identity log group handling: The X-Ray delivery Lambda already creates the shared identity log group idempotently (lines 76-95 of xray-delivery-lambda/index.py). No changes needed.

3. Naming limit enforcement: The construct already truncates runtimeId to 49 chars (line 55: shortId = props.runtimeId.substring(0, 49)) to ensure delivery source/destination names stay under 60 chars. This works for gatewayId too.

4. PR 83 vs PR 86 confusion: Issue description mentions PR 86 for MCP server observability, but git history shows PR 83 (fc32516) added RuntimeObservability to MCP servers. PR 83 is not yet merged to main (exists only in a branch). The current main branch only has observability for the assistant runtime (from PR 80/77).

Open Questions

1. enableIdentityLogs for gateway: Should the gateway instantiation set enableIdentityLogs: true or false? The issue doesn't specify. Assumption: true (omit the prop, use default) to match runtime behavior.

2. Log retention days: Issue specifies 30 days. Should this be configurable via SdlcConfig (like runtime properties) or hardcoded? Assumption: hardcoded to 30 days to match the assistant stack pattern.

3. Gateway workload identity ARN: Does the gateway have a workload-identity ARN like runtimes do? The X-Ray Lambda constructs identity ARN as arn:aws:bedrock-agentcore:{region}:{account}:workload-identity-directory/default/workload-identity/{runtimeId}. For gateway, should it use gatewayId in place of runtimeId? Assumption: yes, the pattern is generic (resourceArn determines the resource, identity uses the same ID).

4. cdk-nag suppressions for gateway stack: The gateway-stack.ts already has broad IAM5 suppressions (line 75). No additional suppressions needed for observability IAM.

Test Command

npm test                     # Run Jest tests (TypeScript)
npx cdk synth --quiet        # Validate CDK synthesis + cdk-nag checks (MUST pass before commit)

Python tests are in project-management/shared/tests/ but not relevant for this CDK change.

Key Dependencies

• aws-cdk-lib: Core CDK library (v2.255.0+)
• cdk-nag: Security/best-practices linting (v2.38.2+)
• Custom resource Lambda: Python 3.12 runtime for X-Ray delivery handler
• CloudWatch Logs Delivery API: Requires logs:PutDeliverySource, logs:CreateDelivery, etc.
• AgentCore Gateway API: Uses gatewayArn and gatewayId from createGateway response

Constraints and Edge Cases

1. Backward compatibility: Default resourceType: "runtime" ensures existing assistant/MCP instantiations continue working without changes.

2. 60-character name limit: Delivery source/destination names have a 60-char limit. The construct already handles this via shortId (49 chars + suffixes like -app-src = 57 chars max).

3. Shared identity log group: Multiple resources (assistant, MCP servers, gateway) all write to the same identity log group. The X-Ray Lambda handler already handles this idempotently.

4. Stack deployment order: Gateway stack deploys before assistant stack, so both can have observability without circular dependencies.

5. IAM wildcard resources: CloudWatch Logs Delivery API requires * resources for cross-service delivery setup. This is already suppressed in the runtime-observability construct (line 176-185) and doesn't need additional suppressions in gateway-stack.

6. No gateway USAGE_LOGS: The issue explicitly states gateway does NOT have USAGE_LOGS. The construct must skip creating all four resources (LogGroup, DeliverySource, DeliveryDestination, Delivery) when resourceType === "gateway".

References

• AWS Docs: Amazon Bedrock AgentCore Gateway
• AWS Docs: Add observability to AgentCore resources
• Codebase: coding-assistants/claude-code/obs.md — observability architecture and query patterns
• Codebase: CONTRIBUTING.md — CDK architecture and extension patterns
• Codebase: CLAUDE.md — project conventions and security rules

[sdlc-ai-developer]

Report

Before implementing, a few points in the spec are ambiguous and would change the generated code. Please clarify:

1. Identity logs for the gateway — Should the gateway's RuntimeObservability instantiation enable identity APPLICATION_LOGS (and identity TRACES)? The deliveries table lists Identity APPLICATION_LOGS as one of the 3 gateway deliveries, but the sample instantiation snippet omits enableIdentityLogs. Confirm the gateway should use the default identity behavior, or whether it should be set explicitly.

2. Log retention configurability — The issue hardcodes logRetentionDays: 30. Should retention stay hardcoded (matching the assistant stack) or be surfaced through SdlcConfig? Confirm hardcoded 30 days is acceptable.

3. Gateway workload-identity ARN — The X-Ray delivery Lambda builds the identity ARN as arn:aws:bedrock-agentcore:{region}:{account}:workload-identity-directory/default/workload-identity/{runtimeId}. For a gateway, should gatewayId be substituted for runtimeId, or does the gateway lack a per-resource workload identity (meaning identity TRACES should be skipped)? This determines whether the identity TRACES delivery is created for the gateway.

[EashanKaushik]

Clarifications

1. enableIdentityLogs for gateway

Use the default (true). Same as runtimes.

2. Log retention days

Hardcode to 30 days. Same pattern as assistant stack. No need for config-driven value.

3. Gateway workload identity ARN

Yes, use gatewayId in place of runtimeId. The pattern is the same:
arn:aws:bedrock-agentcore:{region}:{account}:workload-identity-directory/default/workload-identity/{gatewayId}

4. cdk-nag suppressions

Correct — the existing gateway-stack suppressions are broad enough. No additional suppressions needed.

Proceed with implementation.