Sentinel Infrastructure Deployment Guide

This guide covers the deployment of the Sentinel Cybersecurity Triage Platform infrastructure using AWS CloudFormation with a modular stack approach.

Overview

The Sentinel platform consists of several AWS services working together to provide automated cybersecurity news ingestion, analysis, and triage capabilities. The infrastructure is now split into 5 modular CloudFormation stacks for better management and deployment flexibility.

Stack Architecture

The infrastructure is organized into the following stacks:

Core Stack (01-core-stack.yaml)
- KMS encryption keys
- Artifacts S3 bucket (for Lambda packages)
- Random suffix generator for unique naming
- Foundation resources needed by all other stacks
Storage Stack (02-storage-stack.yaml)
- DynamoDB tables (Articles, Comments, Memory)
- S3 buckets (Content, Traces)
- CloudWatch log groups
- Data persistence layer
Backend Services Stack (03-backend-services-stack.yaml)
- Lambda functions (all processing logic)
- Step Functions state machine
- IAM roles for Lambda execution
- Core application logic
AI/Bedrock Stack (04-ai-bedrock-stack.yaml)
- OpenSearch Serverless collection
- Bedrock Knowledge Base
- Bedrock Agents
- AI/ML components
API & Auth Stack (05-api-auth-stack.yaml)
- Cognito User Pool and Identity Pool
- API Gateway REST API
- Amplify web application
- CloudWatch dashboard
- User-facing components

Architecture Components

Data Storage: DynamoDB tables for articles, comments, and memory
File Storage: S3 buckets for content, artifacts, and traces
Compute: Lambda functions for processing logic
Orchestration: Step Functions for workflow management
AI/ML: Bedrock agents and knowledge bases for intelligent processing
Search: OpenSearch Serverless for vector similarity search
Authentication: Cognito for user management
API: API Gateway for REST endpoints
Frontend: Amplify for web application hosting
Monitoring: CloudWatch for logging and metrics

Prerequisites

Before deploying the infrastructure, ensure you have:

AWS CLI installed and configured with appropriate credentials
Sufficient AWS permissions to create the required resources
AWS Bedrock access enabled in your account and region
Domain/Email setup for SES notifications (optional)
Git repository for Amplify deployment (optional)
jq installed for JSON processing (used in delete script)

Required AWS Permissions

Your AWS user/role needs permissions for:

CloudFormation (full access)
IAM (create/manage roles and policies)
S3 (create/manage buckets)
DynamoDB (create/manage tables)
Lambda (create/manage functions)
Step Functions (create/manage state machines)
Bedrock (access to models and agents)
OpenSearch Serverless (create/manage collections)
Cognito (create/manage user pools)
API Gateway (create/manage APIs)
Amplify (create/manage applications)
CloudWatch (create/manage dashboards and logs)
KMS (create/manage encryption keys)

Deployment Steps

1. Clone and Prepare

git clone <repository-url>
cd sentinel

2. Configure Parameters

The deployment now uses stack-specific parameter files for better organization:

cloudformation/parameters-core.json - Core Stack parameters
cloudformation/parameters-storage.json - Storage Stack parameters
cloudformation/parameters-backend.json - Backend Services parameters
cloudformation/parameters-ai.json - AI/Bedrock Stack parameters
cloudformation/parameters-api-auth.json - API & Auth Stack parameters

You can also use the legacy parameters-prod.json file, but the stack-specific files are recommended.

Key parameters to customize:

SESenderEmail - Email address for notifications
EscalationEmails - Comma-separated list of escalation emails
AlertEmails - Comma-separated list of alert emails
AmplifyRepositoryUrl - Git repository URL for web app
BedrockModelId - Bedrock model for LLM operations
RelevanceThreshold - Minimum relevance score (0.0-1.0)

3. Deploy Infrastructure

Using the Deploy Script (Recommended)

cd cloudformation

# Deploy all stacks
./deploy-stacks.sh

# Or deploy Lambda packages separately
./deploy-lambda-packages.sh -e prod -b <artifacts-bucket-name>

# Deploy Lambda packages and recreate storage stack
./deploy-lambda-packages.sh -e prod -b <artifacts-bucket-name> -r

The deployment process:

Validate all CloudFormation templates
Deploy stacks in the correct dependency order:
- Core Stack (foundation)
- Package and upload Lambda functions from src/lambda/
- Storage Stack (data layer)
- Backend Services Stack (application logic)
- AI/Bedrock Stack (AI components)
- API & Auth Stack (user interface)
Display deployment outputs and next steps

Manual Deployment (Advanced)

If you prefer manual control, deploy stacks in this order:

# 1. Deploy Core Stack
aws cloudformation create-stack \
  --stack-name sentinel-prod-core \
  --template-body file://01-core-stack.yaml \
  --parameters file://parameters-prod.json \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

# Wait for completion
aws cloudformation wait stack-create-complete \
  --stack-name sentinel-prod-core \
  --region us-east-1

# 2. Package and upload Lambda functions
./deploy-lambda-packages.sh -e prod -b <artifacts-bucket-name>

# 3. Deploy Storage Stack
aws cloudformation create-stack \
  --stack-name sentinel-prod-storage \
  --template-body file://02-storage-stack.yaml \
  --parameters file://parameters-prod.json \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

# Continue with remaining stacks...

4. Lambda Function Implementation

The deployment script now uses the actual source code from the src/lambda/ directory. The expected structure is:

src/
├── common/                          # Shared utilities (optional)
│   ├── __init__.py
│   ├── aws_clients.py              # AWS service clients
│   ├── bedrock_utils.py            # Bedrock model utilities
│   └── logging_utils.py            # Logging configuration
└── lambda/
    ├── feed-parser/
    │   ├── lambda_function.py      # Main handler
    │   └── requirements.txt        # Dependencies (optional)
    ├── relevancy-evaluator/
    ├── dedup-tool/
    ├── guardrail-tool/
    ├── storage-tool/
    ├── human-escalation/
    ├── notifier/
    ├── analyst-assistant/
    ├── commentary-api/
    ├── publish-decision/
    ├── query-kb/
    └── opensearch-index-manager/   # Custom resource for OpenSearch

If a function directory doesn't exist, the script creates a placeholder implementation.

5. Verify Deployment

Check the stack outputs for important URLs and identifiers:

# Check all stack outputs
aws cloudformation describe-stacks --stack-name sentinel-prod-core --query 'Stacks[0].Outputs'
aws cloudformation describe-stacks --stack-name sentinel-prod-storage --query 'Stacks[0].Outputs'
aws cloudformation describe-stacks --stack-name sentinel-prod-backend --query 'Stacks[0].Outputs'
aws cloudformation describe-stacks --stack-name sentinel-prod-ai --query 'Stacks[0].Outputs'
aws cloudformation describe-stacks --stack-name sentinel-prod-api-auth --query 'Stacks[0].Outputs'

Post-Deployment Configuration

1. Implement Lambda Functions

Replace the placeholder Lambda functions with actual implementations:

Feed Parser: RSS feed parsing and article extraction
Relevancy Evaluator: AI-powered relevance scoring
Dedup Tool: Semantic deduplication using embeddings
Guardrail Tool: Content policy enforcement
Storage Tool: Article storage and retrieval
Human Escalation: Notification and escalation logic
Notifier: Email and alert notifications
Analyst Assistant: AI chat interface
Commentary API: Article comments and annotations
Publish Decision: Human review decision processing
Query KB: Knowledge base RAG queries

2. Cognito User Setup

Create users in the Cognito User Pool:

# Get User Pool ID from stack outputs
USER_POOL_ID=$(aws cloudformation describe-stacks \
  --stack-name sentinel-prod-api-auth \
  --query 'Stacks[0].Outputs[?OutputKey==`UserPoolId`].OutputValue' \
  --output text)

# Create admin user
aws cognito-idp admin-create-user \
  --user-pool-id $USER_POOL_ID \
  --username admin \
  --user-attributes Name=email,Value=admin@example.com \
  --temporary-password TempPass123! \
  --message-action SUPPRESS

3. SES Email Verification

If using email notifications, verify your sender email in SES:

aws ses verify-email-identity --email-address noreply@yourdomain.com

4. Amplify Repository Connection

Connect your Amplify app to a Git repository:

Go to the AWS Amplify console
Find your app (sentinel-prod-app)
Connect to your Git repository
Configure build settings

5. Configure RSS Feeds

Update the config/feeds.yaml file with your RSS feed sources:

feeds:
  - name: "Krebs on Security"
    url: "https://krebsonsecurity.com/feed/"
    category: "security_news"
    priority: "high"
  - name: "Threatpost"
    url: "https://threatpost.com/feed/"
    category: "threat_intelligence"
    priority: "medium"

6. Update Keywords

Modify config/keywords.yaml to define relevance keywords:

high_priority:
  - "zero-day"
  - "ransomware"
  - "data breach"
  - "vulnerability"

medium_priority:
  - "malware"
  - "phishing"
  - "security update"

low_priority:
  - "security awareness"
  - "best practices"

7. OpenSearch Index Setup

If using AI features, you may need to create indices in OpenSearch:

# Get OpenSearch endpoint from stack outputs
OPENSEARCH_ENDPOINT=$(aws cloudformation describe-stacks \
  --stack-name sentinel-prod-ai \
  --query 'Stacks[0].Outputs[?OutputKey==`OpenSearchCollectionEndpoint`].OutputValue' \
  --output text)

# Create vector index (example)
curl -X PUT "$OPENSEARCH_ENDPOINT/sentinel-vector-index" \
  -H "Content-Type: application/json" \
  -d '{
    "mappings": {
      "properties": {
        "vector": {"type": "knn_vector", "dimension": 1536},
        "text": {"type": "text"},
        "metadata": {"type": "object"}
      }
    }
  }'

Testing the Deployment

1. Test API Endpoints

# Get API Gateway URL from stack outputs
API_URL=$(aws cloudformation describe-stacks \
  --stack-name sentinel-prod-api-auth \
  --query 'Stacks[0].Outputs[?OutputKey==`ApiGatewayUrl`].OutputValue' \
  --output text)

# Test analyst assistant endpoint (will return placeholder response)
curl -X POST "$API_URL/analyst-assistant" \
  -H "Content-Type: application/json" \
  -d '{"message": "What are the latest security threats?"}'

2. Test Step Functions

# Get State Machine ARN
STATE_MACHINE_ARN=$(aws cloudformation describe-stacks \
  --stack-name sentinel-prod-backend \
  --query 'Stacks[0].Outputs[?OutputKey==`IngestionStateMachineArn`].OutputValue' \
  --output text)

# Start execution
aws stepfunctions start-execution \
  --state-machine-arn $STATE_MACHINE_ARN \
  --name test-execution-$(date +%s) \
  --input '{"feedConfigs": [], "batchSize": 10}'

3. Monitor Logs

# View Lambda logs
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/sentinel-prod"

# View Step Functions logs
aws logs describe-log-groups --log-group-name-prefix "/aws/stepfunctions/sentinel-prod"

Stack Management

Updating Stacks

To update individual stacks:

# Update a specific stack
aws cloudformation update-stack \
  --stack-name sentinel-prod-backend \
  --template-body file://03-backend-services-stack.yaml \
  --parameters file://parameters-prod.json \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

Stack Dependencies

Remember the dependency order when updating:

Core Stack (no dependencies)
Storage Stack (depends on Core)
Backend Services Stack (depends on Core, Storage)
AI/Bedrock Stack (depends on Core, Storage)
API & Auth Stack (depends on Backend Services)

Monitoring and Maintenance

CloudWatch Dashboard

Access the pre-configured dashboard:

Go to CloudWatch console
Navigate to Dashboards
Open "sentinel-prod-dashboard"

Key Metrics to Monitor

Lambda Function Errors: Monitor error rates and duration
DynamoDB Throttling: Watch for read/write capacity issues
Step Functions Failures: Track workflow execution success
API Gateway 4xx/5xx Errors: Monitor API health
S3 Storage Costs: Track storage growth
Bedrock API Calls: Monitor LLM usage and costs

Cost Optimization

Review DynamoDB Usage: Consider switching to provisioned capacity for predictable workloads
S3 Lifecycle Policies: Ensure old content transitions to cheaper storage classes
Lambda Memory Optimization: Right-size memory allocation based on usage patterns
Bedrock Usage: Monitor and optimize LLM API calls

Troubleshooting

Common Issues

Stack Creation Fails
- Check IAM permissions
- Verify Bedrock access in your region
- Ensure unique S3 bucket names
- Check cross-stack references
Lambda Functions Timeout
- Increase timeout values in parameters
- Check for network connectivity issues
- Review function logs for bottlenecks
Cross-Stack Reference Errors
- Ensure stacks are deployed in correct order
- Check that exported values exist
- Verify stack names match expected patterns
Bedrock Access Denied
- Ensure Bedrock is enabled in your AWS account
- Check model access permissions
- Verify region availability
OpenSearch Collection Creation Fails
- Check service quotas
- Verify encryption policy configuration
- Ensure proper IAM permissions

Getting Help

Check CloudFormation events for detailed error messages
Review CloudWatch logs for runtime issues
Consult AWS documentation for service-specific troubleshooting
Use AWS Support for complex issues

Cleanup

To remove all infrastructure:

cd cloudformation
./delete-stacks.sh

The script will:

Confirm deletion (type 'DELETE' to proceed)
Empty all S3 buckets
Delete OpenSearch collections
Delete stacks in reverse dependency order
Provide cleanup verification steps

Warning: This will delete all data and cannot be undone. Make sure to backup any important data before cleanup.

Force Deletion (Automated)

For automated cleanup (use with caution):

./delete-stacks.sh --force

Security Considerations

Encryption: All data is encrypted at rest using KMS
Network Security: Resources are deployed with proper security groups
IAM Least Privilege: Roles have minimal required permissions
API Security: API Gateway uses Cognito authentication
Cross-Stack References: Secure resource sharing between stacks
Secrets Management: Use AWS Secrets Manager for sensitive configuration

Scaling Considerations

DynamoDB: Auto-scaling enabled for read/write capacity
Lambda: Concurrent execution limits can be adjusted
Step Functions: Parallel processing limits configurable
API Gateway: Built-in scaling with throttling controls
OpenSearch: Serverless scaling based on usage
Modular Stacks: Individual components can be scaled independently

Backup and Disaster Recovery

DynamoDB: Point-in-time recovery enabled
S3: Versioning and cross-region replication available
Lambda: Code stored in S3 for recovery
Configuration: Infrastructure as Code enables quick recreation
Cross-Stack Exports: Enable easy disaster recovery across stacks

Development Workflow

Lambda Development

Implement functions in src/lambda/<function-name>/
Add dependencies to requirements.txt in each function directory
Test locally using AWS SAM or similar tools
Re-run deployment script to update functions

Stack Updates

Modify CloudFormation templates
Update parameters if needed
Deploy individual stacks or use full deployment script
Monitor CloudFormation events for issues

CI/CD Integration

Consider integrating with:

AWS CodePipeline for automated deployments
GitHub Actions for CI/CD workflows
AWS SAM for Lambda function development and testing

This completes the deployment guide for the modular Sentinel Cybersecurity Triage Platform.

FilesExpand file tree

02-deployment-guide.md

Latest commit

History