diff --git a/README.md b/README.md index 5abe097..9c9dd33 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ export LOGSTORY_CUSTOMER_ID=01234567-0123-4321-abcd-01234567890a export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json export LOGSTORY_REGION=US export LOGSTORY_AUTO_GET=true # Auto-download missing usecases +export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB # Filter usecases (pipe-separated) logstory replay usecase RULES_SEARCH_WORKSHOP ``` @@ -74,6 +75,7 @@ LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json LOGSTORY_REGION=US LOGSTORY_USECASES_BUCKETS=gs://logstory-usecases-20241216,gs://my-custom-bucket LOGSTORY_AUTO_GET=true # Auto-download missing usecases (optional) +LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB # Filter usecases (pipe-separated, optional) ``` Then run commands without additional options: @@ -311,6 +313,11 @@ logstory replay all \ --customer-id=01234567-0123-4321-abcd-01234567890a \ --credentials-path=/path/to/credentials.json +# Replay specific usecases using environment variable filtering +LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB logstory replay all \ + --customer-id=01234567-0123-4321-abcd-01234567890a \ + --credentials-path=/path/to/credentials.json + # Replay a specific usecase logstory replay usecase RULES_SEARCH_WORKSHOP \ --customer-id=01234567-0123-4321-abcd-01234567890a \ @@ -466,6 +473,7 @@ tree /tmp/var/log/logstory/ - `LOGSTORY_CREDENTIALS_PATH`: Path to JSON credentials file - `LOGSTORY_REGION`: SecOps tenant region (default: US) - `LOGSTORY_LOCAL_LOG_DIR`: Base directory for local file output (default: /tmp/var/log/logstory) +- `LOGSTORY_USECASES`: Filter usecases for `replay all` command (pipe-separated, e.g., `NETWORK_ANALYSIS|GITHUB`) ### Command Migration Guide diff --git a/docs/configuration.md b/docs/configuration.md index 59a1939..ba054d2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -26,6 +26,7 @@ export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json export LOGSTORY_REGION=US export LOGSTORY_USECASES_BUCKETS=gs://my-bucket,file:///local/usecases export LOGSTORY_AUTO_GET=true # Auto-download missing usecases +export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB # Filter usecases for 'replay all' command # Now run commands without additional options logstory replay usecase RULES_SEARCH_WORKSHOP # Will auto-download if missing diff --git a/docs/env-file.md b/docs/env-file.md index cb0d7f8..2a9f05d 100644 --- a/docs/env-file.md +++ b/docs/env-file.md @@ -490,6 +490,9 @@ LOGSTORY_REGION=US # Supports: gs://bucket, file:///path, bare-bucket-name LOGSTORY_USECASES_BUCKETS=gs://bucket1,file:///local/usecases +# Optional: Filter usecases for 'replay all' command (pipe-separated) +LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB + # Optional: Local file output directory LOGSTORY_LOCAL_LOG_DIR=/tmp/var/log/logstory diff --git a/docs/index.md b/docs/index.md index 82aa8ab..22b0e5f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -460,6 +460,7 @@ Quick start using Makefile targets: export LOGSTORY_PROJECT_ID=your-gcp-project-id export LOGSTORY_CUSTOMER_ID=your-chronicle-customer-uuid export LOGSTORY_API_TYPE=rest # or 'legacy' +export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB # Optional: filter usecases (pipe-separated) # Deploy to Cloud Run make enable-apis diff --git a/docs/testing-pipe-delimited-usecases.md b/docs/testing-pipe-delimited-usecases.md new file mode 100644 index 0000000..058cb93 --- /dev/null +++ b/docs/testing-pipe-delimited-usecases.md @@ -0,0 +1,268 @@ +# Testing Pipe-Delimited Usecase Filtering in Cloud Run + +## Overview + +This document provides testing instructions for the pipe-delimited usecase filtering functionality in Logstory's Cloud Run deployment. This feature allows you to filter which usecases are processed when using `replay all` by setting the `LOGSTORY_USECASES` environment variable. + +## Feature Description + +The `LOGSTORY_USECASES` environment variable filtering feature allows you to: + +- Use `replay all` command with selective usecase processing +- Avoid gcloud argument parsing issues entirely +- Filter usecases using pipe-separated values (e.g., `NETWORK_ANALYSIS|GITHUB`) +- Maintain backward compatibility - no filtering variable means process all usecases + +## Implementation + +When `LOGSTORY_USECASES` is set, the `replay all` command: +1. Parses the pipe-separated usecase list +2. Validates that all requested usecases exist +3. Only processes the specified usecases instead of all available ones +4. Provides clear feedback about which usecases are being processed + +## Local Testing + +### Prerequisites + +Set up your development environment: + +```bash +# Create and activate virtual environment +python -m venv venv +source venv/bin/activate + +# Install in editable mode +pip install -e . + +# Set required environment variables +export LOGSTORY_CUSTOMER_ID=your-uuid +export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json +export LOGSTORY_REGION=US +export LOGSTORY_API_TYPE=rest +``` + +### Test Cases + +#### 1. Test Normal Behavior (No Filtering) + +```bash +# Should process all available usecases +logstory replay all --local-file-output + +# Expected output: "Processing all usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, ..." +``` + +#### 2. Test Single Usecase Filtering + +```bash +# Filter to single usecase +LOGSTORY_USECASES=NETWORK_ANALYSIS logstory replay all --local-file-output + +# Expected output: "Processing filtered usecases: NETWORK_ANALYSIS" +``` + +#### 3. Test Multiple Usecase Filtering + +```bash +# Filter to multiple usecases +LOGSTORY_USECASES=NETWORK_ANALYSIS\|RULES_SEARCH_WORKSHOP logstory replay all --local-file-output + +# Expected output: "Processing filtered usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP" +``` + +#### 4. Test Error Handling + +```bash +# Test with invalid usecase +LOGSTORY_USECASES=INVALID_USECASE logstory replay all --local-file-output + +# Expected: Error message listing available usecases +``` + +## Cloud Run Testing + +### Prerequisites + +Ensure your Cloud Run environment is set up: + +```bash +# Set required environment variables +export LOGSTORY_API_TYPE=rest +export LOGSTORY_CUSTOMER_ID=your-uuid +export LOGSTORY_PROJECT_ID=your-project-id +export LOGSTORY_REGION=US + +# Build and deploy updated Docker image +make build +make docker-build +``` + +### Test Cases + +#### 1. Test Normal Cloud Run Behavior + +```bash +# Process all usecases (no filtering) +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all" \ + --wait +``` + +#### 2. Test Single Usecase Filtering + +```bash +# Filter to single usecase +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all" \ + --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS" \ + --wait +``` + +#### 3. Test Multiple Usecase Filtering + +```bash +# Filter to multiple usecases using pipe separator +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all" \ + --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP" \ + --wait +``` + +#### 4. Test with Additional Options + +```bash +# Combine filtering with entities and custom timestamp delta +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all,--entities" \ + --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS,LOGSTORY_TIMESTAMP_DELTA=3d" \ + --wait +``` + +## Validation Methods + +### Check Execution Status + +```bash +# Get the latest execution name +EXECUTION_NAME=$(gcloud run jobs executions list \ + --job logstory-replay \ + --region us-central1 \ + --limit 1 \ + --format "value(name)") + +# Check if it completed successfully +gcloud run jobs executions describe $EXECUTION_NAME \ + --region us-central1 \ + --format "value(status.conditions[0].status)" +``` + +### Analyze Logs + +```bash +# Install beta components if needed +gcloud components install beta --quiet + +# View execution logs +gcloud beta run jobs executions logs read $EXECUTION_NAME --region us-central1 +``` + +### Success Indicators + +Look for these patterns in the logs: + +**Filtered Processing:** +``` +Processing filtered usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP +Processing usecase: NETWORK_ANALYSIS, logtype: BRO_JSON +Processing usecase: RULES_SEARCH_WORKSHOP, logtype: POWERSHELL +Successfully posted entries using RestIngestionBackend +``` + +**Normal Processing:** +``` +Processing all usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, THW2 +``` + +**Error Handling:** +``` +Error: Invalid usecases: INVALID_NAME +Available usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, THW2 +``` + +## Troubleshooting + +### Common Issues + +#### Issue: Container Exits with Error +**Solution:** Check execution logs for specific error messages and verify environment variables are set correctly. + +#### Issue: All Usecases Processed Despite Filtering +**Solution:** Verify the Docker image was rebuilt and deployed after code changes. Check that `LOGSTORY_USECASES` environment variable is correctly set. + +#### Issue: Invalid Usecase Error +**Solution:** Run `logstory usecases list-installed` to see available usecases, or check the error message for the list of valid options. + +### Debugging Commands + +```bash +# Check job configuration +gcloud run jobs describe logstory-replay --region us-central1 + +# List recent executions +gcloud run jobs executions list \ + --job logstory-replay \ + --region us-central1 \ + --limit 5 + +# Check environment variables in job +gcloud run jobs describe logstory-replay \ + --region us-central1 \ + --format "value(spec.template.template.spec.template.spec.containers[0].env[])" +``` + +## Benefits + +1. **No gcloud parsing issues** - Environment variables avoid command-line argument parsing problems +2. **Clean command structure** - Simple `logstory,replay,all` arguments +3. **Flexible filtering** - Easy to specify any combination of usecases +4. **Backward compatible** - Existing behavior preserved when no filtering is specified +5. **Clear feedback** - Logs clearly show which usecases are being processed + +## Usage Examples + +### Development Workflow + +```bash +# Test locally first +LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP logstory replay all --local-file-output + +# Deploy to Cloud Run +make build && make docker-build + +# Test in Cloud Run +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all" \ + --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP" \ + --wait +``` + +### Production Usage + +```bash +# Set environment variables in .env file or Cloud Run job configuration +LOGSTORY_USECASES=PRODUCTION_USECASE_1|PRODUCTION_USECASE_2 + +# Deploy with scheduled execution +gcloud run jobs execute logstory-replay \ + --region us-central1 \ + --args "logstory,replay,all" \ + --wait +``` + +This implementation provides a clean, reliable solution for filtering usecases in Cloud Run deployments while maintaining full backward compatibility and avoiding all gcloud argument parsing issues. \ No newline at end of file diff --git a/src/logstory/logstory.py b/src/logstory/logstory.py index e7228fe..ba55ae7 100644 --- a/src/logstory/logstory.py +++ b/src/logstory/logstory.py @@ -41,7 +41,7 @@ def version_callback(value: bool): except Exception: __version__ = "unknown" typer.echo(f"logstory {__version__}") - raise typer.Exit() + raise typer.Exit # Create Typer app and command groups @@ -846,7 +846,7 @@ def replay_all_usecases( ), usecases_bucket: str | None = UsecasesBucketOption, ): - """Replay all usecases.""" + """Replay all usecases (or filtered usecases if LOGSTORY_USECASES is set).""" # Load environment file first (needed for download logic) load_env_file(env_file) @@ -892,7 +892,26 @@ def replay_all_usecases( impersonate_service_account, ) - usecases = get_usecases() + # Get all available usecases + all_usecases = get_usecases() + + # Filter usecases if LOGSTORY_USECASES environment variable is set + filter_usecases = os.getenv("LOGSTORY_USECASES") + if filter_usecases: + # Parse pipe-separated list and filter + requested_usecases = [uc.strip() for uc in filter_usecases.split("|") if uc.strip()] + # Validate that requested usecases exist + invalid_usecases = [uc for uc in requested_usecases if uc not in all_usecases] + if invalid_usecases: + typer.echo(f"Error: Invalid usecases: {', '.join(invalid_usecases)}") + typer.echo(f"Available usecases: {', '.join(sorted(all_usecases))}") + raise typer.Exit(1) + usecases = requested_usecases + typer.echo(f"Processing filtered usecases: {', '.join(usecases)}") + else: + usecases = all_usecases + typer.echo(f"Processing all usecases: {', '.join(usecases)}") + _replay_usecases(usecases, "*", entities, timestamp_delta, local_file_output)