Skip to content

feat: app --region support to metrics collector script #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: app --region support to metrics collector script #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions metrics-collector/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,25 @@ This project is built with Poetry for dependency management. To get started:
3. Alternatively, you can use pip with the provided `requirements.txt`:
`pip install -r requirements.txt`

The install might take a couple of minutes because of the dependencies.
The install might take a couple of minutes because of the dependencies.

## 🏃‍♂️ Usage

Run the metrics collector with a single command:

```bash
# Scan all regions
python -m metrics_collector.utilization_example --start-time 2025-02-19

# Scan only a specific region
python -m metrics_collector.utilization_example --start-time 2025-02-19 --region us-east-1
```

Options:

--start-time: Specify the start time for metric collection (ISO8601 format)
--end-time: Specify the end time (defaults to current time if not provided)
--region: AWS region to scan (if not specified, all regions will be scanned)
--config: Path to a custom configuration file
--output: Custom name for the output CSV file
```
Expand Down Expand Up @@ -161,9 +167,9 @@ This configuration flexibility allows you to tailor the metrics collection to yo

There is a companion project in the making where we will simplify metric visualization. Stay tuned for future updates!

Want to turn your CSV data into stunning visualizations? Check out our companion project DynamoDB Metrics Visualizer to create interactive dashboards and charts!
Want to turn your CSV data into stunning visualizations? Check out our companion project DynamoDB Metrics Visualizer to create interactive dashboards and charts!

The current report will present data in csv
The current report will present data in csv

| Region | Table Name | Read Utilization | Write Utilization |
|--------|------------|------------------|-------------------|
Expand Down Expand Up @@ -196,4 +202,4 @@ We welcome contributions! Please see our Contributing Guide for more details.

This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.

Built with ❤️ by DynamoDB Specialist Solutions Architects.
Built with ❤️ by DynamoDB Specialist Solutions Architects.
17 changes: 11 additions & 6 deletions metrics-collector/metrics_collector/collector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,31 +129,36 @@ async def get_tables_and_metrics(self, region, start_time, end_time):
table_metrics = await asyncio.gather(*tasks)
return region, provisioned_tables, table_metrics

async def collect_all_metrics(self, start_time, end_time):
async def collect_all_metrics(self, start_time, end_time, specific_region=None):
"""
Collect metrics for all provisioned DynamoDB tables across all regions.
Collect metrics for all provisioned DynamoDB tables across all regions or a specific region.

Args:
start_time (datetime): The start time for metric collection.
end_time (datetime): The end time for metric collection.
specific_region (str, optional): If provided, only collect metrics for this region.

Returns:
tuple: A tuple containing all metrics and low utilization tables.
"""
all_metrics = {}
low_utilization_tables = {}

logger.info("Fetching all AWS regions...")
regions = await self.get_all_regions()
logger.info(f"Found {len(regions)} regions.")
if specific_region:
logger.info(f"Collecting metrics for specific region: {specific_region}")
regions = [specific_region]
else:
logger.info("Fetching all AWS regions...")
regions = await self.get_all_regions()
logger.info(f"Found {len(regions)} regions.")

logger.info("Identifying provisioned tables in each region...")
total_provisioned_tables = 0
async for region in tqdm_asyncio(regions, desc="Scanning regions"):
provisioned_tables = await self.get_provisioned_tables(region)
total_provisioned_tables += len(provisioned_tables)

logger.info(f"Found {total_provisioned_tables} provisioned tables across all regions.")
logger.info(f"Found {total_provisioned_tables} provisioned tables across {'all regions' if not specific_region else specific_region}.")

logger.info("Collecting metrics for provisioned tables...")
region_tasks = [self.get_tables_and_metrics(region, start_time, end_time) for region in regions]
Expand Down
7 changes: 6 additions & 1 deletion metrics-collector/metrics_collector/utilization_example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,11 @@ async def main():
parser = argparse.ArgumentParser(description="Collect DynamoDB metrics")
parser.add_argument("--start-time", type=str, help="Start time in ISO8601 format")
parser.add_argument("--end-time", type=str, help="End time in ISO8601 format")
parser.add_argument(
Copy link
Contributor

@tebanieo tebanieo Oct 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create first a constant that includes all the regions.

AWS_REGIONS = [
    "us-east-1", "us-east-2", "us-west-1", "us-west-2",
    "ca-central-1", "eu-west-1", "eu-west-2", "eu-west-3", 
    "eu-central-1", "eu-north-1", "eu-south-1", "ap-south-1",
    "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", 
    "ap-southeast-1", "ap-southeast-2", "ap-southeast-3",
    "sa-east-1", "me-south-1", "af-south-1"
]

Create a validation function

def is_valid_aws_region(region):
    return region in AWS_REGIONS

def validate_aws_region(region):
    if not is_valid_aws_region(region):
        raise argparse.ArgumentTypeError(f"Invalid AWS region: {region}")
    return region

And then add this to the arg parsing.

parser.add_argument('--region', '-r', required=False,
                        type=validate_aws_region,
                        help='AWS region to use')

"--region",
type=str,
help="AWS region to scan (if not specified, all regions will be scanned)",
)
parser.add_argument(
"--storage",
choices=["disk", "memory"],
Expand Down Expand Up @@ -170,7 +175,7 @@ async def main():

logger.info(f"Collecting metrics from {start_time} to {end_time}")

all_metrics, low_utilization_tables = await collector.collect_all_metrics(start_time, end_time)
all_metrics, low_utilization_tables = await collector.collect_all_metrics(start_time, end_time, args.region)

logger.info("Metrics collected and stored successfully.")

Expand Down