Merge pull request duo-labs#4 from duo-labs/master

refresh from master fork

README.md

@@ -4,15 +4,44 @@ CloudMapper
 
 CloudMapper helps you analyze your Amazon Web Services (AWS) environments.  The original purpose was to generate network diagrams and display them in your browser.  It now contains much more functionality, including auditing for security issues.
 
-*Network mapping demo: https://duo-labs.github.io/cloudmapper/*
+- [Network mapping demo](https://duo-labs.github.io/cloudmapper/)
+- [Report demo](https://duo-labs.github.io/cloudmapper/account-data/report.html)
+- [Intro post](https://duo.com/blog/introducing-cloudmapper-an-aws-visualization-tool)
+- [Post to show usage in spotting misconfigurations](https://duo.com/blog/spotting-misconfigurations-with-cloudmapper)
 
-*Report demo: https://duo-labs.github.io/cloudmapper/account-data/report.html*
+# Commands
+
+- `api_endpoints`: List the URLs that can be called via API Gateway.
+- `audit`: Check for potential misconfigurations.
+- `collect`: Collect metadata about an account. More details [here](https://summitroute.com/blog/2018/06/05/cloudmapper_collect/).
+- `find_admins`: Look at IAM policies to identify admin users and roles and spot potential IAM issues. More details [here](https://summitroute.com/blog/2018/06/12/cloudmapper_find_admins/).
+- `find_unused`: Look for unused resources in the account.  Finds unused Security Groups, Elastic IPs, network interfaces, and volumes.
+- `prepare`/`webserver`: See [Network Visualizations](docs/network_visualizations.md)
+- `public`: Find public hosts and port ranges. More details [here](https://summitroute.com/blog/2018/06/13/cloudmapper_public/).
+- `sg_ips`: Get geoip info on CIDRs trusted in Security Groups. More details [here](https://summitroute.com/blog/2018/06/12/cloudmapper_sg_ips/).
+- `stats`: Show counts of resources for accounts. More details [here](https://summitroute.com/blog/2018/06/06/cloudmapper_stats/).
+- `weboftrust`: Show Web Of Trust. More details [here](https://summitroute.com/blog/2018/06/13/cloudmapper_wot/).
+- `report`: Generate HTML report. Includes summary of the accounts and audit findings. More details [here](https://summitroute.com/blog/2019/03/04/cloudmapper_report_generation/).
+- `iam_report`: Generate HTML report for the IAM information of an account. More details [here](https://summitroute.com/blog/2019/03/11/cloudmapper_iam_report_command/).
+
+
+If you want to add your own private commands, you can create a `private_commands` directory and add them there.
 
-*Intro post: https://duo.com/blog/introducing-cloudmapper-an-aws-visualization-tool*
+# Screenshots
 
-*Post to show usage in spotting misconfigurations: https://duo.com/blog/spotting-misconfigurations-with-cloudmapper*
+<img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/ideal_layout.png" width=100% alt="Ideal layout">
+<table border=0>
+<tr><td>
+<img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/report_resources.png" alt="Report screenshot">
+<td><img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/report_findings_summary.png" alt="Findings summary">
+<tr><td>
+<img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/report_findings.png" alt="Findings">
+<td><img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/iam_report-inactive_and_detail.png" alt="IAM report">
+<tr><td>
+<img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/command_line_audit.png" alt="Command-line audit">
+<td><img src="https://raw.githubusercontent.com/duo-labs/cloudmapper/master/docs/images/command_line_public.png" alt="Command-line public command">
+</table>
 
-![Demo screenshot](docs/images/ideal_layout.png "Demo screenshot")
 
 ## Installation
 
@@ -101,22 +130,10 @@ Collecting the data is done as follows:
 python cloudmapper.py collect --account my_account
 ```
 
-# Commands
+### Alternatives
+For network diagrams, you may want to try https://github.com/lyft/cartography or https://github.com/anaynayak/aws-security-viz
 
-- `api_endpoints`: List the URLs that can be called via API Gateway.
-- `audit`: Check for potential misconfigurations.
-- `collect`: Collect metadata about an account. More details [here](https://summitroute.com/blog/2018/06/05/cloudmapper_collect/).
-- `find_admins`: Look at IAM policies to identify admin users and roles and spot potential IAM issues. More details [here](https://summitroute.com/blog/2018/06/12/cloudmapper_find_admins/).
-- `prepare`/`webserver`: See [Network Visualizations](docs/network_visualizations.md)
-- `public`: Find public hosts and port ranges. More details [here](https://summitroute.com/blog/2018/06/13/cloudmapper_public/).
-- `sg_ips`: Get geoip info on CIDRs trusted in Security Groups. More details [here](https://summitroute.com/blog/2018/06/12/cloudmapper_sg_ips/).
-- `stats`: Show counts of resources for accounts. More details [here](https://summitroute.com/blog/2018/06/06/cloudmapper_stats/).
-- `weboftrust`: Show Web Of Trust. More details [here](https://summitroute.com/blog/2018/06/13/cloudmapper_wot/).
-- `report`: Generate HTML report. Includes summary of the accounts and audit findings. More details [here](https://summitroute.com/blog/2019/03/04/cloudmapper_report_generation/).
-- `iam_report`: Generate HTML report for the IAM information of an account. More details [here](https://summitroute.com/blog/2019/03/11/cloudmapper_iam_report_command/).
-
-
-If you want to add your own private commands, you can create a `private_commands` directory and add them there.
+For auditng and other AWS security tools see https://github.com/toniblyx/my-arsenal-of-aws-security-tools
 
 Licenses
 --------

account-data/demo/us-east-1/iam-get-account-authorization-details.json

@@ -117,7 +117,12 @@
                 ],
                 "Version": "2012-10-17"
             },
-            "AttachedManagedPolicies": [],
+            "AttachedManagedPolicies": [
+                {
+                    "PolicyName": "AmazonEC2RoleforSSM",
+                    "PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
+                }
+            ],
             "CreateDate": "2018-11-20T17:14:45+00:00",
             "InstanceProfileList": [],
             "Path": "/service-role/",

audit_config.yaml

@@ -141,20 +141,34 @@ USER_HAS_NOT_USED_ACCESS_KEY_FOR_MAX_DAYS:
   is_global: True
   group: IAM
 
-BAD_MFA_POLICY:
+IAM_BAD_MFA_POLICY:
   title: Incorrect policy used to attempt to enforce MFA
   description: AWS had advised incorrect policies for enforcing MFA which allowed an attacker, if they compromised keys that were protected by this policy, to remove the MFA policy from themselves, or remove the existing MFA device and add their own.
   severity: High
   is_global: True
   group: IAM
 
+IAM_KNOWN_BAD_POLICY:
+  title: Known bad policy used
+  description: AWS has provided flawed policies to customers. These are either deprecated or no longer advised.
+  severity: High
+  is_global: True
+  group: IAM
+
 IAM_NOTACTION_ALLOW:
   title: Use of NotAction in an Allow statement
   description: Using NotAction in an Allow policy almost always results in unwanted actions being allowed and should be avoided.
   severity: Medium
   is_global: True
   group: IAM
 
+IAM_ROLE_ALLOWS_ASSUMPTION_FROM_ANYWHERE:
+  title: IAM role allows assumption from anywhere
+  description: The IAM role's trust policy allows any other account to assume it.
+  severity: High
+  is_global: True
+  group: IAM
+
 IAM_MANAGED_POLICY_UNINTENTIONALLY_ALLOWING_ADMIN:
   title: Managed policy is allowing admin
   description: This finding is primarily for the deprecated AmazonElasticTranscoderFullAccess policy that was found to grant admin privileges.
@@ -265,6 +279,12 @@ EC2_CLASSIC:
   severity: Info
   group: EC2
 
+EC2_OLD:
+  title: Old EC2
+  description: EC2 runnning that was launched more than 365 days ago.
+  severity: Info
+  group: EC2
+
 LAMBDA_PUBLIC:
   title: Lambda is internet accessible
   description: Lambdas should not be publicly callable. Other resources, such as an API Gateway should be used to call the Lambda.

cloudmapper.py

@@ -30,7 +30,7 @@
 import pkgutil
 import importlib
 
-__version__ = "2.5.7"
+__version__ = "2.6.0"
 
 
 def show_help(commands):

commands/find_unused.py

@@ -0,0 +1,14 @@
+from __future__ import print_function
+import json
+
+from shared.common import parse_arguments
+from shared.find_unused import find_unused_resources
+
+
+__description__ = "Find unused resources in accounts"
+
+def run(arguments):
+    _, accounts, config = parse_arguments(arguments)
+    unused_resources = find_unused_resources(accounts)
+
+    print(json.dumps(unused_resources, indent=2, sort_keys=True))

commands/report.py

@@ -84,7 +84,7 @@ def report(accounts, config, args):
             {
                 "name": account["name"],
                 "id": account["id"],
-                "collection_date": get_collection_date(account),
+                "collection_date": get_collection_date(account)[:10],
             }
         )
 

shared/audit.py

@@ -1,5 +1,4 @@
 import json
-from datetime import datetime
 import pyjq
 import traceback
 
@@ -12,6 +11,8 @@
     is_unblockable_cidr,
     is_external_cidr,
     Finding,
+    get_collection_date,
+    days_between
 )
 from shared.query import query_aws, get_parameter_file
 from shared.nodes import Account, Region
@@ -236,14 +237,6 @@ def audit_root_user(findings, region):
 def audit_users(findings, region):
     MAX_DAYS_SINCE_LAST_USAGE = 90
 
-    def days_between(s1, s2):
-        """s1 and s2 are date strings, such as 2018-04-08T23:33:20+00:00 """
-        time_format = "%Y-%m-%dT%H:%M:%S"
-
-        d1 = datetime.strptime(s1.split("+")[0], time_format)
-        d2 = datetime.strptime(s2.split("+")[0], time_format)
-        return abs((d1 - d2).days)
-
     # TODO: Convert all of this into a table
 
     json_blob = query_aws(region.account, "iam-get-credential-report", region)
@@ -635,6 +628,23 @@ def audit_ec2(findings, region):
                 # Ignore EC2's that are off
                 continue
 
+            # Check for old instances
+            if instance.get("LaunchTime", "") != "":
+                MAX_RESOURCE_AGE_DAYS = 365
+                collection_date = get_collection_date(region.account)
+                launch_time = instance["LaunchTime"].split(".")[0]
+                age_in_days = days_between(launch_time, collection_date)
+                if age_in_days > MAX_RESOURCE_AGE_DAYS:
+                    findings.add(Finding(
+                        region, 
+                        "EC2_OLD", 
+                        instance["InstanceId"],
+                        resource_details={
+                            "Age in days": age_in_days
+                        },
+                    ))
+
+            # Check for EC2 Classic
             if "vpc" not in instance.get("VpcId", ""):
                 findings.add(Finding(region, "EC2_CLASSIC", instance["InstanceId"]))
 
@@ -729,7 +739,7 @@ def audit_sg(findings, region):
                     region,
                     "SG_LARGE_CIDR",
                     cidr,
-                    resource_details={"size": ip.size, "security_groups": cidrs[cidr]},
+                    resource_details={"size": ip.size, "security_groups": list(cidrs[cidr])},
                 )
             )
 
@@ -932,7 +942,7 @@ def audit(accounts):
                 if region.name == "us-east-1":
                     audit_s3_buckets(findings, region)
                     audit_cloudtrail(findings, region)
-                    audit_iam(findings, region.account)
+                    audit_iam(findings, region)
                     audit_password_policy(findings, region)
                     audit_root_user(findings, region)
                     audit_users(findings, region)

shared/common.py

@@ -309,18 +309,29 @@ def get_us_east_1(account):
 
     raise Exception("us-east-1 not found")
 
+def iso_date(d):
+    """ Convert ISO format date string such as 2018-04-08T23:33:20+00:00"""
+    time_format = "%Y-%m-%dT%H:%M:%S"
+    return datetime.datetime.strptime(d.split("+")[0], time_format)
+
+def days_between(s1, s2):
+        """s1 and s2 are date strings"""
+        d1 = iso_date(s1)
+        d2 = iso_date(s2)
+        return abs((d1 - d2).days)
 
 def get_collection_date(account):
-    account_struct = Account(None, account)
+    if type(account) is not Account:
+        account = Account(None, account)
+    account_struct = account
     json_blob = query_aws(
         account_struct, "iam-get-credential-report", get_us_east_1(account_struct)
     )
     if not json_blob:
         raise Exception("File iam-get-credential-report.json does not exist or is not well-formed. Likely cause is you did not run the collect command for this account.")
 
     # GeneratedTime looks like "2019-01-30T15:43:24+00:00"
-    # so extract the data part "2019-01-30"
-    return json_blob["GeneratedTime"][:10]
+    return json_blob["GeneratedTime"]
 
 
 def get_access_advisor_active_counts(account, max_age=90):

shared/find_unused.py

@@ -0,0 +1,119 @@
+import pyjq
+
+from shared.common import query_aws, get_regions
+from shared.nodes import Account, Region
+
+
+def find_unused_security_groups(region):
+    # Get the defined security groups, then find all the Security Groups associated with the
+    # ENIs.  Then diff these to find the unused Security Groups.
+    used_sgs = set()
+
+    defined_sgs = query_aws(region.account, "ec2-describe-security-groups", region)
+
+    network_interfaces = query_aws(
+        region.account, "ec2-describe-network-interfaces", region
+    )
+
+    defined_sg_set = {}
+
+    for sg in pyjq.all(".SecurityGroups[]", defined_sgs):
+        defined_sg_set[sg["GroupId"]] = sg
+
+    for used_sg in pyjq.all(
+        ".NetworkInterfaces[].Groups[].GroupId", network_interfaces
+    ):
+        used_sgs.add(used_sg)
+
+    unused_sg_ids = set(defined_sg_set) - used_sgs
+    unused_sgs = []
+    for sg_id in unused_sg_ids:
+        unused_sgs.append(
+            {
+                "id": sg_id,
+                "name": defined_sg_set[sg_id]["GroupName"],
+                "description": defined_sg_set[sg_id].get("Description", ""),
+            }
+        )
+    return unused_sgs
+
+
+def find_unused_volumes(region):
+    unused_volumes = []
+    volumes = query_aws(region.account, "ec2-describe-volumes", region)
+    for volume in pyjq.all('.Volumes[]|select(.State=="available")', volumes):
+        unused_volumes.append({"id": volume["VolumeId"]})
+
+    return unused_volumes
+
+
+def find_unused_elastic_ips(region):
+    unused_ips = []
+    ips = query_aws(region.account, "ec2-describe-addresses", region)
+    for ip in pyjq.all(".Addresses[] | select(.AssociationId == null)", ips):
+        unused_ips.append({"id": ip["AllocationId"], "ip": ip["PublicIp"]})
+
+    return unused_ips
+
+
+def find_unused_network_interfaces(region):
+    unused_network_interfaces = []
+    network_interfaces = query_aws(
+        region.account, "ec2-describe-network-interfaces", region
+    )
+    for network_interface in pyjq.all(
+        '.NetworkInterfaces[]|select(.Status=="available")', network_interfaces
+    ):
+        unused_network_interfaces.append(
+            {"id": network_interface["NetworkInterfaceId"]}
+        )
+
+    return unused_network_interfaces
+
+
+def add_if_exists(dictionary, key, value):
+    if value:
+        dictionary[key] = value
+
+
+def find_unused_resources(accounts):
+    unused_resources = []
+    for account in accounts:
+        unused_resources_for_account = []
+        for region_json in get_regions(Account(None, account)):
+            region = Region(Account(None, account), region_json)
+
+            unused_resources_for_region = {}
+
+            add_if_exists(
+                unused_resources_for_region,
+                "security_groups",
+                find_unused_security_groups(region),
+            )
+            add_if_exists(
+                unused_resources_for_region, "volumes", find_unused_volumes(region)
+            )
+            add_if_exists(
+                unused_resources_for_region,
+                "elastic_ips",
+                find_unused_elastic_ips(region),
+            )
+            add_if_exists(
+                unused_resources_for_region,
+                "network_interfaces",
+                find_unused_network_interfaces(region),
+            )
+
+            unused_resources_for_account.append(
+                {
+                    "region": region_json["RegionName"],
+                    "unused_resources": unused_resources_for_region,
+                }
+            )
+        unused_resources.append(
+            {
+                "account": {"id": account["id"], "name": account["name"]},
+                "regions": unused_resources_for_account,
+            }
+        )
+    return unused_resources