Prowler v3 Importer #7348

finventario · 2023-01-02T18:24:33Z

Scanner Name
In prowler v3, the fields have changed.

See https://github.com/prowler-cloud/prowler/releases/tag/3.0.0 and prowler-cloud/prowler@7b9fae5

Sample File
csv:

ASSESSMENT_START_TIME;FINDING_UNIQUE_ID;PROVIDER;CHECK_ID;CHECK_TITLE;CHECK_TYPE;STATUS;STATUS_EXTENDED;SERVICE_NAME;SUBSERVICE_NAME;SEVERITY;RESOURCE_TYPE;RESOURCE_DETAILS;RESOURCE_TAGS;DESCRIPTION;RISK;RELATED_URL;REMEDIATION_RECOMMENDATION_TEXT;REMEDIATION_RECOMMENDATION_URL;REMEDIATION_RECOMMENDATION_CODE_NATIVEIAC;REMEDIATION_RECOMMENDATION_CODE_TERRAFORM;REMEDIATION_RECOMMENDATION_CODE_CLI;REMEDIATION_RECOMMENDATION_CODE_OTHER;CATEGORIES;DEPENDS_ON;RELATED_TO;NOTES;PROFILE;ACCOUNT_ID;ACCOUNT_NAME;ACCOUNT_EMAIL;ACCOUNT_ARN;ACCOUNT_ORG;ACCOUNT_TAGS;REGION;RESOURCE_ID;RESOURCE_ARN
2023-01-02T17:43:47.486212;prowler-aws-accessanalyzer_enabled_without_findings-xxxxx;aws;accessanalyzer_enabled_without_findings;Check if IAM Access Analyzer is enabled without findings;IAM;FAIL;IAM Access Analyzer is not enabled;accessanalyzer;;low;Other;;[];Check if IAM Access Analyzer is enabled without findings;AWS IAM Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This lets you identify unintended access to your resources and data, which is a security risk. IAM Access Analyzer uses a form of mathematical analysis called automated reasoning, which applies logic and mathematical inference to determine all possible access paths allowed by a resource policy.;https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html;Enable IAM Access Analyzer for all accounts, create analyzer and take action over it is recommendations (IAM Access Analyzer is available at no additional cost).;https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html;;;aws accessanalyzer create-analyzer --analyzer-name <NAME> --type <ACCOUNT|ORGANIZATION>;;;;;;ENV;xxxxxx;;;;;;xxregio;xxxxx;

json:

[{
    "AssessmentStartTime": "2023-01-02T17:22:49.730532",
    "FindingUniqueId": "prowler-aws-accessanalyzer_enabled_without_findings-xxxxx",
    "Provider": "aws",
    "CheckID": "accessanalyzer_enabled_without_findings",
    "CheckTitle": "Check if IAM Access Analyzer is enabled without findings",
    "CheckType": [
        "IAM"
    ],
    "ServiceName": "accessanalyzer",
    "SubServiceName": "",
    "Status": "FAIL",
    "StatusExtended": "IAM Access Analyzer is not enabled",
    "Severity": "low",
    "ResourceType": "Other",
    "ResourceDetails": "",
    "Tags": {
        "Tag1Key": "value",
        "Tag2Key": "value"
    },
    "Description": "Check if IAM Access Analyzer is enabled without findings",
    "Risk": "AWS IAM Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This lets you identify unintended access to your resources and data, which is a security risk. IAM Access Analyzer uses a form of mathematical analysis called automated reasoning, which applies logic and mathematical inference to determine all possible access paths allowed by a resource policy.",
    "RelatedUrl": "https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html",
    "Remediation": {
        "Code": {
            "NativeIaC": "",
            "Terraform": "",
            "CLI": "aws accessanalyzer create-analyzer --analyzer-name <NAME> --type <ACCOUNT|ORGANIZATION>",
            "Other": ""
        },
        "Recommendation": {
            "Text": "Enable IAM Access Analyzer for all accounts, create analyzer and take action over it is recommendations (IAM Access Analyzer is available at no additional cost).",
            "Url": "https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html"
        }
    },
    "Categories": [],
    "DependsOn": [],
    "RelatedTo": [],
    "Notes": "",
    "Profile": "ENV",
    "AccountId": "xxxxx",
    "OrganizationsInfo": null,
    "Region": "xxxxx-regio",
    "ResourceId": "xxxxx",
    "ResourceArn": ""
}]

The text was updated successfully, but these errors were encountered:

security101 · 2023-03-12T15:38:00Z

Using toniblyx/prowler v3.2.4 and DefectDojo v. 2.18.3 I get the following error messages when I import a Prowler result file. The error messages provided by DD vary depending on the filetype:
CSV result file: An exception error occurred during the report import:expected string or bytes-like object, got 'NoneType'
JSON: An exception error occurred during the report import:Expecting property name enclosed in double quotes: line 2 column 1 (char 4)

Any advice?

damiencarol · 2023-03-12T18:40:47Z

@security101 we need a stack track to understand what's going on.

damiencarol · 2023-03-13T08:22:37Z

Made some tests and the format of Prowler changed, this need maintenance work.

security101 · 2023-03-15T12:21:44Z

The prowler team responded to prowler-cloud/prowler#2076

The new format specification is detailed here:

CSV https://docs.prowler.cloud/en/latest/tutorials/reporting/#csv
JSON https://docs.prowler.cloud/en/latest/tutorials/reporting/#json
JSON-ASFF https://docs.prowler.cloud/en/latest/tutorials/reporting/#json-asff

An idea:

Prowler supports as report format json-asff
DD supports json-asff as import (see AWS Security Hub https://defectdojo.github.io/django-DefectDojo/integrations/parsers/file/awssecurityhub/)
May be using the AWS Security Import could solve the issue and might we a recommendation for Prowler v3 reports?

Update: Idea did not work. Got the error message: An exception error occurred during the report import:'list' object has no attribute 'get'

damiencarol · 2023-03-18T15:27:46Z

@security101 I think I should fix the JSON-ASFF parser and extend the Prowler to support v3. Could you push me an ASFF reportin DM/email/Slack?

damiencarol · 2023-03-18T17:40:37Z

I took a look at the different file format and parsers and the best way to fix your issue is to add a parser for ASFF format.
This way, we stay generic and ASFF format is more stable.

@security101 will it be ok for you to use this new one?

security101 · 2023-03-20T13:49:18Z

Dear @damiencarol, yes I think this a good choice and more stable in the future, too.

security101 · 2023-03-20T13:59:58Z

@security101 I think I should fix the JSON-ASFF parser and extend the Prowler to support v3. Could you push me an ASFF reportin DM/email/Slack?

I will provide a report via DM. Upfront the associated stack trace:

django-defectdojo-218-nginx-1         | 2023/03/20 13:50:14 [warn] 7#7: *12 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000010, client: 172.20.0.1, server: , request: "POST /engagement/148/import_scan_results HTTP/1.1", host: "<my_domain>", referrer: "https://<my_domain>/engagement/148/import_scan_results"
django-defectdojo-218-uwsgi-1         | /usr/local/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1358: RuntimeWarning: DateTimeField Test.target_start received a naive datetime (2023-03-20 00:00:00) while time zone support is active.
django-defectdojo-218-uwsgi-1         |   warnings.warn("DateTimeField %s.%s received a naive datetime "
django-defectdojo-218-uwsgi-1         | /usr/local/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1358: RuntimeWarning: DateTimeField Test.target_end received a naive datetime (2023-03-20 00:00:00) while time zone support is active.
django-defectdojo-218-uwsgi-1         |   warnings.warn("DateTimeField %s.%s received a naive datetime "
django-defectdojo-218-uwsgi-1         | [20/Mar/2023 13:50:14] ERROR [dojo.engagement.views:698] Expecting property name enclosed in double quotes: line 2 column 1 (char 4)
django-defectdojo-218-uwsgi-1         | Traceback (most recent call last):
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/engagement/views.py", line 682, in import_scan_results
django-defectdojo-218-uwsgi-1         |     test, finding_count, closed_finding_count, _ = importer.import_scan(scan, scan_type, engagement, user, environment, active=active, verified=verified, tags=tags,
django-defectdojo-218-uwsgi-1         |                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/importers/importer/importer.py", line 305, in import_scan
django-defectdojo-218-uwsgi-1         |     parsed_findings = parser.get_findings(scan, test)
django-defectdojo-218-uwsgi-1         |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/tools/aws_prowler/parser.py", line 29, in get_findings
django-defectdojo-218-uwsgi-1         |     return self.process_json(file, test)
django-defectdojo-218-uwsgi-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/tools/aws_prowler/parser.py", line 123, in process_json
django-defectdojo-218-uwsgi-1         |     deserialized = json.loads(issue)
django-defectdojo-218-uwsgi-1         |                    ^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
django-defectdojo-218-uwsgi-1         |     return _default_decoder.decode(s)
django-defectdojo-218-uwsgi-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
django-defectdojo-218-uwsgi-1         |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
django-defectdojo-218-uwsgi-1         |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
django-defectdojo-218-uwsgi-1         |     obj, end = self.scan_once(s, idx)
django-defectdojo-218-uwsgi-1         |                ^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         | json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 4)
django-defectdojo-218-uwsgi-1         | [pid: 41|app: -|req: -/-] 172.20.0.1 (admin) {74 vars in 1738 bytes} [Mon Mar 20 13:50:14 2023] POST /engagement/148/import_scan_results => generated 145334 bytes in 770 msecs (HTTP/1.1 200) 7 headers in 361 bytes (1 switches on core 0)

awakenine · 2023-04-04T14:14:34Z

@damiencarol, thank you for the parser. I have tested it a bit for prowler, and assume it's better to separate parser for prowlerV3, because some fields are used in a different way in prowler, than in GuardDuty.

Parsing Regular JSON or CSV output is preferable, because ASFF JSON output format miss some important fields (like Risk).

Currently if we use general ASFF parser for Prowler V3 report it creates a lot of PASSED active findings in Dojo that look the same as discovered vulnerabilities in Dojo and only can find Title, Description and Severity missing a lot of useful details.
Below you can see my notes about mapping I think would make sense:

Mapping of "Remediation": { "Recommendation" to Mitigation in Dojo may be also useful as it provides info about the fix.
Resources field can be appended to Description in Dojo just as array.
Description is better to map to both finding title and description in Dojo, because title from PrawlerV3 ASFF is just a check name, which looks confusing.

Here is the sample of prowlerV3 finding in ASFF format

{
    "SchemaVersion": "2018-10-08",
    "Id": "prowler-acm_certificates_expiration_check-HIDDEN",
    "ProductArn": "arn:aws:securityhub:eu-west-1::product/prowler/prowler",
    "RecordState": "ACTIVE",
    "ProductFields": {
      "ProviderName": "Prowler",
      "ProviderVersion": "3.3.2",
      "ProwlerResourceName": "HIDDEN"
    },
    "GeneratorId": "prowler-acm_certificates_expiration_check",
    "AwsAccountId": "HIDDEN",
    "Types": [
      "Data Protection"
    ],
    "FirstObservedAt": "2023-04-01T13:40:45Z",
    "UpdatedAt": "2023-04-01T13:40:45Z",
    "CreatedAt": "2023-04-01T13:40:45Z",
    "Severity": {
      "Label": "HIGH"
    },
    "Title": "Check if ACM Certificates are about to expire in specific days or less",
    "Description": "ACM Certificate for example.com is about to expire in 7 days.",
    "Resources": [
      {
        "Type": "AwsCertificateManagerCertificate",
        "Id": "HIDDEN",
        "Partition": "aws",
        "Region": "eu-west-1"
      }
    ],
    "Compliance": {
      "Status": "FAILED",
      "RelatedRequirements": [
        "CISA your-data-2",
        "SOC2 cc_6_7",
        "GDPR article_32",
        "HIPAA 164_308_a_4_ii_a 164_312_e_1",
        "NIST-800-171-Revision-2 3_13_1 3_13_2 3_13_8 3_13_11",
        "NIST-800-53-Revision-4 ac_4 ac_17_2 sc_12",
        "NIST-800-53-Revision-5 sc_7_12 sc_7_16",
        "NIST-CSF-1.1 ac_5 ds_2",
        "RBI-Cyber-Security-Framework annex_i_1_3",
        "FFIEC d3-pc-im-b-1",
        "FedRamp-Moderate-Revision-4 ac-4 ac-17-2 sc-12",
        "FedRAMP-Low-Revision-4 ac-17 sc-12"
      ],
      "AssociatedStandards": [
        {
          "StandardsId": "CISA"
        },
        {
          "StandardsId": "SOC2"
        },
        {
          "StandardsId": "GDPR"
        },
        {
          "StandardsId": "HIPAA"
        },
        {
          "StandardsId": "NIST-800-171-Revision-2"
        },
        {
          "StandardsId": "NIST-800-53-Revision-4"
        },
        {
          "StandardsId": "NIST-800-53-Revision-5"
        },
        {
          "StandardsId": "NIST-CSF-1.1"
        },
        {
          "StandardsId": "RBI-Cyber-Security-Framework"
        },
        {
          "StandardsId": "FFIEC"
        },
        {
          "StandardsId": "FedRamp-Moderate-Revision-4"
        },
        {
          "StandardsId": "FedRAMP-Low-Revision-4"
        }
      ]
    },
    "Remediation": {
      "Recommendation": {
        "Text": "Monitor certificate expiration and take automated action to renew; replace or remove. Having shorter TTL for any security artifact is a general recommendation; but requires additional automation in place. If not longer required delete certificate. Use AWS config using the managed rule: acm-certificate-expiration-check.",
        "Url": "https://docs.aws.amazon.com/config/latest/developerguide/acm-certificate-expiration-check.html"
      }
    }
  },

security101 · 2023-06-05T16:02:43Z

Dear @damiencarol ,
I see that #8028 is merged into the dev branch. A big thank you to the team. Now if would be of interest if it is already decided/foreseeable in which DD release it will be available? Thank you in advance! Cheers, Andreas

mtesauro · 2023-06-05T19:18:28Z

It's in the current release that happened earlier today: https://github.com/DefectDojo/django-DefectDojo/releases/tag/2.23.0

Search for 8028 on that page and you'll see it in the list of changes.

finventario · 2024-04-09T09:44:10Z

Hi!
It seems with the "stable" version (v4) of prowler, there is an issue again.
Prowler now only supports json-ASFF, json-OCSF and csv.
I suggest to remove the prowler importer or direct users to the AWS Security Hub Scan Findings importer and maybe adopt json-OCSF.

With v4 and json-asff i was able to import the report using the AWS Security Hub Scan Findings importer.

pr3l14t0r · 2024-04-09T15:05:41Z

I suggest to remove the prowler importer or direct users to the AWS Security Hub Scan Findings importer and maybe adopt json-OCSF.

Was also going to say something about this issue. Downside of only taking json-ASFF format is that you can't import prowler scans of other providers like gcp, since prowler just doesn't create an ASFF file for that. Unless you use the Security-Hub integration, which is not always a default.

So long story short: I'd vote for adopting the OCSF format.

kagahd · 2024-06-05T12:41:35Z

I opened PR #10338 which adds a prowler v4 parser for prowler reports in json-OCSF format.

finventario added the Import Scans label Jan 2, 2023

security101 mentioned this issue Mar 12, 2023

Report output compatibility with DefectDojo prowler-cloud/prowler#2076

Closed

damiencarol self-assigned this Mar 12, 2023

damiencarol mentioned this issue Mar 18, 2023

Add parser for AWS Security Finding Format (ASFF) #7838

Merged

10 tasks

anderson-slompo mentioned this issue Apr 28, 2023

Add support to AWS Prowler V3 json format #8028

Merged

awakenine mentioned this issue May 12, 2023

Not able to upload AWS Prowler Scan Report. Give this error:- An exception error occurred during the report import:'NoneType' object has no attribute 'group' #7968

Closed

mtesauro closed this as completed Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prowler v3 Importer #7348

Prowler v3 Importer #7348

finventario commented Jan 2, 2023

security101 commented Mar 12, 2023

damiencarol commented Mar 12, 2023

damiencarol commented Mar 13, 2023

security101 commented Mar 15, 2023 •

edited

Loading

damiencarol commented Mar 18, 2023

damiencarol commented Mar 18, 2023

security101 commented Mar 20, 2023

security101 commented Mar 20, 2023

awakenine commented Apr 4, 2023 •

edited

Loading

security101 commented Jun 5, 2023 •

edited

Loading

mtesauro commented Jun 5, 2023

finventario commented Apr 9, 2024 •

edited

Loading

pr3l14t0r commented Apr 9, 2024

kagahd commented Jun 5, 2024

Prowler v3 Importer #7348

Prowler v3 Importer #7348

Comments

finventario commented Jan 2, 2023

security101 commented Mar 12, 2023

damiencarol commented Mar 12, 2023

damiencarol commented Mar 13, 2023

security101 commented Mar 15, 2023 • edited Loading

damiencarol commented Mar 18, 2023

damiencarol commented Mar 18, 2023

security101 commented Mar 20, 2023

security101 commented Mar 20, 2023

awakenine commented Apr 4, 2023 • edited Loading

security101 commented Jun 5, 2023 • edited Loading

mtesauro commented Jun 5, 2023

finventario commented Apr 9, 2024 • edited Loading

pr3l14t0r commented Apr 9, 2024

kagahd commented Jun 5, 2024

security101 commented Mar 15, 2023 •

edited

Loading

awakenine commented Apr 4, 2023 •

edited

Loading

security101 commented Jun 5, 2023 •

edited

Loading

finventario commented Apr 9, 2024 •

edited

Loading