diff --git a/.gitignore b/.gitignore
index 7d8e92d..f985c5f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -14,7 +14,6 @@ dist/
downloads/
eggs/
.eggs/
-lib/
lib64/
parts/
sdist/
diff --git a/ISSUE_TEMPLATE b/ISSUE_TEMPLATE
new file mode 100644
index 0000000..3e648f9
--- /dev/null
+++ b/ISSUE_TEMPLATE
@@ -0,0 +1,21 @@
+Thanks for taking the time to submit an issue. There's a couple of things that will make this easier for everyone involve. If you could please take the time to complete the list below, it's much appreciated.
+
+- [ ] I'm using Deep Security version ____
+
+- [ ] I'm using the latest version of this repo
+
+- [ ] My Deep Security Manager has a self-signed or non-validating SSL certificate
+
+## Current Output
+
+Please re-run the command using ```--verbose``` and provide the complete output below.
+
+## Addition Details
+
+Can you also please fill in each of the remaining sections.
+
+### Expected behaviour
+
+### Actual behaviour
+
+### Steps to reproduce the behaviour
\ No newline at end of file
diff --git a/README.md b/README.md
index fd7dbf7..fc17fa7 100644
--- a/README.md
+++ b/README.md
@@ -1,102 +1,261 @@
+> This commit has breaking changes that will stay firm moving forward. For original, early adopters please use [release v0.91](https://github.com/deep-security/aws-waf/releases/tag/v0.91)...but move to the current one when you can, it's **way better** ;-)
+
# Deep Security AWS WAF Integration
A simple tool set to help build AWS WAF rule sets from Deep Security.
+## Index
+
+- [Pre-Requisites](#pre-requisites)
+- [Usage](#usage)
+ - [iplists](#usage-iplists)
+ - [sqli](#usage-sqli)
+- [SSL Certificate Validation](#ssl-certificate-validation)
+- [AWS WAF Costs](#aws-waf-costs)
+ - [iplists](#aws-waf-costs-iplists)
+ - [sqli](#aws-waf-costs-sqli)
+
+
+
## Pre-Requisites
```bash
pip install -r requirements.txt
```
-## ip_list_to_set.py
+
-Deep Security can host a number of IP Lists (Policies > Common Objects > Lists > IP Lists). These IP Lists can be very useful as AWS WAF IP Sets for matching rule conditions.
+### Usage
-This utility will convert an IP List (IPv6 and IPv4) from Deep Security to a minimized IP Set for use in an AWS WAF rule.
+The syntax for basic command line usage is available by using the ```--help``` switch.
-### Usage
+```bash
+$ python ds-to-aws-waf.py
+usage: ds-to-aws-waf [COMMAND]
+ For more help on a specific command, type ds-to-aws-waf [COMMAND] --help
+
+ Available commands:
+
+ iplist
+ > Push a Deep Security IP list to an AWS WAF IP Set
+ sqli
+ > Determine which instances protected by Deep Security should also be protected by AWS WAF SQLi rules
+ ...
+```
+
+Each script in this set works under a common structure. There are several shared arguments;
+
+```bash
+ -h, --help show this help message and exit
+ -d DSM, --dsm DSM The address of the Deep Security Manager. Defaults to
+ Deep Security as a Service
+ --dsm-port DSM_PORT The address of the Deep Security Manager. Defaults to
+ an AWS Marketplace/software install (:4119).
+ Automatically configured for Deep Security as a
+ Service
+ -u DSM_USERNAME, --dsm-username DSM_USERNAME
+ The Deep Security username to access the IP Lists
+ with. Should only have read-only rights to IP lists
+ and API access
+ -p DSM_PASSWORD, --dsm-password DSM_PASSWORD
+ The password for the specified Deep Security username.
+ Should only have read-only rights to IP lists and API
+ access
+ -t DSM_TENANT, --dsm-tenant DSM_TENANT
+ The name of the Deep Security tenant/account
+ --ignore-ssl-validation
+ Ignore SSL certification validation. Be careful when
+ you use this as it disables a recommended security
+ check. Required for Deep Security Managers using a
+ self-signed SSL certificate
+ --dryrun Do a dry run of the command. This will not make any
+ changes to your AWS WAF service
+ --verbose Enabled verbose output for the script. Useful for
+ debugging
+```
-Basic command line usage is available by using the ```--help``` switch.
+These core settings allow you to connect to a Deep Security manager or Deep Security as a Service.
```bash
-$ python ip_list_to_set.py --help
->> usage: ip_list_to_set.py [-h] [-d IP_LIST] [-l] [-m DSM] [--dsm-port DSM_PORT]
- -u USERNAME -p PASSWORD [-t TENANT] [--dryrun]
- [--verbose]
+# to connect to your own Deep Security manager
+ds-to-aws-waf [COMMAND] -d 10.1.1.0 -u admin -p USE_RBAC_TO_REDUCE_RISK --ignore-ssl-validation
+
+# to connect to Deep Security as a Service
+ds-to-aws-waf [COMMAND] -u admin -p USE_RBAC_TO_REDUCE_RISK -t MY_ACCOUNT
+```
+
+Each individual command will also have it's own options that allow you to control the behaviour of the command.
+
+You'll notice in the examples, the password is set to USE_RBAC_TO_REDUCE_RISK. In this context, RBAC stands for role based access control.
+
+Currently Deep Security treats API access just like a user logging in. Therefore it is strongly recommended that you create a new Deep Security user for use with this script. This user should have the bare minimum permissions required to complete the tasks.
+
+
+
+### iplists
+
+The iplists command is a simple, two-step process. You must first find the ID of the list in Deep Security and then push that IP list to an AWS WAF IP set.
+
+**Step 1;**
+
+```
+# list the available IP lists in Deep Security
+# ...for Deep Security as a Service
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -t TENANT -l
+
+# ...for another Deep Security manager
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -d DSM_HOSTNAME --ignore-ssl-validation -l
+```
+
+This command will then display a list of IP lists and their associated IDs. You can then use those IDs to push the IP list to AWS WAF as an IP Set.
+
+**Step 2;**
+
+```
+# push a Deep Security IP list to an AWS WAF IP Set
+# ...for Deep Security as a Service
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -t TENANT -i 17
+
+# ...for another Deep Security manager
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -d DSM_HOSTNAME --ignore-ssl-validation -i 17
+```
+
+The complete command syntax is;
+
+```
+ # ./ds-to-aws-waf.py iplist --help
+usage: ds-to-aws-waf.py iplists [-h] [-d DSM] [--dsm-port DSM_PORT] -u
+ DSM_USERNAME -p DSM_PASSWORD [-t DSM_TENANT]
+ [-r AWS_REGION] [--ignore-ssl-validation]
+ [--dryrun] [--verbose] [-l] [-i IP_LIST]
-Deep Security uses the concept of IP Lists to make firewall rules easier to
-administer. The AWS WAF uses a similar concept of IP Sets as rule conditions.
-This utility helps synchronize Deep Security IP Lists with AWS WAF IP Sets.
+Create and update AWS WAF WACL rules based on information from a Deep Security
+installation
optional arguments:
-h, --help show this help message and exit
- -d IP_LIST, --ds IP_LIST
- Specify an IP List within Deep Security as the source
- for the AWS WAF IP Set
- -l, --list List the available Deep Security IP Lists and the AWS
- WAF IP Sets
- -m DSM, --dsm DSM The address of the Deep Security Manager. Defaults to
+ -d DSM, --dsm DSM The address of the Deep Security Manager. Defaults to
Deep Security as a Service
--dsm-port DSM_PORT The address of the Deep Security Manager. Defaults to
- Deep Security as a Service
- -u USERNAME, --username USERNAME
+ an AWS Marketplace/software install (:4119).
+ Automatically configured for Deep Security as a
+ Service
+ -u DSM_USERNAME, --dsm-username DSM_USERNAME
The Deep Security username to access the IP Lists
with. Should only have read-only rights to IP lists
and API access
- -p PASSWORD, --password PASSWORD
+ -p DSM_PASSWORD, --dsm-password DSM_PASSWORD
The password for the specified Deep Security username.
Should only have read-only rights to IP lists and API
access
- -t TENANT, --tenant TENANT
+ -t DSM_TENANT, --dsm-tenant DSM_TENANT
The name of the Deep Security tenant/account
+ -r AWS_REGION, --aws-region AWS_REGION
+ The name of AWS region to connect to
+ --ignore-ssl-validation
+ Ignore SSL certification validation. Be careful when
+ you use this as it disables a recommended security
+ check. Required for Deep Security Managers using a
+ self-signed SSL certificate
--dryrun Do a dry run of the command. This will not make any
changes to your AWS WAF service
--verbose Enabled verbose output for the script. Useful for
debugging
+ -l, --list List the available Deep Security IP Lists and the AWS
+ WAF IP Sets
+ -i IP_LIST, --id IP_LIST
+ Specify an IP List by ID within Deep Security as the
+ source for the AWS WAF IP Set
```
-The first step is to find the ID Deep Security uses for the IP List you want to sync to an AWS WAF IP Set. You can do that using the ```--list``` switch.
+
+
+### sqli
+
+The sqli command contains two parts; the analysis of the workloads on the specified EC2 instances and the creation of an SQLi match condition.
+
+You can run either part separately, though **the creation of the match condition only needs to be run once per account.**
+
+Common usage;
-```bash
-$python ip_list_to_set.py --list -u USERNAME -p PASSWORD -t TENANT
->> Available Deep Security IP Lists
-================================
-1 Ignore Reconnaissance
-2 Network Broadcast
-3 Ingress Filters
-4 Domain Controller(s)
-5 Off Domain IPs
-6 Corporate Network IPs
-...
```
+# create a new SQLi match condition
+# ...for Deep Security as a Service
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -t TENANT --create-match
-Once you know the IP List you want to use as the source, you can pass the ID to the script and have it convert the IP List to an AWS WAF IP Set. For most IP Lists, this conversion works well. However for some lists, you'll see a failure based on the size of the IP List.
+# ...for another Deep Security manager
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -d DSM_HOSTNAME --ignore-ssl-validation --create-match
+```
-Deep Security doesn't enforce the same size limits on IP Lists as AWS WAF does on IP Sets. The script attempts to convert the IP List to the smallest possible number of CIDR blocks but can occasionally still run into the 1000 entry limit for an AWS WAF IP Set.
+To find out which instances should be protected by an AWS WAF SQLi rule;
-In these cases, the best solution is to divide the list within Deep Security and re-run the script with the new ID(s).
+```
+# find out which instances should be protected by an AWS WAF SQLi rule
+# ...for Deep Security as a Service
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -t TENANT -l
-A good practice to adopt is to make a dry run before committing any changes. You can do list using the ```--dryrun``` switch.
+# ...for another Deep Security manager
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -d DSM_HOSTNAME --ignore-ssl-validation -l
+
+# filter those instances by tag and region
+# ...for Deep Security as a Service
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -t TENANT -l --tag Name=Test --tag Environment=PROD -r us-east-1
+
+# ...for another Deep Security manager
+python ds-to-aws-waf.py iplists -u WAF -p PASSWORD -d DSM_HOSTNAME --ignore-ssl-validation -l --tag Name=Test --tag Environment=PROD -r us-east-1
-```bash
-$ python ip_list_to_set.py -d 152 -u USERNAME -p PASSWORD -t TENANT --dryrun
->> ***********************************************************************
-* DRY RUN ENABLED. NO CHANGES WILL BE MADE
-***********************************************************************
-Converted 41 IP List entries to 718 IP Set entries
-Will request the addition of 718 entries in IP Set 9ee53a08-cdaf-4881-a111-3d99b58065e4
-Will update IP Set [AMAZON eu-west-1] with ID [9ee53a08-cdaf-4881-a111-3d99b58065e4]
```
-If you're comfortable with the changes, you can then re-run the script without the ```--dryrun``` switch and commit the changes.
+The complete command syntax is;
-```bash
-$ python ip_list_to_set.py -d 152 -u USERNAME -p PASSWORD -t TENANT --dryrun
->> Converted 41 IP List entries to 718 IP Set entries
-Updated IP Set [AMAZON eu-west-1] with ID [9ee53a08-cdaf-4881-a111-3d99b58065e4]
```
+# ./ds-to-aws-waf.py sqli --help
+usage: ds-to-aws-waf.py sqli [-h] [-d DSM] [--dsm-port DSM_PORT] -u
+ DSM_USERNAME -p DSM_PASSWORD [-t DSM_TENANT]
+ [-r AWS_REGION] [--ignore-ssl-validation]
+ [--dryrun] [--verbose] [-l]
+ [--tag TAGS [TAGS ...]] [--create-match]
-Now the IP Set has been created in AWS WAF and can be used in a AWS WAC WACL rule as a matching condition. AWS was detailed documentation on [the next steps you need to take](http://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html).
+Create and update AWS WAF WACL rules based on information from a Deep Security
+installation
+
+optional arguments:
+ -h, --help show this help message and exit
+ -d DSM, --dsm DSM The address of the Deep Security Manager. Defaults to
+ Deep Security as a Service
+ --dsm-port DSM_PORT The address of the Deep Security Manager. Defaults to
+ an AWS Marketplace/software install (:4119).
+ Automatically configured for Deep Security as a
+ Service
+ -u DSM_USERNAME, --dsm-username DSM_USERNAME
+ The Deep Security username to access the IP Lists
+ with. Should only have read-only rights to IP lists
+ and API access
+ -p DSM_PASSWORD, --dsm-password DSM_PASSWORD
+ The password for the specified Deep Security username.
+ Should only have read-only rights to IP lists and API
+ access
+ -t DSM_TENANT, --dsm-tenant DSM_TENANT
+ The name of the Deep Security tenant/account
+ -r AWS_REGION, --aws-region AWS_REGION
+ The name of AWS region to connect to
+ --ignore-ssl-validation
+ Ignore SSL certification validation. Be careful when
+ you use this as it disables a recommended security
+ check. Required for Deep Security Managers using a
+ self-signed SSL certificate
+ --dryrun Do a dry run of the command. This will not make any
+ changes to your AWS WAF service
+ --verbose Enabled verbose output for the script. Useful for
+ debugging
+ -l, --list List the available EC2 instances
+ --tag TAGS [TAGS ...]
+ Specify the tags to filter the EC2 instances by
+ --create-match Create the SQLi match condition for use in various
+ rules
+```
+
+
## SSL Certificate Validation
@@ -129,4 +288,32 @@ And during execution you may see lines similar to;
.../requests/packages/urllib3/connectionpool.py:789: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
```
-These are expected warnings. Can you tell that we (and the python core teams) are trying to tell you something? If you're interesting in using a valid SSL certificate, you can get one for free from [Let's Encrypt](https://letsencrypt.org), [AWS themselves](https://aws.amazon.com/certificate-manager/) (if your DSM is behind an ELB), or explore commercial options (like the [one from Trend Micro](http://www.trendmicro.com/us/enterprise/cloud-solutions/deep-security/ssl-certificates/)).
\ No newline at end of file
+These are expected warnings. Can you tell that we (and the python core teams) are trying to tell you something? If you're interesting in using a valid SSL certificate, you can get one for free from [Let's Encrypt](https://letsencrypt.org), [AWS themselves](https://aws.amazon.com/certificate-manager/) (if your DSM is behind an ELB), or explore commercial options (like the [one from Trend Micro](http://www.trendmicro.com/us/enterprise/cloud-solutions/deep-security/ssl-certificates/)).
+
+
+
+## AWS WAF Costs
+
+The commands available in this repository are designed to help you build better rule sets for AWS WAF based on what Deep Security understands about your workloads.
+
+There are charges associated with pushing new rules to AWS WAF. **Always check the [AWS WAF pricing page](https://aws.amazon.com/waf/pricing/) for the latest prices.**
+
+AWS WAF charges for each web access control list (WACL), for each rule, and for the number of requests processed. This is in addition to any associated AWS CloudFront charges.
+
+We've done our best to ensure that each command optimizes the changes it makes in AWS WAF in order to reduce your costs. In general, you can run a command with the ```--dryrun``` option to see the results without making changes and before incurring any costs.
+
+
+
+### iplists
+
+The *iplists* command does not create a WACL or rule on your behalf. It creates new IPSet objects that can be used in an AWS WAF rule as a match condition. There are no charges for these IPSets.
+
+
+
+### sqli
+
+The *sqli* command provides recommendation as to which instances should be protected by an rule with an SQLi match set. Additionally, you can ask the command to create an SQLi match set that covers most web applications.
+
+There is no charge for the match set. Charge start when you create a rule using the match set.
+
+The script can be run in ```--dryrun``` to see the end result before pushing the match set to AWS. This can help you get a better idea of what is being created.
\ No newline at end of file
diff --git a/ds-to-aws-waf.py b/ds-to-aws-waf.py
new file mode 100755
index 0000000..20d21dd
--- /dev/null
+++ b/ds-to-aws-waf.py
@@ -0,0 +1,66 @@
+#! /usr/bin/env python
+
+# Standard libraries
+import argparse
+import os
+import urllib2
+import sys
+
+# 3rd party libraries
+import boto3
+import boto3.session
+import netaddr
+
+# project libraries
+import lib.core
+import lib.iplists
+import lib.sqli
+
+def parse_args(str_to_parse=None):
+ """
+ Parse the command line args
+ """
+ cmd = ""
+ if len(sys.argv) > 1:
+ cmd = sys.argv[1]
+
+ return cmd
+
+class Script(lib.core.ScriptContext):
+ def __init__(self, command_to_run):
+ self.command_to_run = command_to_run
+ self.available_commands = {
+ 'iplist':
+ {
+ 'help': 'Push a Deep Security IP list to an AWS WAF IP Set',
+ 'cmd': lib.iplists.run_script,
+ },
+ 'sqli':
+ {
+ 'help': 'Determine which instances protected by Deep Security should also be protected by AWS WAF SQLi rules',
+ 'cmd': lib.sqli.run_script,
+ },
+ }
+
+ if not self.command_to_run in self.available_commands.keys():
+ self.print_help()
+ else:
+ # run a specific command
+ self.available_commands[self.command_to_run]['cmd'](sys.argv[1:])
+
+ def print_help(self):
+ """
+ Print the command line syntax available to the user
+ """
+ self.update_user("usage: ds-to-aws-waf [COMMAND]\n For more help on a specific command, type ds-to-aws-waf [COMMAND] --help\n\n Available commands:\n")
+ for cmd, data in self.available_commands.items():
+ self.update_user(" {}\n > {}".format(cmd, data['help']))
+ self.update_user("")
+
+def main():
+ """
+ Run the script from the command line
+ """
+ context = Script(parse_args())
+
+if __name__ == '__main__': main()
\ No newline at end of file
diff --git a/dsawswaf/ip_list_to_set.py b/dsawswaf/ip_list_to_set.py
deleted file mode 100755
index d817a22..0000000
--- a/dsawswaf/ip_list_to_set.py
+++ /dev/null
@@ -1,409 +0,0 @@
-#! /usr/bin/env python
-
-# Standard libraries
-import argparse
-import os
-import urllib2
-
-# 3rd party libraries
-import boto3
-import boto3.session
-import netaddr
-
-# Project libraries
-import deepsecurity.manager
-
-def parse_args(str_to_parse=None):
- """
- Parse the command line args
- """
- description = """Deep Security uses the concept of IP Lists to make firewall rules easier
- to administer. The AWS WAF uses a similar concept of IP Sets as rule
- conditions.
-
- This utility helps synchronize Deep Security IP Lists with AWS WAF
- IP Sets.
- """
- parser = argparse.ArgumentParser(description=description)
-
- parser.add_argument('-d', '--ds', action='store', dest="ip_list", required=False, help='Specify an IP List within Deep Security as the source for the AWS WAF IP Set')
- parser.add_argument('-l', '--list', action='store_true', required=False, help='List the available Deep Security IP Lists and the AWS WAF IP Sets')
- parser.add_argument('-m', '--dsm', action='store', default='app.deepsecurity.trendmicro.com', required=False, help='The address of the Deep Security Manager. Defaults to Deep Security as a Service')
- parser.add_argument('--dsm-port', action='store', default='443', dest='dsm_port', required=False, help='The address of the Deep Security Manager. Defaults to Deep Security as a Service')
- parser.add_argument('-u', '--username', action='store', required=True, help='The Deep Security username to access the IP Lists with. Should only have read-only rights to IP lists and API access')
- parser.add_argument('-p', '--password', action='store', required=True, help='The password for the specified Deep Security username. Should only have read-only rights to IP lists and API access')
- parser.add_argument('-t', '--tenant', action='store', required=False, default=None, help='The name of the Deep Security tenant/account')
- parser.add_argument('--ignore-ssl-validation', action='store_true', dest='ignore_ssl_validation', required=False, help='Ignore SSL certification validation. Be careful when you use this as it disables a recommended security check. Required for Deep Security Managers using a self-signed SSL certificate')
- parser.add_argument('--dryrun', action='store_true', required=False, help='Do a dry run of the command. This will not make any changes to your AWS WAF service')
- parser.add_argument('--verbose', action='store_true', required=False, help='Enabled verbose output for the script. Useful for debugging')
-
- if str_to_parse:
- return parser.parse_args(str_to_parse)
- else:
- return parser.parse_args()
-
-class ScriptContext():
- """
- Context for IP List to IP Set script.
-
- Using an object makes is easy to avoid any globals and clarifies
- the intention of the script
- """
- def __init__(self, args):
- self.args = args
- self.aws_credentials = None
- self.dsm = None
- self.ip_lists = []
- self.waf = None
- self.ip_sets = []
-
- self.aws_credentials = self._get_aws_credentials()
- self.dsm = self._connect_to_deep_security()
- self.waf = self._connect_to_aws_waf()
- self.ip_lists = self._get_available_ds_lists()
-
- def __del__(self): self.clean_up() # clean up on object destruction
-
- def _log(self, message, priority=False):
- # @TODO add actual logging :-)
- if priority or self.args.verbose:
- print message
-
- def clean_up(self):
- """
- Gracefully dispose of the script's context
- """
- if self.dsm:
- try:
- self.dsm.finish_session()
- except Exception, err: pass
-
- def _get_aws_credentials(self):
- """
- Get a set of AWS credentials from a pre-configured AWS CLI installation
- """
- credentials = None
-
- # check locally for an AWS CLI installation
- aws_credentials_path = [ '{}/.aws/credentials'.format(os.environ['HOME']), "{}\.aws\credentials".format(os.environ['HOME']) ]
- for path in aws_credentials_path:
- if os.path.exists(path) and not credentials:
- self._log("Reading AWS credentials from {}".format(path))
- with open(path) as fh:
- for line in fh:
- if line.startswith('aws_access_key_id'):
- credentials = { 'aws_access_key_id': line.split('=')[-1].strip() }
- elif line.startswith('aws_secret_access_key'):
- credentials['aws_secret_access_key'] = line.split('=')[-1].strip()
-
- return credentials
-
- def _connect_to_deep_security(self):
- dsm = None
- if self.args.ignore_ssl_validation:
- self._log("""***********************************************************************
-* IGNORING SSL CERTIFICATE VALIDATION
-* ===================================
-* You have requested to ignore SSL certificate validation. This is a less secure method
-* of connecting to a Deep Security Manager (DSM). Please ensure that you have other
-* mitigations and security controls in place (like restricting IP space that can access
-* the DSM, implementing least privilege for the Deep Security user/role accessing the
-* API, etc).
-*
-* During script execution, you'll see a number of "InsecureRequestWarning" messages.
-* These are to be expected when operating without validation.
-***********************************************************************""", priority=True)
- try:
- dsm = deepsecurity.manager.Manager(dsm_hostname=self.args.dsm, dsm_port=self.args.dsm_port, username=self.args.username, password=self.args.password, tenant=self.args.tenant, ignore_ssl_validation=self.args.ignore_ssl_validation)
- self._log("Connected to the Deep Security Manager at {}".format(self.args.dsm))
- except Exception, err: pass # @TODO handle this exception gracefully
-
- if not dsm.session_id_rest and not dsm.session_id_soap:
- self._log("Unable to connect to the Deep Security Manager. Please check your settings")
- if not self.args.ignore_ssl_validation:
- self._log("You did not ask to ignore SSL certification validation. This is a common error when connect to a Deep Security Manager that was installed via software or the AWS Marketplace. Please set the flag (--ignore-ssl-validation), check your other settings, and try again")
-
- return dsm
-
- def _connect_to_aws_waf(self):
- waf = None
- try:
- aws = boto3.session.Session(aws_access_key_id=self.aws_credentials['aws_access_key_id'], aws_secret_access_key=self.aws_credentials['aws_secret_access_key'])
- waf = aws.client('waf')
- self._log("Connected to AWS WAF")
- except Exception, err:
- # @TODO handle this exception gracefully
- try:
- waf = boto3.client('waf')
- self._log("Connected to AWS WAF")
- except Exception, err: pass # @TODO handle this exception gracefully
-
- return waf
-
- def _get_available_ds_lists(self):
- """
- Query Deep Security for any existing IP Lists
- """
- ip_lists = None
-
- if self.dsm:
- self.dsm.get_ip_lists()
- ip_lists = self.dsm.ip_lists
- self._log("Cached the available IP Lists from Deep Security")
-
- return ip_lists
-
- def get_ds_list(self, list_id):
- """
- Get the specified IP List from the cached results
- """
- ip_list = None
-
- for key in self.ip_lists.keys():
- if '{}'.format(list_id.strip()) == str(key):
- ip_list = self.ip_lists[key]
- self._log("Found Deep Security IP list [{}]".format(ip_list))
- break
-
- return ip_list
-
- def get_available_aws_sets(self):
- """
- Get a list of the available IP Sets in AWS WAF
- """
- ip_sets = []
- if self.waf:
- response = self.waf.list_ip_sets(Limit=100)
- if response and response.has_key('IPSets'):
- for ip_set in response['IPSets']:
- ip_sets.append(ip_set)
-
- self.ip_sets = ip_sets
-
- return ip_sets
-
- def _get_aws_waf_change_token(self):
- response = self.waf.get_change_token()
- change_token = None
- if response and response.has_key('ChangeToken'):
- change_token = response['ChangeToken']
- self._log("New AWS WAF change token [{}]".format(change_token))
-
- return change_token
-
- def _expand_cidr(self, cidr):
- # AWS WAF IP Sets only accept octets of ['8','16','24','32']
- # find the next largest one to expand the specified CIDR
- # to the smallest possible set
- blocks = []
-
- # yes, you can figure the strata out algorithmically but this
- # is a little more readable and will definitely help with long
- # term maintenance which is way more important!
- strata = { # defines the CIDR octet blocks
- 32: { 'type': 32, 'times': 1, 'size': 1 },
- 31: { 'type': 32, 'times': 2, 'size': 1 },
- 30: { 'type': 32, 'times': 4, 'size': 1 },
- 29: { 'type': 32, 'times': 8, 'size': 1 },
- 28: { 'type': 32, 'times': 16, 'size': 1 },
- 27: { 'type': 32, 'times': 32, 'size': 1 },
- 26: { 'type': 32, 'times': 64, 'size': 1 },
- 25: { 'type': 32, 'times': 128, 'size': 1 },
- 24: { 'type': 24, 'times': 1, 'size': 256 },
- 23: { 'type': 24, 'times': 2, 'size': 256 },
- 22: { 'type': 24, 'times': 4, 'size': 256 },
- 21: { 'type': 24, 'times': 8, 'size': 256 },
- 20: { 'type': 24, 'times': 16, 'size': 256 },
- 19: { 'type': 24, 'times': 32, 'size': 256 },
- 18: { 'type': 24, 'times': 64, 'size': 256 },
- 17: { 'type': 24, 'times': 128, 'size': 256 },
- 16: { 'type': 16, 'times': 1, 'size': 65536 },
- 15: { 'type': 16, 'times': 2, 'size': 65536 },
- 14: { 'type': 16, 'times': 4, 'size': 65536 },
- 13: { 'type': 16, 'times': 8, 'size': 65536 },
- 12: { 'type': 16, 'times': 16, 'size': 65536 },
- 11: { 'type': 16, 'times': 32, 'size': 65536 },
- 10: { 'type': 16, 'times': 64, 'size': 65536 },
- 9: { 'type': 16, 'times': 128, 'size': 65536 },
- 8: { 'type': 8, 'times': 1, 'size': 16777216 },
- 7: { 'type': 8, 'times': 2, 'size': 16777216 },
- 6: { 'type': 8, 'times': 4, 'size': 16777216 },
- 5: { 'type': 8, 'times': 8, 'size': 16777216 },
- 4: { 'type': 8, 'times': 16, 'size': 16777216 },
- 3: { 'type': 8, 'times': 32, 'size': 16777216 },
- 2: { 'type': 8, 'times': 64, 'size': 16777216 },
- 1: { 'type': 8, 'times': 128, 'size': 16777216 },
- }
-
- current_strata = int(cidr.__str__().split('/')[-1])
- for i in range(strata[current_strata]['times']):
- if i == 0:
- current_cidr = netaddr.IPNetwork('{}/{}'.format(cidr[0], strata[current_strata]['type']))
- else:
- index = i * strata[current_strata]['size']
- current_cidr = netaddr.IPNetwork('{}/{}'.format(cidr[index], strata[current_strata]['type']))
-
- blocks.append(current_cidr)
-
- self._log("Expanded CIDR block {} to {} IP Set compatible blocks".format(cidr, len(blocks)))
-
- return blocks
-
- def _parse_ds_addresses(self, ds_list):
- # Accepted DS formats
- # X.X.X.X/1-32 Example: 192.168.2.0/24
- # X.X.X.X/Y.Y.Y.Y Example: 192.168.2.0/255.255.255.0
- # X.X.X.X Example: 192.168.2.33
- # IpV6 Mask Example: 2001:0DB8::CD30:0:0:0:0/60
- # X.X.X.X-Y.Y.Y.Y Example: 192.168.0.2 - 192.168.0.125
- # IPv6-IPv6 Example: FF01::101 - FF01::102
- # IP or Range #Comment Example: 255.255.255.255 #Broadcast IP
- addresses = []
- for address in ds_list.addresses:
- if "#" in address: address = address.split('#').strip() # remove any comments
- if '-' in address:
- try:
- a1, a2 = address.split('-')
- # what's the range between a1 and a2?
- rng = netaddr.IPRange(a1, a2)
- for addr in rng: addresses.append(netaddr.IPNetwork(addr))
- except Exception, err: pass
- else:
- net = netaddr.IPNetwork(address)
- if '{}'.format(net.cidr).split('/')[-1] in ['8','16','24','32']:
- addresses.append(netaddr.IPNetwork(net))
- else:
- for addr in net: addresses.append(netaddr.IPNetwork(addr))
-
- total_set = netaddr.IPSet(addresses)
- total_set.compact()
-
- cidrs = total_set.iter_cidrs()
- waf_compatible = []
- for cidr in cidrs:
- if '{}'.format(cidr).split('/')[-1] in ['8','16','24','32']:
- waf_compatible.append(cidr)
- else:
- waf_compatible += self._expand_cidr(cidr)
-
- self._log("Converted {} IP List entries to {} IP Set entries".format(len(ds_list.addresses), len(waf_compatible)), priority=True)
- return waf_compatible
-
- def _convert_ds_addresses_to_waf(self, ds_list, convert='ignore'):
- """
- convert = One of ['ignore', 'expand', 'upscale']
- """
- # AWS WAF IP Sets only accept CIDR notation with a closing octet of /8 /16 /24 or /32
- # We need to ensure that the specified Deep Security IP List will convert to an IP Set
- addresses = self._parse_ds_addresses(ds_list)
-
- updates = []
- for cidr_network in addresses:
- updates.append(
- {
- 'Action': 'INSERT',
- 'IPSetDescriptor': {
- 'Type': 'IPV{}'.format(cidr_network.version),
- 'Value': cidr_network.cidr.__str__(),
- }
- }
- )
-
- return updates
-
- def create_ip_set(self, ds_list):
- """
- Create an AWS WAF IP Set based on the specified Deep Security IP List
- """
- if self.waf:
- # is there an existing IP Set?
- current_ip_set = None
- if not self.ip_sets: self.get_available_aws_sets()
- for ip_set in self.ip_sets:
- if ip_set.has_key('Name') and ip_set['Name'] == ds_list.name:
- current_ip_set = ip_set['IPSetId']
- break
-
- response = self.waf.get_change_token()
- change_token = self._get_aws_waf_change_token()
-
- if change_token:
- list_created = False
- if not current_ip_set:
- if not self.args.dryrun:
- self._log("Requesting the creation of [{}]".format(ds_list.name))
- response = self.waf.create_ip_set(Name=ds_list.name, ChangeToken=change_token)
- if response:
- self.ip_sets.append(response['IPSet'])
- list_created = True
- current_ip_set = response['IPSet']['IPSetId']
- change_token = self._get_aws_waf_change_token()
- else:
- self._log("Will request the creation of [{}]".format(ds_list.name), priority=True)
-
- if current_ip_set and change_token:
- updates = self._convert_ds_addresses_to_waf(ds_list)
- if len(updates) > 1000:
- self._log("Requested IP List converts to more then the 1000 entry limit for an IP Set. Please split the IP List within Deep Security before syncing with AWS WAF", priority=True)
- else:
- if not self.args.dryrun:
- response = self.waf.update_ip_set(IPSetId=current_ip_set, ChangeToken=change_token, Updates=updates)
- if response and response.has_key('ChangeToken'):
- self._log('Change [{}] requested'.format(response['ChangeToken']))
- else:
- self._log("Will request the addition of {} entries in IP Set {}".format(len(updates), current_ip_set), priority=True)
-
- if not self.args.dryrun:
- msg_verb = "Created" if list_created else "Updated"
- self._log("{} IP Set [{}] with ID [{}]".format(msg_verb, ds_list.name, current_ip_set), priority=True)
- else:
- msg_verb = "create" if list_created else "update"
- self._log("Will {} IP Set [{}] with ID [{}]".format(msg_verb, ds_list.name, current_ip_set), priority=True)
-
- def print_lists(self):
- """
- Pretty print the IP lists from Deep Security and
- the AWS WAF IP Sets
- """
- print "\nAvailable Deep Security IP Lists"
- print "================================"
- if self.ip_lists and len(self.ip_lists) > 0:
- for ds_list_id, ds_list in self.ip_lists.items():
- print "{}\t{}".format(ds_list_id, ds_list.name)
- else:
- print "---\t No IP lists available"
-
-
- print "\nAvailable AWS WAF IP Sets"
- print "========================="
- if self.ip_sets and len(self.ip_sets) > 0:
- for waf_set in self.ip_sets:
- print "{}\t{}".format(waf_set['IPSetId'], waf_set['Name'])
- else:
- print "---\t No AWS WAF WACLs available"
-
-def main():
- """
- Run the script from the command line
- """
- context = ScriptContext(parse_args())
-
- if context.args.list:
- # List the available Deep Security IP Lists and AWS WAF IP Sets
- context.get_available_aws_sets()
- context.print_lists()
-
- elif context.args.ip_list:
- if context.args.dryrun:
- context._log("***********************************************************************", priority=True)
- context._log("* DRY RUN ENABLED. NO CHANGES WILL BE MADE", priority=True)
- context._log("***********************************************************************", priority=True)
- # get the specified Deep Security IP Lists (already cached)
- ip_list = context.get_ds_list(context.args.ip_list)
- # create the IP Set
- if ip_list:
- context.create_ip_set(ip_list)
-
- context.clean_up()
-
-if __name__ == '__main__': main()
\ No newline at end of file
diff --git a/dsawswaf/requests/packages/urllib3/contrib/__init__.py b/lib/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/contrib/__init__.py
rename to lib/__init__.py
diff --git a/lib/core.py b/lib/core.py
new file mode 100644
index 0000000..e864df8
--- /dev/null
+++ b/lib/core.py
@@ -0,0 +1,236 @@
+# standard libraries
+import argparse
+import os
+import urllib2
+
+# 3rd party libraries
+import boto3
+import boto3.session
+
+# project libraries
+import deepsecurity
+
+def get_arg_parser(prog='ds-to-aws-waf.py', description=None, add_help=False):
+ """
+ Create a standardized argument parser
+ """
+ if not description:
+ description = """
+ Create and update AWS WAF WACL rules based on information from a Deep Security installation
+"""
+
+ parser = argparse.ArgumentParser(prog=prog, description=description, add_help=add_help)
+
+ # Deep Security arguments
+ parser.add_argument('-d', '--dsm', action='store', default='app.deepsecurity.trendmicro.com', required=False, help='The address of the Deep Security Manager. Defaults to Deep Security as a Service')
+ parser.add_argument('--dsm-port', action='store', default='4119', dest='dsm_port', required=False, help='The address of the Deep Security Manager. Defaults to an AWS Marketplace/software install (:4119). Automatically configured for Deep Security as a Service')
+ parser.add_argument('-u', '--dsm-username', action='store', dest='dsm_username', required=True, help='The Deep Security username to access the IP Lists with. Should only have read-only rights to IP lists and API access')
+ parser.add_argument('-p', '--dsm-password', action='store', dest='dsm_password', required=True, help='The password for the specified Deep Security username. Should only have read-only rights to IP lists and API access')
+ parser.add_argument('-t', '--dsm-tenant', action='store', dest='dsm_tenant', required=False, default=None, help='The name of the Deep Security tenant/account')
+
+ # AWS arguments
+ parser.add_argument('-r', '--aws-region', action='store', dest='aws_region', required=False, default='us-east-1', help='The name of AWS region to connect to')
+
+ # general structure arguments
+ parser.add_argument('--ignore-ssl-validation', action='store_true', dest='ignore_ssl_validation', required=False, help='Ignore SSL certification validation. Be careful when you use this as it disables a recommended security check. Required for Deep Security Managers using a self-signed SSL certificate')
+ parser.add_argument('--dryrun', action='store_true', required=False, help='Do a dry run of the command. This will not make any changes to your AWS WAF service')
+ parser.add_argument('--verbose', action='store_true', required=False, help='Enabled verbose output for the script. Useful for debugging')
+
+ return parser
+
+class StoreNameValuePairOnEquals(argparse.Action):
+ """
+ Store a set of name value pairs as an argument
+ """
+ def __init__(self, option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None):
+ self.dest = dest
+ argparse.Action.__init__(self, option_strings, dest, nargs=nargs, const=const, default=default, type=type, choices=choices, required=required, help=help, metavar=metavar)
+
+ # cribbed from http://stackoverflow.com/questions/5154716/using-argparse-to-parse-arguments-of-form-arg-val
+ # response by @chepner (http://stackoverflow.com/users/1126841/chepner)
+ def __call__(self, parser, namespace, values, dest, option_string=None):#
+ pairs = {}
+ for val in values:
+ if '=' in val:
+ n, v = val.split('=')
+ pairs[n] = v # matches key:pair
+ else:
+ pairs[v] = '' # matches key:
+
+ attr_key = option_string.strip('-') if option_string else ""
+ if self.dest: attr_key = self.dest
+
+ current_pairs = getattr(namespace, attr_key)
+ if attr_key in dir(namespace) and current_pairs != None:
+ new_pairs = current_pairs.copy()
+ new_pairs.update(pairs)
+ setattr(namespace, attr_key, new_pairs)
+ else:
+ setattr(namespace, attr_key, pairs)
+
+class ScriptContext():
+ """
+ Context for a command line script.
+
+ Using an object makes is easy to avoid any globals and clarifies
+ the intention of the script
+ """
+ def __init__(self, args, parser):
+ self.parser = parser
+ self._passed_args = args
+ self.args = parser.parse_args(self._passed_args)
+ self.dsm = None
+
+ def __del__(self): self.clean_up() # clean up on object destruction
+
+ def clean_up(self):
+ """
+ Gracefully dispose of the script's context
+ """
+ if 'dsm' in dir(self) and self.dsm:
+ try:
+ self.dsm.finish_session()
+ except Exception, err: pass
+
+ def update_user(self, message):
+ """
+ Update the update
+ """
+ print(message)
+
+ def _log(self, msg, err=None, priority=False):
+ """
+ Create a log entry for the specified event
+ """
+ # @TODO add actual logging :-)
+ if priority or self.args.verbose or err:
+ if err:
+ print("{}. Threw an exception:\n{}".format(msg, err))
+ else:
+ print(msg)
+
+ def print_help(self):
+ """
+ Print the command line syntax available to the user
+ """
+ self.parser.print_help()
+
+ def _get_aws_credentials(self):
+ """
+ Get a set of AWS credentials from a pre-configured AWS CLI installation
+ """
+ credentials = None
+
+ # check locally for an AWS CLI installation
+ aws_credentials_path = [ '{}/.aws/credentials'.format(os.environ['HOME']), "{}\.aws\credentials".format(os.environ['HOME']) ]
+ for path in aws_credentials_path:
+ if os.path.exists(path) and not credentials:
+ self._log("Reading AWS credentials from {}".format(path))
+ with open(path) as fh:
+ for line in fh:
+ if line.startswith('aws_access_key_id'):
+ credentials = { 'aws_access_key_id': line.split('=')[-1].strip() }
+ elif line.startswith('aws_secret_access_key'):
+ credentials['aws_secret_access_key'] = line.split('=')[-1].strip()
+
+ return credentials
+
+ def _connect_to_deep_security(self):
+ dsm = None
+ if self.args.ignore_ssl_validation:
+ self._log("""************************************************************************
+* IGNORING SSL CERTIFICATE VALIDATION
+* ===================================
+* You have requested to ignore SSL certificate validation. This is a
+* less secure method of connecting to a Deep Security Manager (DSM).
+* Please ensure that you have other mitigations and security controls
+* in place (like restricting IP space that can access the DSM,
+* implementing least privilege for the Deep Security user/role
+* accessing the API, etc).
+*
+* During script execution, you'll see a number of
+* "InsecureRequestWarning" messages. These are to be expected when
+* operating without validation.
+************************************************************************""", priority=True)
+ try:
+ dsm_port = self.args.dsm_port if not self.args.dsm == 'app.deepsecurity.trendmicro.com' else 443
+ self._log("Attempting to connect to Deep Security at {}:{}".format(self.args.dsm, dsm_port))
+ dsm = deepsecurity.manager.Manager(dsm_hostname=self.args.dsm, dsm_port=dsm_port, username=self.args.dsm_username, password=self.args.dsm_password, tenant=self.args.dsm_tenant, ignore_ssl_validation=self.args.ignore_ssl_validation)
+ self._log("Connected to the Deep Security Manager at {}".format(self.args.dsm))
+ except Exception, err:
+ self._log("Could not connect to the Deep Security", err=err)
+
+ if not dsm.session_id_rest and not dsm.session_id_soap:
+ self._log("Unable to connect to the Deep Security Manager. Please check your settings")
+ if not self.args.ignore_ssl_validation:
+ self._log("You did not ask to ignore SSL certification validation. This is a common error when connect to a Deep Security Manager that was installed via software or the AWS Marketplace. Please set the flag (--ignore-ssl-validation), check your other settings, and try again")
+
+ return dsm
+
+ def _connect_to_aws_waf(self):
+ """
+ Connect to AWS WAF via explicit credentials (shared by the AWS CLI)
+ or an instance role
+ """
+ waf = None
+ try:
+ aws = boto3.session.Session(aws_access_key_id=self.aws_credentials['aws_access_key_id'], aws_secret_access_key=self.aws_credentials['aws_secret_access_key'])
+ waf = aws.client('waf')
+ self._log("Connected to AWS WAF")
+ except Exception, err:
+ self._log("Could not connect to AWS WAF using local CLI credentials", err=err)
+ try:
+ waf = boto3.client('waf')
+ self._log("Connected to AWS WAF")
+ except Exception, err:
+ self._log("Could not connect to AWS WAF using an instance role", err=err)
+
+ return waf
+
+ def _connect_to_aws_ec2(self):
+ """
+ Connect to AWS EC2 via explicit credentials (shared by the AWS CLI)
+ or an instance role
+ """
+ ec2 = None
+ try:
+ aws = boto3.session.Session(aws_access_key_id=self.aws_credentials['aws_access_key_id'], aws_secret_access_key=self.aws_credentials['aws_secret_access_key'], region_name=self.args.aws_region)
+ ec2 = aws.client('ec2')
+ self._log("Connected to AWS EC2")
+ except Exception, err:
+ self._log("Could not connect to AWS EC2 using local CLI credentials", err=err)
+ try:
+ aws = boto3.session.Session(region_name=self.args.aws_region)
+ ec2 = aws.client('ec2')
+ self._log("Connected to AWS EC2")
+ except Exception, err:
+ self._log("Could not connect to AWS EC2 using an instance role", err=err)
+
+ return ec2
+
+ def get_available_aws_sets(self):
+ """
+ Get a list of the available IP Sets in AWS WAF
+ """
+ ip_sets = []
+ if self.waf:
+ response = self.waf.list_ip_sets(Limit=100)
+ if response and response.has_key('IPSets'):
+ for ip_set in response['IPSets']:
+ ip_sets.append(ip_set)
+
+ self.ip_sets = ip_sets
+
+ return ip_sets
+
+ def _get_aws_waf_change_token(self):
+ """
+ Get a new AWS WAF change token (required for any changes)
+ """
+ response = self.waf.get_change_token()
+ change_token = None
+ if response and response.has_key('ChangeToken'):
+ change_token = response['ChangeToken']
+ self._log("New AWS WAF change token [{}]".format(change_token))
+
+ return change_token
\ No newline at end of file
diff --git a/dsawswaf/deepsecurity/__init__.py b/lib/deepsecurity/__init__.py
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/__init__.py
rename to lib/deepsecurity/__init__.py
diff --git a/lib/deepsecurity/application_type.py b/lib/deepsecurity/application_type.py
new file mode 100644
index 0000000..625b093
--- /dev/null
+++ b/lib/deepsecurity/application_type.py
@@ -0,0 +1,30 @@
+class ApplicationType(object):
+ def __init__(self, type_details, manager=None):
+ self.manager = manager
+ self.data = type_details
+ self._process_type_detail(type_details)
+
+ # *****************************************************************
+ # 'Private' methods
+ # *****************************************************************
+ def _process_type_detail(self, type_details):
+ """
+ Convert the most useful type details returned from the API into
+ top level properties
+ """
+ for key, prop in {
+ 'ID': 'id',
+ 'name': 'name',
+ 'description': 'description',
+ 'TBUID': 'tbuid',
+ 'authoritative': 'authoritative',
+ 'direction': 'direction',
+ 'ignoreRecommendations': 'ignore_recommendations',
+ 'protocolIcmp': 'protocol_icmp',
+ 'protocolPortBased': 'protocol_port_based',
+ 'protocolType': 'protocol_type',
+ }.items():
+ try:
+ setattr(self, prop, type_details[key])
+ except Exception, err:
+ if self.manager: self.manager.log("Could not add property [%s] to rule [%s]. Threw exception: %s".format(prop, type_details['name'], err), err=err, level='warning')
\ No newline at end of file
diff --git a/dsawswaf/deepsecurity/cloud_account.py b/lib/deepsecurity/cloud_account.py
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/cloud_account.py
rename to lib/deepsecurity/cloud_account.py
diff --git a/dsawswaf/deepsecurity/computer.py b/lib/deepsecurity/computer.py
old mode 100755
new mode 100644
similarity index 53%
rename from dsawswaf/deepsecurity/computer.py
rename to lib/deepsecurity/computer.py
index ad4ad55..39aac63
--- a/dsawswaf/deepsecurity/computer.py
+++ b/lib/deepsecurity/computer.py
@@ -3,6 +3,14 @@ def __init__(self, host_details, manager=None):
self.manager = manager
self.data = host_details
self._process_host_detail(host_details)
+ self.application_types = {}
+ self.recommended_rules = {
+ 'intrusion_prevention': {},
+ 'firewall': {},
+ 'web_reputation': {},
+ 'integrity_monitoring': {},
+ 'log_inspection': {},
+ }
# *****************************************************************
# 'Private' methods
@@ -17,13 +25,31 @@ def _process_host_detail(self, host_details):
'name': 'hostname',
'description': 'description',
'displayName': 'display_name',
+ 'external': 'external',
+ 'externalID': 'external_id',
+ 'hostGroupID': 'host_group_id',
+ 'hostType': 'host_type',
'platform': 'platform',
'securityProfileID': 'policy_id',
- 'cloudObjectImageId': 'cloud_image_id',
- 'cloudObjectInstanceId': 'cloud_instance_id',
+ 'antiMalwareClassicPatternVersion': 'anti_malware_classic_pattern_version',
+ 'antiMalwareEngineVersion': 'anti_malware_engine_version',
+ 'antiMalwareIntelliTrapExceptionVersion': 'anti_malware_intelli_trap_exception_version',
+ 'antiMalwareIntelliTrapVersion': 'anti_malware_intelli_trap_version',
+ 'antiMalwareSmartScanPatternVersion': 'anti_malware_smart_scan_pattern_version',
+ 'antiMalwareSpywarePatternVersion': 'anti_malware_spyware_pattern_version',
+ 'cloudObjectImageId': 'cloud_object_image_id', # @TODO handle property name change
+ 'cloudObjectInstanceId': 'cloud_object_instance_id', # @TODO handle property name change
+ 'cloudObjectInternalUniqueId': 'cloud_object_internal_unique_id',
'cloudObjectSecurityGroupIds': 'cloud_security_policy',
'cloudObjectType': 'cloud_type',
+ 'componentKlasses': 'component_klasses',
+ 'componentNames': 'component_names',
+ 'componentTypes': 'component_types',
+ 'componentVersions': 'component_versions',
+ 'hostGroupName': 'host_group_name',
+ 'hostInterfaces': 'host_interfaces',
'hostLight': 'status_light',
+
'securityProfileName': 'policy_name',
'lastIPUsed': 'last_ip',
'overallAntiMalwareStatus': 'module_status_anti_malware',
@@ -33,16 +59,41 @@ def _process_host_detail(self, host_details):
'overallLogInspectionStatus': 'module_status_log_inspection',
'overallWebReputationStatus': 'module_status_web_reputation',
'overallStatus': 'overall_status',
+ 'lastAnitMalwareScheduledScan': 'last_anti_malware_scheduled_scan',
+ 'lastAntiMalwareEvent': 'last_anti_malware_event',
+ 'lastAntiMalwareManualScan': 'last_anti_malware_manual_scan',
+ 'lastDpiEvent': 'last_dpi_event',
+ 'lastFirewallEvent': 'last_firewall_event',
+ 'lastIPUsed': 'last_ip_used',
+ 'lastIntegrityMonitoringEvent': 'last_integrity_monitoring_event',
+ 'lastLogInspectionEvent': 'last_log_inspection_event',
+ 'lastWebReputationEvent': 'last_web_reputation_event',
+ 'light': 'light',
+ 'locked': 'locked',
+ 'overallLastRecommendationScan': 'overall_last_recommendation_scan',
+ 'overallLastSuccessfulCommunication': 'overall_last_successful_communication',
+ 'overallLastSuccessfulUpdate': 'overall_last_successful_update',
+ 'overallLastUpdateRequired': 'overall_last_update_required',
+ 'overallVersion': 'overall_version',
+ 'securityProfileName': 'security_profile_name',
+ 'virtualName': 'virtual_name',
+ 'virtualUuid': 'virtual_uuid',
}.items():
try:
- setattr(self, prop, host_details[key])
+ if key in dir(host_details):
+ setattr(self, prop, host_details[key])
+ else:
+ self.manager.log("Property {} is not present in API response".format(key))
except Exception, err:
- if self.manager: self.manager.log.warning("Could not add property [%s] to computer [%s]. Threw exception: %s" % (prop, host_details['name'], err))
+ if self.manager: self.manager.log("Could not add property [%s] to computer [%s]. Threw exception: %s".format(prop, host_details['name'], err), err=err, level='warning')
try:
- self.number_of_interfaces = len(host_details['hostInterfaces'])
+ if 'has_key' in host_details and host_details.has_key('hostInterfaces') and host_details['hostInterfaces'] and type(host_details['hostInterfaces']) == type([]):
+ self.number_of_interfaces = len(host_details['hostInterfaces'])
+ else:
+ self.number_of_interfaces = None
except Exception, err:
- if self.manager: self.manager.log.warning("Could not add property [number_of_interfaces] to computer [%s]. Threw exception: %s" % (host_details['name'], err))
+ if self.manager: self.manager.log("Could not add property [number_of_interfaces] to computer [%s]. Threw exception: %s".format(host_details['name'], err), err=err, level='warning')
# *****************************************************************
# Public methods
diff --git a/dsawswaf/deepsecurity/computer_group.py b/lib/deepsecurity/computer_group.py
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/computer_group.py
rename to lib/deepsecurity/computer_group.py
diff --git a/dsawswaf/deepsecurity/deepsecurity.latest.wsdl.xml b/lib/deepsecurity/deepsecurity.latest.wsdl.xml
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/deepsecurity.latest.wsdl.xml
rename to lib/deepsecurity/deepsecurity.latest.wsdl.xml
diff --git a/lib/deepsecurity/firewall_rule.py b/lib/deepsecurity/firewall_rule.py
new file mode 100644
index 0000000..92cbb3b
--- /dev/null
+++ b/lib/deepsecurity/firewall_rule.py
@@ -0,0 +1,78 @@
+class FirewallRule(object):
+ def __init__(self, rule_details, manager=None):
+ self.manager = manager
+ self.data = rule_details
+ self._process_rule_detail(rule_details)
+
+ # *****************************************************************
+ # 'Private' methods
+ # *****************************************************************
+ def _process_rule_detail(self, rule_details):
+ """
+ Convert the most useful rules details returned from the API into
+ top level properties
+ """
+ for key, prop in {
+ 'ID': 'id',
+ 'name': 'name',
+ 'description': 'description',
+ 'action': 'action',
+ 'anyFlags': 'any_flags',
+ 'destinationIP': 'destination_ip',
+ 'destinationIPListID': 'destination_ip_list_id',
+ 'destinationIPMask': 'destination_ip_mask',
+ 'destinationIPNot': 'destination_ip_not',
+ 'destinationIPRangeFrom': 'destination_ip_range_from',
+ 'destinationIPRangeTo': 'destination_ip_range_to',
+ 'destinationIPType': 'destination_ip_type',
+ 'destinationMAC': 'destination_mac',
+ 'destinationMACListID': 'destination_mac_list_id',
+ 'destinationMACNot': 'destination_mac_not',
+ 'destinationMACType': 'destination_mac_type',
+ 'destinationPortListID': 'destination_port_list_id',
+ 'destinationPortNot': 'destination_port_not',
+ 'destinationPortType': 'destination_port_type',
+ 'destinationPorts': 'destination_ports',
+ 'destinationSingleIP': 'destination_single_ip',
+ 'disabledLog': 'disabled_log',
+ 'frameNot': 'frame_not',
+ 'frameNumber': 'frame_number',
+ 'frameType': 'frame_type',
+ 'icmpCode': 'icmp_code',
+ 'icmpNot': 'icmp_not',
+ 'icmpType': 'icmpType',
+ 'packetDirection': 'packet_direction',
+ 'priority': 'priority',
+ 'protocolNot': 'protocol_not',
+ 'protocolNumber': 'protocol_number',
+ 'protocolType': 'protocol_type',
+ 'raiseAlert': 'raise_alert',
+ 'scheduleID': 'schedule_id',
+ 'sourceIP': 'source_ip',
+ 'sourceIPListID': 'source_ip_list_id',
+ 'sourceIPMask': 'source_ip_mask',
+ 'sourceIPNot': 'source_ip_not',
+ 'sourceIPRangeFrom': 'source_ip_range_from',
+ 'sourceIPRangeTo': 'source_ip_range_to',
+ 'sourceIPType': 'source_ip_type',
+ 'sourceMAC': 'source_mac',
+ 'sourceMACListID': 'source_mac_list_id',
+ 'sourceMACNot': 'source_mac_not',
+ 'sourceMACType': 'source_mac_type',
+ 'sourcePortListID': 'source_port_list_id',
+ 'sourcePortNot': 'source_port_not',
+ 'sourcePortType': 'source_port_type',
+ 'sourcePorts': 'source_ports',
+ 'sourceSingleIP': 'source_single_ip',
+ 'tcpFlagACK': 'tcp_flag_ack',
+ 'tcpFlagFIN': 'tcp_flag_fin',
+ 'tcpFlagPSH': 'tcp_flag_psh',
+ 'tcpFlagRST': 'tcp_flag_rst',
+ 'tcpFlagSYN': 'tcp_flag_syn',
+ 'tcpFlagURG': 'tcp_flag_urg',
+ 'tcpNot': 'tcp_not',
+ }.items():
+ try:
+ setattr(self, prop, rule_details[key])
+ except Exception, err:
+ if self.manager: self.manager.log("Could not add property [%s] to rule [%s]. Threw exception: %s".format(prop, rule_details['name'], err), err=err, level='warning')
\ No newline at end of file
diff --git a/lib/deepsecurity/integrity_monitoring_rule.py b/lib/deepsecurity/integrity_monitoring_rule.py
new file mode 100644
index 0000000..d70eb7b
--- /dev/null
+++ b/lib/deepsecurity/integrity_monitoring_rule.py
@@ -0,0 +1,34 @@
+class IntegrityMonitoringRule(object):
+ def __init__(self, rule_details, manager=None):
+ self.manager = manager
+ self.data = rule_details
+ self._process_rule_detail(rule_details)
+
+ # *****************************************************************
+ # 'Private' methods
+ # *****************************************************************
+ def _process_rule_detail(self, rule_details):
+ """
+ Convert the most useful rules details returned from the API into
+ top level properties
+ """
+ for key, prop in {
+ 'ID': 'id',
+ 'name': 'name',
+ 'description': 'description',
+ 'TBUID': 'tbuid',
+ 'allowOnChange': 'allow_on_change',
+ 'authoritative': 'authoritative',
+ 'content': 'content',
+ 'identifier': 'identifier',
+ 'ignoreRecommendations': 'ignore_recommendations',
+ 'issued': 'issued',
+ 'minAgentVersion': 'min_agent_version',
+ 'minManagerVersion': 'min_manager_version',
+ 'raiseAlert': 'raise_alert',
+ 'severity': 'severity',
+ }.items():
+ try:
+ setattr(self, prop, rule_details[key])
+ except Exception, err:
+ if self.manager: self.manager.log("Could not add property [%s] to rule [%s]. Threw exception: %s".format(prop, rule_details['name'], err), err=err, level='warning')
\ No newline at end of file
diff --git a/lib/deepsecurity/intrusion_prevention_rule.py b/lib/deepsecurity/intrusion_prevention_rule.py
new file mode 100644
index 0000000..314460c
--- /dev/null
+++ b/lib/deepsecurity/intrusion_prevention_rule.py
@@ -0,0 +1,51 @@
+class IntrusionPreventionRule(object):
+ def __init__(self, rule_details, manager=None):
+ self.manager = manager
+ self.data = rule_details
+ self._process_rule_detail(rule_details)
+
+ # *****************************************************************
+ # 'Private' methods
+ # *****************************************************************
+ def _process_rule_detail(self, rule_details):
+ """
+ Convert the most useful rules details returned from the API into
+ top level properties
+ """
+ for key, prop in {
+ 'ID': 'id',
+ 'name': 'name',
+ 'description': 'description',
+ 'TBUID': 'tbuid',
+ 'applicationTypeID': 'application_type_id',
+ 'authoritative': 'authoritative',
+ 'cvssScore': 'cvss_score',
+ 'detectOnly': 'detect_only',
+ 'disableEvent': 'disable_event',
+ 'eventOnPacketDrop': 'event_on_packet_drop',
+ 'eventOnPacketModify': 'event_on_packet_modify',
+ 'identifier': 'identifier',
+ 'ignoreRecommendations': 'ignore_recommendations',
+ 'includePacketData': 'include_packet_data',
+ 'issued': 'issued',
+ 'patternAction': 'pattern_action',
+ 'patternCaseSensitive': 'pattern_case_sensitive',
+ 'patternEnd': 'pattern_end',
+ 'patternIf': 'pattern_if',
+ 'patternPatterns': 'pattern_patterns',
+ 'patternStart': 'pattern_start',
+ 'priority': 'priority',
+ 'raiseAlert': 'raise_alert',
+ 'ruleXML': 'rule_xml',
+ 'scheduleID': 'schedule_id',
+ 'severity': 'severity',
+ 'signatureAction': 'signature_action',
+ 'signatureCaseSensitive': 'signature_case_sensitive',
+ 'templateType': 'template_type',
+ 'cveNumbers': 'cve_numbers',
+ 'msNumbers': 'ms_numbers',
+ }.items():
+ try:
+ setattr(self, prop, rule_details[key])
+ except Exception, err:
+ if self.manager: self.manager.log("Could not add property [%s] to rule [%s]. Threw exception: %s".format(prop, rule_details['name'], err), err=err, level='warning')
\ No newline at end of file
diff --git a/dsawswaf/deepsecurity/ip_list.py b/lib/deepsecurity/ip_list.py
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/ip_list.py
rename to lib/deepsecurity/ip_list.py
diff --git a/lib/deepsecurity/log_inspection_rule.py b/lib/deepsecurity/log_inspection_rule.py
new file mode 100644
index 0000000..a0f7c00
--- /dev/null
+++ b/lib/deepsecurity/log_inspection_rule.py
@@ -0,0 +1,34 @@
+class LogInspectionRule(object):
+ def __init__(self, rule_details, manager=None):
+ self.manager = manager
+ self.data = rule_details
+ self._process_rule_detail(rule_details)
+
+ # *****************************************************************
+ # 'Private' methods
+ # *****************************************************************
+ def _process_rule_detail(self, rule_details):
+ """
+ Convert the most useful rules details returned from the API into
+ top level properties
+ """
+ for key, prop in {
+ 'ID': 'id',
+ 'name': 'name',
+ 'description': 'description',
+ 'TBUID': 'tbuid',
+ 'alertMinSeverity': 'alert_min_severity',
+ 'authoritative': 'authoritative',
+ 'content': 'content',
+ 'files': 'files',
+ 'identifier': 'identifier',
+ 'ignoreRecommendations': 'ignore_recommendations',
+ 'issued': 'issued',
+ 'minAgentVersion': 'min_agent_version',
+ 'minManagerVersion': 'min_manager_version',
+ 'raiseAlert': 'raise_alert',
+ }.items():
+ try:
+ setattr(self, prop, rule_details[key])
+ except Exception, err:
+ if self.manager: self.manager.log("Could not add property [%s] to rule [%s]. Threw exception: %s".format(prop, rule_details['name'], err), err=err, level='warning')
\ No newline at end of file
diff --git a/dsawswaf/deepsecurity/manager.py b/lib/deepsecurity/manager.py
old mode 100755
new mode 100644
similarity index 86%
rename from dsawswaf/deepsecurity/manager.py
rename to lib/deepsecurity/manager.py
index 75c35f6..fff2529
--- a/dsawswaf/deepsecurity/manager.py
+++ b/lib/deepsecurity/manager.py
@@ -13,10 +13,15 @@
import suds
# Project libraries
+import application_type
import cloud_account
import computer
import computer_group
+import firewall_rule
+import integrity_monitoring_rule
+import intrusion_prevention_rule
import ip_list
+import log_inspection_rule
import policy
import soap_https_handler
@@ -55,6 +60,14 @@ def __init__(self, username=None, password=None, tenant='Primary', dsm_hostname=
self.computer_details = {}
self.cloud_accounts = {}
self.ip_lists = {}
+ self.application_types = {}
+ self.rules = {
+ 'intrusion_prevention': {},
+ 'firewall': {},
+ 'web_reputation': {},
+ 'integrity_monitoring': {},
+ 'log_inspection': {},
+ }
# Setup functions
self._debug = debug
@@ -515,7 +528,9 @@ def get_all(self):
- policies
- cloud accounts
- ip lists
+ - application type ids
"""
+ self.get_all_application_types()
self.get_computer_groups()
self.get_computers_with_details()
self.get_policies()
@@ -582,6 +597,11 @@ def get_computers(self):
def get_computers_with_details(self, detail_level='HIGH'):
"""
Get a list of all the Computers managed by Deep Security
+
+ Acceptable values for detail_level are:
+ - HIGH
+ - MEDIUM
+ - LOW
"""
host_filter_type = self.soap_client.factory.create("EnumHostFilterType")
host_details = self.soap_client.factory.create("EnumHostDetailLevel")
@@ -888,6 +908,7 @@ def get_computer_protection_information(self, tenant=None, from_timestamp=None,
'to': to_timestamp,
}
result = self._make_call(call)
+ if not result: tenants = { 'computer_id_key': {} }
data = self._parse_rest_response(result)
# 0--3 hostID_Type elements create an ID for the computers
@@ -940,5 +961,127 @@ def get_computer_protection_information(self, tenant=None, from_timestamp=None,
def get_tenant_overall_usage_information(self, tenant=None, from_timestamp=None, to_timestamp=None):
"""
+ @TODO: implement
+ """
+ pass
+
+ def get_intrusion_prevention_rules(self):
+ """
+ Retrieve all of the intrusion prevention rules
+ """
+ call = self._get_call_structure()
+ call['method'] = 'DPIRuleRetrieveAll'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ }
+ result = self._make_call(call)
+ if result:
+ for obj in result:
+ self.rules['intrusion_prevention'][obj['ID']] = intrusion_prevention_rule.IntrusionPreventionRule(rule_details=obj, manager=self)
+
+ def get_firewall_rules(self):
+ """
+ Retrieve all of the firewall rules
+ """
+ call = self._get_call_structure()
+ call['method'] = 'firewallRuleRetrieveAll'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ }
+ result = self._make_call(call)
+ if result:
+ for obj in result:
+ self.rules['firewall'][obj['ID']] = firewall_rule.FirewallRule(rule_details=obj, manager=self)
+
+ def get_integrity_monitoring_rules(self):
+ """
+ Retrieve all of the integrity monitoring rules
+ """
+ call = self._get_call_structure()
+ call['method'] = 'integrityRuleRetrieveAll'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ }
+ result = self._make_call(call)
+ if result:
+ for obj in result:
+ self.rules['integrity_monitoring'][obj['ID']] = integrity_monitoring_rule.IntegrityMonitoringRule(rule_details=obj, manager=self)
+
+ def get_log_inspection_rules(self):
+ """
+ Retrieve all of the log inspection rules
+ """
+ call = self._get_call_structure()
+ call['method'] = 'logInspectionRuleRetrieveAll'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ }
+ result = self._make_call(call)
+ if result:
+ for obj in result:
+ self.rules['log_inspection'][obj['ID']] = log_inspection_rule.LogInspectionRule(rule_details=obj, manager=self)
+
+ def get_all_application_types(self):
+ """
+ Retrieve all application types from the Deep Security rules database
+ """
+ call = self._get_call_structure()
+ call['method'] = 'applicationTypeRetrieveAll'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ }
+ result = self._make_call(call)
+ if result:
+ for obj in result:
+ self.application_types[obj['ID']] = application_type.ApplicationType(type_details=obj, manager=self)
+
+ def get_all_rules(self):
"""
- pass
\ No newline at end of file
+ Retrieve all of the rules from the Deep Security Manager
+
+ Calls;
+ - get_intrusion_prevention_rules
+ - get_firewall_rules
+ - get_integrity_monitoring_rules
+ - get_log_inspection_rules
+ """
+ self.get_intrusion_prevention_rules()
+ self.get_firewall_rules()
+ self.get_integrity_monitoring_rules()
+ self.get_log_inspection_rules()
+
+ def get_recommended_rules_for_computer(self, computer_id):
+ """
+ Retrieve all of the rules recommend for the specific computer
+ """
+ rule_type_map = {
+ 'APPLICATIONTYPE': 'application_type',
+ 'PAYLOADFILTER': 'intrusion_prevention',
+ 'FIREWALLRULE': 'firewall',
+ 'INTEGRITYRULE': 'integrity_monitoring',
+ 'LOGINSPECTIONRULE': 'log_inspection',
+ }
+ recommendation_rule_types = {
+ 'APPLICATIONTYPE': 1,
+ 'PAYLOADFILTER': 2,
+ 'FIREWALLRULE': 3,
+ 'INTEGRITYRULE': 4,
+ 'LOGINSPECTIONRULE': 5,
+ }
+ for rule_type, rule_type_id in recommendation_rule_types.items():
+ call = self._get_call_structure()
+ call['method'] = 'hostRecommendationRuleIDsRetrieve'
+ call['data'] = {
+ 'sID': self.session_id_soap,
+ 'hostID': computer_id,
+ 'type': rule_type_id,
+ 'onlyunassigned': False,
+ }
+ result = self._make_call(call)
+ if result:
+ recommended_rule_type = rule_type_map[rule_type]
+ if recommended_rule_type == 'application_type':
+ self.computers[computer_id].application_types[obj] = self.application_types[obj] if self.application_types.has_key(obj) else None
+ else:
+ for obj in result:
+ self.computers[computer_id].recommended_rules[recommended_rule_type][obj] = self.rules[recommended_rule_type][obj] if self.rules[recommended_rule_type].has_key(obj) else None
\ No newline at end of file
diff --git a/dsawswaf/deepsecurity/policy.py b/lib/deepsecurity/policy.py
old mode 100755
new mode 100644
similarity index 96%
rename from dsawswaf/deepsecurity/policy.py
rename to lib/deepsecurity/policy.py
index 6db540b..af81ee2
--- a/dsawswaf/deepsecurity/policy.py
+++ b/lib/deepsecurity/policy.py
@@ -33,7 +33,7 @@ def _parse_details(self, policy_details):
'applicationTypeIDs': 'application_types',
'firewallRuleIDs': 'firewall_rules',
'firewallState': 'firewall_state',
- 'integrityRuleIDs': 'integrity_rules',
+ 'integrityRuleIDs': 'integrity_monitoring_rules',
'integrityState': 'integrity_state',
'logInspectionRuleIDs': 'log_inspection_rules',
'logInspectionState': 'log_inspection_state',
diff --git a/dsawswaf/deepsecurity/soap_https_handler.py b/lib/deepsecurity/soap_https_handler.py
old mode 100755
new mode 100644
similarity index 100%
rename from dsawswaf/deepsecurity/soap_https_handler.py
rename to lib/deepsecurity/soap_https_handler.py
diff --git a/lib/iplists.py b/lib/iplists.py
new file mode 100644
index 0000000..d4569f6
--- /dev/null
+++ b/lib/iplists.py
@@ -0,0 +1,284 @@
+# standard libraries
+import argparse
+import os
+import urllib2
+
+# 3rd party libraries
+import boto3
+import boto3.session
+import netaddr
+
+# project libraries
+import core
+import deepsecurity.manager
+
+def run_script(args):
+ # configure the command line args
+ parser = core.get_arg_parser(prog='ds-to-aws-waf.py iplists', add_help=True)
+ parser.add_argument('-l', '--list', action='store_true', required=False, help='List the available Deep Security IP Lists and the AWS WAF IP Sets')
+ # change to i from -d/--ds?
+ parser.add_argument('-i', '--id', action='store', dest="ip_list", required=False, help='Specify an IP List by ID within Deep Security as the source for the AWS WAF IP Set')
+
+ script = Script(args[1:], parser)
+
+ if script.args.list:
+ # List the available Deep Security IP Lists and AWS WAF IP Sets
+ script.connect()
+ script.get_available_aws_sets()
+ script.print_lists()
+
+ elif script.args.ip_list:
+ script.connect()
+ if script.args.dryrun:
+ script._log("***********************************************************************", priority=True)
+ script._log("* DRY RUN ENABLED. NO CHANGES WILL BE MADE", priority=True)
+ script._log("***********************************************************************", priority=True)
+ # get the specified Deep Security IP Lists (already cached)
+ ip_list = script.get_ds_list(script.args.ip_list)
+ # create the IP Set
+ if ip_list:
+ script.create_ip_set(ip_list)
+
+ script.clean_up()
+
+class Script(core.ScriptContext):
+ def __init__(self, args, parser):
+ core.ScriptContext.__init__(self, args, parser)
+ #super(Script, self).__init__(args, parser)
+ self.aws_credentials = None
+ self.dsm = None
+ self.ip_lists = []
+ self.waf = None
+ self.ip_sets = []
+
+ self.aws_credentials = self._get_aws_credentials()
+ self.dsm = None #self._connect_to_deep_security()
+ self.waf = None #self._connect_to_aws_waf()
+ self.ip_lists = None #self._get_available_ds_lists()
+
+ def connect(self):
+ """
+ Connect to Deep Security and AWS WAF
+ """
+ self.dsm = self._connect_to_deep_security()
+ self.waf = self._connect_to_aws_waf()
+ self.ip_lists = self._get_available_ds_lists()
+
+ def _get_available_ds_lists(self):
+ """
+ Query Deep Security for any existing IP Lists
+ """
+ ip_lists = None
+
+ if self.dsm:
+ self.dsm.get_ip_lists()
+ ip_lists = self.dsm.ip_lists
+ self._log("Cached the available IP Lists from Deep Security")
+
+ return ip_lists
+
+ def get_ds_list(self, list_id):
+ """
+ Get the specified IP List from the cached results
+ """
+ ip_list = None
+
+ for key in self.ip_lists.keys():
+ if '{}'.format(list_id.strip()) == str(key):
+ ip_list = self.ip_lists[key]
+ self._log("Found Deep Security IP list [{}]".format(ip_list))
+ break
+
+ return ip_list
+
+ def _expand_cidr(self, cidr):
+ # AWS WAF IP Sets only accept octets of ['8','16','24','32']
+ # find the next largest one to expand the specified CIDR
+ # to the smallest possible set
+ blocks = []
+
+ # yes, you can figure the strata out algorithmically but this
+ # is a little more readable and will definitely help with long
+ # term maintenance which is way more important!
+ strata = { # defines the CIDR octet blocks
+ 32: { 'type': 32, 'times': 1, 'size': 1 },
+ 31: { 'type': 32, 'times': 2, 'size': 1 },
+ 30: { 'type': 32, 'times': 4, 'size': 1 },
+ 29: { 'type': 32, 'times': 8, 'size': 1 },
+ 28: { 'type': 32, 'times': 16, 'size': 1 },
+ 27: { 'type': 32, 'times': 32, 'size': 1 },
+ 26: { 'type': 32, 'times': 64, 'size': 1 },
+ 25: { 'type': 32, 'times': 128, 'size': 1 },
+ 24: { 'type': 24, 'times': 1, 'size': 256 },
+ 23: { 'type': 24, 'times': 2, 'size': 256 },
+ 22: { 'type': 24, 'times': 4, 'size': 256 },
+ 21: { 'type': 24, 'times': 8, 'size': 256 },
+ 20: { 'type': 24, 'times': 16, 'size': 256 },
+ 19: { 'type': 24, 'times': 32, 'size': 256 },
+ 18: { 'type': 24, 'times': 64, 'size': 256 },
+ 17: { 'type': 24, 'times': 128, 'size': 256 },
+ 16: { 'type': 16, 'times': 1, 'size': 65536 },
+ 15: { 'type': 16, 'times': 2, 'size': 65536 },
+ 14: { 'type': 16, 'times': 4, 'size': 65536 },
+ 13: { 'type': 16, 'times': 8, 'size': 65536 },
+ 12: { 'type': 16, 'times': 16, 'size': 65536 },
+ 11: { 'type': 16, 'times': 32, 'size': 65536 },
+ 10: { 'type': 16, 'times': 64, 'size': 65536 },
+ 9: { 'type': 16, 'times': 128, 'size': 65536 },
+ 8: { 'type': 8, 'times': 1, 'size': 16777216 },
+ 7: { 'type': 8, 'times': 2, 'size': 16777216 },
+ 6: { 'type': 8, 'times': 4, 'size': 16777216 },
+ 5: { 'type': 8, 'times': 8, 'size': 16777216 },
+ 4: { 'type': 8, 'times': 16, 'size': 16777216 },
+ 3: { 'type': 8, 'times': 32, 'size': 16777216 },
+ 2: { 'type': 8, 'times': 64, 'size': 16777216 },
+ 1: { 'type': 8, 'times': 128, 'size': 16777216 },
+ }
+
+ current_strata = int(cidr.__str__().split('/')[-1])
+ for i in range(strata[current_strata]['times']):
+ if i == 0:
+ current_cidr = netaddr.IPNetwork('{}/{}'.format(cidr[0], strata[current_strata]['type']))
+ else:
+ index = i * strata[current_strata]['size']
+ current_cidr = netaddr.IPNetwork('{}/{}'.format(cidr[index], strata[current_strata]['type']))
+
+ blocks.append(current_cidr)
+
+ self._log("Expanded CIDR block {} to {} IP Set compatible blocks".format(cidr, len(blocks)))
+
+ return blocks
+
+ def _parse_ds_addresses(self, ds_list):
+ # Accepted DS formats
+ # X.X.X.X/1-32 Example: 192.168.2.0/24
+ # X.X.X.X/Y.Y.Y.Y Example: 192.168.2.0/255.255.255.0
+ # X.X.X.X Example: 192.168.2.33
+ # IpV6 Mask Example: 2001:0DB8::CD30:0:0:0:0/60
+ # X.X.X.X-Y.Y.Y.Y Example: 192.168.0.2 - 192.168.0.125
+ # IPv6-IPv6 Example: FF01::101 - FF01::102
+ # IP or Range #Comment Example: 255.255.255.255 #Broadcast IP
+ addresses = []
+ for address in ds_list.addresses:
+ if "#" in address: address = address.split('#').strip() # remove any comments
+ if '-' in address:
+ try:
+ a1, a2 = address.split('-')
+ # what's the range between a1 and a2?
+ rng = netaddr.IPRange(a1, a2)
+ for addr in rng: addresses.append(netaddr.IPNetwork(addr))
+ except Exception, err: pass
+ else:
+ net = netaddr.IPNetwork(address)
+ if '{}'.format(net.cidr).split('/')[-1] in ['8','16','24','32']:
+ addresses.append(netaddr.IPNetwork(net))
+ else:
+ for addr in net: addresses.append(netaddr.IPNetwork(addr))
+
+ total_set = netaddr.IPSet(addresses)
+ total_set.compact()
+
+ cidrs = total_set.iter_cidrs()
+ waf_compatible = []
+ for cidr in cidrs:
+ if '{}'.format(cidr).split('/')[-1] in ['8','16','24','32']:
+ waf_compatible.append(cidr)
+ else:
+ waf_compatible += self._expand_cidr(cidr)
+
+ self._log("Converted {} IP List entries to {} IP Set entries".format(len(ds_list.addresses), len(waf_compatible)), priority=True)
+ return waf_compatible
+
+ def _convert_ds_addresses_to_waf(self, ds_list, convert='ignore'):
+ """
+ convert = One of ['ignore', 'expand', 'upscale']
+ """
+ # AWS WAF IP Sets only accept CIDR notation with a closing octet of /8 /16 /24 or /32
+ # We need to ensure that the specified Deep Security IP List will convert to an IP Set
+ addresses = self._parse_ds_addresses(ds_list)
+
+ updates = []
+ for cidr_network in addresses:
+ updates.append(
+ {
+ 'Action': 'INSERT',
+ 'IPSetDescriptor': {
+ 'Type': 'IPV{}'.format(cidr_network.version),
+ 'Value': cidr_network.cidr.__str__(),
+ }
+ }
+ )
+
+ return updates
+
+ def create_ip_set(self, ds_list):
+ """
+ Create an AWS WAF IP Set based on the specified Deep Security IP List
+ """
+ self._log("Attempt to create a matching IP Set for Deep Security list #{}".format(ds_list))
+ if self.waf:
+ # is there an existing IP Set?
+ current_ip_set = None
+ if not self.ip_sets: self.get_available_aws_sets()
+ for ip_set in self.ip_sets:
+ if ip_set.has_key('Name') and ip_set['Name'] == ds_list.name:
+ current_ip_set = ip_set['IPSetId']
+ break
+
+ #response = self.waf.get_change_token()
+ change_token = self._get_aws_waf_change_token()
+
+ if change_token:
+ list_created = False
+ if not current_ip_set:
+ if not self.args.dryrun:
+ self._log("Requesting the creation of [{}]".format(ds_list.name))
+ response = self.waf.create_ip_set(Name=ds_list.name, ChangeToken=change_token)
+ if response:
+ self.ip_sets.append(response['IPSet'])
+ list_created = True
+ current_ip_set = response['IPSet']['IPSetId']
+ change_token = self._get_aws_waf_change_token()
+ else:
+ self._log("Will request the creation of [{}]".format(ds_list.name), priority=True)
+
+ if current_ip_set and change_token:
+ updates = self._convert_ds_addresses_to_waf(ds_list)
+ if len(updates) > 1000:
+ self._log("Requested IP List converts to more then the 1000 entry limit for an IP Set. Please split the IP List within Deep Security before syncing with AWS WAF", priority=True)
+ else:
+ if not self.args.dryrun:
+ response = self.waf.update_ip_set(IPSetId=current_ip_set, ChangeToken=change_token, Updates=updates)
+ if response and response.has_key('ChangeToken'):
+ self._log('Change [{}] requested'.format(response['ChangeToken']))
+ else:
+ self._log("Will request the addition of {} entries in IP Set {}".format(len(updates), current_ip_set), priority=True)
+
+ if not self.args.dryrun:
+ msg_verb = "Created" if list_created else "Updated"
+ self._log("{} IP Set [{}] with ID [{}]".format(msg_verb, ds_list.name, current_ip_set), priority=True)
+ else:
+ msg_verb = "create" if list_created else "update"
+ self._log("Will {} IP Set [{}] with ID [{}]".format(msg_verb, ds_list.name, current_ip_set), priority=True)
+
+ def print_lists(self):
+ """
+ Pretty print the IP lists from Deep Security and
+ the AWS WAF IP Sets
+ """
+ print "\nAvailable Deep Security IP Lists"
+ print "================================"
+ if self.ip_lists and len(self.ip_lists) > 0:
+ for ds_list_id, ds_list in self.ip_lists.items():
+ print "{}\t{}".format(ds_list_id, ds_list.name)
+ else:
+ print "---\t No IP lists available"
+
+
+ print "\nAvailable AWS WAF IP Sets"
+ print "========================="
+ if self.ip_sets and len(self.ip_sets) > 0:
+ for waf_set in self.ip_sets:
+ print "{}\t{}".format(waf_set['IPSetId'], waf_set['Name'])
+ else:
+ print "---\t No AWS WAF WACLs available"
diff --git a/dsawswaf/requests/__init__.py b/lib/requests/__init__.py
similarity index 100%
rename from dsawswaf/requests/__init__.py
rename to lib/requests/__init__.py
diff --git a/dsawswaf/requests/adapters.py b/lib/requests/adapters.py
similarity index 100%
rename from dsawswaf/requests/adapters.py
rename to lib/requests/adapters.py
diff --git a/dsawswaf/requests/api.py b/lib/requests/api.py
similarity index 100%
rename from dsawswaf/requests/api.py
rename to lib/requests/api.py
diff --git a/dsawswaf/requests/auth.py b/lib/requests/auth.py
similarity index 100%
rename from dsawswaf/requests/auth.py
rename to lib/requests/auth.py
diff --git a/dsawswaf/requests/cacert.pem b/lib/requests/cacert.pem
similarity index 100%
rename from dsawswaf/requests/cacert.pem
rename to lib/requests/cacert.pem
diff --git a/dsawswaf/requests/certs.py b/lib/requests/certs.py
similarity index 100%
rename from dsawswaf/requests/certs.py
rename to lib/requests/certs.py
diff --git a/dsawswaf/requests/compat.py b/lib/requests/compat.py
similarity index 100%
rename from dsawswaf/requests/compat.py
rename to lib/requests/compat.py
diff --git a/dsawswaf/requests/cookies.py b/lib/requests/cookies.py
similarity index 100%
rename from dsawswaf/requests/cookies.py
rename to lib/requests/cookies.py
diff --git a/dsawswaf/requests/exceptions.py b/lib/requests/exceptions.py
similarity index 100%
rename from dsawswaf/requests/exceptions.py
rename to lib/requests/exceptions.py
diff --git a/dsawswaf/requests/hooks.py b/lib/requests/hooks.py
similarity index 100%
rename from dsawswaf/requests/hooks.py
rename to lib/requests/hooks.py
diff --git a/dsawswaf/requests/models.py b/lib/requests/models.py
similarity index 100%
rename from dsawswaf/requests/models.py
rename to lib/requests/models.py
diff --git a/dsawswaf/requests/packages/__init__.py b/lib/requests/packages/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/__init__.py
rename to lib/requests/packages/__init__.py
diff --git a/dsawswaf/requests/packages/chardet/__init__.py b/lib/requests/packages/chardet/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/__init__.py
rename to lib/requests/packages/chardet/__init__.py
diff --git a/dsawswaf/requests/packages/chardet/big5freq.py b/lib/requests/packages/chardet/big5freq.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/big5freq.py
rename to lib/requests/packages/chardet/big5freq.py
diff --git a/dsawswaf/requests/packages/chardet/big5prober.py b/lib/requests/packages/chardet/big5prober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/big5prober.py
rename to lib/requests/packages/chardet/big5prober.py
diff --git a/dsawswaf/requests/packages/chardet/chardetect.py b/lib/requests/packages/chardet/chardetect.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/chardetect.py
rename to lib/requests/packages/chardet/chardetect.py
diff --git a/dsawswaf/requests/packages/chardet/chardistribution.py b/lib/requests/packages/chardet/chardistribution.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/chardistribution.py
rename to lib/requests/packages/chardet/chardistribution.py
diff --git a/dsawswaf/requests/packages/chardet/charsetgroupprober.py b/lib/requests/packages/chardet/charsetgroupprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/charsetgroupprober.py
rename to lib/requests/packages/chardet/charsetgroupprober.py
diff --git a/dsawswaf/requests/packages/chardet/charsetprober.py b/lib/requests/packages/chardet/charsetprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/charsetprober.py
rename to lib/requests/packages/chardet/charsetprober.py
diff --git a/dsawswaf/requests/packages/chardet/codingstatemachine.py b/lib/requests/packages/chardet/codingstatemachine.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/codingstatemachine.py
rename to lib/requests/packages/chardet/codingstatemachine.py
diff --git a/dsawswaf/requests/packages/chardet/compat.py b/lib/requests/packages/chardet/compat.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/compat.py
rename to lib/requests/packages/chardet/compat.py
diff --git a/dsawswaf/requests/packages/chardet/constants.py b/lib/requests/packages/chardet/constants.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/constants.py
rename to lib/requests/packages/chardet/constants.py
diff --git a/dsawswaf/requests/packages/chardet/cp949prober.py b/lib/requests/packages/chardet/cp949prober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/cp949prober.py
rename to lib/requests/packages/chardet/cp949prober.py
diff --git a/dsawswaf/requests/packages/chardet/escprober.py b/lib/requests/packages/chardet/escprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/escprober.py
rename to lib/requests/packages/chardet/escprober.py
diff --git a/dsawswaf/requests/packages/chardet/escsm.py b/lib/requests/packages/chardet/escsm.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/escsm.py
rename to lib/requests/packages/chardet/escsm.py
diff --git a/dsawswaf/requests/packages/chardet/eucjpprober.py b/lib/requests/packages/chardet/eucjpprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/eucjpprober.py
rename to lib/requests/packages/chardet/eucjpprober.py
diff --git a/dsawswaf/requests/packages/chardet/euckrfreq.py b/lib/requests/packages/chardet/euckrfreq.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/euckrfreq.py
rename to lib/requests/packages/chardet/euckrfreq.py
diff --git a/dsawswaf/requests/packages/chardet/euckrprober.py b/lib/requests/packages/chardet/euckrprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/euckrprober.py
rename to lib/requests/packages/chardet/euckrprober.py
diff --git a/dsawswaf/requests/packages/chardet/euctwfreq.py b/lib/requests/packages/chardet/euctwfreq.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/euctwfreq.py
rename to lib/requests/packages/chardet/euctwfreq.py
diff --git a/dsawswaf/requests/packages/chardet/euctwprober.py b/lib/requests/packages/chardet/euctwprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/euctwprober.py
rename to lib/requests/packages/chardet/euctwprober.py
diff --git a/dsawswaf/requests/packages/chardet/gb2312freq.py b/lib/requests/packages/chardet/gb2312freq.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/gb2312freq.py
rename to lib/requests/packages/chardet/gb2312freq.py
diff --git a/dsawswaf/requests/packages/chardet/gb2312prober.py b/lib/requests/packages/chardet/gb2312prober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/gb2312prober.py
rename to lib/requests/packages/chardet/gb2312prober.py
diff --git a/dsawswaf/requests/packages/chardet/hebrewprober.py b/lib/requests/packages/chardet/hebrewprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/hebrewprober.py
rename to lib/requests/packages/chardet/hebrewprober.py
diff --git a/dsawswaf/requests/packages/chardet/jisfreq.py b/lib/requests/packages/chardet/jisfreq.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/jisfreq.py
rename to lib/requests/packages/chardet/jisfreq.py
diff --git a/dsawswaf/requests/packages/chardet/jpcntx.py b/lib/requests/packages/chardet/jpcntx.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/jpcntx.py
rename to lib/requests/packages/chardet/jpcntx.py
diff --git a/dsawswaf/requests/packages/chardet/langbulgarianmodel.py b/lib/requests/packages/chardet/langbulgarianmodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langbulgarianmodel.py
rename to lib/requests/packages/chardet/langbulgarianmodel.py
diff --git a/dsawswaf/requests/packages/chardet/langcyrillicmodel.py b/lib/requests/packages/chardet/langcyrillicmodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langcyrillicmodel.py
rename to lib/requests/packages/chardet/langcyrillicmodel.py
diff --git a/dsawswaf/requests/packages/chardet/langgreekmodel.py b/lib/requests/packages/chardet/langgreekmodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langgreekmodel.py
rename to lib/requests/packages/chardet/langgreekmodel.py
diff --git a/dsawswaf/requests/packages/chardet/langhebrewmodel.py b/lib/requests/packages/chardet/langhebrewmodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langhebrewmodel.py
rename to lib/requests/packages/chardet/langhebrewmodel.py
diff --git a/dsawswaf/requests/packages/chardet/langhungarianmodel.py b/lib/requests/packages/chardet/langhungarianmodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langhungarianmodel.py
rename to lib/requests/packages/chardet/langhungarianmodel.py
diff --git a/dsawswaf/requests/packages/chardet/langthaimodel.py b/lib/requests/packages/chardet/langthaimodel.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/langthaimodel.py
rename to lib/requests/packages/chardet/langthaimodel.py
diff --git a/dsawswaf/requests/packages/chardet/latin1prober.py b/lib/requests/packages/chardet/latin1prober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/latin1prober.py
rename to lib/requests/packages/chardet/latin1prober.py
diff --git a/dsawswaf/requests/packages/chardet/mbcharsetprober.py b/lib/requests/packages/chardet/mbcharsetprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/mbcharsetprober.py
rename to lib/requests/packages/chardet/mbcharsetprober.py
diff --git a/dsawswaf/requests/packages/chardet/mbcsgroupprober.py b/lib/requests/packages/chardet/mbcsgroupprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/mbcsgroupprober.py
rename to lib/requests/packages/chardet/mbcsgroupprober.py
diff --git a/dsawswaf/requests/packages/chardet/mbcssm.py b/lib/requests/packages/chardet/mbcssm.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/mbcssm.py
rename to lib/requests/packages/chardet/mbcssm.py
diff --git a/dsawswaf/requests/packages/chardet/sbcharsetprober.py b/lib/requests/packages/chardet/sbcharsetprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/sbcharsetprober.py
rename to lib/requests/packages/chardet/sbcharsetprober.py
diff --git a/dsawswaf/requests/packages/chardet/sbcsgroupprober.py b/lib/requests/packages/chardet/sbcsgroupprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/sbcsgroupprober.py
rename to lib/requests/packages/chardet/sbcsgroupprober.py
diff --git a/dsawswaf/requests/packages/chardet/sjisprober.py b/lib/requests/packages/chardet/sjisprober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/sjisprober.py
rename to lib/requests/packages/chardet/sjisprober.py
diff --git a/dsawswaf/requests/packages/chardet/universaldetector.py b/lib/requests/packages/chardet/universaldetector.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/universaldetector.py
rename to lib/requests/packages/chardet/universaldetector.py
diff --git a/dsawswaf/requests/packages/chardet/utf8prober.py b/lib/requests/packages/chardet/utf8prober.py
similarity index 100%
rename from dsawswaf/requests/packages/chardet/utf8prober.py
rename to lib/requests/packages/chardet/utf8prober.py
diff --git a/dsawswaf/requests/packages/urllib3/__init__.py b/lib/requests/packages/urllib3/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/__init__.py
rename to lib/requests/packages/urllib3/__init__.py
diff --git a/dsawswaf/requests/packages/urllib3/_collections.py b/lib/requests/packages/urllib3/_collections.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/_collections.py
rename to lib/requests/packages/urllib3/_collections.py
diff --git a/dsawswaf/requests/packages/urllib3/connection.py b/lib/requests/packages/urllib3/connection.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/connection.py
rename to lib/requests/packages/urllib3/connection.py
diff --git a/dsawswaf/requests/packages/urllib3/connectionpool.py b/lib/requests/packages/urllib3/connectionpool.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/connectionpool.py
rename to lib/requests/packages/urllib3/connectionpool.py
diff --git a/lib/requests/packages/urllib3/contrib/__init__.py b/lib/requests/packages/urllib3/contrib/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/dsawswaf/requests/packages/urllib3/contrib/appengine.py b/lib/requests/packages/urllib3/contrib/appengine.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/contrib/appengine.py
rename to lib/requests/packages/urllib3/contrib/appengine.py
diff --git a/dsawswaf/requests/packages/urllib3/contrib/ntlmpool.py b/lib/requests/packages/urllib3/contrib/ntlmpool.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/contrib/ntlmpool.py
rename to lib/requests/packages/urllib3/contrib/ntlmpool.py
diff --git a/dsawswaf/requests/packages/urllib3/contrib/pyopenssl.py b/lib/requests/packages/urllib3/contrib/pyopenssl.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/contrib/pyopenssl.py
rename to lib/requests/packages/urllib3/contrib/pyopenssl.py
diff --git a/dsawswaf/requests/packages/urllib3/exceptions.py b/lib/requests/packages/urllib3/exceptions.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/exceptions.py
rename to lib/requests/packages/urllib3/exceptions.py
diff --git a/dsawswaf/requests/packages/urllib3/fields.py b/lib/requests/packages/urllib3/fields.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/fields.py
rename to lib/requests/packages/urllib3/fields.py
diff --git a/dsawswaf/requests/packages/urllib3/filepost.py b/lib/requests/packages/urllib3/filepost.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/filepost.py
rename to lib/requests/packages/urllib3/filepost.py
diff --git a/dsawswaf/requests/packages/urllib3/packages/__init__.py b/lib/requests/packages/urllib3/packages/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/packages/__init__.py
rename to lib/requests/packages/urllib3/packages/__init__.py
diff --git a/dsawswaf/requests/packages/urllib3/packages/ordered_dict.py b/lib/requests/packages/urllib3/packages/ordered_dict.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/packages/ordered_dict.py
rename to lib/requests/packages/urllib3/packages/ordered_dict.py
diff --git a/dsawswaf/requests/packages/urllib3/packages/six.py b/lib/requests/packages/urllib3/packages/six.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/packages/six.py
rename to lib/requests/packages/urllib3/packages/six.py
diff --git a/dsawswaf/requests/packages/urllib3/packages/ssl_match_hostname/__init__.py b/lib/requests/packages/urllib3/packages/ssl_match_hostname/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/packages/ssl_match_hostname/__init__.py
rename to lib/requests/packages/urllib3/packages/ssl_match_hostname/__init__.py
diff --git a/dsawswaf/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py b/lib/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py
rename to lib/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py
diff --git a/dsawswaf/requests/packages/urllib3/poolmanager.py b/lib/requests/packages/urllib3/poolmanager.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/poolmanager.py
rename to lib/requests/packages/urllib3/poolmanager.py
diff --git a/dsawswaf/requests/packages/urllib3/request.py b/lib/requests/packages/urllib3/request.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/request.py
rename to lib/requests/packages/urllib3/request.py
diff --git a/dsawswaf/requests/packages/urllib3/response.py b/lib/requests/packages/urllib3/response.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/response.py
rename to lib/requests/packages/urllib3/response.py
diff --git a/dsawswaf/requests/packages/urllib3/util/__init__.py b/lib/requests/packages/urllib3/util/__init__.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/__init__.py
rename to lib/requests/packages/urllib3/util/__init__.py
diff --git a/dsawswaf/requests/packages/urllib3/util/connection.py b/lib/requests/packages/urllib3/util/connection.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/connection.py
rename to lib/requests/packages/urllib3/util/connection.py
diff --git a/dsawswaf/requests/packages/urllib3/util/request.py b/lib/requests/packages/urllib3/util/request.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/request.py
rename to lib/requests/packages/urllib3/util/request.py
diff --git a/dsawswaf/requests/packages/urllib3/util/response.py b/lib/requests/packages/urllib3/util/response.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/response.py
rename to lib/requests/packages/urllib3/util/response.py
diff --git a/dsawswaf/requests/packages/urllib3/util/retry.py b/lib/requests/packages/urllib3/util/retry.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/retry.py
rename to lib/requests/packages/urllib3/util/retry.py
diff --git a/dsawswaf/requests/packages/urllib3/util/ssl_.py b/lib/requests/packages/urllib3/util/ssl_.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/ssl_.py
rename to lib/requests/packages/urllib3/util/ssl_.py
diff --git a/dsawswaf/requests/packages/urllib3/util/timeout.py b/lib/requests/packages/urllib3/util/timeout.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/timeout.py
rename to lib/requests/packages/urllib3/util/timeout.py
diff --git a/dsawswaf/requests/packages/urllib3/util/url.py b/lib/requests/packages/urllib3/util/url.py
similarity index 100%
rename from dsawswaf/requests/packages/urllib3/util/url.py
rename to lib/requests/packages/urllib3/util/url.py
diff --git a/dsawswaf/requests/sessions.py b/lib/requests/sessions.py
similarity index 100%
rename from dsawswaf/requests/sessions.py
rename to lib/requests/sessions.py
diff --git a/dsawswaf/requests/status_codes.py b/lib/requests/status_codes.py
similarity index 100%
rename from dsawswaf/requests/status_codes.py
rename to lib/requests/status_codes.py
diff --git a/dsawswaf/requests/structures.py b/lib/requests/structures.py
similarity index 100%
rename from dsawswaf/requests/structures.py
rename to lib/requests/structures.py
diff --git a/dsawswaf/requests/utils.py b/lib/requests/utils.py
similarity index 100%
rename from dsawswaf/requests/utils.py
rename to lib/requests/utils.py
diff --git a/lib/sqli.patterns b/lib/sqli.patterns
new file mode 100644
index 0000000..eb51a3d
--- /dev/null
+++ b/lib/sqli.patterns
@@ -0,0 +1,3 @@
+database
+web
+http
\ No newline at end of file
diff --git a/lib/sqli.py b/lib/sqli.py
new file mode 100644
index 0000000..7c4339f
--- /dev/null
+++ b/lib/sqli.py
@@ -0,0 +1,307 @@
+# Standard libraries
+import argparse
+import inspect
+import os
+import re
+import urllib2
+
+# 3rd party libraries
+import boto3
+import boto3.session
+import netaddr
+
+# Project libraries
+import core
+import deepsecurity.manager
+
+def run_script(args):
+ # configure the command line args
+ parser = core.get_arg_parser(prog='ds-to-aws-waf.py sqli', add_help=True)
+ parser.add_argument('-l', '--list', action='store_true', required=False, help='List the available EC2 instances')
+ parser.add_argument('--tag', action=core.StoreNameValuePairOnEquals, nargs="+", dest="tags", required=False, help='Specify the tags to filter the EC2 instances by')
+
+ parser.add_argument('--create-match', action='store_true', required=False, dest="create_match", help='Create the SQLi match condition for use in various rules')
+
+ script = Script(args[1:], parser)
+
+ if script.args.list:
+ # List the available EC2 instances and cross reference with Deep Security
+ script.connect()
+ script.get_ec2_instances()
+ script.get_deep_security_info()
+ recommendations = script.compare_ec2_to_deep_security()
+ script.print_recommendations(recommendations)
+
+ if script.args.create_match:
+ script.connect()
+ if script.args.dryrun:
+ script._log("***********************************************************************", priority=True)
+ script._log("* DRY RUN ENABLED. NO CHANGES WILL BE MADE", priority=True)
+ script._log("***********************************************************************", priority=True)
+
+ # create the recommend SQLi match condition
+ script.create_match_condition()
+
+ script.clean_up()
+
+class Script(core.ScriptContext):
+ def __init__(self, args, parser):
+ core.ScriptContext.__init__(self, args, parser)
+ #super(Script, self).__init__(args, parser)
+ self.aws_credentials = None
+ self.dsm = None
+ self.ip_lists = []
+ self.waf = None
+ self.ec2 = None
+ self.instances = {}
+ self.tbuids = []
+ self.patterns = []
+
+ self.aws_credentials = self._get_aws_credentials()
+ self.dsm = None
+ self.waf = None
+
+ self.cache_patterns()
+
+ def connect(self):
+ """
+ Connect to Deep Security and AWS WAF
+ """
+ self.dsm = self._connect_to_deep_security()
+ self.waf = self._connect_to_aws_waf()
+ self.ec2 = self._connect_to_aws_ec2()
+
+ def cache_patterns(self):
+ """
+ Cache the patterns for matching Deep Security rules for SQLi
+ recommendations
+ """
+ CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(inspect.stack()[0][1]))
+ TBUIDS = os.path.join(CURRENT_FILE_DIR, 'sqli.tbuids')
+ PATTERNS = os.path.join(CURRENT_FILE_DIR, 'sqli.patterns')
+
+ if os.path.exists(TBUIDS):
+ self._log("Caching TBUIDS for rule matching")
+ with open(TBUIDS, 'r') as fh:
+ for line in fh: self.tbuids.append(line.strip())
+
+ if os.path.exists(PATTERNS):
+ self._log("Caching patterns for rule matching")
+ with open(PATTERNS, 'r') as fh:
+ for line in fh: self.patterns.append(line.strip())
+
+ def get_ec2_instances(self):
+ """
+ Get a list of EC2 instances from AWS
+ """
+ if self.ec2:
+ # build any filters first
+ filters = None
+ if self.args.tags:
+ filters = []
+ for k, v in self.args.tags.items():
+ filters.append({'Name':'tag:{}'.format(k), 'Values':['{}'.format(v)]})
+
+ self._log("Applying {} filters to the request for EC2 instances".format(len(filters)))
+
+ if filters:
+ response = self.ec2.describe_instances(Filters=filters)
+ else:
+ response = self.ec2.describe_instances()
+
+ if response and response.has_key('Reservations'):
+ for reservation in response['Reservations']:
+ if reservation.has_key('Instances'):
+ for instance in reservation['Instances']:
+ self.instances[instance['InstanceId']] = instance
+
+ def get_deep_security_info(self):
+ """
+ Get all of the relevant information from Deep Security in order
+ to build a smart rule set for AWS WAF
+ """
+ self._log("Requesting information from Deep Security about your deployment", priority=True)
+ if self.dsm:
+ self.dsm.get_all()
+ self._log("Requesting rules from the Deep Security manager. This will take a few seconds...")
+ self.dsm.get_all_rules()
+ self._log("Requesting computers from the Deep Security manager. This will take a few seconds...")
+ self.dsm.get_computers_with_details()
+ self._log("Requested information from the Deep Security manager cached locally")
+
+ def compare_ec2_to_deep_security(self):
+ """
+ Compare the list of EC2 instance returned from AWS vs
+ the list of known instances in Deep Security
+ """
+ ds_instance_map = {}
+ recommendations = {}
+ if self.dsm and self.dsm.computers and self.instances:
+ for computer_id, computer_details in self.dsm.computers.items():
+ ds_instance_map[computer_details.cloud_object_instance_id] = computer_id
+
+ for instance_id, instance_details in self.instances.items():
+ if ds_instance_map.has_key(instance_id):
+ self._log("Deep Security has instance {} in inventory".format(instance_id))
+ recommendations[instance_id] = self.analyze_computer(ds_instance_map[instance_id])
+ else:
+ self._log("Deep Security does not have instance {} in inventory".format(instance_id))
+ recommendations[instance_id] = None
+
+ return recommendations
+
+ def does_rule_match_sqli(self, rule):
+ """
+ Determine if a rule matches the defined parameters for an
+ SQLi recommendation
+ """
+ sqli_recommended = False
+ if 'tbuid' in dir(rule):
+ sqli_recommended = True
+
+ if 'application_type_id' in dir(rule):
+ if self.dsm.application_types.has_key(rule.application_type_id):
+ if self.dsm.application_types[rule.application_type_id].tbuid in self.tbuids:
+ sqli_recommended = True
+
+ for pattern in self.patterns:
+ if 'name' in dir(rule) and 'description' in dir(rule):
+ for attr in [rule.name, rule.description]:
+ try:
+ m = re.search(pattern, attr)
+ if m:
+ sqli_recommended = True
+ except Exception, err: pass # @TODO handle this gracefully
+
+ return sqli_recommended
+
+ def analyze_computer(self, ds_computer_id):
+ """
+ Analyze the specified computer to determine if it should be
+ protected by SQLi rules
+ """
+ self._log("Analyzing computer {}:{}".format(ds_computer_id, self.dsm.computers[ds_computer_id].hostname))
+ recommendation = False
+ self.dsm.get_recommended_rules_for_computer(ds_computer_id)
+ computer = self.dsm.computers[ds_computer_id]
+ sqli_recommendations = []
+
+ # check at the policy level
+ if computer.policy_id:
+ self._log("Computer is protected by Deep Security. Checking rules")
+ for rule_type in [
+ 'integrity_monitoring_rules',
+ 'log_inspection_rules',
+ 'intrusion_prevention_rules'
+ ]:
+ if self.dsm.policies.has_key(computer.policy_id):
+ rule_set = getattr(self.dsm.policies[computer.policy_id], rule_type)
+ if rule_set: # policy has these type of rules applied
+ for rule_id in getattr(self.dsm.policies[computer.policy_id], rule_type)[-1]:
+ rule = self.dsm.rules[rule_type.replace('_rules', '')][rule_id]
+ if self.does_rule_match_sqli(rule): sqli_recommendations.append(rule)
+ else:
+ self._log("Instance {} has no rules of type {} applied".format(computer.cloud_object_instance_id, rule_type))
+ else:
+ self._log("Policy {} is not available for analysis".format(computer.policy_id))
+ else:
+ self._log("Deep Security is aware of the instance but is not protecting it with a policy")
+ recommendation = None
+
+ # now check for any recommendations to the computer
+ for rule_type, rules in computer.recommended_rules.items():
+ self._log("Checking for recommended {} rules".format(rule_type))
+ for rule_id, rule in rules.items():
+ if self.does_rule_match_sqli(rule): sqli_recommendations.append(rule)
+
+ for application_type_id, application_type in computer.application_types.items():
+ if application_type.tbuid in self.tbuids:
+ sqli_recommendations.append(application_type)
+
+ if len(sqli_recommendations) > 1:
+ recommendation = True if len(sqli_recommendations) > 0 else False
+ self._log("Found {} rules indicating this instance should be protected by an SQLi rule set".format(len(sqli_recommendations)))
+
+ return recommendation
+
+ def print_recommendations(self, recommendations):
+ """
+ Print the recommendations for each instance
+ """
+ self._log("************************************************************************", priority=True)
+ self._log("Completed recommendation phase", priority=True)
+ self._log(" Instance\tRecommendation", priority=True)
+ for instance_id, recommendation in recommendations.items():
+ print " {}\t{}".format(instance_id, recommendation, priority=True)
+
+ self._log("************************************************************************", priority=True)
+
+ def create_match_condition(self):
+ """
+ Create the recommend SQLi match condition
+
+ Reference for SQLi match sets is available at http://docs.aws.amazon.com/waf/latest/developerguide/web-acl-sql-conditions.html
+ """
+ MATCH_SET_NAME = "Deep Security SQLi Guidance"
+
+ # does the match set already exist?
+ exists = False
+ response = self.waf.list_sql_injection_match_sets(Limit=100)
+ if response and response.has_key('SqlInjectionMatchSets'):
+ for match_set in response['SqlInjectionMatchSets']:
+ if match_set['Name'] == MATCH_SET_NAME:
+ exists = True
+ break
+
+ if exists:
+ self._log("Desired SQLi match set already exists. No action needed")
+ else:
+ self._log("Attempting to create a new SQLi match set; {}".format(MATCH_SET_NAME))
+ sqli_match_set_updates = [
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'URI', 'Data': 'string' }, 'TextTransformation': 'URL_DECODE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'QUERY_STRING', 'Data': 'string' }, 'TextTransformation': 'URL_DECODE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'QUERY_STRING', 'Data': 'string' }, 'TextTransformation': 'HTML_ENTITY_DECODE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'QUERY_STRING', 'Data': 'string' }, 'TextTransformation': 'LOWERCASE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'BODY', 'Data': 'string' }, 'TextTransformation': 'URL_DECODE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'BODY', 'Data': 'string' }, 'TextTransformation': 'HTML_ENTITY_DECODE' }},
+ { 'Action': 'INSERT', 'SqlInjectionMatchTuple': { 'FieldToMatch': { 'Type': 'BODY', 'Data': 'string' }, 'TextTransformation': 'LOWERCASE' }},
+ ]
+ if not self.args.dryrun:
+ # get a change token
+ change_token = self._get_aws_waf_change_token()
+ if change_token:
+ # create the match set
+ match_set_id = None
+ try:
+ response = self.waf.create_sql_injection_match_set(
+ Name=MATCH_SET_NAME,
+ ChangeToken=change_token
+ )
+ self._log("Created a new SQLi match set: {}".format(MATCH_SET_NAME))
+
+ if response and response.has_key('SqlInjectionMatchSet') and response['SqlInjectionMatchSet'].has_key('SqlInjectionMatchSetId'):
+ match_set_id = response['SqlInjectionMatchSet']['SqlInjectionMatchSetId']
+
+ except Exception, err:
+ self._log("Could not create a new SQLi match set", err=err)
+ return
+
+ if match_set_id:
+ # get another change token
+ change_token = self._get_aws_waf_change_token()
+ if change_token:
+ # update the match set
+ try:
+ response = self.waf.update_sql_injection_match_set(
+ SqlInjectionMatchSetId=match_set_id,
+ ChangeToken=change_token,
+ Updates=sqli_match_set_updates
+ )
+ self._log("Updated SQLi match set; {}".format(MATCH_SET_NAME), priority=True)
+ except Exception, err:
+ self._log("Unable to update SQLi match set", err=err)
+ else:
+ self._log("Would request an AWS WAF change token to create a new SQLi match set", priority=True)
+ self._log(" SQLi match set will contain;", priority=True)
+ for update in sqli_match_set_updates:
+ self._log(" {}".format(update), priority=True)
\ No newline at end of file
diff --git a/lib/sqli.tbuids b/lib/sqli.tbuids
new file mode 100644
index 0000000..c8816e9
--- /dev/null
+++ b/lib/sqli.tbuids
@@ -0,0 +1,187 @@
+56CED14E-AB02-95F0-15C7-69DE24E5F03F
+E1007380-3083-2F78-D5FB-76E4EB5AE47E
+F93C37D9-23FF-77DB-E987-A6FA6FA79F9D
+116D77F5-4361-3FC8-3288-A4B14AD5C7F9
+9E6F6290-B4F3-7BD5-7D11-E65529C4304E
+E0B8CC22-1576-FA15-6EE7-2261EA15849F
+4BC4534D-B40A-C7D1-89D6-DBAA1028187C
+16E4DB9A-8921-CFF2-6B76-0CEF87B32655
+88E1CBC0-97B4-FE30-855C-0EF2EDF598D1
+66429D87-9C3B-665B-6EA7-2A35C75F3105
+BA09982D-DB52-B533-04D5-2CF1AEAD24D5
+3337C255-59FE-BE20-0CC4-48213915A44C
+A8ED3728-886D-B461-2EAC-E94A13124B2A
+23372E7A-2F4A-8D5E-2B51-E1E1D412C105
+A28DEC58-7E45-EEDF-2974-12E9264928B7
+C9AB8614-72C0-E19A-6F9E-DFC3039A3DED
+EB92100A-A339-7AFF-1500-941213B15B70
+57B1CBB3-F1FC-F6D9-3CC0-64AB776E6FF1
+42903DCE-50A9-A031-B230-47750FFDA1D3
+2974A79B-8E39-C3BA-E957-F7DC7E60FF27
+D585A590-F2FA-375A-89F2-CD2FC36FA367
+80679FB1-990C-AD00-69F0-F6932811FBE5
+AD172E45-7447-262A-3511-36ADFF536942
+F5306285-CAFA-82E6-F331-E953302F2C11
+2FDEB90F-F2FA-9951-5BCF-2BA84515D1CF
+A7E6FA2C-BE62-A261-7D4E-04B47C33C64F
+0861D301-627D-0101-A138-58260C8D7AD6
+A13F7C6D-3D91-D529-0E1C-797503B6CFC1
+AF3B6BD8-CA55-3A68-BA2D-37788D6F9E7C
+29C0D6F8-4443-389C-58C2-5E67710D472E
+DC57A8F4-1B9F-1DC6-7036-BE48C56EF342
+8CA1DF55-468C-B51B-7C2E-14043A1433D3
+9E70E86F-1A18-A849-5754-773511FE9A88
+6B28F971-1B6C-89D4-7532-B67866F3D1D6
+228B7B7D-4D3D-EA3A-7E07-18D676A47F41
+C8E61BDF-ED02-B633-D79D-60B371D5A715
+A432B1CE-3699-2545-9F2D-9B9D4D7FCDB4
+8E5275C6-A17F-9B59-D0C1-29952C6421DF
+3B76582A-B957-B7A7-E213-E44712E63E86
+82AEA07F-3BEA-7997-4486-4B52421B36AC
+708E888A-131F-F5CC-8148-05BF4E1D2FA5
+9A2DA167-C977-BB79-C6DE-2DFAD1B89E7A
+3FCAF716-DB34-3535-7138-1554FAC2FC3C
+74BA36E2-8B52-25DD-725F-E69287F1CFD7
+40C1077B-BD60-76FD-EFB5-A6598C268086
+2AE78F68-F861-FFCC-C0B1-1A3E01232ED7
+0F5F4ADD-49EC-0BAF-769A-7475BBFC99F6
+ECAA334C-9FC1-52E7-0B80-43D53DE38F4C
+92FE2EEB-B0F8-D08F-4C84-669C26A5F02C
+063A9D5F-3FC3-BDBA-C206-69446B1F0C12
+C67DA457-FCF9-D039-B12A-41ECD82C8D72
+0A24ECC7-3067-824D-070F-0134EDD9F8AB
+E8342438-19A6-2126-F4F3-6F6A1F6F01DC
+8D049EE0-EB87-9C40-BC5D-BCBDD2988C3C
+D3D364EE-E034-E036-234F-1FE6EBBBC0BA
+E0EE072F-3E6D-4119-1D64-F9ACCC1A3316
+B68A6D96-3DBB-2D23-9512-4DAC92700EB9
+EB894404-1007-7FF4-6957-9D0AB978837B
+A7F9B841-5034-D8CE-12FC-40CD76F0C5CB
+7CCC2F0E-4477-2FC4-5B9F-5A1A35A5420B
+2FF1CE96-B68A-A099-98BA-7647AEFF4F03
+F55E5FD0-8FBE-5FE8-07B2-0234493E9855
+48E20CBF-4ADA-FCC6-61CD-73577DF58C89
+7420EE4B-E624-5A31-092D-934C90013A47
+37BF0885-0B06-E9CA-41B7-3A61464B2888
+0B92F6EE-99DD-04C4-2A55-BB0227EA0410
+E57E079C-8E15-B586-0D83-1CACC10FE6E9
+C1DAC8A1-2CAE-422A-610C-7781DE3EEEB1
+0AC35B7B-546B-C7E7-81FB-29E2276E40C0
+9FCAF76E-C728-59D4-69C3-6BE0C2F90D99
+F47BE48F-CBF5-0F71-453C-F78EFBF1D41A
+ECA6D607-4602-8DB6-ABFB-95B7E5257F42
+81E177BD-F31F-77E4-A9C5-D4B1ED3C7131
+EC211B72-6708-AA7E-1185-E91D53214E0C
+5E735CDC-8259-9104-3E4B-80F8F2CFE1AD
+AF9FA4BF-C99F-734C-0BCC-2A47505A020D
+DB77CBCA-4552-B0F1-8792-F08EA59FE226
+47218305-B243-21B9-49E2-C6EE971516FD
+69CD94C1-E4BB-9F8F-810A-0A3E0232D73B
+269909B0-7D1E-0FA0-FDA3-0D85FE08BE6E
+63E65C25-F09D-BFEB-F485-70DC803B0B04
+B8270E15-5099-5B54-F65D-5DF8608B6F10
+DCAF5EE6-BBF6-813F-B7C2-EE0B8E5C349C
+28480D68-40F5-8665-C78E-D71974E15FE1
+4C71C8BB-03DF-E60D-18AD-A4EA82582DD4
+9F7F0948-CD4A-E670-02AF-0EEDE2F54886
+FF1C780E-52D7-4DA9-9F1D-27C1E54679DE
+FDD6313C-E183-A333-AC50-5B094720FD95
+09657A77-5186-A333-FAF4-58827C9C36C5
+13C9E9DB-0132-487D-F224-CCAFFBCE5ACC
+6C3985BF-7FCA-9BCD-9F8A-D2E34ABEC242
+8605417E-3ABB-F90A-AEE1-DC3938D391ED
+7327B83D-4A36-E53D-266E-23A6CBE8188E
+4D89C5DD-9576-8BCD-795D-FEDB7A8521F3
+08ED30E8-AF90-C6C4-CA46-A2DA230C48F7
+9C81CEEB-D345-FB21-2C6E-C910B8F565E2
+18DD307F-FCD0-C1C1-6054-C848866537C0
+912B2F9F-B5D6-D00C-F496-5AE6D2105F7F
+D086A783-2BE6-00FA-0525-395D537C7398
+0C6F5056-97CC-88BA-121E-B7BE36F2121C
+0589503F-5BF6-07CB-27F4-4070670FA593
+ACD5393B-0436-0FBD-FAA7-00360181297D
+A2C04D87-8AE3-7628-FED7-0F27A624589B
+8CC5E609-CBDD-C2E1-B2E0-2B0D918347EF
+54CFE76A-C5C8-D967-F2BE-27FCF957D47E
+0E73EFD4-9872-DFD8-DF33-6D260FE387B9
+465E9DA9-044D-B600-E585-7301020BEAF1
+DFDA3AD9-16CC-8702-5B2D-80C7578F8440
+913D17C7-0B4F-EFA8-495D-023E5CE93F1F
+43DD41A0-4601-A510-029D-9FE762CD0124
+931DF6DD-2E91-E2CF-4706-50429A7E884B
+1C6884ED-17BC-31BE-D51E-20591FA3128A
+B8E11632-4F21-58C8-D43A-A9D400B2C562
+99167304-2640-8A22-F5B5-649ED2646CDB
+539C3B16-8D2B-07DA-B97E-B7E7BA3183D4
+FCC29CC9-3A9E-3F2F-E8DF-763437848F94
+E73CBEE6-B67A-9EEA-B822-54AF1B12FCA5
+6F8F8F4B-5775-1D59-5F38-B64024C66067
+21F5043D-4F14-3DF5-C073-2EA98603C5AF
+D73C8C2A-841B-985D-B6F7-6DCB5980555D
+4DCF072A-9346-145C-6410-DBC006D7BB3E
+21E4B2E1-4354-88AB-972A-FB114F91A72B
+9A3B113E-EDBD-3ECA-3ECF-BD64E26D7332
+8B4CE660-D614-443B-F157-C34974A98CD7
+D29688F1-1BC6-71CB-5709-A1D2AE10C4EB
+A5950A71-0475-7E27-605D-AA31F38C73B4
+7DAE29A7-441E-272E-4F7C-F3EDC59E0480
+0BD56561-5624-BB66-9206-58AE87394F26
+D5B59B1D-DDC7-395A-E77C-8EF87A70C9B0
+AE23A85C-E34D-7D75-D628-CD36CE762549
+614C0780-972E-8DA5-483B-158D935D1869
+AA0CA20D-8189-5D94-92E0-0BBE95E00ECA
+F8F794E9-1325-937A-CFD2-85813ED40B48
+61287143-BEBE-8251-1AD5-1E99F5CF53D5
+DED9D8A9-2004-3041-8CC7-A1132E564431
+896B0D45-022E-3353-F653-EE6B311833DD
+051620EA-B579-FE52-D4EC-8B1E6BEC2BD1
+9A0B55CA-E364-2C21-F24A-72ACF0EE9AAB
+83D24F72-C785-C661-3D32-2A621874DA3D
+D2B44C62-1419-7F80-9CB9-1D543B127986
+141BAD7C-B29B-B8B3-6E84-1F9ADB0579CF
+FC104216-5DA9-3B8B-6F7C-422ADAFCAFDC
+ADCB580E-9844-C8BA-AB37-91DFB739CCB6
+48EED515-5575-6C9C-5A18-4655F006D614
+8C4848B5-8EBE-DC14-DF9E-3A62AB39D0AF
+EAC852A7-9B04-AA94-4C15-BBDE8CE04C49
+5ABD3746-CE73-E700-1F48-1348326C0C23
+A6C56F87-3490-8B2D-5CC1-B1454B40E8E0
+111BE9DB-B83F-4219-130C-D4D321F2DDE8
+7ACE0F7A-82ED-8B09-863F-6A83282A56B3
+7F8E672C-E62A-C0C4-A9A9-074CC9ADC5F7
+6D107CE3-8E56-9588-4F9B-8EB587825936
+8A0D744A-685C-5DD6-94B8-9A56C50B9A95
+AF40DB70-5FFD-3C0B-227D-A65868777C6A
+1E0BB90B-C6DA-CAD6-B964-8490BD93F803
+231BCCD8-E0EE-FFF7-00AD-3D35DA8371BC
+2C62E294-FB2E-3ACF-4FCF-91F53273F0EF
+2750F003-95E4-2AC0-B7F5-BEF080FE51F0
+869683C4-2033-6A53-4C42-35F45F623FE3
+2681A2B6-24B8-438E-8A7F-D27433ED7433
+455F4D59-CB85-6300-C11B-0883F7961250
+81A88C59-050D-20FC-81CA-DE1B00A62A8E
+A9F8F204-BB7C-51F0-32AE-65E24004A7AD
+A9F940C0-DC2A-3A1D-7882-14A39443B0BE
+E42F027B-C825-72D6-3616-0AAA79D3F06C
+4F4698C1-B7E3-E7C7-C164-CA85D731F29F
+10E6F064-2047-134F-52F7-55009BC255B8
+EB8A4F99-216E-88E7-7001-698E6FC81FFE
+8FC5ED0B-4657-7E6C-4ED5-9B2DF46AE6B5
+6852716F-283C-F17C-7F5F-5230E5BF6BDB
+0A5DF089-0C1E-5097-A25C-4BEEA6A37825
+908623FA-3114-7B31-50EC-A55EC8D8DE32
+E090991C-B24D-BEC9-6358-A94E4F8A1556
+09EB0D0F-B98E-A598-BDD2-FA2FB4F74725
+999AC4E9-F120-9493-EB3B-B52D1C0ECB72
+F56F8EFA-ED64-4E4D-CD5E-002718707F20
+5CE68786-3A65-DEA0-D33E-AA885729BC8C
+0CD538A7-3046-ADFA-9E37-238C129B99F3
+6AA9EA3B-36B2-EA17-4827-CAA8A8ABB6AF
+FE1B64A3-5F50-A232-2FF2-88A90FC03E16
+1211BF87-9714-BB18-9CB8-E4FD331E2203
+E85096D2-5080-B0A0-90D4-8975573D55F6
+B96EB1DF-D3AB-C130-0EAF-3E404E6E7E97
+66792E9A-EA0C-BC9C-F485-A9AFE6C3958A
+1573DA33-90ED-027A-6EE4-C0FFE4A1F93C
+C54098CC-3120-D3A6-C043-3A75F7467885
+BF439F42-7C50-B07D-A629-661B203FF23D
\ No newline at end of file
diff --git a/dsawswaf/suds/__init__.py b/lib/suds/__init__.py
similarity index 100%
rename from dsawswaf/suds/__init__.py
rename to lib/suds/__init__.py
diff --git a/dsawswaf/suds/bindings/__init__.py b/lib/suds/bindings/__init__.py
similarity index 100%
rename from dsawswaf/suds/bindings/__init__.py
rename to lib/suds/bindings/__init__.py
diff --git a/dsawswaf/suds/bindings/binding.py b/lib/suds/bindings/binding.py
similarity index 100%
rename from dsawswaf/suds/bindings/binding.py
rename to lib/suds/bindings/binding.py
diff --git a/dsawswaf/suds/bindings/document.py b/lib/suds/bindings/document.py
similarity index 100%
rename from dsawswaf/suds/bindings/document.py
rename to lib/suds/bindings/document.py
diff --git a/dsawswaf/suds/bindings/multiref.py b/lib/suds/bindings/multiref.py
similarity index 100%
rename from dsawswaf/suds/bindings/multiref.py
rename to lib/suds/bindings/multiref.py
diff --git a/dsawswaf/suds/bindings/rpc.py b/lib/suds/bindings/rpc.py
similarity index 100%
rename from dsawswaf/suds/bindings/rpc.py
rename to lib/suds/bindings/rpc.py
diff --git a/dsawswaf/suds/builder.py b/lib/suds/builder.py
similarity index 100%
rename from dsawswaf/suds/builder.py
rename to lib/suds/builder.py
diff --git a/dsawswaf/suds/cache.py b/lib/suds/cache.py
similarity index 100%
rename from dsawswaf/suds/cache.py
rename to lib/suds/cache.py
diff --git a/dsawswaf/suds/client.py b/lib/suds/client.py
similarity index 100%
rename from dsawswaf/suds/client.py
rename to lib/suds/client.py
diff --git a/dsawswaf/suds/metrics.py b/lib/suds/metrics.py
similarity index 100%
rename from dsawswaf/suds/metrics.py
rename to lib/suds/metrics.py
diff --git a/dsawswaf/suds/mx/__init__.py b/lib/suds/mx/__init__.py
similarity index 100%
rename from dsawswaf/suds/mx/__init__.py
rename to lib/suds/mx/__init__.py
diff --git a/dsawswaf/suds/mx/appender.py b/lib/suds/mx/appender.py
similarity index 100%
rename from dsawswaf/suds/mx/appender.py
rename to lib/suds/mx/appender.py
diff --git a/dsawswaf/suds/mx/basic.py b/lib/suds/mx/basic.py
similarity index 100%
rename from dsawswaf/suds/mx/basic.py
rename to lib/suds/mx/basic.py
diff --git a/dsawswaf/suds/mx/core.py b/lib/suds/mx/core.py
similarity index 100%
rename from dsawswaf/suds/mx/core.py
rename to lib/suds/mx/core.py
diff --git a/dsawswaf/suds/mx/encoded.py b/lib/suds/mx/encoded.py
similarity index 100%
rename from dsawswaf/suds/mx/encoded.py
rename to lib/suds/mx/encoded.py
diff --git a/dsawswaf/suds/mx/literal.py b/lib/suds/mx/literal.py
similarity index 100%
rename from dsawswaf/suds/mx/literal.py
rename to lib/suds/mx/literal.py
diff --git a/dsawswaf/suds/mx/typer.py b/lib/suds/mx/typer.py
similarity index 100%
rename from dsawswaf/suds/mx/typer.py
rename to lib/suds/mx/typer.py
diff --git a/dsawswaf/suds/options.py b/lib/suds/options.py
similarity index 100%
rename from dsawswaf/suds/options.py
rename to lib/suds/options.py
diff --git a/dsawswaf/suds/plugin.py b/lib/suds/plugin.py
similarity index 100%
rename from dsawswaf/suds/plugin.py
rename to lib/suds/plugin.py
diff --git a/dsawswaf/suds/properties.py b/lib/suds/properties.py
similarity index 100%
rename from dsawswaf/suds/properties.py
rename to lib/suds/properties.py
diff --git a/dsawswaf/suds/reader.py b/lib/suds/reader.py
similarity index 100%
rename from dsawswaf/suds/reader.py
rename to lib/suds/reader.py
diff --git a/dsawswaf/suds/resolver.py b/lib/suds/resolver.py
similarity index 100%
rename from dsawswaf/suds/resolver.py
rename to lib/suds/resolver.py
diff --git a/dsawswaf/suds/sax/__init__.py b/lib/suds/sax/__init__.py
similarity index 100%
rename from dsawswaf/suds/sax/__init__.py
rename to lib/suds/sax/__init__.py
diff --git a/dsawswaf/suds/sax/attribute.py b/lib/suds/sax/attribute.py
similarity index 100%
rename from dsawswaf/suds/sax/attribute.py
rename to lib/suds/sax/attribute.py
diff --git a/dsawswaf/suds/sax/date.py b/lib/suds/sax/date.py
similarity index 100%
rename from dsawswaf/suds/sax/date.py
rename to lib/suds/sax/date.py
diff --git a/dsawswaf/suds/sax/document.py b/lib/suds/sax/document.py
similarity index 100%
rename from dsawswaf/suds/sax/document.py
rename to lib/suds/sax/document.py
diff --git a/dsawswaf/suds/sax/element.py b/lib/suds/sax/element.py
similarity index 100%
rename from dsawswaf/suds/sax/element.py
rename to lib/suds/sax/element.py
diff --git a/dsawswaf/suds/sax/enc.py b/lib/suds/sax/enc.py
similarity index 100%
rename from dsawswaf/suds/sax/enc.py
rename to lib/suds/sax/enc.py
diff --git a/dsawswaf/suds/sax/parser.py b/lib/suds/sax/parser.py
similarity index 100%
rename from dsawswaf/suds/sax/parser.py
rename to lib/suds/sax/parser.py
diff --git a/dsawswaf/suds/sax/text.py b/lib/suds/sax/text.py
similarity index 100%
rename from dsawswaf/suds/sax/text.py
rename to lib/suds/sax/text.py
diff --git a/dsawswaf/suds/servicedefinition.py b/lib/suds/servicedefinition.py
similarity index 100%
rename from dsawswaf/suds/servicedefinition.py
rename to lib/suds/servicedefinition.py
diff --git a/dsawswaf/suds/serviceproxy.py b/lib/suds/serviceproxy.py
similarity index 100%
rename from dsawswaf/suds/serviceproxy.py
rename to lib/suds/serviceproxy.py
diff --git a/dsawswaf/suds/soaparray.py b/lib/suds/soaparray.py
similarity index 100%
rename from dsawswaf/suds/soaparray.py
rename to lib/suds/soaparray.py
diff --git a/dsawswaf/suds/store.py b/lib/suds/store.py
similarity index 100%
rename from dsawswaf/suds/store.py
rename to lib/suds/store.py
diff --git a/dsawswaf/suds/sudsobject.py b/lib/suds/sudsobject.py
similarity index 100%
rename from dsawswaf/suds/sudsobject.py
rename to lib/suds/sudsobject.py
diff --git a/dsawswaf/suds/transport/__init__.py b/lib/suds/transport/__init__.py
similarity index 100%
rename from dsawswaf/suds/transport/__init__.py
rename to lib/suds/transport/__init__.py
diff --git a/dsawswaf/suds/transport/http.py b/lib/suds/transport/http.py
similarity index 100%
rename from dsawswaf/suds/transport/http.py
rename to lib/suds/transport/http.py
diff --git a/dsawswaf/suds/transport/https.py b/lib/suds/transport/https.py
similarity index 100%
rename from dsawswaf/suds/transport/https.py
rename to lib/suds/transport/https.py
diff --git a/dsawswaf/suds/transport/options.py b/lib/suds/transport/options.py
similarity index 100%
rename from dsawswaf/suds/transport/options.py
rename to lib/suds/transport/options.py
diff --git a/dsawswaf/suds/umx/__init__.py b/lib/suds/umx/__init__.py
similarity index 100%
rename from dsawswaf/suds/umx/__init__.py
rename to lib/suds/umx/__init__.py
diff --git a/dsawswaf/suds/umx/attrlist.py b/lib/suds/umx/attrlist.py
similarity index 100%
rename from dsawswaf/suds/umx/attrlist.py
rename to lib/suds/umx/attrlist.py
diff --git a/dsawswaf/suds/umx/basic.py b/lib/suds/umx/basic.py
similarity index 100%
rename from dsawswaf/suds/umx/basic.py
rename to lib/suds/umx/basic.py
diff --git a/dsawswaf/suds/umx/core.py b/lib/suds/umx/core.py
similarity index 100%
rename from dsawswaf/suds/umx/core.py
rename to lib/suds/umx/core.py
diff --git a/dsawswaf/suds/umx/encoded.py b/lib/suds/umx/encoded.py
similarity index 100%
rename from dsawswaf/suds/umx/encoded.py
rename to lib/suds/umx/encoded.py
diff --git a/dsawswaf/suds/umx/typed.py b/lib/suds/umx/typed.py
similarity index 100%
rename from dsawswaf/suds/umx/typed.py
rename to lib/suds/umx/typed.py
diff --git a/dsawswaf/suds/wsdl.py b/lib/suds/wsdl.py
similarity index 100%
rename from dsawswaf/suds/wsdl.py
rename to lib/suds/wsdl.py
diff --git a/dsawswaf/suds/wsse.py b/lib/suds/wsse.py
similarity index 100%
rename from dsawswaf/suds/wsse.py
rename to lib/suds/wsse.py
diff --git a/dsawswaf/suds/xsd/__init__.py b/lib/suds/xsd/__init__.py
similarity index 100%
rename from dsawswaf/suds/xsd/__init__.py
rename to lib/suds/xsd/__init__.py
diff --git a/dsawswaf/suds/xsd/deplist.py b/lib/suds/xsd/deplist.py
similarity index 100%
rename from dsawswaf/suds/xsd/deplist.py
rename to lib/suds/xsd/deplist.py
diff --git a/dsawswaf/suds/xsd/doctor.py b/lib/suds/xsd/doctor.py
similarity index 100%
rename from dsawswaf/suds/xsd/doctor.py
rename to lib/suds/xsd/doctor.py
diff --git a/dsawswaf/suds/xsd/query.py b/lib/suds/xsd/query.py
similarity index 100%
rename from dsawswaf/suds/xsd/query.py
rename to lib/suds/xsd/query.py
diff --git a/dsawswaf/suds/xsd/schema.py b/lib/suds/xsd/schema.py
similarity index 100%
rename from dsawswaf/suds/xsd/schema.py
rename to lib/suds/xsd/schema.py
diff --git a/dsawswaf/suds/xsd/sxbase.py b/lib/suds/xsd/sxbase.py
similarity index 100%
rename from dsawswaf/suds/xsd/sxbase.py
rename to lib/suds/xsd/sxbase.py
diff --git a/dsawswaf/suds/xsd/sxbasic.py b/lib/suds/xsd/sxbasic.py
similarity index 100%
rename from dsawswaf/suds/xsd/sxbasic.py
rename to lib/suds/xsd/sxbasic.py
diff --git a/dsawswaf/suds/xsd/sxbuiltin.py b/lib/suds/xsd/sxbuiltin.py
similarity index 100%
rename from dsawswaf/suds/xsd/sxbuiltin.py
rename to lib/suds/xsd/sxbuiltin.py