Fix requierments.txt file

[cgaudit]

Hi All,
Does someone has a fix file of requierments.txt which works?
BR

[GastroGee]

I switched requests to 2.25.0 and urllib3 to 1.25.6

[cgaudit]

Hi @GastroGee Thank's
I change it but pyjq is still erroring...
#888 (comment)

[OlaSegha]

Ah i see, so i was building a docker image and here is some adjustment with my dockerfile
Note, i installed pyjq with my dockerfile ...

# Dockerize https://github.com/duo-labs/cloudmapper
# This project pulls Cloudmapper and initializes it in a docker container.

FROM python:3.7-slim-buster

EXPOSE 8000
WORKDIR /opt/cloudmapper

## basic installs
RUN apt-get update
RUN apt-get install autoconf automake build-essential libtool python3.7-dev python3-tk jq awscli -y
RUN apt-get install git jq -y

## Clone cloudmapper from duo release page
RUN git clone https://github.com/duo-labs/cloudmapper.git .

## The default requirements file has lots of dependency issues, removing it. 
RUN rm requirements.txt 

## Passing in our own tested requirement file that works 
COPY requirements.txt ./requirements.txt

## Install pip and others 
RUN pip install --upgrade pip setuptools wheel
RUN pip install PEP517
RUN pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install
RUN pip install --upgrade awscli
RUN pip install pyjq

VOLUME /opt/cloudmapper/web 
VOLUME /opt/cloudmapper/account-data 

COPY entrypoint.sh ./entrypoint.sh


# COPY .env ./.env

##Pass in my config file 
COPY config.json ./config.json

CMD ["bash","entrypoint.sh"]

and then my requirements file is this

astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2020.4.5.1
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==1.3.2.20201103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

hope that helps

[w0rmr1d3r]

Thank you @OlaSegha , can you open a PR with that fix so that we can all take a look?

Cheers!

[cgaudit]

Hi @OlaSegha,
Thank's!

I tried to reinstalled it on my local Linux VM after I updated the requierments.txt file with you code and run the
pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install

l still got the pyjq error:

An error occurred while installing pyjq==2.3.1 --hash=sha256:5f3cd45b856ae4b42a1058d2e91e13d84df5ea188aedc3ae397dc0e01c270db9 --hash=sha256:45f7688d8191a4043e7c4839d6cabff8f16abc98fe48c3a4a0cc2da05c62096d! Will try again.

Is it necessary to install it on a docker and how would you suggest to do it the quickest way?

Any Idea why?

@w0rmr1d3r did it worked for you? Is it install locally on your Linux?

BR

[w0rmr1d3r]

Hello @cgaudit !
I'm not running a Linux 😅 , however, since this project gave me a lot of issues installing it and the maintainers aren't reviewing the PRs with the fixes proposed, I switched to another tool which works.

This one -> https://github.com/toniblyx/prowler

See if it helps with what you want to scan or do.

Cheers!

[oichokabu]

Hi All,

I'm sorry for the similar question.
The requirements.txt is not correct, could you please tell me the correct file or how to fix it?

I have run this command.

pip install -r requirements.txt

However, I encountered the following error.

ERROR: botocore 1.17.6 has requirement urllib3<1.26,>=1.20; python_version != "3.4", but you'll have urllib3 1.26.5 which is incompatible.
ERROR: requests 2.22.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 1.26.5 which is incompatible.

I followed @OlaSegha's advice to check requirements.txt but it did not lead me to a solution😢

Please help me from this error.

[OlaSegha]

@oicho014 @cgaudit

So i just attempted this on a linux VM, my virtual env is python3.7

First the requirements file is wrong, i removed pyjq 2.1.0 from the list .... and updated urllib3 to 1.25.6.

astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2020.4.5.1
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==1.3.2.20201103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

after which i just ran

pip install pyjq==2.1.0

[oichokabu]

I tried what you said, and the build went through.
Now I can sleep soundly!
Thanks, @OlaSegha🎉

[cgaudit]

Hi
@OlaSegha

I tried to rerun it but unfortunately got an error:

ERROR: Failed building wheel for pyjq
Running setup.py clean for pyjq
Failed to build matplotlib pyjq
Installing collected packages: botocore, s3transfer, PyYAML, pyflakes, pycodestyle, pandas, mccabe, matplotlib, isort, idna, chardet, certifi, boto3, astroid, typed-ast, seaborn, requests, pylint, pyjq, policyuniverse, parliament, nose, mock, coverage, autopep8, autoflake
Attempting uninstall: botocore
Found existing installation: botocore 1.21.6
Uninstalling botocore-1.21.6:
Successfully uninstalled botocore-1.21.6
Attempting uninstall: s3transfer
Found existing installation: s3transfer 0.5.0
Uninstalling s3transfer-0.5.0:
Successfully uninstalled s3transfer-0.5.0
Attempting uninstall: PyYAML
Found existing installation: PyYAML 5.3.1
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I have python 3.92 installed.

Not sure why..

BR

[OlaSegha]

@cgaudit Well the toplevel README says that the installation is known to work with python3.7, so that might be contributing to the issue.

[w0rmr1d3r]

@cgaudit are you running on a Mac or Linux?
I had problems with the Mac installation regarding Python 3.9

[cgaudit]

Hi
@w0rmr1d3r sorry for the delay... I'm running on Linux

[cgaudit]

Hi
I run it using ubuntu with 3.7 on the demo and it succeeded, thank's!

1. Is it nessecery to add external CIDR names to the config.json file or it can be ignored? If not how can I get it?
2. Should I truncate any existing folder or it will build new folder?
   BR

[GastroGee]

To question 1; It is not necessary... if u do not have external CIDRs then u dont have to add it to ur config.json. External CIDRs wld be CIDRs assigned to internal networks within ur organization like HQ CIDR, datacenter etc

[cgaudit]

It seems that there are external CIDR's. Do I need all IPv4 and their names?
Is their names also required or it is sufficient to add only the IP's
"1.1.1.1/32": {"name": "SF Office"}
?

Anyway...

1. I don't have MFA
2. Got error when running without CIDR's:
   python3 cloudmapper.py prepare --config config.json.xxx --account xxx
   Traceback (most recent call last): File "cloudmapper.py", line 72, in <module> main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 833, in run prepare(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 662, in prepare cytoscape_json = build_data_structure(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 420, in build_data_structure for region_json in get_regions(account, outputfilter): File "/home/hunter/cloudmapper-main/shared/common.py", line 161, in get_regions regions = pyjq.all(".Regions[]{}".format(region_filter), region_data) File "/home/hunter/cloudmapper-main/venv/lib/python3.8/site-packages/pyjq.py", line 49, in all return compile(script, vars, library_paths).all(_get_value(value, url, opener)) File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561) _pyjq.ScriptRuntimeError: Cannot iterate over null (null)