Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Instructions for running tox with docker don't work #1817

Open
bmispelon opened this issue Dec 10, 2024 · 1 comment · May be fixed by #1828
Open

Instructions for running tox with docker don't work #1817

bmispelon opened this issue Dec 10, 2024 · 1 comment · May be fixed by #1828
Labels
help-needed Help needed

Comments

@bmispelon
Copy link
Member

As mentionned by Pat on Discord (not sure what their github username is), our current instructions for running tox with docker don't seem to work:

Full log ``` $ docker compose exec web tox py312-tests: install_deps> python -I -m pip install -r /usr/src/app/requirements/tests.txt Collecting django-push@ git+https://github.com/brutasse/django-push.git@22fda99641cfbd2f3075a723d92652a8e38220a5 (from -r /usr/src/app/requirements/common.txt (line 6)) Cloning https://github.com/brutasse/django-push.git (to revision 22fda99641cfbd2f3075a723d92652a8e38220a5) to /tmp/pip-install-xbk4s6l8/django-push_869cd076d70d473da4c86b60e8cecf7d Resolved https://github.com/brutasse/django-push.git to commit 22fda99641cfbd2f3075a723d92652a8e38220a5 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting Babel==2.16.0 (from -r /usr/src/app/requirements/common.txt (line 1)) Downloading babel-2.16.0-py3-none-any.whl.metadata (1.5 kB) Collecting django-contact-form==5.1.1 (from -r /usr/src/app/requirements/common.txt (line 2)) Downloading django_contact_form-5.1.1-py3-none-any.whl.metadata (2.7 kB) Collecting django-countries==7.6.1 (from -r /usr/src/app/requirements/common.txt (line 3)) Downloading django_countries-7.6.1-py3-none-any.whl.metadata (31 kB) Collecting django-hosts==5.1 (from -r /usr/src/app/requirements/common.txt (line 4)) Downloading django_hosts-5.1-py3-none-any.whl.metadata (4.7 kB) Collecting django-money==3.5.3 (from -r /usr/src/app/requirements/common.txt (line 5)) Downloading django_money-3.5.3-py3-none-any.whl.metadata (18 kB) Collecting django-read-only==1.18.0 (from -r /usr/src/app/requirements/common.txt (line 7)) Downloading django_read_only-1.18.0-py3-none-any.whl.metadata (7.1 kB) Collecting django-recaptcha==4.0.0 (from -r /usr/src/app/requirements/common.txt (line 8)) Downloading django_recaptcha-4.0.0-py3-none-any.whl.metadata (13 kB) Collecting django-registration-redux==2.13 (from -r /usr/src/app/requirements/common.txt (line 9)) Downloading django_registration_redux-2.13-py2.py3-none-any.whl.metadata (5.1 kB) Collecting Django==5.0.9 (from -r /usr/src/app/requirements/common.txt (line 10)) Downloading Django-5.0.9-py3-none-any.whl.metadata (4.1 kB) Collecting docutils==0.21.2 (from -r /usr/src/app/requirements/common.txt (line 11)) Downloading docutils-0.21.2-py3-none-any.whl.metadata (2.8 kB) Collecting feedparser==6.0.11 (from -r /usr/src/app/requirements/common.txt (line 12)) Downloading feedparser-6.0.11-py3-none-any.whl.metadata (2.4 kB) Collecting Jinja2==3.1.4 (from -r /usr/src/app/requirements/common.txt (line 13)) Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB) Collecting libsass==0.23.0 (from -r /usr/src/app/requirements/common.txt (line 14)) Downloading libsass-0.23.0-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (4.2 kB) Collecting Markdown==3.7 (from -r /usr/src/app/requirements/common.txt (line 15)) Downloading Markdown-3.7-py3-none-any.whl.metadata (7.0 kB) Collecting Pillow==11.0.0 (from -r /usr/src/app/requirements/common.txt (line 16)) Downloading pillow-11.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (9.1 kB) Collecting psycopg==3.2.3 (from psycopg[c]==3.2.3->-r /usr/src/app/requirements/common.txt (line 17)) Downloading psycopg-3.2.3-py3-none-any.whl.metadata (4.3 kB) Collecting Pygments==2.18.0 (from -r /usr/src/app/requirements/common.txt (line 18)) Downloading pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB) Collecting pykismet3==0.1.1 (from -r /usr/src/app/requirements/common.txt (line 19)) Downloading pykismet3-0.1.1-py2.py3-none-any.whl.metadata (2.9 kB) Collecting requests==2.32.3 (from -r /usr/src/app/requirements/common.txt (line 20)) Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Collecting sorl-thumbnail==12.11.0 (from -r /usr/src/app/requirements/common.txt (line 21)) Downloading sorl_thumbnail-12.11.0-py3-none-any.whl.metadata (10 kB) Collecting Sphinx==8.1.3 (from -r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinx-8.1.3-py3-none-any.whl.metadata (6.4 kB) Collecting stripe==3.1.0 (from -r /usr/src/app/requirements/common.txt (line 23)) Downloading stripe-3.1.0-py2.py3-none-any.whl.metadata (2.6 kB) Collecting time-machine==2.16.0 (from -r /usr/src/app/requirements/common.txt (line 24)) Downloading time_machine-2.16.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting django-debug-toolbar==4.4.6 (from -r /usr/src/app/requirements/dev.txt (line 2)) Downloading django_debug_toolbar-4.4.6-py3-none-any.whl.metadata (3.9 kB) Collecting isort==5.13.2 (from -r /usr/src/app/requirements/dev.txt (line 3)) Downloading isort-5.13.2-py3-none-any.whl.metadata (12 kB) Collecting pre-commit~=4.0.1 (from -r /usr/src/app/requirements/dev.txt (line 4)) Downloading pre_commit-4.0.1-py2.py3-none-any.whl.metadata (1.3 kB) Collecting watchdog==6.0.0 (from -r /usr/src/app/requirements/dev.txt (line 5)) Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl.metadata (44 kB) Collecting coverage==7.6.9 (from -r /usr/src/app/requirements/tests.txt (line 2)) Downloading coverage-7.6.9-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.2 kB) Collecting requests-mock==1.12.1 (from -r /usr/src/app/requirements/tests.txt (line 3)) Downloading requests_mock-1.12.1-py2.py3-none-any.whl.metadata (4.1 kB) Collecting tox<5.0.0 (from -r /usr/src/app/requirements/tests.txt (line 4)) Downloading tox-4.23.2-py3-none-any.whl.metadata (3.7 kB) Collecting asgiref (from django-countries==7.6.1->-r /usr/src/app/requirements/common.txt (line 3)) Downloading asgiref-3.8.1-py3-none-any.whl.metadata (9.3 kB) Collecting typing-extensions (from django-countries==7.6.1->-r /usr/src/app/requirements/common.txt (line 3)) Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) Collecting setuptools (from django-money==3.5.3->-r /usr/src/app/requirements/common.txt (line 5)) Using cached setuptools-75.6.0-py3-none-any.whl.metadata (6.7 kB) Collecting py-moneyed<3.1,>=2.0 (from django-money==3.5.3->-r /usr/src/app/requirements/common.txt (line 5)) Downloading py_moneyed-3.0-py3-none-any.whl.metadata (3.2 kB) Collecting sqlparse>=0.3.1 (from Django==5.0.9->-r /usr/src/app/requirements/common.txt (line 10)) Downloading sqlparse-0.5.3-py3-none-any.whl.metadata (3.9 kB) Collecting sgmllib3k (from feedparser==6.0.11->-r /usr/src/app/requirements/common.txt (line 12)) Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting MarkupSafe>=2.0 (from Jinja2==3.1.4->-r /usr/src/app/requirements/common.txt (line 13)) Downloading MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Collecting charset-normalizer<4,>=2 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading charset_normalizer-3.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB) Collecting idna<4,>=2.5 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading idna-3.10-py3-none-any.whl.metadata (10 kB) Collecting urllib3<3,>=1.21.1 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB) Collecting certifi>=2017.4.17 (from requests==2.32.3->-r /usr/src/app/requirements/common.txt (line 20)) Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB) Collecting sphinxcontrib-applehelp>=1.0.7 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_applehelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-devhelp>=1.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_devhelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-htmlhelp>=2.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-jsmath>=1.0.1 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl.metadata (1.4 kB) Collecting sphinxcontrib-qthelp>=1.0.6 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_qthelp-2.0.0-py3-none-any.whl.metadata (2.3 kB) Collecting sphinxcontrib-serializinghtml>=1.1.9 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl.metadata (2.4 kB) Collecting snowballstemmer>=2.2 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading snowballstemmer-2.2.0-py2.py3-none-any.whl.metadata (6.5 kB) Collecting alabaster>=0.7.14 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading alabaster-1.0.0-py3-none-any.whl.metadata (2.8 kB) Collecting imagesize>=1.3 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading imagesize-1.4.1-py2.py3-none-any.whl.metadata (1.5 kB) Collecting packaging>=23.0 (from Sphinx==8.1.3->-r /usr/src/app/requirements/common.txt (line 22)) Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB) Collecting python-dateutil (from time-machine==2.16.0->-r /usr/src/app/requirements/common.txt (line 24)) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting psycopg-c==3.2.3 (from psycopg[c]==3.2.3->-r /usr/src/app/requirements/common.txt (line 17)) Downloading psycopg_c-3.2.3.tar.gz (598 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 598.9/598.9 kB 3.2 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error

Running command git clone --filter=blob:none --quiet https://github.com/brutasse/django-push.git /tmp/pip-install-xbk4s6l8/django-push_869cd076d70d473da4c86b60e8cecf7d
Running command git rev-parse -q --verify 'sha^22fda99641cfbd2f3075a723d92652a8e38220a5'
Running command git fetch -q https://github.com/brutasse/django-push.git 22fda99641cfbd2f3075a723d92652a8e38220a5
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
running dist_info
creating /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info
writing /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-qwn9zhkf/psycopg_c.egg-info/SOURCES.txt'
couldn't run 'pg_config' --includedir: [Errno 2] No such file or directory: 'pg_config'
error: [Errno 2] No such file or directory: 'pg_config'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
py312-tests: exit 1 (21.41 seconds) /usr/src/app> python -I -m pip install -r /usr/src/app/requirements/tests.txt pid=28
py312-tests: FAIL ✖ in 21.53 seconds
py312-flake8: commands[0]> flake8
py312-flake8: OK ✔ in 2.4 seconds
py312-black: OK ✔ in 0.02 seconds
py312-isort: commands[0]> make isort-check
python -m isort --check accounts aggregator blog contact dashboard djangoproject docs foundation fundraising legacy members releases svntogit tracdb
py312-tests: FAIL code 1 (21.53 seconds)
py312-flake8: OK (2.39=setup[0.01]+cmd[2.38] seconds)
py312-black: OK (0.01 seconds)
py312-isort: OK (0.87=setup[0.01]+cmd[0.85] seconds)
evaluation failed :( (24.98 seconds)

</details
@bmispelon bmispelon added the help-needed Help needed label Dec 10, 2024
@pbratkowski
Copy link
Contributor

pbratkowski commented Dec 10, 2024 •
edited
Loading

The [edit: first part of the] issue occurs due to tox attempting to install requirements in isolation, but the docker build process purging required packages.

djangoproject.com/Dockerfile

Lines 37 to 41 in 75f3fde

&& apt-get purge --assume-yes --auto-remove \
gcc \
libc6-dev \
libpq-dev \
zlib1g-dev \

This creates a smaller image, which is good for production but does affect development.

I think the most elegant solution would be to use Docker's multi-stage builds, with base, dev, and production targets. Package installation could then be done in the base image, while unnecessary packages could be purged in production.

pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 10, 2024
@pbratkowski
Implemented Docker multi-stage build
d4f8800 
This solves the first part of django#1817, where tox fails to install test
dependencies due to missing packages.

- Added target inputs to GitHub workflows.
- Added stages to Dockerfile.
- Moved package removal to production stage.
- Added target dev to compose file.
@pbratkowski pbratkowski mentioned this issue Dec 10, 2024
Closed
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 11, 2024
@pbratkowski
Implemented Docker multi-stage build
c93147a 
This solves the first part of django#1817, where tox fails to install test
dependencies due to missing packages.

- Added stages to Dockerfile.
- Moved package removal to production stage.
- Added target dev to compose file.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 13, 2024
@pbratkowski
Implemented Docker multi-stage build
5424f72 
This solves the first part of django#1817, where tox fails to install test
dependencies due to missing packages.

- Added stages to Dockerfile.
- Moved package removal to production stage.
- Added target dev to compose file.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 13, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
19356bc 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 13, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
9fbab4a 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 14, 2024
@pbratkowski
Added 'KEEP_DEPENDENCIES' Docker argument
48f434d 
As per django#1817, tox is missing build dependencies and cannot run tests
in containers. This commit added an argument to make purging packages
optional, allowing to optimize image for deployment, while also
keeping the required packages in dev environments.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 14, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
b99166a 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 15, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
ff3aaa3 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 16, 2024
@pbratkowski
Added 'KEEP_DEPENDENCIES' Docker argument
5560493 
As per django#1817, tox is missing build dependencies and cannot run tests
in containers. This commit added an argument to make purging packages
optional, allowing to optimize image for deployment, while also
keeping the required packages in dev environments.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 16, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
9470ac4 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
@pbratkowski pbratkowski linked a pull request Dec 16, 2024 that will close this issue
Draft
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 16, 2024
@pbratkowski
Added 'KEEP_DEPENDENCIES' Docker argument
d689893 
As per django#1817, tox is missing build dependencies and cannot run tests
in containers. This commit added an argument to make purging packages
optional, allowing to optimize image for deployment, while also
keeping the required packages in dev environments.
pbratkowski added a commit to pbratkowski/djangoproject.com that referenced this issue Dec 16, 2024
@pbratkowski
Fixed django#1817 -- Added tracdb container
65b2438 
Added 'tracdb' container to compose file. Simplified docker settings to
remove duplication. Added support for 'secrets.json' in the docker dev
stage. Added the 'DJANGOPROJECT_DATA_DIR' env var to tox.
@bmispelon bmispelon mentioned this issue Dec 18, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help-needed Help needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix #1817 docker dx pbratkowski/djangoproject.com
2 participants
@bmispelon @pbratkowski