🐳 Containerize CI (openvinotoolkit#616)

* Add dockerfile + readme

* Update workflow

* break ci into two stages + add shared memory command to readme + dockerfile precommit check

* Add images for cuda 10 and 11

* Address codacy issues

.ci/README.md

@@ -0,0 +1,61 @@
+# Guide to Setting up the CI using the Docker images
+
+## Steps
+
+1. Build the docker image using the Dockerfile in the .ci directory.
+   Make sure you are in the root directory of `anomalib`.
+
+   ```bash
+   sudo docker build --build-arg HTTP_PROXY="$http_proxy" --build-arg \
+   HTTPS_PROXY="$https_proxy" --build-arg NO_PROXY="$no_proxy" \
+   . -t anomalib-ci -f .ci/cuda11.4.Dockerfile
+   ```
+
+   Here, `anomalib-ci` is the name of the image.
+
+1. Create and start a container
+
+   ```bash
+   sudo docker run --gpus all \
+   --shm-size=256M\
+    -i -t --mount type=bind,source=<path-to-datasets>,target=/home/user/datasets,readonly\
+    -d --name anomalib-ci-container anomalib-ci
+   ```
+
+   Note: `--gpus all` is required for the container to have access to the GPUs.
+   `-d` flag ensure that the container is detached when it is created.
+   `mount` is required to ensure that tests have access to the dataset.
+
+1. Enter the container by
+
+   ```bash
+   sudo docker exec -it  anomalib-ci-container /bin/bash
+   ```
+
+1. Install github actions runner in the container by navigating to [https://github.com/openvinotoolkit/anomalib/settings/actions/runners/new](https://github.com/openvinotoolkit/anomalib/settings/actions/runners/new)
+
+   For example:
+
+   ```bash
+   mkdir actions-runner && cd actions-runner
+
+   curl -o actions-runner-linux-x64-2.296.1.tar.gz -L https://github.com/actions/runner/releases/download/v2.296.1/actions-runner-linux-x64-2.296.1.tar.gz
+
+   tar xzf ./actions-runner-linux-x64-2.296.1.tar.gz
+
+   rm actions-runner-linux-x64-2.296.1.tar.gz
+
+   ./config.sh --url https://github.com/openvinotoolkit/anomalib --token <enter-your-token-here>
+   ```
+
+   Follow the instructions on the screen to complete the installation.
+
+1. Now the container is ready. Type `exit` to leave the container.
+
+1. Start github actions runner in detached mode in the container and set the
+   codacy token and the anomalib dataset environment variables.
+
+   ```bash
+   sudo docker exec -d anomalib-ci-container /bin/bash -c \
+   "export ANOMALIB_DATASET_PATH=/home/user/datasets;export CODACY_PROJECT_TOKEN=<codacy-project-token> && /home/user/actions-runner/run.sh"
+   ```

.ci/cuda10.2.Dockerfile

@@ -0,0 +1,74 @@
+#########################################################
+## Python Environment with CUDA
+#########################################################
+ARG HTTP_PROXY
+ARG HTTPS_PROXY
+ARG NO_PROXY
+
+FROM nvidia/cuda:10.2-devel-ubuntu18.04 AS python_base_cuda10.2
+LABEL maintainer="Anomalib Development Team"
+
+# Setup proxies
+
+ENV http_proxy=$HTTP_PROXY
+ENV https_proxy=$HTTPS_PROXY
+ENV no_proxy=$NO_PROXY
+ENV DEBIAN_FRONTEND="noninteractive"
+
+# Update system and install wget
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+        curl=7.58.0-2ubuntu3.20 \
+        wget=1.19.4-1ubuntu2 \
+        ffmpeg=7:3.4.2-2 \
+        libpython3.8=3.8.0-3ubuntu1~18.04.2 \
+        npm=3.5.2-0ubuntu4 \
+        ruby=1:2.5.1 \
+        software-properties-common=0.96.24.32.18 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install latest git for github actions
+RUN add-apt-repository ppa:git-core/ppa &&\
+    apt-get update && \
+    apt-get install --no-install-recommends -y git=1:2.38.0-0ppa1~ubuntu18.04.1 &&\
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Prettier requires atleast nodejs 10
+RUN curl -sL https://deb.nodesource.com/setup_14.x > nodesetup.sh && \
+    bash - nodesetup.sh && \
+    apt-get install --no-install-recommends -y nodejs=14.20.1-1nodesource1 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Create a non-root user
+RUN useradd -m user
+USER user
+
+# Install Conda
+RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > ~/miniconda.sh -s && \
+    bash ~/miniconda.sh -b -p /home/user/conda && \
+    rm ~/miniconda.sh
+ENV PATH "/home/user/conda/bin:${PATH}"
+RUN conda install python=3.8
+
+
+#########################################################
+## Anomalib Development Env
+#########################################################
+
+FROM python_base_cuda10.2 as anomalib_development_env
+
+# Install all anomalib requirements
+COPY ./requirements/base.txt /tmp/anomalib/requirements/base.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/base.txt
+
+COPY ./requirements/openvino.txt /tmp/anomalib/requirements/openvino.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/openvino.txt
+
+# Install other requirements related to development
+COPY ./requirements/dev.txt /tmp/anomalib/requirements/dev.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/dev.txt
+
+WORKDIR /home/user

Dockerfile → .ci/cuda11.4.Dockerfile

@@ -1,34 +1,58 @@
 #########################################################
 ## Python Environment with CUDA
 #########################################################
+ARG HTTP_PROXY
+ARG HTTPS_PROXY
+ARG NO_PROXY
 
-FROM nvidia/cuda:11.4.0-devel-ubuntu20.04 AS python_base_cuda
-LABEL MAINTAINER="Anomalib Development Team"
+FROM nvidia/cuda:11.4.0-devel-ubuntu20.04 AS python_base_cuda11.4
+LABEL maintainer="Anomalib Development Team"
+
+# Setup proxies
+
+ENV http_proxy=$HTTP_PROXY
+ENV https_proxy=$HTTPS_PROXY
+ENV no_proxy=$NO_PROXY
+ENV DEBIAN_FRONTEND="noninteractive"
 
 # Update system and install wget
 RUN apt-get update && \
-    DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -y \
+    apt-get install --no-install-recommends -y \
+        curl=7.68.0-1ubuntu2.13 \
         wget=1.20.3-1ubuntu2 \
         ffmpeg=7:4.2.7-0ubuntu0.1 \
         libpython3.8=3.8.10-0ubuntu1~20.04.5 \
-        git=1:2.25.1-1ubuntu3.5 \
-        sudo=1.8.31-1ubuntu1.2 && \
+        nodejs=10.19.0~dfsg-3ubuntu1 \
+        npm=6.14.4+ds-1ubuntu2 \
+        ruby=1:2.7+1 \
+        software-properties-common=0.99.9.8 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install latest git for github actions
+RUN add-apt-repository ppa:git-core/ppa &&\
+    apt-get update && \
+    apt-get install --no-install-recommends -y git=1:2.38.0-0ppa1~ubuntu20.04.1 &&\
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+# Create a non-root user
+RUN useradd -m user
+USER user
+
 # Install Conda
-RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh --quiet && \
-    bash ~/miniconda.sh -b -p /opt/conda && \
+RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > ~/miniconda.sh -s && \
+    bash ~/miniconda.sh -b -p /home/user/conda && \
     rm ~/miniconda.sh
-ENV PATH "/opt/conda/bin:${PATH}"
+ENV PATH "/home/user/conda/bin:${PATH}"
 RUN conda install python=3.8
 
 
 #########################################################
 ## Anomalib Development Env
 #########################################################
 
-FROM python_base_cuda as anomalib_development_env
+FROM python_base_cuda11.4 as anomalib_development_env
 
 # Install all anomalib requirements
 COPY ./requirements/base.txt /tmp/anomalib/requirements/base.txt
@@ -38,17 +62,7 @@ COPY ./requirements/openvino.txt /tmp/anomalib/requirements/openvino.txt
 RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/openvino.txt
 
 # Install other requirements related to development
-RUN apt-get update && \
-    DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -y \
-        nodejs=10.19.0~dfsg-3ubuntu1 \
-        npm=6.14.4+ds-1ubuntu2 \
-        ruby=1:2.7+1 && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
 COPY ./requirements/dev.txt /tmp/anomalib/requirements/dev.txt
 RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/dev.txt
 
-# Install anomalib
-COPY . /anomalib
-WORKDIR /anomalib
-RUN pip install --no-cache-dir -e .
+WORKDIR /home/user

.devcontainer/devcontainer.json

@@ -5,10 +5,10 @@
   // Sets the run context to one level up instead of the .devcontainer folder.
   "context": "..",
   // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
-  "dockerFile": "../Dockerfile",
+  "dockerFile": "../.ci/cuda11.4.Dockerfile",
   // Set *default* container specific settings.json values on container create.
   "settings": {
-    "python.pythonPath": "/opt/conda/bin/python",
+    "python.pythonPath": "/home/user/conda/bin/python",
     "python.formatting.provider": "black",
     "editor.formatOnSave": true,
     "python.formatting.blackArgs": ["--line-length", "120"],

.dockerignore

@@ -1,4 +1,5 @@
 *.dot
+.git
 .idea
 .pytest_cache
 .tox

.github/workflows/nightly.yml

@@ -12,16 +12,12 @@ jobs:
       max-parallel: 1
     if: github.ref == 'refs/heads/main'
     steps:
-      - name: Print GPU status
-        run: nvidia-smi
       - name: CHECKOUT REPOSITORY
         uses: actions/checkout@v2
       - name: Install Tox
         run: pip install tox
       - name: Coverage
-        run: |
-          export ANOMALIB_DATASET_PATH=/media/data1/datasets/
-          tox -e nightly
+        run: tox -e nightly
       - name: Upload coverage result
         uses: actions/upload-artifact@v2
         with:

.github/workflows/pre_merge.yml

@@ -7,24 +7,27 @@ on:
   workflow_dispatch: # run on request (no need for PR)
 
 jobs:
+  Code-Quality-Checks:
+    runs-on: [self-hosted, linux, x64]
+    steps:
+      - name: CHECKOUT REPOSITORY
+        uses: actions/checkout@v2
+      - name: Install Tox
+        run: pip install tox
+      - name: Code quality checks
+        run: tox -e pre-commit
   Tox:
     runs-on: [self-hosted, linux, x64]
+    needs: Code-Quality-Checks
     strategy:
       max-parallel: 1
     steps:
-      - name: Print GPU status
-        run: nvidia-smi
       - name: CHECKOUT REPOSITORY
         uses: actions/checkout@v2
       - name: Install Tox
         run: pip install tox
-      - name: Code quality checks
-        run: tox -e pre-commit
       - name: Coverage
-        run: |
-          export ANOMALIB_DATASET_PATH=/media/data1/datasets/
-          export CUDA_VISIBLE_DEVICES=3
-          tox -e pre_merge
+        run: tox -e pre_merge
       - name: Upload coverage report
         run: |
           # If the workflow is triggered from PR then it gets the commit id from the PR.

.pre-commit-config.yaml

@@ -82,3 +82,10 @@ repos:
     rev: 2.1.6
     hooks:
       - id: markdownlint
+
+  - repo: https://github.com/AleksaC/hadolint-py
+    rev: v2.10.0
+    hooks:
+      - id: hadolint
+        name: Lint Dockerfiles
+        description: Runs hadolint to lint Dockerfiles