AzureML Compute Job failed to start

[Nanthini10]

Unable to use dockerfile to create an Environment in the explain-model-on-amlcompute.ipynb notebook.

The following is the code

from azureml.core import Workspace, Environment
myenv = Environment(name="rapidsenv")

myenv.register(workspace=ws)
myenv.docker.enabled=True

dockerfile = r"""
FROM rapidsai/rapidsai-cloud-ml:0.17-cuda11.0-base-ubuntu18.04-py3.8
RUN echo "success" # I'd like to install different packages here
"""
myenv.docker.base_image = None
myenv.docker.base_dockerfile = dockerfile
myenv.python.user_managed_dependencies=True


from azureml.core import Run
from azureml.core import ScriptRunConfig

src = ScriptRunConfig(source_directory=project_folder, 
                      script='train_explain.py',
                      compute_target=gpu_cluster,
                      environment=myenv) 
run = experiment.submit(config=src)
run

This is what shows up on the Portal.

AzureMLCompute job failed.
JobContainerConfigFailed: Container configuration failed unexpectedly
JobContainerConfigFailed: Container configuration failed unexpectedly
err: Docker exec failure when attempting to execute 'containerSetup' task. Error: exit status 126. Indicates a command was found but not executable, likely due to permission issues or missing dependencies.
Reason: Docker exec failure when attempting to execute 'containerSetup' task. Error: exit status 126. Indicates a command was found but not executable, likely due to permission issues or missing dependencies.
Info: Failed to prepare an environment for the job execution: Job environment preparation failed on 10.0.0.4 with err exit status 1.

How do I resolve this issue?

[vizhur]

Environment was created and image was built. Job failed to start on a specified compute

[vobornij]

Make sure the image has all required dependencies, see
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-custom-docker-image

Also, it seems Azure ML expects shell-like argument handling, you might need to add something like this to your Dockerfile.

ENTRYPOINT []
CMD ["/bin/bash"]

[vivram]

Can you share the runid for the failed run?

[manuelreyesgomez]

Hello, I am having exactly the same error

The environment is "created" and the failure comes when the Job is supposed to start

However I had executed the same job on a different environment and it has worked fine, the issue seems to be while docker running the container specified on the new environment:

environment_name = 'tlt_env_n'
python_interpreter = '/usr/bin/python'
conda_packages = ['matplotlib', 'jupyterlab']

### CREATE OR RETRIEVE THE ENVIRONMENT

if environment_name not in ws.environments:
env = Environment(name=environment_name)
env.docker.enabled = True
env.docker.base_image = None
env.docker.base_dockerfile = 'FROM nvcr.io/nvidia/ds_deployment_demo:20.06\n RUN pip install nvidia-pyindex && \\n pip install nvidia-tlt'
env.python.interpreter_path = python_interpreter
env.python.user_managed_dependencies = True
conda_dep = CondaDependencies()
for conda_package in conda_packages:
conda_dep.add_conda_package(conda_package)
env.python.conda_dependencies = conda_dep
env.register(workspace=ws)

The base container has "/bin/sh -c 'jupyter…" as command, wondering if that could be the issue

@vivram Run ID: tlt-mrg-exp3_1632936100_eb157ff6

Please, any insight would be greatly appreciated, thanks