Support encryption of image layers (imgcrypt)

[rarrais]

Currently all code contained within images generated by Rigel can be accessed.
This is not ideal as ROS applications may contain sensitive data (like access credentials), proprietary algorithms (e.g., for navigation), AI models (e.g., for object detection), etc.

Therefore, a mechanism to encrypt the layers of generated container images is ideal, thus safeguarding any intellectual property. There are already tools such as imgcrypt that can be used to do just this.

The goal of this issue is therefore to assess how this encryption mechanism works and to integrate it with the existing Rigel rigel.core.BuildXPlugin plugin.

[gsilva00]

Update

Podman

I started by setting up Podman and trying its encryption. It turns out that the Ubuntu 22.04 release is Podman 3.4.4, and the --encryption-key=<method>:<key> argument for commands like podman push are supported in later versions. I tried to compile Podman's latest version (5.4.2) from source, but it needs another dependency from the same developing community, called netavark. This would also need to be compiled from the source and I believe getting them to work together wouldn't be easy. Only if necessary will I dive into it.

On imgcrypt

I can confirm what I've said before: this library only works directly with the containerd application. It is an extension to it.
I will however make a correction to what I previously stated. As in Podman, it is possible to encrypt individual layers (with the --layer <num> flag in the sudo ctr-enc images encrypt ... command.

I have been creating an image with docker buildx build with the docker-container driver (as used in Rigel), exporting the image into a tarball and importing the tarball to the containerd runtime / storage system.
This transferring process is not ideal at all to implement in Rigel, but this is just for testing. I don't know if there is a better way of doing it, but that's what I am going to search next, as well as possible ways include it into the existing Rigel pipeline.
I have serious doubts about this, however, due to imgcrypt's lack of popularity and widespread support. Especially considering that we add an additional layer of abstraction of top of this already-lacking interaction between imgcrypt and the Docker CLI, in Rigel's use of Python on Whales (which wraps the Docker CLI to more cleanly use it in Python).

Going back to the testing, surprisingly, I've been getting inconsistent and rather nonsensical results in terms of allowing me to run an encrypted image without providing and key, as if it was unencrypted. I've double- and triple-checked several times if the images are actually encrypted, deleted them and encrypted them again and the results still don't make sense. I need to test more to take any more conclusions

Questions

Simultaneously, I have been searching online and I have been seeing strong push-back against storing secret / inaccessible stuff in an image:

• https://www.reddit.com/r/docker/comments/1ewssds/how_do_i_protect_my_code_in_a_docker_image/
• https://forums.docker.com/t/container-security-encryption/137699
• https://forums.docker.com/t/how-easy-hard-is-it-to-break-in-docker-containers/141161
• https://forums.docker.com/t/in-the-docker-container-we-need-to-be-able-to-exec-into-it-and-view-the-application-code-and-dockerfile-but-i-want-to-set-up-a-password-for-the-users-to-restrict-access/143841
• https://forums.docker.com/t/how-to-lock-the-containers-code/138343

I have, however, seen an image being encrypted and the constraints that that puts on running it.
So, I'm afraid I'm misunderstanding the extent of what encrypting the layers actually is / does.

In that same line of thinking, many questions arose beyond the initial description of this issue:

• what exactly are we trying to encrypt?
• when/where will that encrypted data be used?
• when will it be decrypted, if at all?
• how will it be decrypted? how will we be passing the necessary keys?
• are we hoping to use the encrypted data from some layer in some other part of the container (like in a running application), simultaneously hiding sensitive information / keeping it encrypted from the user and providing full functionality to said application?
• or are we just hoping to not allow image functionality to unauthorized users?

cc @rarrais @MisterOwlPT

[rarrais]

Hi @gsilva00 , thank you for the in depth analysis.

I started by setting up Podman and trying its encryption. It turns out that the Ubuntu 22.04 release is Podman 3.4.4, and the --encryption-key=: argument for commands like podman push are supported in later versions. I tried to compile Podman's latest version (5.4.2) from source, but it needs another dependency from the same developing community, called netavark. This would also need to be compiled from the source and I believe getting them to work together wouldn't be easy. Only if necessary will I dive into it.

Can't you create a VM for this test with a newer distro and Podman version? Check Vagrant and VirtualBox. They work great together. Also just a note that users might not have the latest Podman version, so we have to account for that.

As to your questions, it really depends on what the Robotair costumer wants to do with it, so we should prepare it for the majority of scenarios. Nevertheless, for the time being and for the scope of this action, we just want to support image/layer encryption through Rigel, to the extent that it is possible for an initial Proof-of-Concept.

The fine details about the multitude of ways in which an encryption/decryption can be performed and the management of their associated keys can come at a later stage. However, please document all findings in that regard as to help us tailor the approach.

[robotairio]

@gsilva00 This is used for the encryption this is one solution which can be used for encryption.

Also check out this video from dockercon

I can briefly answer some of your questions based on my understanding of the codemeter.

what exactly are we trying to encrypt?
when/where will that encrypted data be used?
when will it be decrypted, if at all?
how will it be decrypted? how will we be passing the necessary keys?

We will encrypt the application code that is in the container and will be encrypted using the CI pipelines.
The encrypted code will be decrypted during the run-time of the containers and will be decrypted using the Codemeter container which contains the decryption key to do it.

Also refer this

[rarrais]

@gsilva00 , after the Easter break, please update the status of this issue.

[gsilva00]

Update

Podman

By setting up a Ubuntu 24.04.2 LTS VM, I was able to use the Podman encryption and decryption features (added in v4.4.0-RC1), as 4.9.3 is available for it and only 3.4.4 for my host's Ubuntu 22.04.5 LTS.

Podman does not allow for Encryption At-Rest (encrypting and storing the container).

However, Podman allows for:

Encryption in transit

We encrypt the image/layer as we're pushing the image to a registry with the --encryption-key flag:

# Can also add --encrypt-layer=[layerIndex] to the push command to specify the layer to encrypt (instead of the whole image). 
# The layer index can be negative - works exactly in the same way as Python's negative indexing. 
# --encrypt-layer=[-1] refers to the topmost / most recently created layer
$ podman push --encryption-key=[protocol:keyfile] [image] 

And the image stays encrypted in the registry.

Then we provide the decryption key with the and --decryption-key flag, and can pull the image:

$ podman pull --decryption-key=[protocol:keyfile] [image]

Secrets

Passwords, API keys, certificates, etc. stored in a protected directory in the host and that can be mounted at runtime so that containers have a read-only access to them at runtime.

This allows for:

• Protection of secret information when pushing the images to a public registry
• Avoid unneeded exposure of secrets through environment variables
• Storing secrets securely in the host to later be mounted in the container

About the DockerCon19 video

This is the presentation of IBM's proposal for a New Mediatype for Encrypted Layers of a Container Image by Brandon Lum. This proposal started 7+ years ago and is still a work in-progress for Docker CLI's implementation as per this PR (which is related to this issue) and its related issues/PRs (which I won't mention to avoid polluting their references meaninglessly).

Containerd's imgcrypt

As of Oct 31, 2022, the DockerHub registry supports any OCI artifact, including the Encrypted Layer mediatype used by Containerd's imgcrypt - as per my testing, cannot confirm with 100% certainty as I cannot find official documentation on it.
In any case, this allows the storage of images encrypted using the implementation of this proposal in DockerHub.

This also allows for encryption and decryption of an image, both in entirety and per layer, as mentioned before.

As a non-core subproject of containerd, its encryption features are not included in the main project, which means that Docker has no access to them, even if containerd is its container runtime.

After watching the DockerCon video, I've found out that Brandom Lum is one of the primary contributors and one of the maintainers of imgcrypt.

Integration into Rigel

From what I can understand, given that there is no Docker CLI integration, due to waiting on ongoing replacement of Moby's backend with containerd, as mentioned in this comment, there is no support in Docker or BuildKit for the encryption features, and therefore no API from Python-On-Whales to use in Rigel.

Conclusion

In terms of the end goal of this issue, any of the encryption/secret solutions described only address the first out of the 2 discussed approaches:

1. Encrypt before pushing to registry and pull and decrypt to use the container
2. Allow for normal use of the container and its applications, but without allowing access to secrets/source code/etc used by the applications in the container

Given these problems, how can I progress with the actual implementation? Which of these tools other than Docker can be integrated with the Rigel pipeline? And how?

[rarrais]

Hi @gsilva00 , thanks for the very in-depth analysis.

One question: is the encryption in transit exclusive to Podman, i.e., it is not available in Docker?

If it is exclusive to Podman, I would say it is not something we should support, as the majority of the user base will be most likely using Docker.

As per your concluding question, I would say that we should focus on topic 2, i.e., implementing the secrets feature allowing users to store sensitive data in a protected directory.

[gsilva00]

Hi,

Both the encryption in transit and the secret feature are exclusive to Podman.
Docker currently lacks any built-in encryption features. This makes it impossible to integrate into our current infrastructure, as far as I understand.

Docker has a similar feature to Podman Secrets called Docker Secrets.
But Docker Secrets is exclusive to Docker Swarm. This is a mode in Docker to manage multiple Docker Engine instances (multiple Docker daemons) in clusters, usually across multiple machines. I don't think this is what we're using or intending to use.

[rarrais]

Hi,

Both the encryption in transit and the secret feature are exclusive to Podman. Docker currently lacks any built-in encryption features. This makes it impossible to integrate into our current infrastructure, as far as I understand.

Hi @gsilva00. Thanks, agreed.

Docker has a similar feature to Podman Secrets called Docker Secrets. But Docker Secrets is exclusive to Docker Swarm. This is a mode in Docker to manage multiple Docker Engine instances (multiple Docker daemons) in clusters, usually across multiple machines. I don't think this is what we're using or intending to use.

Have you checked build-level secrets? I thought this is what you were referring to.

[gsilva00]

I hadn't checked that yet! It does seems more promising.
I will investigate and provide more feedback as soon as I can.