possibility to configure base docker image for functions

[cellerich]

🚀 Feature

Why the Issue

My use case is a Python function which makes use of Pandas and Numpy, two widely used libraries when it comes to data analysis and manipulation. When I "preload" those libraries trough requirements.txt it bloats up the .appwrite directory and the resulting tar file to 120MB. So I have to extend the _APP_STORAGE_LIMIT variable to be able to load the function at all.
Even with that "workaround" the function will not run because the Python base image is python:3.8-alpine and is missing some libraries for Pandas to run.

Proposal

Would it be possible to have a configuration option to set the base image of functions to something else (public or private registry).
One possibility would be trough environment variables. Or any other means of "Admin" configuration would do.
I understand that the image would need some extra care that in the case of Python the following would be set:

WORKDIR /usr/local/src/
ENV PYTHONPATH "${PYTHONPATH}:/usr/local/src/.appwrite"

Have you spent some time to check if this issue has been raised before?

yes

Have you read the Code of Conduct?

yes

Pitch

I think the more people are using AppWrite and the phenomenal functions the more we will see some limitations of the fixed base images. Therefore I think it might be a good enhancement to allow Administrators of AppWrite installations to extend or alter the predefined base images.

[eldadfux]

This is something we have discussed internally and seems like an interesting idea that will allow huge flexibility.

Three main things we would probably need to achieve this feature:

• Securely store credentials of custom docker registries (can be add as an Images tab alongside Functions)
• Storing custom images info on project level
• Adding ability to control the warmup of a single/multiple functions workers from the project level

Open questions:

• This might open the way for a lot of disk usage if used with a lot of images, is there any limits we should apply?
• Should we add UI components to allow visibility for the current disk space consumed by warmed up images?

@cellerich what were the missing Alpine libraries in your use-case?

[cellerich]

@eldadfux there is a whole discussion about Alpine not the perfect fit to run Pandas on Stackoverflow

However one of the suggestions is to add the py3-pandas@community package and do not include Pandas in the requirements.txt file.

Currently I will try it out and make my own image with the following dockerfile:

FROM python:3.8-alpine

LABEL maintainer="cello@cellerich.ch"

RUN echo "@community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories \
&& apk add py3-pandas@community
ENV PYTHONPATH="/usr/lib/python3.8/site-packages"

RUN apk add tar

RUN mkdir /usr/local/src

WORKDIR /usr/local/src/

ENV PYTHONPATH "${PYTHONPATH}:/usr/local/src/.appwrite"

Will report my findings later...

[cellerich]

@eldadfux i made a working docker image an pushed it to docker hub:
-> https://hub.docker.com/r/cellerich/env-python-3-8-pandas

[cellerich]

with Pandas installed the image gets bigger:

compared with the current Python environment image:

I don't know how many Appwrite users are depending on Pandas with their Python functions but it would be helpful if you could maybe change the base image to include the package?

[JaviBonilla]

Deployment of custom Docker images would be a great addition to Appwrite. The installation of some Python packages require compilation of C/C++ code or the installation of external libraries. This cannot be done without installing the appropriate software in the Docker image.

Most cloud platforms support both approaches, for example in Google Cloud, you can deploy traditional functions (Cloud Function) and custom Docker images (Cloud Run).

[stnguyen90]

Related issue for Python: open-runtimes/open-runtimes#86

[jj10133]

I'm currently trying to use uber's h3 library however the build is failing during deploying the function

[sheecegardezi]

I am currently using the self-hosted version and have a specific requirement for installing certain modules in the container where the Python function is running. To address this, it would be highly beneficial if I could edit the base image used to create the Python function environment.
As a workaround (we are providing a requirement.txt file during build time, which installs the required packages in the function) similarly, we could have a startup.sh script that installs some packages, edits environment variables, and performs other tasks during the build process.
Maybe we can create a pipy package that runs cmds on OS as part of its setup and that can be used to initialize the system.

[devloic]

Yes, custom images definitely needed please. In my case I need the flutter sdk to be able to build flutter apps for a "CI" like tool.

[ZachHandley]

Trying to use Llama Index in a function, depends on pandas, anything I can do to help speed this up?

[binaryfire]

We ended up moving to OpenFaas for functions. Custom images are essential for serverless applications:

https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html
https://cloud.google.com/compute/docs/images/create-custom

Hope Appwrite adds this at some point. Would be better to not have to run a separate service of course

[ZachHandley]

Okay so the new function runtimes I got all excited about this getting fixed, and yet we still have to use OpenRuntimes Images, please let us use custom images as Torch is unable to be installed on Alpine linux and I'm really salty. I am tired of being limited. Just open it up and let us use custom docker images as bases.

[ZachHandley]

Not to mention the Python Openruntime uses Flask 🤮

[elivesay]

Any updates on this? Any other way to use Torch in App Write?