SentinelSpawner for JupyterHub

SentinelSpawner is an extension of JupyterHub’s KubeSpawner that handles selected unrecoverable errors during the notebook spawn phase.

It monitors Kubernetes events generated during notebook startup, scans event messages for known error patterns, and halts the spawn process when an unrecoverable issue occurs. Instead of displaying raw technical logs, SentinelSpawner presents a clear, user-friendly error message—and, in some cases, offers guidance on how to resolve the issue before attempting to spawn again. SentinelSpawner also integrates with Prometheus and artificial intelligence (AI) tools to analyze the internal cluster state, further improving the accuracy and usefulness of the feedback provided to users.

Important Notice

SentinelSpawner was developed and validated on the e-INFRA CZ infrastructure, and some of its advanced features (notably Prometheus-based analysis and AI-assisted suggestions) assume the presence of services similar to those available in that environment.

However, SentinelSpawner itself is not tied to e-INFRA CZ and remains a pure extension of KubeSpawner. It can be deployed on any infrastructure where KubeSpawner is used and should correctly detect and handle the configured unrecoverable spawn errors.

If Prometheus and/or an AI endpoint are not configured, SentinelSpawner will gracefully fall back to a generic user-facing message (e.g., requesting the user to lower their resource requirements).

Build

SentinelSpawner is not a standalone binary. It is imported as a Python class and runs within the Hub component of JupyterHub, specifically in the Zero to JupyterHub with Kubernetes (Z2JH) distribution. It functions solely as an extension of KubeSpawner, which serves as the default spawner in that distribution.

To build the Python wheel package, ensure you have the build module installed:

python3 -m pip install build

Then, from the directory containing the pyproject.toml file, run:

python3 -m build

On success, this will produce a wheel file, for example:

dist/sentinel_spawner-1.0.0-py3-none-any.whl

The package name and version can be modified in the pyproject.toml file.

The provided Dockerfile is configured to copy this wheel into the CERIT development version of JupyterHub’s container image and install it using pip from the generated wheel distribution.

If desired, you can comment out the CERIT image and uncomment the official JupyterHub image. Note that when deploying outside the e-INFRA CZ environment, additional configuration (such as authentication for external services) may be required for full functionality.

Build the Docker image with:

docker build -t <image-name> .  # replace <image-name> with your actual image name

Push it to a remote registry with:

docker push <image-name>  # replace <image-name> with your actual image name

Deploy

See deployment/values.yaml for an example configuration illustrating the necessary adjustments to enable SentinelSpawner.

Certain features of SentinelSpawner are configured via environment variables, including:

• the Prometheus endpoint address,
• the AI inference endpoint,
• and the contact email shown in user-facing error messages.

When deploying outside of e-INFRA, simply setting these endpoints may not be sufficient, as external services may require additional authentication (e.g., API keys or tokens), which is environment-specific and not handled automatically by SentinelSpawner.