Skip to content

This repository provides a development environment for data engineering labs.

Notifications You must be signed in to change notification settings

flynn3103/de-labs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Engineering Labs

This repository provides a development environment for data engineering labs.

Prerequisites

  • Python 3.12 or higher
  • k3d (for local Kubernetes cluster)
  • kubectl (for Kubernetes cluster management)
  • uv (Python package installer)

Installation

  1. Clone the repository:

    git clone <repository-url>
    cd de-labs
  2. Create a virtual environment and install dependencies using uv:

    uv venv
    source .venv/bin/activate  # On Unix/macOS
    # OR
    .venv\Scripts\activate  # On Windows
    
    uv pip install .

Cluster Setup

  1. Create a local Kubernetes cluster:

    make create-k3d-cluster

    This will create a k3d cluster with 1 server and 2 agent nodes.

  2. Initialize the namespace:

    make init-namespace

Deploying Services

Deploy All Services

make deploy svc=minio,spark

Deploy Individual Services

# Deploy MinIO only
make deploy svc=minio

# Deploy Spark only
make deploy svc=spark

Undeploying Services

Undeploy All Services

make undeploy svc=minio,spark

Undeploy Individual Services

# Undeploy MinIO only
make undeploy svc=minio

# Undeploy Spark only
make undeploy svc=spark

Cleaning Up

  1. Remove all resources from the current namespace:

    make clean-all
  2. Delete the k3d cluster:

    make delete-k3d-cluster

Troubleshooting

  1. If you encounter issues with the cluster context, ensure you're using the correct kubectl context:

    kubectl config use-context k3d-de-labs
  2. To verify the cluster status:

    make check-k3d-cluster

About

This repository provides a development environment for data engineering labs.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published