Skip to content

BUILDING.md

Latest commit

 

History

History
90 lines (70 loc) · 3.56 KB

BUILDING.md

File metadata and controls

90 lines (70 loc) · 3.56 KB
Raw

Build Instructions

YALS contains two components:

  1. TypeScript code: Universally buildable on any OS
  2. C++ bindings: Requires an OS-specific C++ compiler and additional setup

The C++ bindings need to be built to integrate the llama.cpp library and provide the necessary "glue" required by YALS.

Prerequisites

To get started, install the following prerequisites:

  • Deno
  • A C++ compiler:
    • Windows: Visual Studio 2022 build tools
    • macOS: Xcode command-line tools (xcode-select --install)
    • Linux: GCC (sudo apt install build-essential)
  • CMake:
    • Windows: Installed with Visual Studio build tools
    • macOS (homebrew): brew install cmake
    • Linux: sudo apt install cmake (For Ubuntu 22.04, follow this askubuntu answer to install the latest version)
  • Ninja (Makes builds faster)
    • Windows: winget install -e --id Ninja-build.Ninja
    • macOS (homebrew): brew install ninja
    • Linux: sudo apt install ninja-build
  • sccache (optional, but speeds up subsequent builds)
  • Rust(Used for improved grammar parsing via LLGuidance)

Building

Clone the repository and navigate to the project folder:

git clone https://github.com/theroyallab/YALS.git
cd YALS

All build commands are encapsulated in Deno tasks, similar to npm scripts in NodeJS.

Note

Unlike llama.cpp and its derivatives, YALS uses an extremely fast grammar tool called llguidance for JSON schemas, Regex, and lark grammars.

Due to an extra dependency being required for users systems, llguidance is off by default, but it is highly recommended to turn it on at build time for improved grammar handling.

To enable it, set LLGUIDANCE=1 in your shell before invoking the deno task.

To build the C++bindings:

  • Windows: deno task bindings-win
  • macOS/Linux: deno task bindings

This will invoke CMake to build the bindings and copy the resulting shared libraries to the lib folder.

Optionally, environment variables can be set for certain architectures when building (ex. CUDA):

  • MAX_JOBS: Number of parallel jobs (defaults to the number of CPU cores)
  • LLAMACPP_REPO: Point to a custom repository for llama.cpp (Here be dragons!)
  • LLAMACPP_TAG: Set a specific tag for llama.cpp (Here be dragons!)
  • GGML_CUDA=1: Enables CUDA support
  • CMAKE_CUDA_ARCHITECTURES: Specifies CUDA compute capabilities (defaults to native if using CMake > 3.24)
  • GGML_VULKAN=1: Enables Vulkan Support
  • GGML_HIP=1: Enables HIP ROCM Support (Requires specifying DAMDGPU_TARGETS, Linux only)
  • AMDGPU_TARGETS: Specify ROCM target (example: gfx1030)
  • GGML_SYCL=1: Enables SYCL support for Intel ARC GPUs (Make sure to run source /opt/intel/oneapi/setvars.sh to setup the build environment)
  • LLGUIDANCE=1: (Recommended) Enable llguidance for grammars. Requires Rust on the system. (default 0)

Running

To start the server with necessary permissions:

deno task start

With full permissions (useful for testing new features):

deno run -A main.ts

Packaging

Note

Note: All YALS commits are built via GitHub Actions, so manual packaging is typically unnecessary unless you need to distribute builds with a custom build configuration.

To create a distributable binary:

  1. Run: deno task build to package all TypeScript code into a standalone binary
  2. Zip the following files and directories:
    • YALS(.exe)
    • lib/
    • models/
    • templates/
    • config_sample.yml
  3. Distribute the archive, and the recipient can simply extract and run it.