Creating a self-contained MicroHs binary with embedded libraries

[shakfu]

This document describes how to create a self-contained MicroHs-based application that embeds all MicroHaskell libraries and can compile programs to standalone executables without any external file dependencies.

Background

mhs-midi is a MIDI-oriented music programming environment built on MicroHs. It is one of several language implementations that compile into convenient, self-contained executables. In its initial form, mhs-midi required extensive manual configuration, including specifying multiple directories such as MHSDIR, which points to the original MicroHs installation. To simplify this setup, a Python wrapper script was introduced. This script handled environment configuration and served as a REPL and compiler frontend for running and compiling MIDI-focused Haskell programs, while also integrating with the C/C++ libremidi library via FFI.

Despite these improvements, mhs-midi remained an outlier: relocating it without friction was still difficult. Ideally, it should be distributable as a single executable that users can download and run immediately, without installing MicroHs or configuring MHSDIR. To explore this possibility, an experiment was designed with the following objectives:

1. Single-binary distribution — no external files or dependencies required
2. Full featured — REPL, run, and compile modes all work
3. No MicroHs source changes — achieve embedding via C-level interception
4. Standalone compilation — user programs can be compiled into independent executables

The Approach: Virtual Filesystem with fmemopen

MicroHs reads library files through the mhs_fopen FFI function in eval.c. The proposed approach intercepts this function to serve embedded files from memory.

Step 1: Embed Files as C Arrays

A script converts all .hs files to a C header. We provide both Python and C implementations:

# scripts/embed_libs.py (simplified)
def escape_c_string(content: bytes) -> str:
    """Escape raw bytes for C string literal."""
    result = []
    for byte in content:
        if byte == ord('\\'):
            result.append('\\\\')
        elif byte == ord('"'):
            result.append('\\"')
        elif byte == ord('\n'):
            result.append('\\n')
        elif byte < 32 or byte > 126:
            # Non-printable: use octal escape
            result.append(f'\\{byte:03o}')
        else:
            result.append(chr(byte))
    return ''.join(result)

# Generate header
for vfs_path, full_path, content in files:
    escaped = escape_c_string(content)
    f.write(f'    {{ "{vfs_path}", "{escaped}", {len(content)} }},\n')

This generates a header like:

// mhs_embedded_libs.h (generated)
typedef struct {
    const char* path;
    const char* content;
    size_t length;
} EmbeddedFile;

static const EmbeddedFile embedded_files[] = {
    { "lib/Prelude.hs", "module Prelude...", 12345 },
    { "lib/Data/List.hs", "module Data.List...", 5678 },
    // ... 240 more entries
    { NULL, NULL, 0 }  // Sentinel
};

C Implementation (for MicroHs Integration)

Provide a pure C implementation (scripts/embed_libs.c) suitable for integration into MicroHs itself:

// Core byte escaping - handles UTF-8 correctly
static int escape_byte(unsigned char byte, char* buf) {
    switch (byte) {
        case '\\': return sprintf(buf, "\\\\");
        case '"':  return sprintf(buf, "\\\"");
        case '\n': return sprintf(buf, "\\n");
        default:
            if (byte < 32 || byte > 126)
                return sprintf(buf, "\\%03o", byte);  // Octal escape
            buf[0] = byte; buf[1] = '\0';
            return 1;
    }
}

Usage:

# Compile the tool
gcc -o embed_libs scripts/embed_libs.c

# Generate header with all dependencies
./embed_libs mhs_embedded.h lib/ \
    --runtime src/runtime/ \
    --header midi_ffi.h \
    --lib liblibremidi.a \
    --lib libmidi_ffi.a

Output:

Collecting .hs files from lib/...
Collecting runtime files from src/runtime/...
Embedding headers:
  midi_ffi.h (2100 bytes)
Embedding libraries:
  liblibremidi.a (1143392 bytes)
  libmidi_ffi.a (18256 bytes)
Generated: mhs_embedded.h (265 files, 2447946 bytes)

The C implementation has no dependencies beyond libc and is portable to any POSIX system.

Step 2: Virtual Filesystem Using fmemopen

The VFS serves embedded files as FILE* streams:

// vfs.c
#include "mhs_embedded_libs.h"

#define VFS_VIRTUAL_ROOT "/mhs-embedded"

static const EmbeddedFile* find_embedded_file(const char* rel_path) {
    for (const EmbeddedFile* ef = embedded_files; ef->path; ef++) {
        if (strcmp(ef->path, rel_path) == 0) {
            return ef;
        }
    }
    return NULL;
}

FILE* vfs_fopen(const char* path, const char* mode) {
    // Check if path starts with our virtual root
    if (strncmp(path, VFS_VIRTUAL_ROOT, strlen(VFS_VIRTUAL_ROOT)) == 0) {
        const char* rel_path = path + strlen(VFS_VIRTUAL_ROOT) + 1;
        const EmbeddedFile* ef = find_embedded_file(rel_path);
        if (ef) {
            // Serve from memory - no disk access!
            return fmemopen((void*)ef->content, ef->length, "r");
        }
        return NULL;  // File not found in VFS
    }
    // Fall back to real filesystem
    return fopen(path, mode);
}

Step 3: Intercept MicroHs FFI

As MicroHs calls mhs_fopen for all file operations, rename the original and provide an override:

# scripts/patch_eval_vfs.py
# Renames mhs_fopen to mhs_fopen_orig in eval.c
# Adds forward declaration for our override

// mhs_ffi_override.c
// Forward declaration of original (renamed by patch script)
from_t mhs_fopen_orig(int s);

// Our override that checks VFS first
from_t mhs_fopen(int s) {
    const char* path = mhs_to_Ptr(s, 0);
    const char* mode = mhs_to_Ptr(s, 1);

    // Try VFS first
    FILE* result = vfs_fopen(path, mode);

    return mhs_from_Ptr(s, 2, result);
}

Step 4: Set MHSDIR to Virtual Root

At startup, point MHSDIR to the virtual filesystem:

// main.c
int main(int argc, char** argv) {
    vfs_init();
    setenv("MHSDIR", "/mhs-embedded", 1);
    return mhs_main(argc, argv);
}

MicroHs now constructs paths like /mhs-embedded/lib/Prelude.hs, which the VFS intercepts and serves from memory.

Challenge 1: UTF-8 Encoding Bug

Symptom: After loading ~163 of 185 modules, we got ERR: getb_utf8.

Root Cause: The embedding script had two bugs:

# BUG 1: Wrong length - counts Unicode characters, not bytes
content = file.read_text()
len(content)  # Returns 4971 characters, but file has 4995 bytes!

# BUG 2: Invalid octal escapes for high Unicode codepoints
f'\\{ord(char):03o}'  # ord('...') = 8801 -> '\21141' (invalid!)

The file Data/Bifunctor.hs contains the Unicode character ... (U+2261). Python's len() on a string counts code points, not bytes. And C octal escapes only support values 0-377 (0-255).

Solution: Work with bytes, not strings:

def escape_c_string(content: bytes) -> str:
    for byte in content:  # Iterate bytes, not characters
        if byte > 126:
            result.append(f'\\{byte:03o}')  # Always valid 0-255

Challenge 2: Compilation to Executable

When users compile with -o program (not -o program.c), MicroHs invokes cc which needs real files on disk. The VFS only works within the MicroHs process.

Solution: Detect compilation mode and extract to temp directory:

static int needs_extraction(int argc, char** argv) {
    for (int i = 1; i < argc; i++) {
        if (strncmp(argv[i], "-o", 2) == 0) {
            const char* output = (argv[i][2]) ? argv[i] + 2 : argv[i + 1];
            if (output) {
                size_t len = strlen(output);
                // .c output works with VFS, executable needs extraction
                if (len >= 2 && strcmp(output + len - 2, ".c") == 0)
                    return 0;
                return 1;
            }
        }
    }
    return 0;
}

int main(int argc, char** argv) {
    vfs_init();

    if (needs_extraction(argc, argv)) {
        char* temp_dir = vfs_extract_to_temp();
        setenv("MHSDIR", temp_dir, 1);
        // ... run mhs_main ...
        vfs_cleanup_temp(temp_dir);
    } else {
        setenv("MHSDIR", "/mhs-embedded", 1);
        // ... run mhs_main ...
    }
}

Challenge 3: Linking MIDI Libraries

For our MIDI project, compiled executables need to link against libremidi and other libraries. We embed the static libraries (.a files) and inject linker flags:

if (linking_midi) {
    // Extract includes .a files to temp_dir/lib/
    new_argv[j++] = "-optl";
    new_argv[j++] = lib_path;  // e.g., /tmp/mhs-xxx/lib/libmidi_ffi.a

    #ifdef __APPLE__
    new_argv[j++] = "-optl"; new_argv[j++] = "-framework";
    new_argv[j++] = "-optl"; new_argv[j++] = "CoreMIDI";
    #endif
}

Generalizing for Other Projects

This approach turned out to be quite useful, and can be adapted for any MicroHs-based application:

1. Minimal Standalone (REPL/Run only)

If you only need REPL and -r (run) modes, you just need:

• embed_libs.py or embed_lib.c - embed your .hs libraries
• vfs.c - the fmemopen-based VFS
• mhs_ffi_override.c - intercept mhs_fopen
• patch_eval_vfs.py - rename original mhs_fopen

This gives you a ~1MB overhead for the MicroHs standard library.

2. Full Standalone (with Compilation)

To support -o executable:

• Also embed src/runtime/*.c and src/runtime/*.h
• Add vfs_extract_to_temp() function
• Detect -o without .c suffix and extract

3. With Additional Libraries

If your application has C dependencies:

• Embed static libraries (.a files) as binary content
• Inject -optl flags when compiling to executable
• Include any required headers

Potential MicroHs Enhancement

A potential enhancement to MicroHs itself could be a compile-time option to embed libraries:

mhs --embed-libs=./lib --embed-runtime -o my_repl MyRepl.hs

This would generate a single C file with embedded libraries, eliminating the need for external VFS machinery. The generated REPL would be truly standalone.

The C implementation of embed_libs.c (~500 lines, no dependencies) could be integrated directly into MicroHs to provide this functionality. It handles:

• Recursive directory traversal for .hs and .hs-boot files
• Runtime C/H file embedding (--runtime)
• Static library embedding (--lib)
• Header file embedding (--header)
• Proper UTF-8 byte escaping
• String literal chunking for C89 compatibility

Results

Binary	Size	Capabilities
mhs-midi	782KB	Requires MHSDIR (auto-detected)
mhs-midi-standalone	3.2MB	Fully self-contained

The standalone binary:

• Embeds 273 files (~2.5MB of content)
• Starts REPL in ~0.5s (with -C caching)
• Can compile HelloMidi.hs to 859KB standalone executable
• Works on macOS and Linux (Windows has separate MicroHs build issues)

Files

scripts/
    embed_libs.py         # Convert files to C header (Python)
    embed_libs.c          # Convert files to C header (C, for MicroHs integration)
    patch_eval_vfs.py     # Patch eval.c for override

projects/mhs-midi/
    vfs.c, vfs.h          # Virtual filesystem
    mhs_ffi_override.c    # FFI intercept
    mhs_midi_standalone_main.c  # Entry point

Acknowledgments

Thanks to Lennart Augustsson for creating MicroHs, which makes this kind of embedding possible through its use of combinators, clean FFI design and single-file C output.

The exploratory work to develop the standalone mhs-midi implementation was greatly accelerated by the use of claude-code.

References

• MicroHs - A small Haskell compiler
• fmemopen(3) - Memory stream interface
• midi-langs - MIDI programming languages including mhs-midi

[augustss]

Amazing hack! 👍

Your use case is a bit unusual. You want to build custom mhs which does not depend on external files. I can see how this could be useful in other situations too. I might add some support for that, actually.

Until then, I think you should publish your work on hackage. I'm not sure exactly how, though.

I think you could make something different by embedding entire .pkg files with mhs and preloading it. That should be faster.

[shakfu]

Thank you for your positive feedback.

I agree, it was an unusual use-case. Indeed, if I had been developing it in isolation, I probably wouldn't have wanted to go in that direction. But also the nice thing about MicroHs is that it is not so complex that it would dissuade you up front from trying this "hack".

I think you could make something different by embedding entire .pkg files with mhs and preloading it. That should be faster.

Thanks for the advice I will look into the .pkg file structure. Presumably they are already compressed.

Also fyi, as an addendum to my post above: I subsequently managed to compress the embedded Haskell Modules, application modules, ffi_wrapper and and static library using zstd.

The approach I took is that prior to compilation, there is one off compression step of these files to embed them in the c-header, and for the runtime decompression, a vfs variant with zstd decompression support was created. This requires adding a 900k single-file zstd decompressor (zstddeclib.c) to the code base.

The results are promising: binary size reduced from 3.2 MB to 1.3 MB with embedded files compressed 5.2x (2.5 MB to 367 KB including 112 KB dictionary).

[augustss]

I converted the issue to a discussion.

[shakfu]

Thanks. That makes sense.

[shakfu]

@augustss wrote

I think you could make something different by embedding entire .pkg files with mhs and preloading it. That should be faster.

Do you mean macOS .pkg files here?

[augustss]

No, I mean the .pkg files from MHS.

[shakfu]

Understood. Thanks. I'll look more closely into those.

[shakfu]

I'm not able to build a .pkg for some reason.

I tried with this following script run from the root of the midi-langs project ...

#!/usr/bin/env bash

THIRDPARTY=`pwd`/thirdparty
MHS_MIDI=`pwd`/projects/mhs-midi

MHSDIR=${THIRDPARTY}/MicroHs ${THIRDPARTY}/MicroHs/bin/mhs \
	-a ${MHS_MIDI}/lib \
	-P music.pkg \
		${MHS_MIDI}/lib/Music.hs

Note that Music.hs is a self-contained pure module.

.. and I get this error:

~/projects/midi-langs/thirdparty/MicroHs/bin/mhs: uncaught exception: error: no location: undefined export: main

[augustss]

You need something like this mhs -Pmusic -i${MHS_MIDI}/lib Music -o music.pkg. The name after -P is the package name, unrelated to the file name.

[shakfu]

Thanks for your advice. I managed to get a package compiled after some time. There was a tricky bug that prevented one of the files from being compiled into the package which turned out to be caused by a name collision of an inner function pattern used in one of my functions with pattern ErrorCall :: String -> ErrorCall in Control/Exception/Internal.hs:

thirdparty/MicroHs/bin/mhs: uncaught exception: error: "projects/mhs-midi/lib/MidiPerform.hs": line 402, col 54:
  found:    pattern
  expected: { . LQIdent ( UQIdent [ literal _primitive @ (# \ case let if QualDo do mdo QSymOper ` :: ∷ where ; eof

After figuring this out, the compilation worked well with this script:

#!/usr/bin/env bash

# Note -P <entry> must be of form <name>-<semver>

PKG_VER=0.1.0
PKG_NAME=music
THIRDPARTY=`pwd`/thirdparty
MHS_MIDI=`pwd`/projects/mhs-midi

MHSDIR=${THIRDPARTY}/MicroHs ${THIRDPARTY}/MicroHs/bin/mhs -P${PKG_NAME}-${PKG_VER} -i${MHS_MIDI}/lib \
	Async Midi MidiPerform Music MusicPerform \
	-o ${PKG_NAME}.pkg

I think embedding the .pkg format into an executable would probably work, but would only make sense if you could compile the standard library into the same format. I have no idea if this is possible.

[augustss]

When you do make install, there will be a base.pkg installed which is the base library.
You can then pass a -p flag to force a package to be loaded on start.

[arossato]

There was a tricky bug that prevented one of the files from being compiled into the package which turned out to be caused by a name collision of an inner function pattern used in one of my functions with pattern ErrorCall :: String -> ErrorCall in Control/Exception/Internal.hs:

thirdparty/MicroHs/bin/mhs: uncaught exception: error: "projects/mhs-midi/lib/MidiPerform.hs": line 402, col 54:
  found:    pattern
  expected: { . LQIdent ( UQIdent [ literal _primitive @ (# \ case let if QualDo do mdo QSymOper ` :: ∷ where ; eof

I reported it yesterday on the issue tracker of your project...

A really nice project. And thanks for this guide too.

[augustss]

Yes, pattern is a reserved word in MicroHs. That is not going to change.

[shakfu]

I'll try to experiment over the weekend with embedding .pkg using the same process. Let's see where it goes.

[augustss]

The advantage of using the .pkg file is that startup would be much faster.

[shakfu]

Embedding Precompiled .pkg Files in a Standalone MicroHs Binary

This document provides some details about the development of MHS_USE_PKG mode for the standalone mhs-midi variants (mhs-midi-pkg, mhs-midi-pkg-zstd), self-contained MicroHs REPLs with embedded MIDI support.

The goal was to improve on the earlier demonstration of embedding the base haskell source files, app source files, MicroHS runtime, and statically compiled FFI dependencies.

Summary

The standalone mhs-midi binaries embed Haskell source files using a Virtual Filesystem (VFS), but this requires parsing and compiling ~274 modules on first run, resulting in a ~20 second cold-start delay. This document provides details about the development of MHS_USE_PKG mode, which embeds precompiled .pkg files instead of source files.

Key Results:

• 20x faster cold start: Reduced from ~20s to ~1s
• Full functionality preserved: REPL, compilation to executables, and all MIDI features work identically
• Self-contained: No external dependencies required
• The use of .pkg improves startup time if a .mshcache is not available. If it subsequently avialable then using then startup times for the other variants are almost equal.

Implementation Challenges Solved:

1. Virtual directory operations for package discovery (MicroHs scans directories, not just files)
2. Module-to-package mapping via .txt files (185+ mapping files needed)
3. Hybrid embedding for compilation support (runtime source files still required for cc)

When to Use:

Build Mode	Cold Start	Warm Cache	Binary Size
Default (.hs embedding)	~20s	~0.5s	~3.3 MB
MHS_USE_PKG	~1s	~0.95s	~4.8 MB
MHS_USE_ZSTD	~20s	~0.5s	~1.3 MB
MHS_USE_PKG + MHS_USE_ZSTD	~1s	~0.95s	~3.1M MB

• MHS_USE_PKG + MHS_USE_ZSTD: Best for distribution: with best balance of fast startup and small size (~1s startup, ~3.1M MB binary)
• Default mode: Suitable for development with persistent .mhscache (faster warm starts, ~3.3 MB binary)

Background: The Cold Start Problem

The standalone mhs-midi binaries embed all necessary Haskell source files (~274 .hs files from MicroHs/lib and custom MIDI libraries) using a Virtual Filesystem (VFS). On first run, MicroHs must parse and compile every module before presenting the REPL prompt. This takes approximately 20 seconds.

Subsequent runs load from .mhscache in ~0.5 seconds, but the cold-start penalty is problematic for:

• First-time users evaluating the tool
• CI/CD pipelines without persistent caches
• Distribution to end users

MicroHs Package System

MicroHs supports precompiled packages via the -P flag during compilation:

PKG_VER=0.1.0
PKG_NAME=music
THIRDPARTY=`pwd`/thirdparty
MHS_MIDI=`pwd`/projects/mhs-midi

MHSDIR=${THIRDPARTY}/MicroHs ${THIRDPARTY}/MicroHs/bin/mhs \
    -P${PKG_NAME}-${PKG_VER} \
    -i${MHS_MIDI}/lib \
    -o ${PKG_NAME}-${PKG_VER}.pkg \
    Async Midi MidiPerform Music MusicPerform

This creates a .pkg file containing serialized, already-compiled modules.

To pre-compile the base package or MicroHs standard library

cd path/to/MicroHs
make install

Then it becomes possible to load packages at runtime with -p:

~/.mcabal/bin/mhs -pbase-0.15.2.0

# OR

~/.mcabal/bin/mhs -pbase

The key point is that in the default installation of the mhs binary, MHSDIR is set automatically. If MHSDIR is overridden by passing it as an environment variable, it changes not just the package search path but also where MicroHs looks for source files, causing the above commands to fail.

When using mhs as a local dev binary with MHSDIR set, you can specify the package path explicitly:

MHSDIR=./thirdparty/MicroHs ./thirdparty/MicroHs/bin/mhs \
  -a"$HOME/.mcabal/mhs-0.15.2.0" -pbase

The -a flag sets the package search path (looks in <path>/packages/), and -p loads named packages.

The Local Dev Binary Problem

However, there's a critical limitation: the local dev binary still recompiles all modules from source, even when packages are preloaded:

Installed binary (works correctly):

~/.mcabal/bin/mhs -pbase
Loading package /Users/sa/.mcabal/mhs-0.15.2.0/packages/base-0.15.2.0.pkg
Type ':quit' to quit, ':help' for help
>

Local dev binary (recompiles everything despite -p flag):

MHSDIR=./thirdparty/MicroHs ./thirdparty/MicroHs/bin/mhs -a"$HOME/.mcabal/mhs-0.15.2.0" -pbase
Welcome to interactive MicroHs, version 0.15.2.0
    importing done Data.Bool_Type, 3ms
    importing done Data.Ordering_Type, 2ms
    importing done Primitives, 148ms
    ... (continues loading all modules from source)

When MHSDIR is overridden, it changes not just the package search path but also where MicroHs looks for source files. The compiler prefers source files from MHSDIR/lib/ over preloaded package contents.

This limitation is why the VFS embedding approach was necessary - it's the only way to achieve fast startup with a fully self-contained binary that doesn't depend on external paths.

Implementation Strategy

The existing VFS intercepts mhs_fopen FFI calls to serve embedded files via fmemopen(). Simply embedding .pkg files and using -a/-p flags wouldn't work because of the local dev binary limitation described above.

The solution was to embed packages AND intercept all file operations so MicroHs has no access to external source files - forcing it to use the preloaded packages.

Challenge 1: Package Discovery via Directory Operations

Initial testing with embedded .pkg files failed immediately:

Package not found: base

Debugging revealed that MicroHs discovers packages by scanning the packages/ directory:

-- From MicroHs/src/MicroHs/Packages.hs
findPkgs :: FilePath -> IO [FilePath]
findPkgs path = do
    let pkgdir = path </> "packages"
    fs <- getDirectoryContents pkgdir
    return [pkgdir </> f | f <- fs, ".pkg" `isSuffixOf` f]

This calls opendir/readdir/closedir - not fopen. The VFS needed to intercept directory operations.

Solution: Virtual Directory Listing

Extended mhs_ffi_override.c to intercept directory operations:

typedef struct {
    int index;
    int is_virtual;
} VirtualDir;

void* mhs_opendir(const char* path) {
    if (is_virtual_packages_dir(path)) {
        VirtualDir* vd = malloc(sizeof(VirtualDir));
        vd->index = 0;
        vd->is_virtual = 1;
        return vd;
    }
    return opendir_orig(path);
}

struct dirent* mhs_readdir(void* dirp) {
    VirtualDir* vd = dirp;
    if (vd->is_virtual) {
        // Return embedded package names from table
        if (embedded_packages[vd->index].path) {
            static struct dirent de;
            strcpy(de.d_name, embedded_packages[vd->index++].filename);
            return &de;
        }
        return NULL;
    }
    return readdir_orig(dirp);
}

Packages were now discovered. But loading still failed.

Challenge 2: Module-to-Package Mapping

Module not found: Prelude

MicroHs doesn't scan package contents to find modules. Instead, it uses .txt mapping files:

~/.mcabal/mhs-0.15.2.0/
    Prelude.txt          # Contains: "base-0.15.2.0.pkg"
    Data/List.txt        # Contains: "base-0.15.2.0.pkg"
    Control/Monad.txt    # Contains: "base-0.15.2.0.pkg"
    ...

When searching for module Data.List, MicroHs looks for Data/List.txt and reads the package name from it.

Solution: Embed .txt Mapping Files

Updated mhs-embed.c (with --pkg-mode) to collect and embed all .txt files from the MicroHs installation (approximately 190 files):

def collect_txt_files(base_dir: Path) -> list[tuple[str, Path]]:
    """Collect all .txt module mapping files."""
    txt_files = []
    for txt_path in base_dir.rglob("*.txt"):
        if "packages" in txt_path.parts:
            continue  # Skip .txt inside package directories
        rel_path = txt_path.relative_to(base_dir)
        txt_files.append((str(rel_path), txt_path))
    return sorted(txt_files)

For custom modules (Midi, Music, etc.), synthetic .txt entries are generated:

def generate_music_txt_files(pkg_name: str, modules: list[str]):
    return [(f"{module}.txt", pkg_name.encode()) for module in modules]

The VFS serves these as small strings:

static const char txt_Prelude_txt[] = "base-0.15.2.0.pkg";
static const char txt_Data_List_txt[] = "base-0.15.2.0.pkg";
// ...

Challenge 3: Compilation Support

With packages loading correctly, the REPL started in ~1 second. But the compilation test failed:

mhs-midi-pkg -o /tmp/test /tmp/Test.hs
# Error: Cannot find runtime files for cc

MicroHs compiles Haskell to C, then invokes cc with runtime source files (eval.c, main.c, headers). In package mode, these weren't embedded.

Solution: Hybrid Embedding

The solution embeds both precompiled packages AND runtime source files:

def collect_runtime_files(base_dir: Path) -> list[tuple[str, Path]]:
    """Collect runtime source files from packages/mhs-X.Y.Z/data/src/runtime/"""
    runtime_files = []
    packages_dir = base_dir / "packages"
    for mhs_dir in packages_dir.iterdir():
        if mhs_dir.name.startswith("mhs-"):
            runtime_dir = mhs_dir / "data" / "src" / "runtime"
            if runtime_dir.exists():
                for file_path in runtime_dir.rglob("*"):
                    if file_path.is_file():
                        rel_path = file_path.relative_to(runtime_dir)
                        vfs_path = f"src/runtime/{rel_path}"
                        runtime_files.append((vfs_path, file_path))
    return sorted(runtime_files)

Static libraries are also embedded for linking:

# CMake handles this automatically, but the equivalent command is:
./mhs-embed output.h --pkg-mode \
    --pkg packages/base-0.15.2.0.pkg=build/mcabal/mhs-0.15.2.0/packages/base-0.15.2.0.pkg \
    --pkg packages/music-0.1.0.pkg=build/music-0.1.0.pkg \
    --txt-dir build/mcabal/mhs-0.15.2.0 \
    --music-modules music-0.1.0.pkg:Async,Midi,MidiPerform,Music,MusicPerform \
    --lib build/libmidi_ffi.a \
    --lib build/liblibremidi.a \
    --header src/midi_ffi.h

During compilation, the VFS extracts these to a temporary directory for cc to consume.

Final Embedded Content (MHS_USE_PKG)

Category	Count	Size
.pkg packages	2	2.7 MB
.txt module mappings	190	3.2 KB
Runtime source files	33	1.5 MB
Static libraries	4	1.2 MB
Header files	1	2.1 KB

Performance Comparison

Startup Time

Build Mode	Cold Start	Warm Cache	Binary Size
Default (.hs embedding)	~20s	~0.5s	~3.3 MB
MHS_USE_PKG	~1s	~0.95s	~4.8 MB
MHS_USE_ZSTD	~20s	~0.5s	~1.3 MB
MHS_USE_PKG + MHS_USE_ZSTD	~1s	~0.95s	~3.1M MB

Key insight: .mhscache is more efficient than .pkg for warm starts because it's a single pre-processed blob. The PKG mode's value is eliminating the 20-second cold-start penalty. Note that MHS_USE_ZSTD compresses source files but still requires parsing and compilation, so cold-start time remains ~20s.

Binary Size

Build Mode	Binary Size
Default (.hs embedding)	~3.3 MB
MHS_USE_PKG	~4.8 MB
MHS_USE_ZSTD	~1.3 MB
MHS_USE_PKG + MHS_USE_ZSTD	~3.1 MB

Feature Matrix

Feature	Default	MHS_USE_PKG	MHS_USE_ZSTD	Both
Fast cold start	No	Yes	No	Yes
Smallest binary	No	No	Yes	No
Compile to executable	Yes	Yes	Yes	Yes
No external dependencies	Yes	Yes	Yes	Yes

Error Messages and Debugging

Error messages and debugging information are functionally identical across all build modes:

• Runtime errors in user code: File, line, and column are shown identically
• Type errors: Full constraint information preserved
• Stack traces: MicroHs shows the error location only (no full call stack in any mode)

The only difference is in errors originating from Prelude/library code:

# Source embedding mode (VFS path):
"/mhs-embedded/lib/Data/List.hs",389:11: head: empty list

# PKG mode (original compile-time path):
"/Users/.../thirdparty/MicroHs/lib/Data/List.hs",389:11: head: empty list

Line and column numbers are preserved in both cases. There is no degradation in debugging capability when using precompiled packages.

Build Instructions

# Build all variants
make mhs-midi-all

# Or build specific variants:
make mhs-midi-src        # Source embedding (uncompressed)
make mhs-midi-src-zstd   # Source embedding (compressed, smallest)
make mhs-midi-pkg        # Package embedding (fast startup)
make mhs-midi-pkg-zstd   # Package + compression (recommended)

# Or using cmake directly:
cmake --build build --target mhs-midi-all
cmake --build build --target mhs-midi-pkg-zstd

Variant Summary:

• mhs-midi-src: Embeds .hs source files, ~20s cold start, ~3.3MB
• mhs-midi-src-zstd: Compressed source, ~20s cold start, ~1.3MB (smallest)
• mhs-midi-pkg: Precompiled packages, ~1s cold start, ~4.8MB
• mhs-midi-pkg-zstd: Compressed packages, ~1s cold start, ~3.1MB (recommended)

Prerequisites for MHS_USE_PKG

The package mode requires MicroHs to be built (but not installed):

cd thirdparty/MicroHs
make

The CMake build handles everything else automatically:

• Builds base-0.15.2.0.pkg locally in build/mhs-base-src/dist-mcabal/
• Installs packages to build/mcabal/ (not ~/.mcabal)
• Generates module mapping .txt files that mhs-embed.c --pkg-mode reads

This keeps the source tree clean and avoids modifying user's home directory.

Conclusion

Embedding precompiled .pkg files required solving three unexpected challenges:

1. Virtual directory operations for package discovery
2. Module-to-package .txt mapping files
3. Hybrid embedding for compilation support

The result is a ~20x improvement in cold-start time (20s to 1s) while maintaining full functionality including the ability to compile Haskell programs to standalone executables.

For distribution to end users, mhs-midi-pkg-zstd is recommended.

[shakfu]

@augustss I posted a full description of steps to embed .pkg files in mhs-midi above. This is currently working and tested on linux and macOS. Windows compiles the non-embedded case but not the embedded ones because it doesn't support fmemopen.

[shakfu]

@augustss and @arossato

To help in making this method more general for other MicroHs users, I have created a project, mh-embed, which creates MicroHs standalones outside the context of my midi-langs project.

So far it can create source-based standalones only. I will add .pkg support later.

[augustss]

I'm considering adding zip files as "file system" to mhs. Then we could just embed an entire zip file.

[shakfu]

Yes, that would work. The tricky part is not having an equivalent to fmemopen on windows (I struggled with that). A fallback could be to unzip to tmp folder or files, but decompression to file-readable memory is best.

I googled this and found the following:

• https://github.com/Arryboom/fmemopen_windows: an actual implementation 😄
• https://github.com/Cibiv/PDA/blob/master/fmemopen.c: include tmp file fallback for windows.
• https://github.com/openSUSE/libsolv/blob/master/win32/fmemopen.c: another tmp-based implementation

[augustss]

We can do without fmemopen, I think. The file abstraction used in the runtime is a BFILE. It can be different things, eg, a FILE, but also a memory buffer.

[shakfu]

I understand now. You mean BFILE defined in MicroHs/src/runtime/bfile.c

[augustss]

Yes

[shakfu]

@augustss fyi, I have added to mhs-embed the .pkg-based examples. So now you can do:

  make example-pkg-zstd   Build example-pkg-zstd standalone binary
  make example-pkg        Build example-pkg standalone binary
  make example-src-zstd   Build example-src-zstd standalone binary
  make example-src        Build example-src standalone binary
  make example            Build example REPL binary only