Skip to content

Commit 9359372

Browse files
copyleftdevclaude
andcommitted
docs: cross-platform installation for macOS, Linux, and Windows
Installation docs previously assumed Linux. Now covers all platforms: README.md: - New Prerequisites section with per-platform build dep table (macOS/Ubuntu/Fedora/Arch/Windows/Docker) - Documents CMake + Go requirement for BoringSSL compilation book/src/getting-started/installation.md: - Full instructions for macOS (Homebrew), Linux (Ubuntu/Fedora/Arch), Windows (native MSVC and WSL2), and Docker - Chrome installation per platform - Troubleshooting section: BoringSSL build failures, Chrome not found, openssl-sys conflicts book/src/contributing/development-setup.md: - Cross-references Installation for platform setup - IDE setup table (VS Code, JetBrains, Neovim, Emacs) - Platform-specific notes for macOS, Windows, Linux Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 777f15b commit 9359372

53 files changed

Lines changed: 691 additions & 154 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,21 @@ A deterministic crawl kernel. Same input + same seed = identical crawl, identica
44

55
Not a crawler. Not a Wayback clone. The canonical memory layer of the web. See [CLAUDE.md](CLAUDE.md) for the full philosophy.
66

7+
## Prerequisites
8+
9+
- **Rust 1.86+** — install via [rustup.rs](https://rustup.rs/)
10+
- **CMake 3+, Go 1.19+, C compiler** — required for BoringSSL (TLS impersonation)
11+
- **Chrome/Chromium** — optional, for `--browser` capture mode
12+
13+
| Platform | Build deps |
14+
|---|---|
15+
| macOS | `xcode-select --install && brew install cmake go` |
16+
| Ubuntu/Debian | `sudo apt install build-essential cmake golang-go libclang-dev` |
17+
| Fedora | `sudo dnf install gcc gcc-c++ cmake golang clang-devel` |
18+
| Arch | `sudo pacman -S cmake go clang` |
19+
| Windows | Visual Studio Build Tools (C++ workload) + [CMake](https://cmake.org/download/) + [Go](https://go.dev/dl/) |
20+
| Any (Docker) | `docker build -t palimpsest .` — no local toolchain needed |
21+
722
## Quick Start
823

924
```bash

book/src/contributing/development-setup.md

Lines changed: 40 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,15 @@
22

33
## Prerequisites
44

5-
- **Rust 1.86+**`rustup update stable`
6-
- **cargo** — included with Rust
7-
- **git** — for source checkout
8-
- **Chrome/Chromium** — optional, for browser capture tests
5+
| Tool | Version | Why |
6+
|---|---|---|
7+
| Rust | 1.86+ stable | `rustup update stable` |
8+
| CMake | 3.x+ | BoringSSL compilation |
9+
| Go | 1.19+ | BoringSSL compilation |
10+
| C compiler | gcc, clang, or MSVC | BoringSSL compilation |
11+
| Git | any | Source checkout |
12+
13+
See [Installation](../getting-started/installation.md) for platform-specific setup (macOS, Linux, Windows).
914

1015
## Clone and Build
1116

@@ -15,10 +20,12 @@ cd palimpsest
1520
cargo build --workspace
1621
```
1722

23+
First build takes 2-4 minutes (BoringSSL compiles from source). Subsequent builds are incremental.
24+
1825
## Running Tests
1926

2027
```bash
21-
# Full test suite (excludes long-running scale tests)
28+
# Full test suite (288 tests, excludes long-running scale tests)
2229
cargo test --workspace
2330

2431
# Simulation tests only
@@ -30,27 +37,51 @@ cargo test -p palimpsest-sim --test scale_test
3037
# Stress test (10K pages)
3138
cargo test -p palimpsest-sim --test stress_test
3239

40+
# Stealth regression tests (requires Chrome + network access)
41+
cargo test -p palimpsest-fetch --test stealth_test -- --ignored --nocapture --test-threads=1
42+
3343
# Single crate
3444
cargo test -p palimpsest-frontier
3545
```
3646

3747
## Pre-Commit Checks
3848

39-
Before submitting a PR, run:
49+
Before submitting a PR:
4050

4151
```bash
42-
cargo fmt --check # Formatting
52+
cargo fmt --check # Formatting
4353
cargo clippy -- -D warnings # Lints (must be warning-free)
44-
cargo test --workspace # All tests pass
54+
cargo test --workspace # All tests pass
4555
```
4656

4757
## IDE Setup
4858

49-
`rust-analyzer` is recommended. The workspace `Cargo.toml` at the project root configures all 15 crates automatically.
59+
**rust-analyzer** is recommended for all editors. The workspace `Cargo.toml` at the project root configures all 15 crates automatically.
60+
61+
| Editor | Setup |
62+
|---|---|
63+
| VS Code | Install `rust-analyzer` extension |
64+
| JetBrains (CLion/RustRover) | Built-in Rust support |
65+
| Neovim | `mason.nvim` → install `rust-analyzer` |
66+
| Emacs | `lsp-mode` + `rustic` |
5067

5168
## Docker Testing
5269

5370
```bash
5471
docker build -t palimpsest .
5572
docker run palimpsest --help
5673
```
74+
75+
## Platform Notes
76+
77+
### macOS
78+
79+
BoringSSL builds cleanly with Xcode command line tools + Homebrew CMake + Go. No special flags needed.
80+
81+
### Windows (MSVC)
82+
83+
Requires Visual Studio Build Tools with the "Desktop development with C++" workload. CMake and Go must be in `PATH`. WSL2 is the recommended alternative for a smoother experience.
84+
85+
### Linux
86+
87+
All major distributions work. Ensure `cmake`, `go`, and `clang` (or `gcc`) are installed. See [Installation](../getting-started/installation.md) for distro-specific package commands.

book/src/getting-started/installation.md

Lines changed: 183 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,33 +2,177 @@
22

33
## Prerequisites
44

5-
- **Rust 1.86+** (stable toolchain)
6-
- **Git** for source checkout
7-
- **Chrome or Chromium** (optional, for browser capture mode)
8-
- **Docker** (optional, for containerized deployment)
5+
| Dependency | Required | Notes |
6+
|---|---|---|
7+
| Rust 1.86+ | Yes | Stable toolchain via [rustup](https://rustup.rs/) |
8+
| Git | Yes | Source checkout |
9+
| C compiler + CMake | Yes | BoringSSL build (via wreq) |
10+
| Go 1.19+ | Yes | BoringSSL build (via wreq) |
11+
| Chrome or Chromium | Optional | Browser capture mode (`--browser`) |
12+
| Docker | Optional | Containerized deployment |
913

10-
## Building from Source
14+
### Why the C/Go toolchain?
15+
16+
Palimpsest uses [wreq](https://github.com/0x676e67/wreq) with BoringSSL for TLS fingerprint impersonation. BoringSSL is compiled from source during `cargo build`, which requires CMake, a C compiler, and Go.
17+
18+
## macOS
1119

1220
```bash
21+
# Install Rust
22+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
23+
source $HOME/.cargo/env
24+
25+
# Install build dependencies (Xcode command line tools + CMake + Go)
26+
xcode-select --install
27+
brew install cmake go
28+
29+
# Clone and build
1330
git clone https://github.com/copyleftdev/palimpsest.git
1431
cd palimpsest
1532
cargo build --release
33+
34+
# Verify
35+
./target/release/palimpsest --help
1636
```
1737

18-
The binary is placed at `target/release/palimpsest`.
38+
**Chrome for browser capture:**
39+
```bash
40+
# Chrome is usually at:
41+
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
1942

20-
## Verifying the Installation
43+
# Or install via Homebrew:
44+
brew install --cask google-chrome
45+
```
46+
47+
## Linux (Ubuntu/Debian)
2148

2249
```bash
50+
# Install Rust
51+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
52+
source $HOME/.cargo/env
53+
54+
# Install build dependencies
55+
sudo apt update
56+
sudo apt install -y build-essential cmake golang-go pkg-config libclang-dev
57+
58+
# Clone and build
59+
git clone https://github.com/copyleftdev/palimpsest.git
60+
cd palimpsest
61+
cargo build --release
62+
63+
# Verify
2364
./target/release/palimpsest --help
2465
```
2566

26-
You should see the full subcommand list: `crawl`, `replay`, `history`, `extract`, `shadow-compare`, `serve`, `worker`, `api`, `stats`, `migrate`.
67+
**Chrome for browser capture:**
68+
```bash
69+
# Install Chrome
70+
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg
71+
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
72+
sudo apt update && sudo apt install -y google-chrome-stable
73+
74+
# Verify
75+
google-chrome --version
76+
```
77+
78+
## Linux (Fedora/RHEL)
79+
80+
```bash
81+
# Install Rust
82+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
83+
source $HOME/.cargo/env
84+
85+
# Install build dependencies
86+
sudo dnf install -y gcc gcc-c++ cmake golang clang-devel pkg-config
87+
88+
# Clone and build
89+
git clone https://github.com/copyleftdev/palimpsest.git
90+
cd palimpsest
91+
cargo build --release
92+
```
93+
94+
## Linux (Arch)
95+
96+
```bash
97+
sudo pacman -S rust cmake go clang pkg-config
98+
git clone https://github.com/copyleftdev/palimpsest.git
99+
cd palimpsest
100+
cargo build --release
101+
```
102+
103+
## Windows
104+
105+
### Option A: Native (MSVC)
106+
107+
```powershell
108+
# 1. Install Rust from https://rustup.rs/ (choose MSVC toolchain)
109+
110+
# 2. Install Visual Studio Build Tools (C/C++ workload)
111+
# https://visualstudio.microsoft.com/visual-cpp-build-tools/
112+
113+
# 3. Install CMake
114+
# https://cmake.org/download/ (add to PATH during install)
115+
116+
# 4. Install Go
117+
# https://go.dev/dl/ (add to PATH during install)
118+
119+
# 5. Clone and build
120+
git clone https://github.com/copyleftdev/palimpsest.git
121+
cd palimpsest
122+
cargo build --release
123+
124+
# 6. Verify
125+
.\target\release\palimpsest.exe --help
126+
```
127+
128+
### Option B: WSL2 (recommended)
129+
130+
Windows Subsystem for Linux gives you a full Linux environment. Follow the Linux (Ubuntu/Debian) instructions above inside WSL2:
131+
132+
```powershell
133+
# Install WSL2 with Ubuntu
134+
wsl --install -d Ubuntu
135+
136+
# Then inside the WSL2 terminal, follow the Linux instructions
137+
```
138+
139+
### Option C: Docker (any platform)
140+
141+
If you don't want to install build tools, Docker works on all platforms:
142+
143+
```bash
144+
docker build -t palimpsest .
145+
docker run palimpsest --help
146+
docker run -v ./output:/data palimpsest crawl https://example.com -d 2 -o /data
147+
```
148+
149+
See [Docker Deployment](../operations/docker.md) for the full compose setup.
150+
151+
## Verifying the Build
152+
153+
After building, you should see all 10 subcommands:
154+
155+
```
156+
$ palimpsest --help
157+
Usage: palimpsest <COMMAND>
158+
159+
Commands:
160+
crawl Start a crawl with seed URLs
161+
replay Reconstruct a captured URL from artifacts
162+
history Show capture history for a URL
163+
extract Extract text and RAG chunks from captured content
164+
shadow-compare Compare against legacy crawler WARC files
165+
serve Start a distributed frontier server
166+
worker Connect to a frontier server and crawl
167+
api Start the retrieval API server
168+
stats Print workspace statistics
169+
migrate Run storage migrations
170+
```
27171

28172
## Running the Test Suite
29173

30174
```bash
31-
# All tests (excludes long-running scale tests)
175+
# All tests (288, excludes long-running scale tests)
32176
cargo test --workspace
33177

34178
# Simulation framework only
@@ -39,28 +183,45 @@ cargo test -p palimpsest-sim --test scale_test
39183

40184
# Stress test (10K pages)
41185
cargo test -p palimpsest-sim --test stress_test
186+
187+
# Stealth regression tests (requires Chrome + network)
188+
cargo test -p palimpsest-fetch --test stealth_test -- --ignored --nocapture --test-threads=1
42189
```
43190

44-
## Docker
191+
## Troubleshooting
45192

46-
```bash
47-
docker build -t palimpsest .
193+
### BoringSSL build fails
48194

49-
# Verify
50-
docker run palimpsest --help
195+
The most common build issue. Check:
196+
197+
```bash
198+
cmake --version # Need 3.x+
199+
go version # Need 1.19+
200+
clang --version # Or gcc — need a C compiler
51201
```
52202

53-
See [Docker Deployment](../operations/docker.md) for the full compose setup.
203+
On macOS, ensure Xcode command line tools are installed: `xcode-select --install`
204+
205+
On Windows, ensure Visual Studio Build Tools include the "Desktop development with C++" workload.
54206

55-
## Browser Capture
207+
### Chrome not found (browser capture)
56208

57-
Browser capture requires Chrome or Chromium accessible in `PATH`. Palimpsest communicates via the Chrome DevTools Protocol (CDP).
209+
Palimpsest looks for Chrome/Chromium in `PATH`. If installed in a non-standard location:
58210

59211
```bash
60-
# Verify Chrome is available
61-
google-chrome --version
62-
# or
63-
chromium --version
212+
# macOS — add to PATH
213+
export PATH="/Applications/Google Chrome.app/Contents/MacOS:$PATH"
214+
215+
# Windows — add to PATH
216+
set PATH=%PATH%;C:\Program Files\Google\Chrome\Application
217+
```
218+
219+
### `openssl-sys` linker errors
220+
221+
Palimpsest uses BoringSSL (via wreq), not OpenSSL. If you see `openssl-sys` errors, another dependency may be pulling it in. Check with:
222+
223+
```bash
224+
cargo tree -i openssl-sys
64225
```
65226

66-
No additional configuration is needed — the `--browser` flag on the `crawl` command handles the rest.
227+
If present, the `boring-sys2` crate's `prefix-symbols` feature should prevent symbol conflicts on Linux. On macOS this is not typically an issue.

docs/book/404.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
const path_to_root = "";
3838
const default_light_theme = "light";
3939
const default_dark_theme = "coal";
40-
window.path_to_searchindex_js = "searchindex-4f269067.js";
40+
window.path_to_searchindex_js = "searchindex-d84f7c65.js";
4141
</script>
4242
<!-- Start loading toc.js asap -->
4343
<script src="toc-7980dfd2.js"></script>

docs/book/appendix/api-reference.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
const path_to_root = "../";
3737
const default_light_theme = "light";
3838
const default_dark_theme = "coal";
39-
window.path_to_searchindex_js = "../searchindex-4f269067.js";
39+
window.path_to_searchindex_js = "../searchindex-d84f7c65.js";
4040
</script>
4141
<!-- Start loading toc.js asap -->
4242
<script src="../toc-7980dfd2.js"></script>

docs/book/appendix/error-taxonomy.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
const path_to_root = "../";
3737
const default_light_theme = "light";
3838
const default_dark_theme = "coal";
39-
window.path_to_searchindex_js = "../searchindex-4f269067.js";
39+
window.path_to_searchindex_js = "../searchindex-d84f7c65.js";
4040
</script>
4141
<!-- Start loading toc.js asap -->
4242
<script src="../toc-7980dfd2.js"></script>

docs/book/appendix/glossary.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
const path_to_root = "../";
3737
const default_light_theme = "light";
3838
const default_dark_theme = "coal";
39-
window.path_to_searchindex_js = "../searchindex-4f269067.js";
39+
window.path_to_searchindex_js = "../searchindex-d84f7c65.js";
4040
</script>
4141
<!-- Start loading toc.js asap -->
4242
<script src="../toc-7980dfd2.js"></script>

0 commit comments

Comments
 (0)