Skip to content

Commit e6f9689

Browse files
asmacdoclaude
andcommitted
Update ReproNim integration doc to match implementation
- Add "very rough draft" status note - Document available placeholders: {img}, {img_path}, {cmd} - Add runtime-only profiles option (no image, require --image) - Update examples to use name:version format (mriqc:23.1.0) - Fix placeholder usage: oci:{img_path} for apptainer, {img} for docker - Add datalad save commands to workflow examples - Simplify Docker profile example 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 88489d6 commit e6f9689

1 file changed

Lines changed: 77 additions & 46 deletions

File tree

docs/design/image-container-refactor-repronim.md

Lines changed: 77 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
This document describes how ReproNim/containers can leverage the refactored datalad-container architecture.
44

5+
STATUS: Very rough draft!
6+
57
---
68

79
## New Capabilities
@@ -26,6 +28,10 @@ With native format storage and versioned image directories, ReproNim can provide
2628
└── image/
2729
```
2830

31+
**Image naming:** Use `name:version` format (like Docker tags):
32+
- `mriqc:23.1.0``.datalad/containers/images/mriqc/23.1.0/image/`
33+
- `fmriprep:24.1.0``.datalad/containers/images/fmriprep/24.1.0/image/`
34+
2935
### Benefits of OCI format for ReproNim:
3036

3137
- **Layer deduplication** - Many neuroimaging containers share base layers
@@ -37,26 +43,51 @@ With native format storage and versioned image directories, ReproNim can provide
3743

3844
## 2. Execution Profiles
3945

40-
ReproNim ships curated base profiles alongside images. Users extend these for their specific needs.
46+
ReproNim can ship curated execution profiles alongside images. Users extend these for their specific needs.
47+
48+
### Available Placeholders
49+
50+
- `{img}` - Docker image name (`datalad-container/mriqc:23.1.0`)
51+
- `{img_path}` - OCI directory path (`.datalad/containers/images/mriqc/23.1.0/image`)
52+
- `{cmd}` - command arguments
4153

4254
### ReproNim Base Profiles
4355

56+
**Option A: Profiles with image (ready to use)**
57+
4458
```yaml
4559
# .datalad/containers/profiles/mriqc.yaml
4660
# Base MRIQC profile - sane defaults for most users
4761

48-
image: mriqc/23.1.0
49-
exec: apptainer exec --cleanenv {img} {cmd}
62+
image: mriqc:23.1.0
63+
exec: apptainer exec --cleanenv oci:{img_path} {cmd}
5064
```
5165
5266
```yaml
5367
# .datalad/containers/profiles/fmriprep.yaml
5468
# Base fMRIPrep profile
5569

56-
image: fmriprep/23.2.0
57-
exec: apptainer exec --cleanenv {img} {cmd}
70+
image: fmriprep:23.2.0
71+
exec: apptainer exec --cleanenv oci:{img_path} {cmd}
5872
```
5973
74+
**Option B: Runtime-only profiles (user specifies image)**
75+
76+
```yaml
77+
# .datalad/containers/profiles/apptainer-default.yaml
78+
# No image - user must provide --image
79+
80+
exec: apptainer exec oci:{img_path} {cmd}
81+
```
82+
83+
```yaml
84+
# .datalad/containers/profiles/docker-default.yaml
85+
86+
exec: docker run --rm --user $(id -u):$(id -g) -v $(pwd):/work -w /work {img} {cmd}
87+
```
88+
89+
Usage: `datalad containers-run --profile apptainer-default --image mriqc:23.1.0 ...`
90+
6091
### User Extensions
6192

6293
Users create their own profiles that extend ReproNim's base:
@@ -65,17 +96,17 @@ Users create their own profiles that extend ReproNim's base:
6596
# my-analysis/.datalad/containers/profiles/mriqc-mylab.yaml
6697
6798
extends: inputs/containers/.datalad/containers/profiles/mriqc.yaml
68-
exec: apptainer exec --cleanenv --nv --bind /scratch:/scratch --bind /data/mylab:/input {img} {cmd}
99+
exec: apptainer exec --cleanenv --nv --bind /scratch:/scratch --bind /data/mylab:/input oci:{img_path} {cmd}
69100
```
70101

71102
```yaml
72103
# my-analysis/.datalad/containers/profiles/mriqc-gpu.yaml
73104
74105
extends: inputs/containers/.datalad/containers/profiles/mriqc.yaml
75-
exec: apptainer exec --cleanenv --nv {img} {cmd}
106+
exec: apptainer exec --cleanenv --nv oci:{img_path} {cmd}
76107
```
77108

78-
**Key point:** ReproNim provides the base. Users clobber `exec` with their environment-specific settings. No runtime-specific profiles (apptainer-gpu, podman-default, etc.) - users know what they need.
109+
**Key point:** ReproNim provides the base. Users clobber `exec` with their environment-specific settings (clobber semantics - child completely replaces parent's exec, no merging).
79110

80111
---
81112

@@ -129,12 +160,14 @@ Provenance (source URL, digest, fetch time) is stored in git commits, not separa
129160
datalad clone https://github.com/ReproNim/containers inputs/containers
130161
131162
# List available images and profiles
132-
datalad containers-images -d inputs/containers
163+
datalad containers-list -d inputs/containers
133164
datalad containers-profiles -d inputs/containers
134165
135-
# Run with base profile
136-
datalad containers-run -d inputs/containers --profile mriqc \
137-
mriqc /bids /outputs participant
166+
# Run with base profile (profile specifies image)
167+
datalad containers-run --profile mriqc -- mriqc /bids /outputs participant
168+
169+
# Or use runtime-only profile with explicit image
170+
datalad containers-run --profile apptainer-default --image mriqc:23.1.0 -- mriqc /bids /outputs participant
138171
```
139172

140173
### Creating a Lab-Specific Profile
@@ -144,34 +177,40 @@ datalad containers-run -d inputs/containers --profile mriqc \
144177
mkdir -p .datalad/containers/profiles
145178
cat > .datalad/containers/profiles/mriqc-mylab.yaml << 'EOF'
146179
extends: inputs/containers/.datalad/containers/profiles/mriqc.yaml
147-
exec: apptainer exec --cleanenv --nv --bind /scratch:/scratch --bind /gpfs/mylab:/data {img} {cmd}
180+
exec: apptainer exec --cleanenv --nv --bind /scratch:/scratch --bind /gpfs/mylab:/data oci:{img_path} {cmd}
148181
EOF
149182

183+
# Save to dataset
184+
datalad save -m "Add mriqc-mylab profile" .datalad/containers/profiles/
185+
150186
# Use it
151-
datalad containers-run --profile mriqc-mylab \
152-
mriqc /data/bids /data/outputs participant
187+
datalad containers-run --profile mriqc-mylab -- mriqc /data/bids /data/outputs participant
153188
```
154189

155190
### One-Off Override
156191

157192
```bash
158-
# Use base profile but override exec for this run
159-
datalad containers-run -d inputs/containers --profile mriqc \
160-
--exec "apptainer exec --cleanenv --nv {img} {cmd}" \
161-
mriqc /bids /outputs participant
193+
# Use base profile but override exec for this run (adds GPU support)
194+
datalad containers-run --profile mriqc \
195+
--exec "apptainer exec --cleanenv --nv oci:{img_path} {cmd}" \
196+
-- mriqc /bids /outputs participant
197+
198+
# Use base profile but override image (use newer version)
199+
datalad containers-run --profile mriqc --image mriqc:24.0.0 \
200+
-- mriqc /bids /outputs participant
162201
```
163202

164203
### Using a Specific Version
165204

166205
```bash
167206
# Create profile for newer version
168207
cat > .datalad/containers/profiles/mriqc-24.yaml << 'EOF'
169-
image: mriqc/24.0.0
170-
exec: apptainer exec --cleanenv {img} {cmd}
208+
image: mriqc:24.0.0
209+
exec: apptainer exec --cleanenv oci:{img_path} {cmd}
171210
EOF
172211

173-
datalad containers-run --profile mriqc-24 \
174-
mriqc /bids /outputs participant
212+
datalad save -m "Add mriqc-24 profile" .datalad/containers/profiles/
213+
datalad containers-run --profile mriqc-24 -- mriqc /bids /outputs participant
175214
```
176215

177216
---
@@ -242,12 +281,15 @@ exec: >-
242281
--contain
243282
-H code/containers/binds/HOME
244283
-B code/containers/binds/zoneinfo/UTC:/etc/localtime
245-
-B {pwd}
246-
--pwd {pwd}
247-
{img}
284+
oci:{img_path}
248285
{cmd}
249286
```
250287
288+
**Available placeholders:**
289+
- `{img}` - Docker image name (for docker/podman)
290+
- `{img_path}` - OCI directory path (for apptainer with `oci:` prefix)
291+
- `{cmd}` - command arguments
292+
251293
### Upstream RFE: Advanced Profile Features
252294

253295
To fully replace the shim, datalad-container would need these profile extensions:
@@ -298,7 +340,7 @@ With these extensions, a profile could fully replace the shim:
298340
```yaml
299341
# .datalad/containers/profiles/mriqc-repronim.yaml
300342
301-
image: mriqc/23.1.0
343+
image: mriqc:23.1.0
302344
303345
pre-run: code/containers/scripts/setup-env.sh
304346
post-run: code/containers/scripts/cleanup.sh
@@ -315,10 +357,7 @@ exec: >-
315357
-B {env.TMPDIR}:/tmp
316358
-B {env.TMPDIR}/var:/var/tmp
317359
-B code/containers/binds/zoneinfo/UTC:/etc/localtime
318-
-B {pwd}
319-
--pwd {pwd}
320-
-W {env.TMPDIR}
321-
{img}
360+
oci:{img_path}
322361
{cmd}
323362
```
324363

@@ -330,35 +369,27 @@ Instead of runtime detection, use a separate profile:
330369
# .datalad/containers/profiles/mriqc-repronim-docker.yaml
331370
# For non-Linux systems (macOS, Windows with Docker)
332371
333-
image: mriqc/23.1.0
372+
image: mriqc:23.1.0
334373
335374
pre-run: code/containers/scripts/setup-env-docker.sh
336375
337376
env:
338-
SINGULARITYENV_MPLCONFIGDIR: /tmp/mpl-config
377+
MPLCONFIGDIR: /tmp/mpl-config
339378
340379
exec: >-
341380
docker run
342-
--privileged
343381
--rm
344-
-e UID={env.UID}
345-
-e GID={env.GID}
346-
-v {env.TMPDIR}:{env.TMPDIR}
347-
-v {pwd}:{pwd}
348-
-v code/containers/binds/HOME:{env.BHOME}
349-
-w {pwd}
350-
repronim/containers:latest
351-
exec
352-
--cleanenv
353-
-H {env.BHOME}
354-
-B {env.TMPDIR}:/tmp
355-
--pwd {pwd}
382+
--user $(id -u):$(id -g)
383+
-v $(pwd):/work
384+
-w /work
356385
{img}
357386
{cmd}
358387
```
359388

360389
Users on macOS would use `--profile mriqc-repronim-docker` explicitly.
361390

391+
**Note:** The Docker profile uses `{img}` (the Docker image name like `datalad-container/mriqc:23.1.0`) while apptainer profiles use `{img_path}` (the OCI directory path).
392+
362393
### Summary: Upstream RFE for datalad-container
363394

364395
To enable ReproNim to replace `singularity_cmd` with profiles:

0 commit comments

Comments
 (0)