Skip to content

Commit 92fc0b1

Browse files
vsochvsoch
committed
testing github deploy from weekend
0 parents  commit 92fc0b1

File tree

8 files changed

+232
-0
lines changed

8 files changed

+232
-0
lines changed

‎.github/workflows/builder.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: singularity-deploy
2+
3+
on:
4+
push:
5+
branches:
6+
- "main"
7+
8+
jobs:
9+
release:
10+
name: Create Release
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Checkout code
14+
uses: actions/checkout@v2
15+
- name: Define Repository Name and Release Version
16+
run: |
17+
repo=$(echo "${GITHUB_REPOSITORY/\//-}")
18+
release=$(cat VERSION)
19+
echo "reponame=$repo" >> $GITHUB_ENV
20+
echo "release_tag=$release" >> $GITHUB_ENV
21+
- name: Create Release
22+
id: create_release
23+
uses: actions/create-release@v1
24+
env:
25+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
26+
with:
27+
tag_name: ${{ env.release_tag }}
28+
release_name: Release ${{ env.release_tag }}
29+
draft: false
30+
prerelease: false
31+
- uses: eWaterCycle/setup-singularity@v6
32+
with:
33+
singularity-version: 3.6.4
34+
- name: Build the singularity container
35+
run: |
36+
repo=$(echo "${GITHUB_REPOSITORY/\//-}")
37+
38+
# For each Singularity* container, build based on the prefix (tag)
39+
for recipe in $(ls Singularity*); do
40+
echo "Building $recipe"
41+
tag=$(echo "${recipe/Singularity\./}")
42+
# If we find empty, use latest
43+
if [ "$tag" == "Singularity" ]; then
44+
tag=latest
45+
fi
46+
# Build the container and name by tag
47+
echo "Tag is $tag."
48+
container="$repo:$tag.sif"
49+
singularity build --fakeroot container.sif Singularity
50+
if [ "$?" == "0" ]; then
51+
echo "Successfully built container $container."
52+
mv container.sif "$container"
53+
else
54+
echo "There was an issue building $container."
55+
fi
56+
done
57+
- name: Upload Release Assets
58+
env:
59+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
60+
upload_url: ${{ steps.create_release.outputs.upload_url }}
61+
tag_name: ${{ env.release_tag }}
62+
run: |
63+
hub release edit $(find . -type f -name "*.sif" -printf "-a %p ") -m "" "$tag_name"

‎.github/workflows/test.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: singularity-deploy-test
2+
3+
# run on creation of a release tag
4+
on:
5+
pull_request: []
6+
7+
jobs:
8+
builder:
9+
name: Test Container Builds
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout code
13+
uses: actions/checkout@v2
14+
- uses: eWaterCycle/setup-singularity@v6
15+
with:
16+
singularity-version: 3.6.4
17+
- name: Build the singularity containers
18+
run: |
19+
repo=$(echo "${GITHUB_REPOSITORY/\//-}")
20+
21+
# For each Singularity* container, build based on the prefix (tag)
22+
for recipe in $(ls Singularity*); do
23+
echo "Building $recipe"
24+
tag=$(echo "${recipe/Singularity\./}")
25+
# If we find empty, use latest
26+
if [ "$tag" == "Singularity" ]; then
27+
tag=latest
28+
fi
29+
# Build the container and name by tag
30+
echo "Tag is $tag."
31+
container="$repo:$tag.sif"
32+
singularity build --fakeroot container.sif Singularity
33+
if [ "$?" == "0" ]; then
34+
echo "Successfully built container $container."
35+
mv container.sif "$container"
36+
else
37+
echo "There was an issue building $container."
38+
fi
39+
done
40+
- name: Run Additional Tests
41+
run: |
42+
echo "Run any additional tests here."

‎README.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Singularity Deploy
2+
3+
[![GitHub actions status](https://github.com/singularityhub/singularity-deploy/workflows/singularity-deploy/badge.svg?branch=main)](https://github.com/singularityhub/singularity-deploy/actions?query=branch%3Amain+workflow%3Asingularity-deploy)
4+
[![GitHub actions status](https://github.com/singularityhub/singularity-deploy/workflows/singularity-deploy-test/badge.svg?branch=main)](https://github.com/singularityhub/singularity-deploy/actions?query=branch%3Amain+workflow%3Asingularity-deploy-test)
5+
6+
Wouldn't it be nice to build Singularity images without a registry proper,
7+
and just keep them alongside the GitHub codebase? This is now possible!
8+
This small repository provides an example to get you started. It will
9+
build one or more images (whatever Singularity.* files that are present at
10+
the root) and then release them as assets to your GitHub repository so
11+
that they can be programatically obtained. Note that assets are limited to
12+
2 GB in size, which is still fairly good. You can use
13+
it as a template for your own recipes as is, or modify it for your custom
14+
use case. Instructions are below!
15+
16+
## Getting Started
17+
18+
### 1. Template or Fork
19+
20+
If you haven't already, template or fork this repository. You can then clone
21+
your fork:
22+
23+
```bash
24+
$ git clone [email protected]:<username>/singularity-deploy
25+
```
26+
27+
You likely want to name the repository by the container. For example, if I would
28+
have created a container on Docker Hub or similar with the name `vsoch/salad`,
29+
here I'd call the repository `salad`. You obviously are limited to your username
30+
or an organizational namespace.
31+
32+
### 1. Write your Singularity Recipe(s)
33+
34+
First, you should write your container recipe(s) in the present working directory.
35+
For good practice, when you are updating recipes you should checkout a new branch
36+
and open a pull request, as the repository comes with a workflow to trigger on a PR
37+
to [test your container build](.github/workflows/test.yml). You can add any additional
38+
tests that that you might need. By default, any Singularity.* file will be automatically detected.
39+
If there is no extension (the name Singularity), the name used will be "latest."
40+
You can use these tags across multiple releases of your containers. For example,
41+
these files would generate packages with sifs named as follows:
42+
43+
- [Singularity](Singularity) maps to [https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.latest.sif](https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.latest.sif)
44+
- [Singularity.pokemon](Singularity.pokemon) maps to [https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.pokemon.sif](https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.pokemon.sif)
45+
- [Singularity.salad](Singularity.salad) maps to [https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.latest.sif](https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.latest.sif)
46+
47+
For each name, you can see the direct download URL contains the repository (singularityhub/singularity-deploy),
48+
You should not use any `:` characters in either your container tag (the GitHub extension) or
49+
the GitHub tags (the release tags) as this might interfere with parsing.
50+
The GitHub release tag (0.0.1 in the example above) is discussed next.
51+
52+
### 2. Update the VERSION file
53+
54+
Any time that you prepare new container recipes, you should update the [VERSION](VERSION)
55+
file. The way that this repository works is to generate a release based on the
56+
string in `VERSION`. A version is just a tag, so it could be something like
57+
`0.0.1` or `0.0.1-slim`. Keep in mind that GitHub releases cannot have duplicated
58+
names, so you should not repeat the same tag. Do not use `:` in your tag names.
59+
If you do need to re-release a tag (not recommended if a user might be using it and then it's changed) you can manually delete
60+
the release and the tag in the GitHub interface. This is a nice structure because it
61+
means you can have containers with different names under the same tag. In the example
62+
above, we have each of "deploy," "latest," and "salad" released under tag 0.0.1.
63+
This is how it looks on GitHub:
64+
65+
![img/releases.png](img/releases.png)
66+
67+
### 3. How to Develop
68+
69+
As we mentioned previously, the container builds will be tested on a pull request,
70+
and the release will trigger on merge into your main branch (main). See the [.github/workflows/builder.yml](.github/workflows/builder.yml))
71+
to edit this. The idea is that you can:
72+
73+
1. Develop your container via a development branch
74+
2. Open a pull request to test the container (the [.github/workflows/test.yml](.github/workflows/test.yml))
75+
3. On merge, your container will be released!
76+
77+
### 4. How to pull
78+
79+
Technically, Singularity can pull just knowing the URL. For example:
80+
81+
```bash
82+
$ singularity pull https://github.com/singularityhub/singularity-deploy/releases/download/0.0.1/singularityhub-singularity-deploy.latest.sif
83+
```
84+
85+
However, the [singularity-hpc](singularity-hpc) tool (will be) designed to be able to parse and handle
86+
these container uris automatically. For the containers here, you could do:
87+
88+
```bash
89+
$ shpc pull gh://singularityhub/singularity-deploy/0.0.1:latest
90+
$ shpc pull gh://singularityhub/singularity-deploy/0.0.1:salad
91+
$ shpc pull gh://singularityhub/singularity-deploy/0.0.1:pokemon
92+
```
93+
94+
or write the container URI into a registry entry:
95+
96+
```
97+
gh: singularityhub/singularity-deploy
98+
latest:
99+
latest: "0.0.1"
100+
tags:
101+
"latest": "0.0.1"
102+
"salad": "0.0.1"
103+
"pokemon": "0.0.1"
104+
maintainer: "@vsoch"
105+
url: https://github.com/singularityhub/singularity-deploy
106+
```
107+
108+
(This part is still under development!)

‎Singularity‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Bootstrap: docker
2+
From: busybox
3+
4+
%post
5+
6+
echo "this is a tiny example container."

‎Singularity.pokemon‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Bootstrap: docker
2+
From: vanessa/pokemon
3+
4+
%post
5+
6+
echo "hello this is pokemon build."

‎Singularity.salad‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Bootstrap: docker
2+
From: vanessa/salad
3+
4+
%post
5+
6+
echo "hello salad, I am spoon."

‎VERSION‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
0.0.1

‎img/releases.png‎

53.6 KB
Loading

0 commit comments

Comments
 (0)