Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixup install.sh output. #142

Merged
merged 1 commit into from
Feb 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 5 additions & 13 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,6 @@ concurrency:
group: CI-${{ github.ref }}
# Queue on all branches and tags, but only cancel overlapping PR burns.
cancel-in-progress: ${{ github.ref != 'refs/heads/main' && !startsWith(github.ref, 'refs/tags/') }}
env:
# Work around SIGSEGVs and other errors under some qemu targets.
UV_CONCURRENT_BUILDS: 1
jobs:
org-check:
name: Check GitHub Organization
Expand Down Expand Up @@ -115,14 +112,14 @@ jobs:
path: .mypy_cache
# We're using a key suffix / restore-keys prefix trick here to get an updatable cache.
# See: https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
key: ${{ matrix.docker-platform || format('{0}-{1}', matrix.os, runner.arch) }}-a-scie-lift-mypy-v1-${{ github.run_id }}
restore-keys: ${{ matrix.docker-platform || format('{0}-{1}', matrix.os, runner.arch) }}-a-scie-lift-mypy-v1
key: ${{ matrix.image || format('{0}-{1}', matrix.os, runner.arch) }}-a-scie-lift-mypy-v1-${{ github.run_id }}
restore-keys: ${{ matrix.image || format('{0}-{1}', matrix.os, runner.arch) }}-a-scie-lift-mypy-v1
- name: Check Formatting & Lints
if: matrix.image == ''
run: |
"${UV}" run dev-cmd ci --skip test
- name: Check Formatting & Lints
if: matrix.image != ''
if: matrix.image == 'debian' && matrix.arch == 'amd64'
run: |
"${UV}" run dev-cmd docker -- --image ${{ matrix.image }} --arch ${{ matrix.arch }} \
ci --skip test
Expand All @@ -143,13 +140,8 @@ jobs:
run: |
"${UV}" run dev-cmd test -- -vvs
- name: Unit Tests
if: matrix.image != ''
if: matrix.image != '' && matrix.arch != 'ppc64le'
run: |
if [ "${{ matrix.arch }}" = "ppc64le" ]; then
echo "Skipping tests on ppc64le."
exit 0
fi

"${UV}" run dev-cmd docker -- --image ${{ matrix.image }} --arch ${{ matrix.arch }} \
test -- -vvs
- name: Build & Package
Expand All @@ -166,7 +158,7 @@ jobs:
run: |
"${UV}" run dev-cmd doc linkcheck
- name: Generate Doc Site
if: matrix.image != ''
if: matrix.image == 'debian' && matrix.arch == 'amd64'
run: |
"${UV}" run dev-cmd docker -- --image ${{ matrix.image }} --arch ${{ matrix.arch }} \
doc linkcheck
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ on:
defaults:
run:
shell: bash
env:
# Work around SIGSEGVs and other errors under some qemu targets.
UV_CONCURRENT_BUILDS: 1
jobs:
org-check:
name: Check GitHub Organization
Expand Down
16 changes: 8 additions & 8 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@

set -eu

COLOR_RED="\x1b[31m"
COLOR_GREEN="\x1b[32m"
COLOR_YELLOW="\x1b[33m"
COLOR_RESET="\x1b[0m"
COLOR_RED="\e[31m"
COLOR_GREEN="\e[32m"
COLOR_YELLOW="\e[33m"
COLOR_RESET="\e[0m"

log() {
echo -e "$@" >&2
printf "$@\n" >&2
}

die() {
Expand Down Expand Up @@ -107,9 +107,9 @@ fetch() {
curl --proto '=https' --tlsv1.2 -SfL --progress-bar -o "${dest}" "${url}"
}

ensure_cmd $([ "${OS}" == "macos" ] && echo "shasum" || echo "sha256sum")
ensure_cmd $([ "${OS}" = "macos" ] && echo "shasum" || echo "sha256sum")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the colored logging not working - fixes above - there were a few warnings / errors from [[ and == bashisms missed in the 1st round.

sha256() {
if [[ "${OS}" == "macos" ]]; then
if [ "${OS}" = "macos" ]; then
shasum --algorithm 256 "$@"
else
sha256sum "$@"
Expand Down Expand Up @@ -204,7 +204,7 @@ done

ARCH="$(determine_arch)"
VARIANT="$(determine_variant)"
DIRSEP=$([ "${OS}" == "windows" ] && echo "\\" || echo "/")
DIRSEP=$([ "${OS}" = "windows" ] && echo "\\" || echo "/")
EXE_EXT=$([ "${OS}" = "windows" ] && echo ".exe" || echo "")

INSTALL_DEST="${INSTALL_PREFIX}${DIRSEP}science${EXE_EXT}"
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ include = ["science*"]
[dependency-groups]
dev = [
"ansicolors",
"coloredlogs",
"dev-cmd",
"docutils",
"mypy",
Expand Down Expand Up @@ -77,6 +78,10 @@ follow_untyped_imports = true
module = "click_log"
follow_untyped_imports = true

[[tool.mypy.overrides]]
module = "coloredlogs"
follow_untyped_imports = true

[[tool.mypy.overrides]]
module = ["colors.*"]
follow_untyped_imports = true
Expand Down
187 changes: 128 additions & 59 deletions scripts/docker/uv.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,69 @@
# Copyright 2025 Science project contributors.
# Licensed under the Apache License, Version 2.0 (see LICENSE).

import hashlib
import json
import logging
import os
import shlex
import shutil
import subprocess
import sys
from argparse import ArgumentParser
from pathlib import Path
from typing import Any

import coloredlogs

from science.platform import CURRENT_OS, Os

logger = logging.getLogger(__name__)


def fingerprint_path(path: Path) -> tuple[str, str]:
if path.is_dir():
tree: dict[str, str] = {}
for r, _, files in os.walk(path):
root = Path(r)
for f in files:
file_path = root / f
tree[str(file_path.relative_to(path))] = hashlib.sha256(
file_path.read_bytes()
).hexdigest()

fingerprint = hashlib.sha256(json.dumps(tree, sort_keys=True).encode()).hexdigest()
else:
fingerprint = hashlib.sha256(path.read_bytes()).hexdigest()

return str(path.resolve().relative_to(Path().resolve())), fingerprint


def fingerprint_paths(*paths: Path) -> str:
return hashlib.sha256(
json.dumps(dict(fingerprint_path(path) for path in paths), sort_keys=True).encode()
).hexdigest()


def image_exists(image_name: str) -> bool:
result = subprocess.run(
args=["docker", "image", "ls", "-q", image_name], capture_output=True, text=True
)
return result.returncode == 0 and bool(result.stdout.strip())


def main() -> Any:
if CURRENT_OS is Os.Windows:
return "This script does not work on Windows yet."

coloredlogs.install(
fmt="%(levelname)s %(message)s",
field_styles={
**coloredlogs.DEFAULT_FIELD_STYLES,
# Default is bold black, we switch to gray; c.f:
# https://coloredlogs.readthedocs.io/en/latest/api.html#available-text-styles-and-colors
"levelname": {"bold": True, "color": 8},
},
)

parser = ArgumentParser()
parser.add_argument("--image", default="debian", choices=["alpine", "debian"])
parser.add_argument(
Expand All @@ -34,6 +82,12 @@ def main() -> Any:
"arm/v6",
],
)
parser.add_argument(
"--inspect",
default=False,
action="store_true",
help="Instead of running `uv run dev-cmd` against the extra args, drop into a shell in the image for inspection.",
)
options, args = parser.parse_known_args()

platform = f"linux/{options.arch}"
Expand All @@ -52,69 +106,84 @@ def main() -> Any:

parent_dir = Path(__file__).parent
arch_tag = options.arch.replace("/", "-")
base_image = f"a-scie/lift/base:{arch_tag}"

# The type-ignores for os.get{uid,gid} cover Windows which we explicitly fail-fast for above.
subprocess.run(
args=[
"docker",
"buildx",
"build",
"--build-arg",
f"UID={os.getuid()}", # type:ignore[attr-defined]
"--build-arg",
f"GID={os.getgid()}", # type:ignore[attr-defined]
"--platform",
platform,
"--tag",
base_image,
str(parent_dir / options.image),
],
check=True,
)
dev_image_context = parent_dir / "uv"
fingerprint = fingerprint_paths(Path("pyproject.toml"), Path("uv.lock"), dev_image_context)
dev_image = f"a-scie/lift/dev:{arch_tag}-{fingerprint}"
if not image_exists(dev_image):
base_image_context = parent_dir / options.image
fingerprint = fingerprint_paths(base_image_context)
base_image = f"a-scie/lift/base:{arch_tag}-{fingerprint}"
if not image_exists(base_image):
# The type-ignores for os.get{uid,gid} cover Windows which we explicitly fail-fast for above.
subprocess.run(
args=[
"docker",
"buildx",
"build",
"--build-arg",
f"UID={os.getuid()}", # type:ignore[attr-defined]
"--build-arg",
f"GID={os.getgid()}", # type:ignore[attr-defined]
"--platform",
platform,
"--tag",
base_image,
str(base_image_context),
],
check=True,
)

dev_image = f"a-scie/lift/dev:{arch_tag}"
ephemeral_build_context = parent_dir / "ephemeral-build-context"
ephemeral_build_context.mkdir(parents=True, exist_ok=True)
shutil.copy(Path("pyproject.toml"), ephemeral_build_context)
shutil.copy(Path("uv.lock"), ephemeral_build_context)

ephemeral_build_context = parent_dir / "ephemeral-build-context"
ephemeral_build_context.mkdir(parents=True, exist_ok=True)
shutil.copy(Path("pyproject.toml"), ephemeral_build_context)
shutil.copy(Path("uv.lock"), ephemeral_build_context)
subprocess.run(
args=[
"docker",
"buildx",
"build",
"--build-arg",
f"BASE_IMAGE={base_image}",
"--build-context",
f"ephemeral={ephemeral_build_context}",
"--platform",
platform,
"--tag",
dev_image,
str(dev_image_context),
],
check=True,
)

subprocess.run(
args=[
"docker",
"buildx",
"build",
"--build-arg",
f"BASE_IMAGE={base_image}",
"--build-context",
f"ephemeral={ephemeral_build_context}",
"--platform",
platform,
"--tag",
dev_image,
str(parent_dir / "uv"),
],
check=True,
)
docker_run_args = [
"docker",
"run",
"--rm",
"-e",
"FORCE_COLOR",
"-e",
"SCIENCE_AUTH_API_GITHUB_COM_BEARER",
"-v",
f"{Path().absolute()}:/code",
"--platform",
platform,
]
if options.inspect:
if args:
logger.warning(f"Ignoring extra args in --inspect mode: {shlex.join(args)}")
docker_run_args.append("--interactive")
docker_run_args.append("--tty")
docker_run_args.append("--entrypoint")
docker_run_args.append("sh" if options.image == "alpine" else "bash")
docker_run_args.append(dev_image)
docker_run_args.append("-i")
else:
docker_run_args.append(dev_image)
docker_run_args.extend(args)

subprocess.run(
args=[
"docker",
"run",
"-e",
"FORCE_COLOR",
"-e",
"SCIENCE_AUTH_API_GITHUB_COM_BEARER",
"-v",
f"{Path().absolute()}:/code",
"--platform",
platform,
dev_image,
*args,
],
check=True,
)
subprocess.run(args=docker_run_args, check=True)


if __name__ == "__main__":
Expand Down
7 changes: 6 additions & 1 deletion scripts/docker/uv/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,12 @@ COPY --from=ephemeral uv.lock .
RUN chown -R build:build /code

USER build

# Work around SIGSEGVs and other errors under some qemu targets.
ENV UV_CONCURRENT_BUILDS=1
# Silence warning about needing to copy since we know /code/.venv lives on a bind mount.
ENV UV_LINK_MODE=copy

RUN uv sync --frozen --no-install-project && rm pyproject.toml uv.lock

ENTRYPOINT ["uv", "run", "dev-cmd"]
ENTRYPOINT ["uv", "run", "--frozen", "dev-cmd"]
Loading
Loading