Repository navigation

Flow pipeline #721

Workflow file for this run

	# Flow
	# Copyright 2023 Akamai Technologies, Inc.
	#
	# Licensed under the Apache License, Version 2.0 (the
	# "License"); you may not use this file except in
	# compliance with the License. You may obtain a copy
	# of the License at
	#
	# https://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in
	# writing, software distributed under the License is
	# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
	# CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing
	# permissions and limitations under the License.

	name: Flow pipeline

	# TODO: Code reuse somehow versus Flow-IPC workflow counterpart file? Could apply to major parts of this file.
	# The copy/pasting and "see comment in other file" types of comments are a bit unseemly.

	# See comment in Flow-IPC workflow counterpart. We follow the same philosophy here.

	on:
	pull_request:
	branches:
	- main
	push:
	branches:
	- main
	tags:
	- v*
	workflow_dispatch:
	inputs:
	job-filter:
	description: >-
	If not blank: only jobs whose name contains this substring shall really run (e.g.:
	"clang-15-relwithdebinfo-asan-cxx17" = 1 cell; "asan" = the ASAN cells; "deps-newest" = the
	newest-dependency-versions cells; "doc" = the doc job).
	Others complete ~instantly with all steps skipped.
	required: false
	default: ''
	type: string

	jobs:
	# See comments in Flow-IPC workflow counterpart. We use same techniques.
	set-vars:
	runs-on: ubuntu-latest
	steps:
	- name: Set variables/constants for subsequent use
	run: \|
	# Set variables/constants for subsequent use.
	outputs:
	doc-commit-message: (Commit by workflow script) Update generated documentation.

	# Impetus behind this is to gate whether the real jobs below actually need to run. Can of course also compute
	# other things as needed.
	# TODO: Already mentioned in a TODO above, but this stuff here is just copy/paste galore. Should really figure out
	# something to avoid it in this `setup` area.
	setup:
	needs: set-vars
	runs-on: ubuntu-latest
	steps:
	- name: Checkout VERSION file and tag history
	uses: actions/checkout@v6
	with:
	sparse-checkout: VERSION
	fetch-depth: 0 # Get all history regarding tags. We'll need that for VERSION checking below.
	- name: Grab repository version from `VERSION` file
	id: repo_version
	run: \|
	# Grab repository version from `VERSION` file.
	if [ ! -e VERSION ]; then
	echo '::error ::No VERSION file in repository root. One must exist.'
	exit 1
	fi
	VERSION=`cat VERSION`
	echo "Determined version: [$VERSION]."
	echo "version=$VERSION" >> $GITHUB_OUTPUT
	- name: (On release creation only) Check repository version against release tag name
	if: success() && (!cancelled()) && startsWith(github.ref, 'refs/tags/v')
	run: \|
	# Check repository version against release tag name.
	VERSION=${{ steps.repo_version.outputs.version }}
	TAG_VERSION=${GITHUB_REF#refs/tags/v}
	if [ "$VERSION" = "$TAG_VERSION" ]; then
	echo ":notice ::Tag name and repo version are an exact match. Looks good."
	elif [[ "$TAG_VERSION" == "$VERSION-"* ]]; then
	echo "::notice ::Tag name [v$TAG_VERSION] starts with repo version (pre-release). Allowed."
	else
	echo "::error ::VERSION file version v[$VERSION] must = release tag name [v$TAG_VERSION]."
	echo "::error ::Suggest deleting release and tag, fixing VERSION if needed, and re-creating release/tag."
	exit 1
	fi
	# TODO: Would be nice to also grab release name and ensure it equals tag name v<...>.
	# Could be a separate step or just execute here if the above succeeded so far. Requires API fetch, so it's
	# a few lines.
	- name: (Except on release creation) Check repository version against already-existing release tag names
	if: success() && (!cancelled()) && (!startsWith(github.ref, 'refs/tags/v'))
	run: \|
	# Check repo version against release tag name.
	# Do not fail on account of VERSION in this path however... just warn or gently inform.
	VERSION=${{ steps.repo_version.outputs.version }}
	if git rev-parse "v$VERSION" >/dev/null 2>&1; then
	echo "::warning ::Tag [v$VERSION] already exists. The repo/VERSION file will require "\
	"a bump, before the next release. However this should be done just ahead of release creation and must "\
	"follow semantic versioning conventions, especially regarding when to bump major/minor/patch components. "\
	"When ready to do this consult [https://semver.org/] as needed."
	elif git tag \| grep -q "^v$VERSION-"; then
	echo "::notice ::Since pre-release tag(s) [v$VERSION-*] already exist(s), but no tag "\
	"[v$VERSION] exists, this repo state is likely intended for a later pre-release or official release. "\
	"Cool! However, reminder: "\
	"The repo/VERSION version should be decided/finalized just ahead of release creation and must "\
	"follow semantic versioning conventions, especially regarding when to bump major/minor/patch components. "\
	"When ready to do this consult [https://semver.org/] as needed."
	else
	echo "::notice ::Since no tag [v$VERSION] or [v$VERSION-*] exists, this repo state "\
	"is likely intended for a future release. Cool! However, reminder: "\
	"The repo/VERSION version should be decided/finalized just ahead of release creation and must "\
	"follow semantic versioning conventions, especially regarding when to bump major/minor/patch components. "\
	"When ready to do this consult [https://semver.org/] as needed."
	fi
	- id: compute_proceed_else_not
	name: Compute whether for main jobs to proceed or not
	# See comments in Flow-IPC workflow counterpart. We use same techniques.
	run: \|
	# Compute whether for main jobs to proceed or not.
	TMP_MSG=/tmp/flow-pipeline-head-cmt-msg.txt
	cat <<'FLOW_PIPELINE_HEAD_CMD_MSG_EOF' > $TMP_MSG
	${{ github.event.head_commit.message }}
	FLOW_PIPELINE_HEAD_CMD_MSG_EOF
	if [ '${{ github.ref }}' != 'refs/heads/main' ] \|\| \
	[ '${{ github.event_name }}' != 'push' ] \|\| \
	! { head --lines=1 $TMP_MSG \| fgrep -xq '${{ needs.set-vars.outputs.doc-commit-message }}'; }; then
	echo 'proceed-else-not=true' >> $GITHUB_OUTPUT
	else
	echo 'proceed-else-not=false' >> $GITHUB_OUTPUT
	echo 'The real jobs will not run: earlier `doc-and-release` job checked-in generated docs to `main`.'
	echo 'That is not a source change and requires no actual pipeline to execute.'
	fi
	outputs:
	proceed-else-not: ${{ steps.compute_proceed_else_not.outputs.proceed-else-not }}

	build-and-test:
	needs: [setup, set-vars]
	if: \|
	needs.setup.outputs.proceed-else-not == 'true'

	strategy:
	fail-fast: false

	# Please read notes in the similar place in Flow-IPC workflow counterpart regarding the matrix's setup,
	# especially the bulk of it that discusses how the C++ standard setting (17, 20, etc.) is handled.
	# We omit that large comment here. (The shorter inline comments are simply repeated.)
	matrix:
	# First axis in basic 3D matrix is the 2 possible C++ standards.
	# For simplicity the Conan-profile-configuring script step simply keys off the `id` instead of our defining
	# other properties as for the other 2 more complex axes below (compiler, build config).
	cxx-std:
	- id: cxx17
	- id: cxx20
	# Second axis = the many compilers (gcc multiple versions, clang multiple versions).
	compiler:
	- id: gcc-9
	name: gcc
	version: 9
	c-path: /usr/bin/gcc-9
	cpp-path: /usr/bin/g++-9
	install: True
	- id: gcc-10
	name: gcc
	version: 10
	c-path: /usr/bin/gcc-10
	cpp-path: /usr/bin/g++-10
	- id: gcc-11
	name: gcc
	version: 11
	c-path: /usr/bin/gcc-11
	cpp-path: /usr/bin/g++-11
	- id: gcc-13
	name: gcc
	version: 13
	c-path: /usr/bin/gcc-13
	cpp-path: /usr/bin/g++-13
	install: True
	- id: clang-13
	name: clang
	version: 13
	c-path: /usr/bin/clang-13
	cpp-path: /usr/bin/clang++-13
	- id: clang-15
	name: clang
	version: 15
	c-path: /usr/bin/clang-15
	cpp-path: /usr/bin/clang++-15
	- id: clang-16
	name: clang
	version: 16
	c-path: /usr/bin/clang-16
	cpp-path: /usr/bin/clang++-16
	install: True
	- id: clang-17
	name: clang
	version: 17
	c-path: /usr/bin/clang-17
	cpp-path: /usr/bin/clang++-17
	install: True
	# Last axis are the many build/test configs.
	build-test-cfg:
	- id: debug
	conan-profile-build-type: Debug
	- id: release
	conan-profile-build-type: Release
	- id: relwithdebinfo
	conan-profile-build-type: RelWithDebInfo
	- id: minsizerel
	conan-profile-build-type: MinSizeRel
	# The sanitized (-*san) build/test configs follow. Here we follow Flow-IPC workflow counterpart.
	# This can be a non-trivial topic -- though by definition Flow's situation is simpler than Flow-IPC's,
	# as the latter includes the former entirely -- so if changing or trying to understand sanitizer-related
	# stuff, please check the Flow-IPC workflow counterpart file, particularly comments near each thing.
	# Keeping comments here light.
	#
	# Note that, as of this writing, MSAN support is present in Flow-IPC workflow counterpart but is disabled
	# for reasons explained therein. We therefore omit it here entirely; no need to have it taking up space here
	# too.
	- id: relwithdebinfo-asan
	conan-profile-build-type: RelWithDebInfo
	conan-profile-custom-conf: \|
	tools.build:cflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
	tools.build:cxxflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
	tools.build:sharedlinkflags = ["-fsanitize=address"]
	tools.build:exelinkflags = ["-fsanitize=address"]
	conan-profile-custom-settings: \|
	compiler.sanitizer = address
	conan-custom-settings-defs: \| # Could we not copy/paste these 3x?
	data = { 'compiler':
	{ 'gcc': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] },
	'clang': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] } } }
	sanitizer-name: asan
	no-lto: True
	- id: relwithdebinfo-ubsan
	conan-profile-build-type: RelWithDebInfo
	conan-profile-custom-conf: \|
	tools.build:cflags = ["-fsanitize=undefined", "-fno-omit-frame-pointer"]
	tools.build:cxxflags = ["-fsanitize=undefined", "-fno-omit-frame-pointer"]
	tools.build:sharedlinkflags = ["-fsanitize=undefined"]
	tools.build:exelinkflags = ["-fsanitize=undefined"]
	conan-profile-custom-settings: \|
	compiler.sanitizer = undefined
	conan-custom-settings-defs: \|
	data = { 'compiler':
	{ 'gcc': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] },
	'clang': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] } } }
	sanitizer-name: ubsan
	no-lto: True
	- id: relwithdebinfo-tsan
	conan-profile-build-type: RelWithDebInfo
	conan-profile-custom-conf: \|
	tools.build:cflags = ["-fsanitize=thread"]
	tools.build:cxxflags = ["-fsanitize=thread"]
	tools.build:sharedlinkflags = ["-fsanitize=thread"]
	tools.build:exelinkflags = ["-fsanitize=thread"]
	conan-profile-custom-settings: \|
	compiler.sanitizer = thread
	conan-custom-settings-defs: \|
	data = { 'compiler':
	{ 'gcc': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] },
	'clang': { 'sanitizer': ['None', 'address', 'thread', 'memory', 'undefined'] } } }
	sanitizer-name: tsan
	no-lto: True
	# 4th axis: dependency versions (`floor` = conanfile.py-contained verions = supported minimums;
	# `newest` = "early-warning" runs with newest conan-center versions of key dependencies). Explained in
	# Flow-IPC workflow counterpart. For Flow the only such dependency is Boost. Bump the `newest` version
	# when conan-center gets a newer one. (TODO: Some kind of reminder system to inform maintainer(s).)
	deps:
	- id: floor
	name-suffix: ''
	- id: newest
	name-suffix: -deps-newest
	conan-profile-replace-requires: \|
	boost/*: boost/1.90.0

	# Remove certain values from the 4D matrix so far.
	# The overall strategy is outlined in a large-ish comment above (or in Flow-IPC pipeline file;
	# referred to earlier above).
	exclude:
	# Firstly the exclusions from the C++17 sub-matrix (the other being the C++20 sub-matrix).
	# Again, these exclusions are explained in Flow-IPC workflow counterpart.
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-9 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-10 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-11 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-13 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx17 }
	compiler: { id: clang-13 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-9 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-10 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-11 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-13 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: clang-13 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-9 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-10 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-11 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: gcc-13 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	- cxx-std: { id: cxx17 }
	compiler: { id: clang-13 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	# Now the exclusions from the C++20 sub-matrix.
	# Again, these exclusions are explained in Flow-IPC workflow counterpart.
	# - First, simply exclude all gccs but gcc-13 and all clangs but clang-17.
	- cxx-std: { id: cxx20 }
	compiler: { id: gcc-9 }
	- cxx-std: { id: cxx20 }
	compiler: { id: gcc-10 }
	- cxx-std: { id: cxx20 }
	compiler: { id: gcc-11 }
	- cxx-std: { id: cxx20 }
	compiler: { id: clang-13 }
	- cxx-std: { id: cxx20 }
	compiler: { id: clang-15 }
	- cxx-std: { id: cxx20 }
	compiler: { id: clang-16 }
	# - Second, among the remaining C++20 configs eliminate all but debug and relwithdebinfo ones.
	- cxx-std: { id: cxx20 }
	build-test-cfg: { id: release }
	- cxx-std: { id: cxx20 }
	build-test-cfg: { id: minsizerel }
	- cxx-std: { id: cxx20 }
	build-test-cfg: { id: relwithdebinfo-asan }
	- cxx-std: { id: cxx20 }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- cxx-std: { id: cxx20 }
	build-test-cfg: { id: relwithdebinfo-tsan }
	# Now the exclusions from the `deps: newest` sub-matrix: keep only gcc-13 and clang-17, only debug and
	# relwithdebinfo, only C++17 (net: 4 cells). Explained in Flow-IPC workflow counterpart.
	- deps: { id: newest }
	compiler: { id: gcc-9 }
	- deps: { id: newest }
	compiler: { id: gcc-10 }
	- deps: { id: newest }
	compiler: { id: gcc-11 }
	- deps: { id: newest }
	compiler: { id: clang-13 }
	- deps: { id: newest }
	compiler: { id: clang-15 }
	- deps: { id: newest }
	compiler: { id: clang-16 }
	- deps: { id: newest }
	build-test-cfg: { id: release }
	- deps: { id: newest }
	build-test-cfg: { id: minsizerel }
	- deps: { id: newest }
	build-test-cfg: { id: relwithdebinfo-asan }
	- deps: { id: newest }
	build-test-cfg: { id: relwithdebinfo-ubsan }
	- deps: { id: newest }
	build-test-cfg: { id: relwithdebinfo-tsan }
	- deps: { id: newest }
	cxx-std: { id: cxx20 }

	# Not using ubuntu-latest, so as to avoid surprises with OS upgrades and such.
	runs-on: ubuntu-22.04

	# The GitHub default job timeout is 6 hours -- extreme. Bound it to something reasonable, just in case
	# something hangs.
	timeout-minutes: 120

	name: ${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}-${{ matrix.cxx-std.id }}${{ matrix.deps.name-suffix }}

	env:
	# Whether this matrix cell shall really run, per the workflow_dispatch job-filter input. See comments in
	# Flow-IPC workflow counterpart. We use same techniques.
	cell-selected: ${{ ((github.event.inputs.job-filter \|\| '') == '') \|\| contains(format('{0}-{1}-{2}{3}', matrix.compiler.id, matrix.build-test-cfg.id, matrix.cxx-std.id, matrix.deps.name-suffix), github.event.inputs.job-filter) }}
	build-dir: ${{ github.workspace }}/build/${{ matrix.build-test-cfg.conan-profile-build-type }}
	install-dir: ${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local
	# For the remaining env entries please see comments in Flow-IPC workflow counterpart. We use same techniques.
	san-suppress-cfg-file: ${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
	san-suppress-cfg-in-file1: sanitize/${{ matrix.build-test-cfg.sanitizer-name }}/suppressions_${{ matrix.compiler.name }}.cfg
	san-suppress-cfg-in-file2: sanitize/${{ matrix.build-test-cfg.sanitizer-name }}/suppressions_${{ matrix.compiler.name }}_${{ matrix.compiler.version }}.cfg
	setup-tests-env: \|
	if [ '${{ matrix.build-test-cfg.sanitizer-name }}' = asan ]; then
	export SAN_SUPP=1
	export SAN_SUPP_CFG=${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
	export ASAN_OPTIONS='disable_coredump=0'
	export LSAN_OPTIONS="suppressions=$SAN_SUPP_CFG"
	echo "ASAN_OPTIONS = [$ASAN_OPTIONS]."
	echo "LSAN_OPTIONS = [$LSAN_OPTIONS]."
	elif [ '${{ matrix.build-test-cfg.sanitizer-name }}' = ubsan ]; then
	export SAN_SUPP=1
	export SAN_SUPP_CFG=${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
	export UBSAN_OPTIONS="disable_coredump=0 print_stacktrace=1 suppressions=$SAN_SUPP_CFG"
	echo "UBSAN_OPTIONS = [$UBSAN_OPTIONS]."
	elif [ '${{ matrix.build-test-cfg.sanitizer-name }}' = tsan ]; then
	export SAN_SUPP=1
	export SAN_SUPP_CFG=${{ github.workspace }}/install/${{ matrix.build-test-cfg.conan-profile-build-type }}/usr/local/bin/san_suppressions.cfg
	export TSAN_OPTIONS="disable_coredump=0 second_deadlock_stack=1 suppressions=$SAN_SUPP_CFG"
	echo "TSAN_OPTIONS = [$TSAN_OPTIONS]."
	fi
	# TODO: Add more *SAN here.
	if [ "$SAN_SUPP" != '' ]; then
	echo 'Sanitizer [${{ matrix.build-test-cfg.sanitizer-name }}] suppressions cfg contents:'
	echo '[[[ file--'
	cat $SAN_SUPP_CFG
	echo '--file ]]]'
	fi
	setup-run-env: \|
	if [ '${{ matrix.build-test-cfg.sanitizer-name }}' = asan ]; then
	export BUILT_CMD_PREFIX='setarch -R'
	echo 'Shall use `setarch -R` to disable ASLR to avoid ASAN-built binary misbehavior.'
	elif [ '${{ matrix.build-test-cfg.sanitizer-name }}' = tsan ]; then
	export BUILT_CMD_PREFIX='setarch -R'
	echo 'Shall use `setarch -R` to disable ASLR to avoid TSAN-built binary misbehavior.'
	fi

	steps:
	- name: Update available software list for apt-get
	if: env.cell-selected == 'true'
	run: \|
	# Update available software list for apt-get.
	lsb_release -a
	sudo apt-get update

	- name: Install clang compiler
	if: \|
	(env.cell-selected == 'true')
	&& matrix.compiler.install && (matrix.compiler.name == 'clang')
	run: \|
	# Install clang compiler.
	wget https://apt.llvm.org/llvm.sh
	chmod u+x llvm.sh
	sudo ./llvm.sh ${{ matrix.compiler.version }}

	- name: Install gcc compiler
	if: \|
	(env.cell-selected == 'true')
	&& matrix.compiler.install && (matrix.compiler.name == 'gcc')
	run: \|
	# Install gcc compiler.
	sudo apt-get install -y software-properties-common
	sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
	sudo apt-get update
	sudo apt-get install -y gcc-${{ matrix.compiler.version }} g++-${{ matrix.compiler.version }}

	- name: Checkout `flow` repository
	if: env.cell-selected == 'true'
	uses: actions/checkout@v6

	- name: Prepare Python virtual environment containing Conan
	if: env.cell-selected == 'true'
	run: \|
	# Prepare Python virtual environment containing Conan.
	# (See comments in Flow-IPC workflow counterpart's analogous step. We use same techniques.)
	pip install virtualenv
	virtualenv ${{ github.workspace }}/conan_virtual_env
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	pip install pip-tools
	pip-compile ${{ github.workspace }}/.github/workflows/requirements.in
	pip-sync ${{ github.workspace }}/.github/workflows/requirements.txt

	- name: Add custom settings for Conan packages
	if: \|
	(env.cell-selected == 'true')
	&& matrix.build-test-cfg.conan-custom-settings-defs
	run: \|
	# Add custom settings for Conan packages.
	# Conan keeps custom settings in settings_user.yml, an overlay merged onto the built-in settings.yml;
	# so simply write the overlay dict from our matrix higher-up.
	# (`conan config home` auto-creates the home dir if needed.)
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	CONAN_USER_SETTINGS_PATH=$(conan config home)/settings_user.yml
	python -c "
	import yaml
	${{ matrix.build-test-cfg.conan-custom-settings-defs }}
	with open('$CONAN_USER_SETTINGS_PATH', 'w') as file:
	yaml.dump(data, file)
	"

	# Note: The C[XX]FLAGS and linker-flags supplied via ${{ matrix.build-test-cfg.conan-profile-custom-conf }}
	# affect not just our objects/libs/executables but also 3rd party dependency builds. For the interesting
	# implications please see notes atop the analogous step in Flow-IPC workflow counterpart.
	- name: Create Conan profile
	if: env.cell-selected == 'true'
	run: \|
	# Create Conan profile.
	cat <<'EOF' > conan_profile
	[settings]
	compiler = ${{ matrix.compiler.name }}
	compiler.version = ${{ matrix.compiler.version }}
	compiler.cppstd = ${{ ((matrix.cxx-std.id == 'cxx20') && '20') \|\| '17' }}
	# TODO: Consider testing with LLVM-libc++ also (with clang anyway).
	compiler.libcxx = libstdc++11
	arch = x86_64
	os = Linux
	build_type = ${{ matrix.build-test-cfg.conan-profile-build-type }}
	${{ matrix.build-test-cfg.conan-profile-custom-settings }}

	[conf]
	tools.build:compiler_executables = {"c": "${{ matrix.compiler.c-path }}", "cpp": "${{ matrix.compiler.cpp-path }}"}
	${{ matrix.build-test-cfg.conan-profile-custom-conf }}

	[buildenv]
	CC = ${{ matrix.compiler.c-path }}
	CXX = ${{ matrix.compiler.cpp-path }}

	[options]
	flow/*:build = True
	flow/*:doc = False
	# `0` here as of this writing would effectively mean `17` too. Specifying `17` explicitly has the same
	# effect in the end but goes down a slightly different code-path in FlowLikeCodeGenerate.cmake. It's not
	# a huge deal... but it's a little nice to cover more paths, so let's do the `0` path when applicable.
	flow/*:build_cxx_std = ${{ ((matrix.cxx-std.id == 'cxx20') && '20') \|\| '0' }}
	EOF
	# `deps: newest` cells only: swap the conanfile.py-pinned dependency versions for the newest ones.
	# See Flow-IPC workflow counterpart.
	if [ '${{ matrix.deps.id }}' = newest ]; then
	cat <<'EOF' >> conan_profile

	[replace_requires]
	${{ matrix.deps.conan-profile-replace-requires }}
	EOF
	fi

	# Cache the Conan-built dependencies (Boost et al): a fresh runner otherwise rebuilds them from source every
	# run. See explanatory comments atop the analogous steps in Flow-IPC workflow counterpart. We use same
	# techniques (including the restore/save split, with the save placed right after the dep-install step, so
	# even a run whose later steps fail still primes the cache; and the flag-snapshot guard in the dep-install
	# step, which makes the restore-keys fallback safe -- see the Caution comment there).
	- name: Restore Conan package cache
	if: env.cell-selected == 'true'
	id: conan-cache-restore
	uses: actions/cache/restore@v6
	with:
	path: ~/.conan2/p
	key: conan2-data-${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}-${{ matrix.cxx-std.id }}${{ matrix.deps.name-suffix }}-${{ hashFiles('conan_profile', 'conanfile.py') }}
	restore-keys: \|
	conan2-data-${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}-${{ matrix.cxx-std.id }}${{ matrix.deps.name-suffix }}-

	- name: Install Flow dependencies with Conan using the profile
	if: env.cell-selected == 'true'
	run: \|
	# Install Flow dependencies with Conan using the profile.
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	${{ env.setup-run-env }}
	# Flag-snapshot guard. For explanation see comments in Flow-IPC workflow counterpart -- both in its
	# analogous step and atop its conan-cache-restore step. We use same techniques.
	if [ '${{ steps.conan-cache-restore.outputs.cache-hit }}' != 'true' ]; then
	flag_sections() { awk '/^\[/{on=0} /^\[(conf\|buildenv)\]/{on=1} on' "$1"; }
	SNAPSHOT=~/.conan2/p/.profile-snapshot
	if [ ! -f $SNAPSHOT ] \|\| ! diff <(flag_sections conan_profile) <(flag_sections $SNAPSHOT); then
	echo 'Restored cache (if any) lacks a flag-match guarantee; evicting its packages (they will rebuild).'
	conan remove '*' -c 2> /dev/null \|\| true
	fi
	fi
	# At least `configure` mini-programs are built with our build-settings, so $BUILT_CMD_PREFIX is required.
	$BUILT_CMD_PREFIX conan install . -pr:a ./conan_profile --build=missing \|\| FAILED=yep
	# Dep build failure is easy to diagnose from stdout/err, unless it happens during a `configure`;
	# then it'll often print something insane like "checking size of void *... 0" and suggest looking at
	# config.log... which will usually reveal the true problem, like a seg-fault or TSAN-triggered problem
	# running the configure-generated test program. So let's get all the `config.log`s we can find
	# and put them where later the log artifact tarball will be sourced; recreate the same dir structure as
	# in the Conan dep build area, but include only the `config.log` files. Note: This is probably only truly
	# useful if indeed the above command fails; but can't hurt to include it regardless.
	SRC=~/.conan2/p
	DST=${{ env.install-dir }}/bin/logs/dep_build/conan-data
	find $SRC -name config.log \| while read -r config_log; do
	# Remove the $SRC part of the path to get a relative path.
	REL_PATH="${config_log#$SRC/}"
	# Determine the directory path within $DST where the config.log should be copied.
	DST_DIR="$DST/${REL_PATH%/*}"
	mkdir -p "$DST_DIR"
	cp -v "$config_log" "$DST_DIR"
	done
	# Prune the source/build trees, which Conan retains after building from source; only the
	# built packages and recipes are needed by this (or later) runs. Keeps cache size down in next step;
	# cache is LRU-based but limited enough to matter in practice as of this writing.
	conan cache clean '*' --source --build
	if [ "$FAILED" != yep ]; then
	# Record the flags this cache's contents were built under, for the flag-snapshot guard (see above).
	cp conan_profile ~/.conan2/p/.profile-snapshot
	fi
	[ "$FAILED" != yep ]

	# (See comment atop the Restore counterpart above.) Skipped on an exact-key restore: the cache would be
	# bit-identical to what is already stored.
	- name: Save Conan package cache
	if: \|
	(env.cell-selected == 'true')
	&& (steps.conan-cache-restore.outputs.cache-hit != 'true')
	uses: actions/cache/save@v6
	with:
	path: ~/.conan2/p
	key: ${{ steps.conan-cache-restore.outputs.cache-primary-key }}

	- name: Build library and demos/tests with Conan
	if: env.cell-selected == 'true'
	run: \|
	# Build libraries and demos/tests with Conan.
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	${{ env.setup-run-env }}
	# Prefixing with $BUILT_CMD_PREFIX here too, in case the build ends up running some binary built with our
	# build-settings. (In Flow-IPC workflow counterpart it definitely does -- the capnp compiler binary.)
	# (`conan build` recomputes the dep-graph and hence needs the profile, just like `conan install`.)
	$BUILT_CMD_PREFIX conan build . -pr:a ./conan_profile

	- name: Install built targets with CMake
	if: env.cell-selected == 'true'
	run: \|
	# Install built targets with CMake.
	cmake --install ${{ env.build-dir }} --prefix ${{ env.install-dir }}
	# Save runner space: blow away build dir after install.
	rm -rf ${{ env.build-dir }}

	# From now on use !cancelled() to try to run any test/demo that exists regardless
	# of preceding failures if any. Same-ish (always()) with the log-upload at the end.
	# Worst-case they'll all fail in super-basic ways and immediately; no extra harm done.

	# From now on save logs in install-dir/bin/logs. They will be tarred up and uploaded
	# as artifacts at the end. Please see comments in Flow-IPC workflow counterpart.
	# We use the same techniques and reasoning. Those may seem overkill given the small number
	# of tests/demos below and their small amount of output (as of this writing); but this can
	# and will grow. The techniques will still apply.

	# Here, as in the other test below, we assemble a suppressions file in case this is a sanitized run.
	# See comments in Flow-IPC workflow counterpart (atop its first test step). We use same techniques.

	- name: Run test/demo [NetFlow echo]
	if: \|
	(env.cell-selected == 'true')
	&& (!cancelled())
	run: \|
	# Run test/demo [NetFlow echo].
	cd ${{ env.install-dir }}/bin
	OUT_DIR=logs/net_flow_echo
	mkdir -p $OUT_DIR
	SUPP_DIR_A=${{ github.workspace }}/src
	{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
	> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } \|\| true
	${{ env.setup-tests-env }}
	${{ env.setup-run-env }}
	$BUILT_CMD_PREFIX ./net_flow_echo_srv.exec 8888 localhost > $OUT_DIR/srv.console.log 2>&1 &
	sleep 1
	if $BUILT_CMD_PREFIX ./net_flow_echo_cli.exec 1 127.0.01 8888 'Hello world!' > $OUT_DIR/cli.console.log 2>&1; \
	then EC=0; else EC=$?; fi
	echo "Client finished with code [$EC]."
	echo 'Server may print exception due to SIGTERM interruption; that is fine.'
	kill `pidof $PWD/net_flow_echo_srv.exec`
	# The executables also produce Flow logs with these hard-coded names in CWD.
	touch flow_echo_srv.log # Create if needed just in case.
	touch flow_echo_cli.log
	mv flow_echo_srv.log $OUT_DIR/srv.log
	mv flow_echo_cli.log $OUT_DIR/cli.log
	[ $EC -eq 0 ]

	- name: Run unit tests
	if: \|
	(env.cell-selected == 'true')
	&& (!cancelled())
	run: \|
	# Run unit tests.
	cd ${{ env.install-dir }}/bin
	# Some newline issues with the possible additional args; so need to make a wrapper script
	# and then redirect, as desired, its output.
	cat <<'EOF' > ${{ env.install-dir }}/bin/run_unit_test.sh
	$BUILT_CMD_PREFIX ./libflow_unit_test.exec "$@"
	EOF
	RUN_IT='/usr/bin/bash -e ${{ env.install-dir }}/bin/run_unit_test.sh'
	OUT_DIR=logs/libflow_unit_test
	mkdir -p $OUT_DIR
	SUPP_DIR_A=${{ github.workspace }}/src
	# As of this writing there are TSAN suppressions for this test specifically. TODO: Revisit them; and then this.
	# Update: Now there are also ASAN (LSAN) suppressions. These are likely permanent as of this writing.
	# Reminder: the following construction handles suppression file(s) from any relevant sanitizer type (if any).
	SUPP_DIR_OWN=${{ github.workspace }}/test/suite/unit_test
	{ cat $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_A/${{ env.san-suppress-cfg-in-file2 }} \
	$SUPP_DIR_OWN/${{ env.san-suppress-cfg-in-file1 }} $SUPP_DIR_OWN/${{ env.san-suppress-cfg-in-file2 }} \
	> ${{ env.san-suppress-cfg-file }} 2> /dev/null; } \|\| true
	${{ env.setup-tests-env }}
	${{ env.setup-run-env }}
	# Sensitive benchmarks in this setting should run and be warned about, if they "fail," but they should not
	# fail the test.
	$RUN_IT --do-not-fail-benchmarks > $OUT_DIR/console.log 2>&1

	- name: Check test/demo logs for non-fatal sanitizer error(s)
	if: \|
	(env.cell-selected == 'true')
	&& (!cancelled()) && (matrix.build-test-cfg.sanitizer-name == 'ubsan')
	run: \|
	# Check test/demo logs for non-fatal sanitizer error(s).
	cd ${{ env.install-dir }}/bin/logs
	# grep returns 0 if 1+ found, 1 if none found, 2+ on error. So check results explicitly instead of -e.
	# Namely, for us, the only OK result is an empty stdout and empty stderr.
	# Otherwise either grep failed (unlikely) or found 1+ problems; either way redirection target will
	# be not-empty, and we will force failure.
	{ grep 'SUMMARY: UndefinedBehaviorSanitizer:' `find . -type f` > san_failure_summaries.txt 2>&1; } \|\| true
	if [ -s san_failure_summaries.txt ]; then
	echo 'Error(s) found. Pipeline will fail. Failures summarized below.'
	echo 'Please peruse uploaded log artifacts, resolve the issues, and run pipeline again.'
	echo '[[[ file--'
	cat san_failure_summaries.txt
	echo '--file ]]]'
	false
	fi
	echo 'No errors found in logs.'

	- name: Package test/demo logs tarball
	if: \|
	(env.cell-selected == 'true')
	&& always()
	run: \|
	# Package test/demo logs tarball.
	cd ${{ env.install-dir }}/bin
	tar cvzf logs.tgz logs
	rm -rf logs # Save runner space.

	- name: Upload test/demo logs (please inspect if failure(s) seen above)
	if: \|
	(env.cell-selected == 'true')
	&& always()
	uses: actions/upload-artifact@v7
	with:
	name: flow-test-logs-${{ matrix.compiler.id }}-${{ matrix.build-test-cfg.id }}-${{ matrix.cxx-std.id }}${{ matrix.deps.name-suffix }}
	path: ${{ env.install-dir }}/bin/logs.tgz

	# TODO: See the crash-debuggability TODO at the analogous spot in Flow-IPC workflow counterpart; it applies
	# here equally.

	doc-and-release:
	needs: [setup, set-vars]
	# Re. job-filter: this job's matrix has exactly 1 cell, so we can just use its (constant) name directly --
	# and at job level, unlike in build-and-test. (Keep the literal in sync with `name` below.)
	if: \|
	(needs.setup.outputs.proceed-else-not == 'true')
	&& (((github.event.inputs.job-filter \|\| '') == '')
	\|\| contains('doc-clang-15-release', github.event.inputs.job-filter))

	strategy:
	fail-fast: false
	matrix:
	compiler:
	# Pick a reasonably modern but pre-installed compiler for building Doxygen/etc.
	- id: clang-15
	name: clang
	version: 15
	c-path: /usr/bin/clang-15
	cpp-path: /usr/bin/clang++-15
	build-cfg:
	- id: release
	conan-profile-build-type: Release
	conan-preset: release

	runs-on: ubuntu-22.04

	name: doc-${{ matrix.compiler.id }}-${{ matrix.build-cfg.id }}

	steps:
	- name: Update available software list for apt-get
	run: sudo apt-get update

	- name: Install Flow dependencies (like Graphviz) with apt-get
	run: sudo apt-get install -y graphviz

	- name: Checkout `flow` repository
	uses: actions/checkout@v6
	with:
	# Avoids the duplicate-authorization-error in (not-frequently-invoked) doc_check_in step later down.
	persist-credentials: false

	- name: Prepare Python virtual environment containing Conan
	run: \|
	# Prepare Python virtual environment containing Conan.
	# (See comment in the build-and-test job's analogous step.)
	pip install virtualenv
	virtualenv ${{ github.workspace }}/conan_virtual_env
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	pip install pip-tools
	pip-compile ${{ github.workspace }}/.github/workflows/requirements.in
	pip-sync ${{ github.workspace }}/.github/workflows/requirements.txt

	# See comments in Flow-IPC workflow counterpart. We use same techniques.
	- name: (On release creation only) Signal Pages about release (web site should update)
	if: success() && (!cancelled()) && startsWith(github.ref, 'refs/tags/v')
	run: \|
	# Signal Pages about release (web site should update).
	curl --fail-with-body -X POST \
	-H 'Accept: application/vnd.github.v3+json' \
	-H 'Authorization: token ${{ secrets.GIT_BOT_PAT }}' \
	'https://api.github.com/repos/Flow-IPC/flow-ipc.github.io/dispatches' \
	-d '{"event_type": "flow-sync-doc-event", "client_payload": {"version": "${{ github.ref_name }}"}}'

	- name: Create Conan profile
	run: \|
	# Create Conan profile.
	cat <<EOF > conan_profile
	[settings]
	compiler = ${{ matrix.compiler.name }}
	compiler.version = ${{ matrix.compiler.version }}
	compiler.cppstd = 17
	compiler.libcxx = libstdc++11
	arch = x86_64
	os = Linux
	build_type = ${{ matrix.build-cfg.conan-profile-build-type }}

	[conf]
	tools.build:compiler_executables = { "c": "${{ matrix.compiler.c-path }}", "cpp": "${{ matrix.compiler.cpp-path }}" }

	[buildenv]
	CC = ${{ matrix.compiler.c-path }}
	CXX = ${{ matrix.compiler.cpp-path }}

	[options]
	flow/*:build = False
	flow/*:doc = True
	EOF

	# Cache Conan-built deps (here: mainly Doxygen). See longer explanation atop the analogous steps in the
	# build-and-test job. (Its flag-snapshot guard is deliberately not replicated here; see reasoning in
	# Flow-IPC workflow counterpart's analogous spot.)
	- name: Restore Conan package cache
	id: conan-cache-restore
	uses: actions/cache/restore@v6
	with:
	path: ~/.conan2/p
	key: conan2-data-doc-${{ hashFiles('conan_profile', 'conanfile.py') }}
	restore-keys: \|
	conan2-data-doc-

	- name: Install Flow dependencies (like Doxygen) with Conan using the profile
	run: \|
	# Install Flow dependencies (like Doxygen) with Conan using the profile.
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	conan install . -pr:a ./conan_profile --build=missing
	# (See comment in the build-and-test job's analogous step.)
	conan cache clean '*' --source --build

	- name: Save Conan package cache
	if: steps.conan-cache-restore.outputs.cache-hit != 'true'
	uses: actions/cache/save@v6
	with:
	path: ~/.conan2/p
	key: ${{ steps.conan-cache-restore.outputs.cache-primary-key }}

	- name: Generate code documentation using Conan and Doxygen
	run: \|
	# Generate code documentation using Conan and Doxygen.
	source ${{ github.workspace }}/conan_virtual_env/bin/activate
	# (`conan build` recomputes the dep-graph and hence needs the profile, just like `conan install`.)
	conan build . -pr:a ./conan_profile

	- name: Create documentation tarball (full docs, API-only docs, landing page)
	run: \|
	# Create documentation tarball (full docs, API-only docs, landing page).
	cd ${{ github.workspace }}/doc/flow_doc
	${{ github.workspace }}/tools/doc/stage_generated_docs.sh \
	${{ github.workspace }}/build/${{ matrix.build-cfg.conan-profile-build-type }}

	- name: Upload documentation tarball
	uses: actions/upload-artifact@v7
	with:
	name: flow-doc
	path: ${{ github.workspace }}/doc/flow_doc.tgz

	# See comments in Flow-IPC workflow counterpart. We use same techniques.
	- name: (`main` branch only) Check-in generated documentation directly into source control
	id: doc_check_in
	if: success() && (!cancelled()) && (github.ref == 'refs/heads/main')
	run: \|
	# Check-in generated documentation directly into source control.
	echo 'generated/ docs have been added or replaced locally; mirroring this into checked-in tree.'
	git config user.name github-actions
	git config user.email github-actions@github.com
	git config --local http.https://github.com/.extraheader \
	"AUTHORIZATION: basic $(echo -n x-access-token:${{ secrets.GIT_BOT_PAT }} \| base64 -w 0)"
	git config --local http.version HTTP/1.1
	cd ${{ github.workspace }}/doc/flow_doc
	git rm -r --cached generated \|\| echo 'No generated/ currently checked in; no problem.'
	git add generated
	git commit -m '${{ needs.set-vars.outputs.doc-commit-message }}'
	git push origin main

	# See comments in Flow-IPC workflow counterpart. We use same techniques.
	- name: (`main` branch only) Signal Pages that we have updated the generated docs (web site should update)
	if: success() && (!cancelled()) && (steps.doc_check_in.outcome != 'skipped')
	run: \|
	# Signal Pages that we have updated the generated docs (web site should update).
	curl --fail-with-body -X POST \
	-H 'Accept: application/vnd.github.v3+json' \
	-H 'Authorization: token ${{ secrets.GIT_BOT_PAT }}' \
	'https://api.github.com/repos/Flow-IPC/flow-ipc.github.io/dispatches' \
	-d '{"event_type": "flow-sync-doc-event"}'

Navigation Menu

Repository navigation

Flow pipeline #721

Workflow file

Flow pipeline #721

Uh oh!

Workflow file for this run