How to reproduce jobs locally? -> improve logs

[DavHau]

I just had a failing instance of ofborg-eval-lib-tests: https://gist.github.com/GrahamcOfBorg/85196b733b08a4c1c29909c2a1a5af42

... and I'm having a hard time trying to debug this.

Several things are missing here:

1. Which commands were executed: The logs do not state which command was executed.
2. Definition unclear: grepping for ofborg-eval-lib-tests in either nixpkgs or nixos/ofborg leads to zero results. Where is the job defined?
3. Reproduce locally: There are no instructions in the logs how I can reproduce that specific job locally.

All of the above should be easy fixes and would make debugging this stuff a lot easier.

EDIT: I just noticed that it displays the command in github next to the job. So that helps. But still it would be nice to have this clearly stated in the logs as well. Also the command displayed in github nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix doesn't actually seem to work and fails with:

error: store path 'q7375xm60a582561sq84mxyx13j79cjb-.version' starts with illegal character '.'

[hraban]

Related, though not exactly the same as your question: I'd love an easy, one-shot way to run an entire hermetic environment exactly equal to the ofborg runners that run as nixos/nixpkgs CI. E.g. a full nixos config so you can repro CI failures locally and iterate on it before pushing a fix.

[raboof]

I agree this would be very helpful.

I just noticed that it displays the command in github next to the job. So that helps. But still it would be nice to have this clearly stated in the logs as well. Also the command displayed in github nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix doesn't actually seem to work and fails

Additionally, it seems GitHub doesn't show the ' characters, making this rather confusing.

[mweinelt]

Most of the commands ofBorg runs can be seen with proper tokenization in the code here:

ofborg/ofborg/src/tasks/eval/nixpkgs.rs

Lines 433 to 562 in eb120ff

	fn evaluation_checks(&self) -> Vec<EvalChecker> {
	// the value that's passed as the nixpkgs arg
	let nixpkgs_arg_value = format!(
	"{{ outPath=./.; revCount=999999; shortRev=\"{}\"; rev=\"{}\"; }}",
	&self.job.pr.head_sha[0..7],
	&self.job.pr.head_sha,
	);
	vec![
	EvalChecker::new(
	"package-list",
	nix::Operation::QueryPackagesJson,
	vec![String::from("--file"), String::from(".")],
	self.nix.clone(),
	),
	EvalChecker::new(
	"package-list-with-aliases",
	nix::Operation::QueryPackagesJson,
	vec![
	String::from("--file"),
	String::from("."),
	String::from("--arg"),
	String::from("config"),
	String::from("{ allowAliases = true; }"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"lib-tests",
	nix::Operation::Build,
	vec![
	String::from("--arg"),
	String::from("pkgs"),
	String::from("import ./. {}"),
	String::from("./lib/tests/release.nix"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixos",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./nixos/release-combined.nix"),
	String::from("-A"),
	String::from("tested"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixos-options",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./nixos/release.nix"),
	String::from("-A"),
	String::from("options"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixos-manual",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./nixos/release.nix"),
	String::from("-A"),
	String::from("manual"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixpkgs-manual",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./pkgs/top-level/release.nix"),
	String::from("-A"),
	String::from("manual"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixpkgs-tarball",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./pkgs/top-level/release.nix"),
	String::from("-A"),
	String::from("tarball"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"nixpkgs-unstable-jobset",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value.clone(),
	String::from("./pkgs/top-level/release.nix"),
	String::from("-A"),
	String::from("unstable"),
	],
	self.nix.clone(),
	),
	EvalChecker::new(
	"darwin",
	nix::Operation::Instantiate,
	vec![
	String::from("--arg"),
	String::from("nixpkgs"),
	nixpkgs_arg_value,
	String::from("./pkgs/top-level/release.nix"),
	String::from("-A"),
	String::from("darwin-tested"),
	],
	self.nix.clone(),
	),
	]
	}