fast import: basic python test

[NanoBjorn]

We did not have any tests on fast_import binary yet.

In this PR I have introduced:

• FastImport class and tools for testing in python
• basic test that runs fast import against vanilla postgres and checks that data is there

Should be merged after #10251

[jcsp]

Perhaps makes sense to make this an extension to test_pgdata_import_smoke, which has this comment currently:

    # We have a Postgres data directory now.
    # Make a localfs remote storage that looks like how after `fast_import` ran.
    # TODO: actually exercise fast_import here

[NanoBjorn]

Just thought that it might not worth doing it all in the same test due to increased complexity -> more complicated debugging if something is failing. Right now it is using vanilla postgres pgdata as we do in fast import, so it should be enough for testing pageserver part.

Moved mine simple test to the same file, but will keep test_pgdata_import_smoke the same. Also added a todo to test full import flow separately, which will complement test_pgdata_import_smoke and test_fast_import_binary, but not complicate any of those.

[NanoBjorn]

WDYT?

[VladLazar]

I agree with John that extending test_pgdata_import_smoke would be more valuable.
If I'm reading the TODO after your new test correctly, it would be completed by extending test_pgdata_import_smoke with what you have in your new test.

[NanoBjorn]

Agreed in DM that I will keep test_pgdata_import_smoke as is and add another one with s3 + pageserver + fast import binary, but if you don't mind I would do it in #10407 because it will anyway need something close

[github-actions]

7359 tests run: 6977 passed, 0 failed, 382 skipped (full report)

---

Flaky tests (6)

Postgres 17

• test_metrics_normal_work: release-x86-64
• test_timeline_ancestor_detach_idempotent_success[shards_initial_after2]: release-x86-64

Postgres 16

• test_physical_replication_config_mismatch_max_locks_per_transaction: release-arm64

Postgres 15

• test_metrics_normal_work: release-x86-64
• test_detached_receives_flushes_while_being_detached: release-arm64

Postgres 14

• test_metrics_normal_work: release-arm64

Code coverage* (full report)

• functions: 33.6% (8432 of 25073 functions)
• lines: 49.1% (70545 of 143652 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
fd09d31 at 2025-01-17T16:33:10.731Z :recycle:}

[VladLazar]

nit: you can match instead and avoid the unwrap

let pg_port = match args.pg_port {
  Some(port) => port,
  None {
    info!("...");
    5432
  }
};

[NanoBjorn]

Changed to a little shorter version with unwrap_or_else

[VladLazar]

nit: why is the copy needed?

[NanoBjorn]

extra_env is passed as dict by reference, so it seems better to freeze it this way at the moment of initialization

basically copied from

neon/test_runner/fixtures/neon_cli.py

Lines 177 to 180 in fd09d31

	if extra_env is None:
	env_vars = {}
	else:
	env_vars = extra_env.copy()

[VladLazar]

I agree with John that extending test_pgdata_import_smoke would be more valuable.
If I'm reading the TODO after your new test correctly, it would be completed by extending test_pgdata_import_smoke with what you have in your new test.