running the test suite against an installed librnp (and/or against the binaries rnp and rnpkeys)

[dkg]

For the debian testing infrastructure it's generally desirable to be able to run the test suite against the installed version of the package.

In particular, I would like to be able to run the test suite against whatever system-installed librnp is there. But I'm not enough of a cmake guru to know how to make this happen easily. The main examples I can see are in the ci/ folder, but those seem to expect general system privileges, or installing software from the wider Internet.

Is there a way that I am missing that I can run some substantial portion of the test suite against an existing, installed librnp-dev package (that package would contain headers, pkg-config info, and the appropriate .so)?

If there isn't a clear way to do that, is that something that would be desirable upstream? having a proper "autopkgtest" for the debian package would make it easier to get new (non-buggy) versions of rnp into debian testing more quickly. I could of course hand-write such a test and ship it in debian on my own, but if there are parts of the upstream RNP test suite that should be run against the installed library, i'd prefer to source them directly from upstream.

Note that this sort of test has to test the installed version -- so it's likely going to be limited to tests at the library (librnp) or binary (rnp or rnpkeys) API boundaries. It should not include any internal unit tests that require a rebuild of the source.

For example, src/tests/support.h seems to #include several files that are not part of the system librnp-dev installation (e.g. "../rnp/fficli.h"), which makes it difficult to know what code the test is running against.

As a first pass, i could just try building all of src/examples/*.c, but i'm not sure how to exercise them to test that they behave as expected. so it'd be great if there was something more comprehensive for testing the installed library, as well as some sort of binary-boundary test that just exercises rnp and rnpkeys directly.

If you have any questions about this situation, i'm happy to try to dig up answers for them. If you know of something easy i can already do (i've only done a cursory skim, i haven't looked in depth), please let me know.

[dkg]

fwiw, i've uploaded some pretty simple scripts that try to exercise the stuff in src/examples/*.c as a first debian autopkgtest. it would be great to have an easy way to trigger tests that exercise rnp and rnpkeys directly (e.g. the cli subset of the test suite) against the versions already found in $PATH. If that's already there, i'd appreciate a pointer to it!

[ni4]

Unfortunately we do not have an easy way to do that right now. While there are numerous FFI tests (which tests API) and CLI tests (which do self-tests and tests against the GnuPG), those are tightly entangled.

[ronaldtse]

I agree with @dkg that we should have a test suite that exercises across the API and CLI that runs independently of external software. Maybe we can take some effort to refactor existing tests so that the tests can run by themselves and can also test against GnuPG when asked to?

[ni4]

The first things which comes to head are:

• extract from rnp_tests something like ffi_tests, which links only to the lib (on path) and doesn't use any internal headers.
• add some command like parameter, like --installation to the cli-tests.py, so it will run executables which are on PATH, and do GnuPG passes only if it is found.

[dkg]

fwiw, i don't mind at all if there are tests that depend on non-RNP tooling (like GnuPG), as long as the necessary tooling is clearly identified, and dependent tests can be easily disabled if the given tool is unavailable.

My goal is to test rnp itself as installed, not as locally built.

I like @ni4's suggestion of making the ffi tests and cli tests configurable so that they can point to either the system-installed copy of rnp (and can be built and run without building any of librnp or rnp itself), or the locally-built copy.

[ronaldtse]

@antonsviridenko do you have time for this? Thanks!

[antonsviridenko]

ok

[antonsviridenko]

@dkg

Note that this sort of test has to test the installed version -- so it's likely going to be limited to tests at the library (librnp) or binary (rnp or rnpkeys) API boundaries. It should not include any internal unit tests that require a rebuild of the source.

Building the source for testing will be definitely required and seems like autopkgtest allows that:

build-needed

The tests need to be run from a built source tree. The test runner
will build the source tree (honouring the source package's build
dependencies), before running the tests. However, the tests are
not entitled to assume that the source package's build
dependencies will be installed when the test is run.

Tests binary should be built that is not normally built and installed in release builds and it should be linked with installed version of librnp (now it just links with the static librnp from the build tree).

[ni4]

Building the source for testing will be definitely required and seems like autopkgtest allows that:

Are you talking about the tests sources or RNP sources? We may separate FFI tests from the library tests (which use sources), so part of the tests would do only calls on already built RNP via the FFI interface (like CLI does).

[antonsviridenko]

About sources from this repo in general.