From time to time, a test developer might want to use openQA to execute a test suite from a different test harness than openQA, but still use openQA to setup test scenarios and prepare the environment for a test suite run; for this case openQA has the ability to process logs from external harnesses, and display the results integrated within the job results of the webUI.
One could say that a Test Harness is supported if its output is compatible with the available {parser-format}, such as LTP, and also xUnit or JUnit, but this can be easily extended to include more formats, such as RSpec or TAP.
The requirements to use this functionality, are quite simple:
-
The test harness must produce a compatible format with supported {parser-format}.
-
The test results can be uploaded via
testapi::parse_extra_logwithin an openQA tests. -
The test results can also be uploaded via web Web Api endpoint.
openQA will store these results in its own internal format for easier presentation, but still will allow the original file to be downloaded.
If a test developer wishes to use the functional interface, after finishing the
execution of the the testing too, calling testapi::parse_extra_log with the
location to a the file generated.
From within a common openQA test distribution, a developer can use parse_extra_log
to upload a text file that contains a supported test output:
script_run('prove --verbose --formatter=TAP::Formatter::JUnit t/28-logging.t > junit-logging.xml');
parse_extra_log('XUnit','junit-logging.xml');Current parser formats:
-
OpenQA::Parser::Format::TAP,
-
OpenQA::Parser::Format::JUnit
-
OpenQA::Parser::Format::LTP
-
OpenQA::Parser::Format::XUnit,
The parser is a base class that acts as a serializer/deserializer for the elements inside of it, it allows to be extended so new formats can be easily added.
The base class is exposing 4 Mojo::Collections available, according to what openQA would require to map the results correctly, 1 extra collection is provided for arbitrary data that can be exposed. The collections represents respectively: test results, test definition and test output.
In structured data mode, elements of the collections are objects. They can be
of any type, even though subclassing or objects of type of OpenQA::Parser::Result
are preferred.
One thing to keep in mind, is that in case deeply nested objects need to be parsed
like hash of hashes, array of hashes, they would need to subclass OpenQA::Parser::Result
or OpenQA::Parser::Results respectively.
As an example, JUnit format can be parsed this way:
use OpenQA::Parser::Format::JUnit;
my $parser_result = OpenQA::Parser::Format::JUnit->new->load("file.xml");
# Now we can access to parsed tests as seen by openQA:
$parser_result->tests->each(sub {
my $test = shift;
print "Test name: ".$test->name;
});
my @all = $parser_result->tests->each;
my @tests = $parser->tests->search(name => qr/1_running_upstream_tests/);
my $first = $parser->tests->search(name => qr/1_running_upstream_tests/)->first();
my $binary_data = $parser->serialize();
# Now, we can also store $binary_data and retrieve it later.
my $new_parser_from_binary = OpenQA::Parser::Format::JUnit->new->deserialize($binary_data);
# thus this works as expected:
$new_parser_from_binary->tests->each( sub {
my $test = shift;
print "Test name: ".$test->name;
});
# We can also serialize all to JSON
my $json_serialization = $parser->to_json;
# save it and access it later
my $from_json = OpenQA::Parser::Format::JUnit->from_json($json_serialization);It is important to know that openQA’s internal mapping for test results works operating almost entirely on the filesystem, leaving only the test modules to be registered into the database, this leads to the following relation: A test module’s name is used to create a file with details (details-$testmodule.json), that will contain a reference to step details, which is a collection of references to files, using a field "text" as tie in, and expecting a filename.