SSH execution fails with "Argument list too long" when environment variables are very large

[thrix]

Problem

When running tests via SSH, tmt fails with the error:

[Errno 7] Argument list too long: 'ssh'

This occurs when the total size of environment variables exceeds the Linux kernel's ARG_MAX limit.

Root Cause

tmt passes environment variables to the remote host by converting them to shell export statements and including them directly in the SSH command line:

https://github.com/teemtee/tmt/blob/7686aaac35cc3d5d2ba95e4faa6a54f1b23db9a8/tmt/steps/provision/__init__.py#L3115-L3117

remote_commands: ShellScript = ShellScript.from_scripts(
    self._prepare_environment(env).to_shell_exports()
)

The entire environment is then passed as a single argument to SSH:

ssh_command += [self._ssh_guest, remote_command]

When environment variables are very large (e.g., a SNAPSHOT_b64 variable containing Base64-encoded JSON with 60+ container image references), the total command line exceeds the OS limit and Python's subprocess raises OSError: [Errno 7] Argument list too long.

Reproduction

This was observed in a Testing Farm run where the SNAPSHOT_b64 environment variable contained a massive Base64-encoded JSON with many container image references:

• Pipeline log: https://artifacts.osci.redhat.com/testing-farm/b7a6ea45-3218-49ee-8802-70c68ffed5f1/pipeline.log

The relevant error from the log:

provision
    ...
    fail: [Errno 7] Argument list too long: 'ssh'
    
The exception was caused by 1 earlier exceptions

Cause number 1:

    provision step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        [Errno 7] Argument list too long: 'ssh'

Potential Solutions

This is a fundamental OS-level limitation. Possible approaches to address it:

1. Write environment to a file and transfer it: Instead of passing environment in the command line, write exports to a temporary file, rsync it to the guest, and source it before running commands.

2. Use SSH's SendEnv/AcceptEnv: SSH can forward specific environment variables, but this requires sshd configuration on the guest which may not be available.

3. Stream environment via stdin: Pass large environment variables through SSH's stdin instead of command line arguments.

4. Detect and warn about oversized environments: Add a check for environment size and provide a clear error message suggesting users reduce the environment size.

5. Environment variable compression/chunking: For specific large variables, compress or split them.

None of these solutions are trivial to implement, and some require changes to the guest's SSH configuration which may not be possible in all environments.

Environment

• tmt version: 1.65.0
• Provision method: connect (Testing Farm)