Add testing of Filebeat version 9.2 #3857

PavelShilin89 · 2025-10-23T22:56:20Z

Type of Change

Documentation update and test

Description of the Change

Added Filebeat 9.2 support to documentation and integration tests.
Changes:
Added new section "Configuration for Filebeat 9.2+"
Updated supported version from "9.0" to "9.2"
Updated integration test to handle Filebeat 9.2 with filestream input
Added prospector.scanner.fingerprint.enabled: false for small files
Key difference in 9.2: Filebeat 9.2 requires disabling fingerprint feature for files < 1024 bytes.

Related Issue

Add testing of Filebeat version 9.2 #3856

…lebeat-versions.rec

…ype.

github-actions · 2025-10-24T05:09:37Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 503s
👉 Check Action Results for commit 6f0366e

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-jina-remote.rec

test/clt-tests/mcl/auto-embeddings-jina-remote.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
    local file1="$1" file2="$2"

    awk '
    NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
    {
        dot += a[FNR]*$1
        sumb2 += $1*$1
    }
    END {
        print dot / (sqrt(suma2) * sqrt(sumb2))
    }' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
+--------------+
| record_count |
+--------------+
- |            1 |
+ |            0 |
+--------------+
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt

SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)

echo "Cosine similarity: $SIMILARITY"

RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
    if (sim > 0.99)
        print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
    else
        print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')

echo "$RESULT"

rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ Cosine similarity: -nan
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
- single_field_md5: #!/[0-9a-f]{32}/!#
+ multi_field_md5: d41d8cd98f00b204e9800998ecf8427e
- SUCCESS: Different FROM specifications produce different vectors
+ single_field_md5: d41d8cd98f00b204e9800998ecf8427e
+ INFO: FROM field comparison result
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK

sanikolaev

discussed in slack, pls update the docs

…ns.rec

github-actions · 2025-10-27T23:15:09Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2486s
👉 Check Action Results for commit cd52762

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-10-28T00:48:42Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2308s
👉 Check Action Results for commit 60cc7e2

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

This reverts commit c3f155e.

github-actions · 2025-10-28T07:35:27Z

clt

❌ CLT tests in test/clt-tests/sharding/cluster/ test/clt-tests/sharding/functional/ test/clt-tests/sharding/replication/ test/clt-tests/sharding/syntax/
✅ OK: 18
❌ Failed: 1
⏳ Duration: 341s
👉 Check Action Results for commit ea8d26a

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/sharding/replication/create-table-5-nodes-60-shards.rec

test/clt-tests/sharding/replication/create-table-5-nodes-60-shards.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=5
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
export CLUSTER_NAME=c
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for port in 1306 2306 3306 4306 5306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: Node synced"; done && cat /tmp/status_1306.log
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create table c:t(id bigint) shards='60' rf='3' timeout='60'"
––– output –––
+ ERROR 1064 (42000) at line 1: Waiting timeout exceeded.
––– input –––
for i in 1 2 3 4 5; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep 'Table: system.t' | sort -V
––– output –––
OK
––– input –––
for i in 1 2 3 4 5; do mysql -h0 -P${i}306 -e "desc t\G" | grep 'Create Table'; done
––– output –––
OK
––– input –––
for i in 1 2 3 4 5; do mysql -h0 -P${i}306 -e "show tables from system\G" | grep 'Table: system.t' | wc -l; done
––– output –––
OK

github-actions · 2025-10-28T07:51:44Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2413s
👉 Check Action Results for commit ea8d26a

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

…-versions.rec Co-authored-by: Sergey Nikolaev <[email protected]>

github-actions · 2025-10-28T10:40:22Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 479s
👉 Check Action Results for commit 66ddff6

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-jina-remote.rec

test/clt-tests/mcl/auto-embeddings-jina-remote.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
    local file1="$1" file2="$2"

    awk '
    NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
    {
        dot += a[FNR]*$1
        sumb2 += $1*$1
    }
    END {
        print dot / (sqrt(suma2) * sqrt(sumb2))
    }' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt

SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)

echo "Cosine similarity: $SIMILARITY"

RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
    if (sim > 0.99)
        print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
    else
        print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')

echo "$RESULT"

rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
- multi_field_md5: #!/[0-9a-f]{32}/!#
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
- single_field_md5: #!/[0-9a-f]{32}/!#
+ multi_field_md5: 4c3c28143432366c75071062f7709762
- SUCCESS: Different FROM specifications produce different vectors
+ single_field_md5: d41d8cd98f00b204e9800998ecf8427e
+ SUCCESS: Different FROM specifications produce different vectors
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK

…ticoresoftware/manticoresearch into test/added-test-filebeat-9.2

github-actions · 2025-10-28T10:57:29Z

test_support_filebeat_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-filebeat-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 381s
👉 Check Action Results for commit 3aa28e3

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y curl jq > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
timeout 420 bash -c 'echo "[]" > /tmp/filebeat_tags.json; page=1; attempts=0; max_attempts=3; while [ $attempts -lt $max_attempts ]; do attempts=$((attempts+1)); if curl -s --fail --max-time 10 "https://hub.docker.com/v2/repositories/elastic/filebeat/tags/?page_size=1000&page=$page" | tee /tmp/page.json | jq -e ".next" > /dev/null; then jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; page=$((page+1)); attempts=0; else break; fi; done; jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; VERSIONS=$(cat /tmp/filebeat_tags.json | grep -E "^([7-9]|[1-9][0-9]+).[0-9]+.[0-9]+$" | grep -E "^(7.(1[7-9]|[2-9][0-9])|[8-9].[0-9]+|9.[0-9]+|[1-9][0-9]+.[0-9]+).[0-9]+$" | sed -E "s/^([0-9]+.[0-9]+).[0-9]+$/\1/" | grep -v "rc|beta|alpha" | sort -V | uniq); echo "$VERSIONS"; mkdir -p /tmp/filebeat_cache; echo "Preparation done"; for version in $VERSIONS; do archive="/tmp/filebeat_cache/filebeat-${version}.0-linux-x86_64.tar.gz"; echo ">>> Checking Filebeat $version ..."; if [ -f "$archive" ] && gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo ">>> Downloading Filebeat $version ..."; wget -q --timeout=30 "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${version}.0-linux-x86_64.tar.gz" -O "$archive" && { if gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo "✗ Archive for $version is corrupted"; rm -f "$archive"; fi; }; fi; done'
––– output –––
OK
––– input –––
set +H && mkdir -p /tmp/filebeat_cache && echo "Preparation done"
––– output –––
OK
––– input –––
cat << 'EOF' > /tmp/filebeat-single-test.sh
#!/usr/bin/env bash
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "✗ Usage: $0 <filebeat_version>" >&2
  return 1 2>/dev/null || exit 1
fi

version="$1"
full_version="${version}.0"
echo ">>> Testing Filebeat version: $version"

# Prepare test log
echo -e "2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > /var/log/dpkg.log

log_lines=$(wc -l < /var/log/dpkg.log)
if [ "$log_lines" -eq 5 ]; then
  echo "✓ Log file has 5 lines"
else
  echo "✗ Error: Expected 5 lines, got $log_lines" >&2
  return 1 2>/dev/null || exit 1
fi

# Check Manticore availability
if ! curl -s localhost:9308/cli_json -d 'SHOW TABLES' | jq -e '.[0].data' > /dev/null; then
  echo "✗ Error: Manticore Search unavailable" >&2
  return 1 2>/dev/null || exit 1
fi
echo "✓ Manticore Search available"

# Create table
mysql -h0 -P9306 -e "
DROP TABLE IF EXISTS dpkg_log;
CREATE TABLE dpkg_log (
  id BIGINT,
  message TEXT INDEXED STORED,
  host JSON,
  agent JSON,
  input JSON,
  log JSON,
  ecs JSON,
  \`@timestamp\` TEXT INDEXED STORED
);"

# Install Filebeat
mkdir -p /usr/share/filebeat /tmp/fb-data-${version}
tar -xzf "/tmp/filebeat_cache/filebeat-${full_version}-linux-x86_64.tar.gz" -C /usr/share/filebeat
FB_DIR="/usr/share/filebeat/filebeat-${full_version}-linux-x86_64"

# Clean previous registry data
rm -rf /tmp/fb-data-${version}/*

skip_filebeat=0


# For all 9.x versions, use filestream with fingerprint disabled
if [[ "$version" =~ ^9\. ]]; then
  echo ">>> Testing Filebeat $version with filestream input and fingerprint disabled..."

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: filestream
  id: dpkg-filestream-input
  enabled: true
  paths: ["/var/log/dpkg.log"]
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true

      if [ "$row_count" -eq 5 ]; then
        echo "✓ Row count check for $version: $row_count rows"
        structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
        has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
        has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
        if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
          echo "✓ Structure check for $version: passed"
          echo "✓ Filebeat version $version tested successfully"
          skip_filebeat=1
        fi
      fi
      break
    fi
    sleep 1
  done

  if [ "$skip_filebeat" -eq 0 ]; then
    kill $FB_PID 2>/dev/null || true
    wait $FB_PID 2>/dev/null || true
    mysql -h0 -P9306 -e "TRUNCATE TABLE dpkg_log" 2>/dev/null || true
    echo "⚠ Real Filebeat $version failed, falling back to simulation..."
  fi
fi

# For versions with compatibility issues that couldn't be fixed above, use simulation
if [ "$skip_filebeat" -eq 0 ] && ([[ "$version" == "9.0" ]] || [[ "$version" == "9.1" ]] || [[ "$version" == "7.17" ]] || [[ "$version" == "8.0" ]] || [[ "$version" == "8.1" ]]); then
  echo ">>> Using alternative approach for Filebeat $version..."

  # Instead of running Filebeat, simulate with direct data insertion via MySQL
  mysql -h0 -P9306 -e "BEGIN"
  timestamp=$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")

  # Insert each log line
  line_count=0
  while IFS= read -r line; do
    line_count=$((line_count + 1))
    id=$(($(date +%s) * 1000 + line_count))
    escaped_line=$(echo "$line" | sed 's/"/\\"/g')
    mysql -h0 -P9306 -e "INSERT INTO dpkg_log VALUES ($id, '$escaped_line', '{\"name\":\"testhost\"}', '{}', '{}', '{}', '{}', '$timestamp')"
    echo "✓ Inserted log line $line_count"
  done < /var/log/dpkg.log

  mysql -h0 -P9306 -e "COMMIT"

  # Check row count
  row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+')
  if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -eq 5 ]; then
    echo "✓ Filebeat $version simulation: inserted all logs"
    echo "✓ Row count check for $version: $row_count rows"

    # Structure check
    structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
    has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
    has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")

    if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
      echo "✓ Structure check for $version: passed"
      echo "✓ Filebeat version $version tested successfully"
      skip_filebeat=1
    else
      echo "✗ Structure check failed: missing required fields"
      return 1 2>/dev/null || exit 1
    fi
  else
    echo "✗ Error: Expected 5 rows, got $row_count"
    return 1 2>/dev/null || exit 1
  fi
fi

# If we've already processed with alternative approach, skip standard Filebeat run
if [ "$skip_filebeat" -eq 1 ]; then
  return 0 2>/dev/null || exit 0
fi
# Determine major version to choose configuration type
major_version=$(echo "$version" | cut -d'.' -f1)
minor_version=$(echo "$version" | cut -d'.' -f2)

# For version 9.2 and higher, use filestream input
if [ "$major_version" -eq 9 ] && [ "$minor_version" -ge 2 ]; then
  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: filestream
  id: dpkg-filestream-input
  enabled: true
  paths: ["/var/log/dpkg.log"]
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML
# For versions 8.1-8.10: allow_older_versions WITHOUT compression_level
# (compression is OFF by default until 8.11)
elif [[ "$major_version" -eq 8 ]] && [[ "$minor_version" -ge 1 ]] && [[ "$minor_version" -le 10 ]]; then
  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths: ["/var/log/dpkg.log"]
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML
# For versions 8.11+: compression is ON by default since 8.11, so we need compression_level: 0
elif [[ "$major_version" -eq 8 ]] && [[ "$minor_version" -ge 11 ]]; then
  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths: ["/var/log/dpkg.log"]
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML
else
  # For versions before 8.1
  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths: ["/var/log/dpkg.log"]
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML
fi

# Start Filebeat (only for non-problematic versions)
echo ">>> Starting Filebeat..."

if [[ "$version" =~ ^8\.1[7-9]$ || "$version" =~ ^8\.[2-9][0-9]$ || "$version" == "9.2" ]]; then
  # For newer versions (8.17+ and 9.2), use the 'run' command
  if "${FB_DIR}/filebeat" help 2>&1 | grep -q "run"; then
    "${FB_DIR}/filebeat" run -e -c "${FB_DIR}/filebeat.yml" > "/tmp/fb-log-${version}.txt" 2>&1 &
  else
    "${FB_DIR}/filebeat" -e -c "${FB_DIR}/filebeat.yml" > "/tmp/fb-log-${version}.txt" 2>&1 &
  fi
else
  # For older versions
  "${FB_DIR}/filebeat" -e -c "${FB_DIR}/filebeat.yml" > "/tmp/fb-log-${version}.txt" 2>&1 &
fi

FB_PID=$!

# Wait for Filebeat to start and logs to be published
echo ">>> Waiting for Filebeat to publish events..."
sleep 6

# Wait for data to appear (max 30 seconds)
max_wait=30
elapsed=0
while [ $elapsed -lt $max_wait ]; do
  row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
  if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
    break
  fi
  sleep 2
  elapsed=$((elapsed + 2))
done

# Stop Filebeat gracefully
kill "$FB_PID" 2>/dev/null || true
sleep 2

# Verify results
row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+')
if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -eq 5 ]; then
  echo "✓ Filebeat $version processed logs"
  echo "✓ Row count check for $version: $row_count rows"

  # Structure check
  structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
  has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
  has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")

  if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
    echo "✓ Structure check for $version: passed"
    echo "✓ Filebeat version $version tested successfully"
  else
    echo "✗ Structure check failed for $version: missing required fields"
    return 1 2>/dev/null || exit 1
  fi
else
  echo "✗ Error for $version: Expected 5 rows, got $row_count"
  echo "Filebeat log:"
  cat "/tmp/fb-log-${version}.txt" | tail -50
  return 1 2>/dev/null || exit 1
fi
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/filebeat-single-test.sh
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 7.17
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.0
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.1
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.2
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.3
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.4
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.5
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.6
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.7
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.8
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.9
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.10
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.11
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.12
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.13
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.14
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.15
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.16
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.17
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.18
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.19
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.0
––– output –––
>>> Testing Filebeat version: 9.0
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Testing Filebeat 9.0 with fingerprint disabled...
+ >>> Testing Filebeat 9.0 with filestream input and fingerprint disabled...
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
✓ Filebeat 9.0 processed logs
✓ Row count check for 9.0: 5 rows
✓ Structure check for 9.0: passed
- ✓ Filebeat version 9.0 tested successfully with fingerprint disabled
+ ✓ Filebeat version 9.0 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.1
––– output –––
>>> Testing Filebeat version: 9.1
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Testing Filebeat 9.1 with fingerprint disabled...
+ >>> Testing Filebeat 9.1 with filestream input and fingerprint disabled...
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
✓ Filebeat 9.1 processed logs
✓ Row count check for 9.1: 5 rows
✓ Structure check for 9.1: passed
- ✓ Filebeat version 9.1 tested successfully with fingerprint disabled
+ ✓ Filebeat version 9.1 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.2
––– output –––
>>> Testing Filebeat version: 9.2
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Starting Filebeat...
+ >>> Testing Filebeat 9.2 with filestream input and fingerprint disabled...
- >>> Waiting for Filebeat to publish events...
+ >>> Starting Filebeat...
- ✓ Filebeat 9.2 processed logs
+ >>> Waiting for Filebeat to publish events...
- ✓ Row count check for 9.2: 5 rows
+ ✓ Filebeat 9.2 processed logs
- ✓ Structure check for 9.2: passed
+ ✓ Row count check for 9.2: 5 rows
- ✓ Filebeat version 9.2 tested successfully
+ ✓ Structure check for 9.2: passed
+ ✓ Filebeat version 9.2 tested successfully
––– input –––
rm -rf /tmp/fb-data-* /tmp/fb-log-*.txt /tmp/page.json /tmp/filebeat_tags.json
––– output –––
OK

github-actions · 2025-10-28T11:01:35Z

Linux debug test results

1 135 tests 1 081 ✅ 47m 59s ⏱️
1 suites 54 💤
1 files 0 ❌

Results for commit 7b8a6d8.

♻️ This comment has been updated with latest results.

github-actions · 2025-10-28T11:13:04Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 437s
👉 Check Action Results for commit 3aa28e3

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-jina-remote.rec

test/clt-tests/mcl/auto-embeddings-jina-remote.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
cosine_similarity() {
    local file1="$1" file2="$2"

    awk '
    NR==FNR { a[NR]=$1; suma2+=$1*$1; next }
    {
        dot += a[FNR]*$1
        sumb2 += $1*$1
    }
    END {
        print dot / (sqrt(suma2) * sqrt(sumb2))
    }' "$file1" "$file2"
}
––– output –––
OK
––– input –––
export -f cosine_similarity
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_invalid_model (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/invalid-model-name-12345' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_valid_model_no_api_key (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v2-base-en' FROM = 'title') " 2>&1
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_remote (title TEXT, content TEXT, description TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title, content' API_KEY='${JINA_API_KEY}') "; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_remote"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(1, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'advanced AI research')"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Failed to send request to remote model
+ 1
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as record_count FROM test_jina_remote WHERE id=1"
––– output –––
+--------------+
| record_count |
+--------------+
- |            1 |
+ |            0 |
+--------------+
––– input –––
mysql -h0 -P9306 -e "INSERT INTO test_jina_remote (id, title, content, description) VALUES(2, 'machine learning algorithms', 'deep neural networks and artificial intelligence', 'different description')"

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector1.txt

mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=2" | \
    grep -v embedding | \
    sed 's/[0-9]\+\(\.[0-9]\+\)\?/\n&\n/g' | \
    grep -E '^[0-9]+(\.[0-9]+)?$' | \
    awk '{printf "%.5f\n", $1}' > /tmp/vector2.txt

SIMILARITY=$(cosine_similarity /tmp/vector1.txt /tmp/vector2.txt)

echo "Cosine similarity: $SIMILARITY"

RESULT=$(awk -v sim="$SIMILARITY" 'BEGIN {
    if (sim > 0.99)
        print "SUCCESS: Same FROM fields produce similar vectors (similarity: " sim ")"
    else
        print "FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: " sim ")"
}')

echo "$RESULT"

rm -f /tmp/vector1.txt /tmp/vector2.txt
––– output –––
- Cosine similarity: #!/(1|0\.[0-9]+)/!#
+ Cosine similarity: -nan
- SUCCESS: Same FROM fields produce similar vectors (similarity: #!/(1|0\.[0-9]+)/!#)
+ FAIL: Different vectors (FROM does not include description field and should not change generated vector value) (similarity: -nan)
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_title_only (title TEXT, content TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/jina-embeddings-v4' FROM = 'title' API_KEY='${JINA_API_KEY}') "; mysql -h0 -P9306 -e "INSERT INTO test_jina_title_only (id, title, content) VALUES(1, 'machine learning algorithms', 'completely different content here')"; MD5_MULTI=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_remote WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); MD5_SINGLE=$(mysql -h0 -P9306 -e "SELECT embedding FROM test_jina_title_only WHERE id=1" | grep -v embedding | md5sum | awk '{print $1}'); echo "multi_field_md5: $MD5_MULTI"; echo "single_field_md5: $MD5_SINGLE"; if [ "$MD5_MULTI" != "$MD5_SINGLE" ]; then echo "SUCCESS: Different FROM specifications produce different vectors"; else echo "INFO: FROM field comparison result"; fi
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_jina_invalid_field (title TEXT, embedding FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME = 'jina/text-embedding-ada-002' FROM = 'nonexistent_field') " 2>&1
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "INSERT INTO test_jina_no_from (id, title, embedding) VALUES(1, 'test title', '(0.1, 0.2, 0.3, 0.4, 0.5)')"; echo "insert_result: $?"; else echo "insert_result: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if mysql -h0 -P9306 -e "SHOW TABLES LIKE 'test_jina_no_from'" | grep -q test_jina_no_from; then mysql -h0 -P9306 -e "SHOW CREATE TABLE test_jina_no_from"; else echo "table_structure: skipped (table not created)"; fi
––– output –––
OK
––– input –––
if [ -n "$JINA_API_KEY" ] && [ "$JINA_API_KEY" != "dummy_key_for_testing" ]; then echo "API key is available for testing"; else echo "API key not available - using dummy for error testing"; fi
––– output –––
OK

github-actions · 2025-10-28T11:27:28Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2161s
👉 Check Action Results for commit 3aa28e3

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-10-28T12:19:04Z

test_support_filebeat_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-filebeat-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 380s
👉 Check Action Results for commit ea2d781

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y curl jq > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
timeout 420 bash -c 'echo "[]" > /tmp/filebeat_tags.json; page=1; attempts=0; max_attempts=3; while [ $attempts -lt $max_attempts ]; do attempts=$((attempts+1)); if curl -s --fail --max-time 10 "https://hub.docker.com/v2/repositories/elastic/filebeat/tags/?page_size=1000&page=$page" | tee /tmp/page.json | jq -e ".next" > /dev/null; then jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; page=$((page+1)); attempts=0; else break; fi; done; jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; VERSIONS=$(cat /tmp/filebeat_tags.json | grep -E "^([7-9]|[1-9][0-9]+).[0-9]+.[0-9]+$" | grep -E "^(7.(1[7-9]|[2-9][0-9])|[8-9].[0-9]+|9.[0-9]+|[1-9][0-9]+.[0-9]+).[0-9]+$" | sed -E "s/^([0-9]+.[0-9]+).[0-9]+$/\1/" | grep -v "rc|beta|alpha" | sort -V | uniq); echo "$VERSIONS"; mkdir -p /tmp/filebeat_cache; echo "Preparation done"; for version in $VERSIONS; do archive="/tmp/filebeat_cache/filebeat-${version}.0-linux-x86_64.tar.gz"; echo ">>> Checking Filebeat $version ..."; if [ -f "$archive" ] && gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo ">>> Downloading Filebeat $version ..."; wget -q --timeout=30 "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${version}.0-linux-x86_64.tar.gz" -O "$archive" && { if gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo "✗ Archive for $version is corrupted"; rm -f "$archive"; fi; }; fi; done'
––– output –––
OK
––– input –––
set +H && mkdir -p /tmp/filebeat_cache && echo "Preparation done"
––– output –––
OK
––– input –––
cat << 'EOF' > /tmp/filebeat-single-test.sh
#!/usr/bin/env bash
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "✗ Usage: $0 <filebeat_version>" >&2
  return 1 2>/dev/null || exit 1
fi

version="$1"
full_version="${version}.0"
echo ">>> Testing Filebeat version: $version"

# Prepare test log
echo -e "2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > /var/log/dpkg.log

log_lines=$(wc -l < /var/log/dpkg.log)
if [ "$log_lines" -eq 5 ]; then
  echo "✓ Log file has 5 lines"
else
  echo "✗ Error: Expected 5 lines, got $log_lines" >&2
  return 1 2>/dev/null || exit 1
fi

# Check Manticore availability
if ! curl -s localhost:9308/cli_json -d 'SHOW TABLES' | jq -e '.[0].data' > /dev/null; then
  echo "✗ Error: Manticore Search unavailable" >&2
  return 1 2>/dev/null || exit 1
fi
echo "✓ Manticore Search available"

# Create table
mysql -h0 -P9306 -e "
DROP TABLE IF EXISTS dpkg_log;
CREATE TABLE dpkg_log (
  id BIGINT,
  message TEXT INDEXED STORED,
  host JSON,
  agent JSON,
  input JSON,
  log JSON,
  ecs JSON,
  \`@timestamp\` TEXT INDEXED STORED
);"

# Install Filebeat
mkdir -p /usr/share/filebeat /tmp/fb-data-${version}
tar -xzf "/tmp/filebeat_cache/filebeat-${full_version}-linux-x86_64.tar.gz" -C /usr/share/filebeat
FB_DIR="/usr/share/filebeat/filebeat-${full_version}-linux-x86_64"

# Clean previous registry data
rm -rf /tmp/fb-data-${version}/*

skip_filebeat=0


# For all 9.x versions, use filestream with fingerprint disabled
if [[ "$version" =~ ^9\. ]]; then
  echo ">>> Testing Filebeat $version with filestream input and fingerprint disabled..."

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: filestream
  id: dpkg-filestream-input
  enabled: true
  paths: ["/var/log/dpkg.log"]
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true

      if [ "$row_count" -eq 5 ]; then
        echo "✓ Row count check for $version: $row_count rows"
        structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
        has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
        has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
        if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
          echo "✓ Structure check for $version: passed"
          echo "✓ Filebeat version $version tested successfully"
          skip_filebeat=1
        fi
      fi
      break
    fi
    sleep 1
  done

  if [ "$skip_filebeat" -eq 0 ]; then
    kill $FB_PID 2>/dev/null || true
    wait $FB_PID 2>/dev/null || true
    mysql -h0 -P9306 -e "TRUNCATE TABLE dpkg_log" 2>/dev/null || true
    echo "✗ Error: Filebeat $version failed to process logs" >&2
    return 1 2>/dev/null || exit 1
  fi
fi

# For all other versions (8.x), use the standard approach
if [ "$skip_filebeat" -eq 0 ]; then
  if [[ "$version" =~ ^8\.[1-9]$ || "$version" =~ ^8\.[1-9][0-9]+$ ]]; then
    # For versions 8.1 and higher, add allow_older_versions option
    ALLOW_OLDER="allow_older_versions: true"
  else
    ALLOW_OLDER=""
  fi

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  ${ALLOW_OLDER}

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true
      break
    fi
    sleep 1
  done

  # Final verification
  row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
  if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $version: $row_count rows"
    structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
    has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
    has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
    if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
      echo "✓ Structure check for $version: passed"
      echo "✓ Filebeat version $version tested successfully"
    else
      echo "✗ Structure check for $version: failed" >&2
      return 1 2>/dev/null || exit 1
    fi
  else
    echo "✗ Row count check for $version: expected 5, got $row_count" >&2
    return 1 2>/dev/null || exit 1
  fi
fi
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/filebeat-single-test.sh
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 7.17
––– output –––
>>> Testing Filebeat version: 7.17
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 7.17 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   399 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 7.17: 5 rows
+ ✗ Row count check for 7.17: expected 5, got 0
- ✓ Structure check for 7.17: passed
- ✓ Filebeat version 7.17 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.0
––– output –––
>>> Testing Filebeat version: 8.0
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 8.0 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   544 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 8.0: 5 rows
+ ✗ Row count check for 8.0: expected 5, got 0
- ✓ Structure check for 8.0: passed
- ✓ Filebeat version 8.0 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.1
––– output –––
>>> Testing Filebeat version: 8.1
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 8.1 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   689 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 8.1: 5 rows
+ ✗ Row count check for 8.1: expected 5, got 0
- ✓ Structure check for 8.1: passed
- ✓ Filebeat version 8.1 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.2
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.3
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.4
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.5
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.6
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.7
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.8
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.9
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.10
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.11
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.12
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.13
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.14
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.15
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.16
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.17
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.18
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.19
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.0
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.1
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.2
––– output –––
OK
––– input –––
rm -rf /tmp/fb-data-* /tmp/fb-log-*.txt /tmp/page.json /tmp/filebeat_tags.json
––– output –––
OK

github-actions · 2025-10-29T07:56:43Z

test_support_filebeat_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-filebeat-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 330s
👉 Check Action Results for commit d9dc133

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y curl jq > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
timeout 420 bash -c 'echo "[]" > /tmp/filebeat_tags.json; page=1; attempts=0; max_attempts=3; while [ $attempts -lt $max_attempts ]; do attempts=$((attempts+1)); if curl -s --fail --max-time 10 "https://hub.docker.com/v2/repositories/elastic/filebeat/tags/?page_size=1000&page=$page" | tee /tmp/page.json | jq -e ".next" > /dev/null; then jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; page=$((page+1)); attempts=0; else break; fi; done; jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; VERSIONS=$(cat /tmp/filebeat_tags.json | grep -E "^([7-9]|[1-9][0-9]+).[0-9]+.[0-9]+$" | grep -E "^(7.(1[7-9]|[2-9][0-9])|[8-9].[0-9]+|9.[0-9]+|[1-9][0-9]+.[0-9]+).[0-9]+$" | sed -E "s/^([0-9]+.[0-9]+).[0-9]+$/\1/" | grep -v "rc|beta|alpha" | sort -V | uniq); echo "$VERSIONS"; mkdir -p /tmp/filebeat_cache; echo "Preparation done"; for version in $VERSIONS; do archive="/tmp/filebeat_cache/filebeat-${version}.0-linux-x86_64.tar.gz"; echo ">>> Checking Filebeat $version ..."; if [ -f "$archive" ] && gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo ">>> Downloading Filebeat $version ..."; wget -q --timeout=30 "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${version}.0-linux-x86_64.tar.gz" -O "$archive" && { if gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo "✗ Archive for $version is corrupted"; rm -f "$archive"; fi; }; fi; done'
––– output –––
OK
––– input –––
set +H && mkdir -p /tmp/filebeat_cache && echo "Preparation done"
––– output –––
OK
––– input –––
cat << 'EOF' > /tmp/filebeat-single-test.sh
#!/usr/bin/env bash
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "✗ Usage: $0 <filebeat_version>" >&2
  return 1 2>/dev/null || exit 1
fi

version="$1"
full_version="${version}.0"
echo ">>> Testing Filebeat version: $version"

# Prepare test log
echo -e "2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > /var/log/dpkg.log

log_lines=$(wc -l < /var/log/dpkg.log)
if [ "$log_lines" -eq 5 ]; then
  echo "✓ Log file has 5 lines"
else
  echo "✗ Error: Expected 5 lines, got $log_lines" >&2
  return 1 2>/dev/null || exit 1
fi

# Check Manticore availability
if ! curl -s localhost:9308/cli_json -d 'SHOW TABLES' | jq -e '.[0].data' > /dev/null; then
  echo "✗ Error: Manticore Search unavailable" >&2
  return 1 2>/dev/null || exit 1
fi
echo "✓ Manticore Search available"

# Create table
mysql -h0 -P9306 -e "
DROP TABLE IF EXISTS dpkg_log;
CREATE TABLE dpkg_log (
  id BIGINT,
  message TEXT INDEXED STORED,
  host JSON,
  agent JSON,
  input JSON,
  log JSON,
  ecs JSON,
  \`@timestamp\` TEXT INDEXED STORED
);"

# Install Filebeat
mkdir -p /usr/share/filebeat /tmp/fb-data-${version}
tar -xzf "/tmp/filebeat_cache/filebeat-${full_version}-linux-x86_64.tar.gz" -C /usr/share/filebeat
FB_DIR="/usr/share/filebeat/filebeat-${full_version}-linux-x86_64"

# Clean previous registry data
rm -rf /tmp/fb-data-${version}/*

skip_standard_test=0


# For all 9.x versions, use filestream with fingerprint disabled
if [[ "$version" =~ ^9\. ]]; then
  echo ">>> Testing Filebeat $version with filestream input and fingerprint disabled..."

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: filestream
  id: dpkg-filestream-input
  enabled: true
  paths: ["/var/log/dpkg.log"]
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true

      if [ "$row_count" -eq 5 ]; then
        echo "✓ Row count check for $version: $row_count rows"
        structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
        has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
        has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
        if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
          echo "✓ Structure check for $version: passed"
          echo "✓ Filebeat version $version tested successfully"
          skip_standard_test=1
        fi
      fi
      break
    fi
    sleep 1
  done

  if [ "$skip_standard_test" -eq 0 ]; then
    kill $FB_PID 2>/dev/null || true
    wait $FB_PID 2>/dev/null || true
    mysql -h0 -P9306 -e "TRUNCATE TABLE dpkg_log" 2>/dev/null || true
    echo "✗ Error: Filebeat $version failed to process logs" >&2
    return 1 2>/dev/null || exit 1
  fi
fi

# For all other versions (8.x), use the standard approach
if [ "$skip_standard_test" -eq 0 ]; then
  if [[ "$version" =~ ^8\.[1-9]$ || "$version" =~ ^8\.[1-9][0-9]+$ ]]; then
    # For versions 8.1 and higher, add allow_older_versions option
    ALLOW_OLDER="allow_older_versions: true"
  else
    ALLOW_OLDER=""
  fi

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  ${ALLOW_OLDER}

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true
      break
    fi
    sleep 1
  done

  # Final verification
  row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
  if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $version: $row_count rows"
    structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
    has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
    has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
    if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
      echo "✓ Structure check for $version: passed"
      echo "✓ Filebeat version $version tested successfully"
    else
      echo "✗ Structure check for $version: failed" >&2
      return 1 2>/dev/null || exit 1
    fi
  else
    echo "✗ Row count check for $version: expected 5, got $row_count" >&2
    return 1 2>/dev/null || exit 1
  fi
fi
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/filebeat-single-test.sh
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 7.17
––– output –––
>>> Testing Filebeat version: 7.17
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 7.17 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   399 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 7.17: 5 rows
+ ✗ Row count check for 7.17: expected 5, got 0
- ✓ Structure check for 7.17: passed
- ✓ Filebeat version 7.17 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.0
––– output –––
>>> Testing Filebeat version: 8.0
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 8.0 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   544 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 8.0: 5 rows
+ ✗ Row count check for 8.0: expected 5, got 0
- ✓ Structure check for 8.0: passed
- ✓ Filebeat version 8.0 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.1
––– output –––
>>> Testing Filebeat version: 8.1
✓ Log file has 5 lines
✓ Manticore Search available
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
- ✓ Filebeat 8.1 processed logs
+ /tmp/filebeat-single-test.sh: line 155:   689 Aborted                 (core dumped) "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1
- ✓ Row count check for 8.1: 5 rows
+ ✗ Row count check for 8.1: expected 5, got 0
- ✓ Structure check for 8.1: passed
- ✓ Filebeat version 8.1 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.2
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.3
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.4
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.5
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.6
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.7
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.8
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.9
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.10
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.11
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.12
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.13
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.14
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.15
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.16
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.17
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.18
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.19
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.0
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.1
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.2
––– output –––
OK
––– input –––
rm -rf /tmp/fb-data-* /tmp/fb-log-*.txt /tmp/page.json /tmp/filebeat_tags.json
––– output –––
OK

github-actions · 2025-10-29T08:32:29Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2486s
👉 Check Action Results for commit d9dc133

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-10-29T10:01:26Z

test_support_filebeat_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-filebeat-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 301s
👉 Check Action Results for commit 9661e3a

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y curl jq > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
timeout 420 bash -c 'echo "[]" > /tmp/filebeat_tags.json; page=1; attempts=0; max_attempts=3; while [ $attempts -lt $max_attempts ]; do attempts=$((attempts+1)); if curl -s --fail --max-time 10 "https://hub.docker.com/v2/repositories/elastic/filebeat/tags/?page_size=1000&page=$page" | tee /tmp/page.json | jq -e ".next" > /dev/null; then jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; page=$((page+1)); attempts=0; else break; fi; done; jq -r ".results[].name" /tmp/page.json >> /tmp/filebeat_tags.json; VERSIONS=$(cat /tmp/filebeat_tags.json | grep -E "^([7-9]|[1-9][0-9]+).[0-9]+.[0-9]+$" | grep -E "^(7.(1[7-9]|[2-9][0-9])|[8-9].[0-9]+|9.[0-9]+|[1-9][0-9]+.[0-9]+).[0-9]+$" | sed -E "s/^([0-9]+.[0-9]+).[0-9]+$/\1/" | grep -v "rc|beta|alpha" | sort -V | uniq); echo "$VERSIONS"; mkdir -p /tmp/filebeat_cache; echo "Preparation done"; for version in $VERSIONS; do archive="/tmp/filebeat_cache/filebeat-${version}.0-linux-x86_64.tar.gz"; echo ">>> Checking Filebeat $version ..."; if [ -f "$archive" ] && gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo ">>> Downloading Filebeat $version ..."; wget -q --timeout=30 "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${version}.0-linux-x86_64.tar.gz" -O "$archive" && { if gzip -t "$archive" >/dev/null 2>&1; then echo "✓ Archive for $version is OK"; else echo "✗ Archive for $version is corrupted"; rm -f "$archive"; fi; }; fi; done'
––– output –––
OK
––– input –––
set +H && mkdir -p /tmp/filebeat_cache && echo "Preparation done"
––– output –––
OK
––– input –––
cat << 'EOF' > /tmp/filebeat-single-test.sh
#!/usr/bin/env bash
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "✗ Usage: $0 <filebeat_version>" >&2
  return 1 2>/dev/null || exit 1
fi

version="$1"
full_version="${version}.0"
echo ">>> Testing Filebeat version: $version"

# Prepare test log
echo -e "2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > /var/log/dpkg.log

log_lines=$(wc -l < /var/log/dpkg.log)
if [ "$log_lines" -eq 5 ]; then
  echo "✓ Log file has 5 lines"
else
  echo "✗ Error: Expected 5 lines, got $log_lines" >&2
  return 1 2>/dev/null || exit 1
fi

# Check Manticore availability
if ! curl -s localhost:9308/cli_json -d 'SHOW TABLES' | jq -e '.[0].data' > /dev/null; then
  echo "✗ Error: Manticore Search unavailable" >&2
  return 1 2>/dev/null || exit 1
fi
echo "✓ Manticore Search available"

# Create table
mysql -h0 -P9306 -e "
DROP TABLE IF EXISTS dpkg_log;
CREATE TABLE dpkg_log (
  id BIGINT,
  message TEXT INDEXED STORED,
  host JSON,
  agent JSON,
  input JSON,
  log JSON,
  ecs JSON,
  \`@timestamp\` TEXT INDEXED STORED
);"

# Install Filebeat
mkdir -p /usr/share/filebeat /tmp/fb-data-${version}
tar -xzf "/tmp/filebeat_cache/filebeat-${full_version}-linux-x86_64.tar.gz" -C /usr/share/filebeat
FB_DIR="/usr/share/filebeat/filebeat-${full_version}-linux-x86_64"

# Clean previous registry data
rm -rf /tmp/fb-data-${version}/*

skip_standard_test=0


# For all 9.x versions, use filestream with fingerprint disabled
if [[ "$version" =~ ^9\. ]]; then
  echo ">>> Testing Filebeat $version with filestream input and fingerprint disabled..."

  cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: filestream
  id: dpkg-filestream-input
  enabled: true
  paths: ["/var/log/dpkg.log"]
  prospector.scanner.check_interval: 1s
  prospector.scanner.fingerprint.enabled: false

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true

      if [ "$row_count" -eq 5 ]; then
        echo "✓ Row count check for $version: $row_count rows"
        structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
        has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
        has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
        if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
          echo "✓ Structure check for $version: passed"
          echo "✓ Filebeat version $version tested successfully"
          skip_standard_test=1
        fi
      fi
      break
    fi
    sleep 1
  done

  if [ "$skip_standard_test" -eq 0 ]; then
    kill $FB_PID 2>/dev/null || true
    wait $FB_PID 2>/dev/null || true
    mysql -h0 -P9306 -e "TRUNCATE TABLE dpkg_log" 2>/dev/null || true
    echo "✗ Error: Filebeat $version failed to process logs" >&2
    return 1 2>/dev/null || exit 1
  fi
fi

# For all other versions (7.17, 8.x), use the standard approach
if [ "$skip_standard_test" -eq 0 ]; then
  # Versions 7.17.0, 8.0.0, 8.1.0 crash on glibc 2.35+ due to missing rseq syscall support
  # Fixed in 7.17.2+ and 8.2.0+
  # See: https://github.com/elastic/beats/issues/30576
  # See: https://github.com/elastic/beats/pull/30620
  if [[ "$version" =~ ^(7\.17|8\.0|8\.1)$ ]]; then
    echo ">>> Using special configuration for Filebeat $version (glibc 2.35+ compatibility fix)..."

    cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  $(if [[ "$version" =~ ^8\.[1-9]$ ]]; then echo "allow_older_versions: true"; fi)

# Fix for glibc 2.35+ rseq syscall issue
seccomp:
  default_action: allow
  syscalls:
    - action: allow
      names:
        - rseq

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  elif [[ "$version" =~ ^8\.[1-9]$ || "$version" =~ ^8\.[1-9][0-9]+$ ]]; then
    # For versions 8.1 and higher (except 8.0, 8.1 which are handled above), add allow_older_versions option
    cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0
  allow_older_versions: true

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML

  else
    # For versions before 8.1
    cat > "${FB_DIR}/filebeat.yml" <<YML
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dpkg.log
  close_eof: true
  scan_frequency: 1s

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  index: "dpkg_log"
  compression_level: 0

path.data: /tmp/fb-data-${version}
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
YML
  fi

  echo ">>> Starting Filebeat..."
  "${FB_DIR}/filebeat" run -c "${FB_DIR}/filebeat.yml" > /tmp/fb-log-${version}.txt 2>&1 &
  FB_PID=$!

  echo ">>> Waiting for Filebeat to publish events..."
  for i in {1..30}; do
    row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
    if [[ "$row_count" =~ ^[0-9]+$ ]] && [ "$row_count" -ge 5 ]; then
      echo "✓ Filebeat $version processed logs"
      kill $FB_PID 2>/dev/null || true
      wait $FB_PID 2>/dev/null || true
      break
    fi
    sleep 1
  done

  # Final verification
  row_count=$(mysql -N -s -h0 -P9306 -e "SELECT COUNT(*) FROM dpkg_log" 2>/dev/null | grep -o '[0-9]\+' || echo "0")
  if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $version: $row_count rows"
    structure=$(curl -s localhost:9308/cli_json -d 'DESCRIBE dpkg_log' | jq -c '[.[0].data[]] | sort_by(.Field)')
    has_timestamp=$(echo "$structure" | grep -q "\"Field\":\"@timestamp\"" && echo "1" || echo "0")
    has_message=$(echo "$structure" | grep -q "\"Field\":\"message\"" && echo "1" || echo "0")
    if [ "$has_timestamp" = "1" ] && [ "$has_message" = "1" ]; then
      echo "✓ Structure check for $version: passed"
      echo "✓ Filebeat version $version tested successfully"
    else
      echo "✗ Structure check for $version: failed" >&2
      return 1 2>/dev/null || exit 1
    fi
  else
    echo "✗ Row count check for $version: expected 5, got $row_count" >&2
    return 1 2>/dev/null || exit 1
  fi
fi
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/filebeat-single-test.sh
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 7.17
––– output –––
>>> Testing Filebeat version: 7.17
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Using special configuration for Filebeat 7.17 (glibc 2.35compatibility fix)...
+ >>> Using special configuration for Filebeat 7.17 (glibc 2.35+ compatibility fix)...
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
✓ Filebeat 7.17 processed logs
✓ Row count check for 7.17: 5 rows
✓ Structure check for 7.17: passed
✓ Filebeat version 7.17 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.0
––– output –––
>>> Testing Filebeat version: 8.0
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Using special configuration for Filebeat 8.0 (glibc 2.35compatibility fix)...
+ >>> Using special configuration for Filebeat 8.0 (glibc 2.35+ compatibility fix)...
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
✓ Filebeat 8.0 processed logs
✓ Row count check for 8.0: 5 rows
✓ Structure check for 8.0: passed
✓ Filebeat version 8.0 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.1
––– output –––
>>> Testing Filebeat version: 8.1
✓ Log file has 5 lines
✓ Manticore Search available
- >>> Using special configuration for Filebeat 8.1 (glibc 2.35compatibility fix)...
+ >>> Using special configuration for Filebeat 8.1 (glibc 2.35+ compatibility fix)...
>>> Starting Filebeat...
>>> Waiting for Filebeat to publish events...
✓ Filebeat 8.1 processed logs
✓ Row count check for 8.1: 5 rows
✓ Structure check for 8.1: passed
✓ Filebeat version 8.1 tested successfully
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.2
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.3
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.4
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.5
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.6
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.7
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.8
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.9
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.10
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.11
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.12
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.13
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.14
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.15
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.16
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.17
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.18
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 8.19
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.0
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.1
––– output –––
OK
––– input –––
timeout 60 bash /tmp/filebeat-single-test.sh 9.2
––– output –––
OK
––– input –––
rm -rf /tmp/fb-data-* /tmp/fb-log-*.txt /tmp/page.json /tmp/filebeat_tags.json
––– output –––
OK

…ticoresoftware/manticoresearch into test/added-test-filebeat-9.2

github-actions · 2025-10-29T11:00:08Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2038s
👉 Check Action Results for commit 109e774

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-10-29T18:57:23Z

test_kafka

❌ CLT tests in test/clt-tests/integrations/kafka/test-integration-
✅ OK: 2
❌ Failed: 1
⏳ Duration: 610s
👉 Check Action Results for commit ec601e9

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/kafka/test-integration-kafka-ms.rec

test/clt-tests/integrations/kafka/test-integration-kafka-ms.rec

––– input –––
(dockerd > /var/log/dockerd.log 2>&1 &) > /dev/null
––– output –––
OK
––– input –––
if timeout 30 grep -qm1 'API listen on /var/run/docker.sock' <(tail -n 0 -f /var/log/dockerd.log); then echo 'Done'; else echo 'Timeout failed'; fi
––– output –––
OK
––– input –––
docker ps
––– output –––
OK
––– input –––
KAFKA_VERSION="4.1.0"
echo "Using Kafka version: $KAFKA_VERSION"
––– output –––
OK
––– input –––
docker network create app-network --driver bridge > /dev/null; echo $?
––– output –––
OK
––– input –––
docker run -it --network=app-network --platform linux/amd64 --name manticore -d ghcr.io/manticoresoftware/manticoresearch:test-kit-latest bash > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
docker exec manticore stdbuf -oL searchd --logdebugvv > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
docker run -it -d --network=app-network --name kafka --platform linux/amd64 \
-v ./test/clt-tests/integrations/kafka/import.sh:/import.sh \
-v ./test/clt-tests/integrations/kafka/dump.json:/tmp/dump.json \
-e KAFKA_NODE_ID=1 \
-e KAFKA_PROCESS_ROLES=broker,controller \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \
-e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 \
-e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \
apache/kafka:$KAFKA_VERSION > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
for i in $(seq 1 60); do
    if docker exec kafka /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1; then
        echo "Kafka ready"
        break
    fi
    sleep 3
done
––– output –––
OK
––– input –––
docker exec kafka /opt/kafka/bin/kafka-topics.sh --create --topic my-data --partitions 4 --bootstrap-server localhost:9092 2>&1 | grep -o 'Created topic my-data\.' | head -n 1
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, is_active bool) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka (id bigint, name text, short_name text, received_at text, size multi, is_active bool);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table TO destination_kafka AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, is_active FROM kafka;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW SOURCES;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW SOURCE kafka;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW MVS;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW MV view_table;"
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT * FROM destination_kafka ORDER BY id ASC;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "DROP SOURCE kafka;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "DROP table destination_kafka;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW TABLES;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_destination"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, location json, is_active bool) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_destination' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka (id bigint, name text, short_name text, received_at text, size multi, lat float, lon float, distance float, is_active bool);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_destination TO destination_kafka AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, location.lat as lat, location.lon as lon, GEODIST(lat, lon, 49.0, 3.0, {in=degrees, out=m}) AS distance, is_active FROM kafka"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT id, name, short_name, received_at, size, lat, lon, is_active AS distance FROM destination_kafka ORDER BY id ASC;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_metadata"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_metadata (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, metadata json) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_metadata' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_metadata (id bigint, name text, short_name text, received_at text, size multi, views int, info text);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_metadata TO destination_kafka_metadata AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, metadata.views as views, metadata.info as info FROM kafka_metadata WHERE views > 1000;"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka_metadata" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_metadata;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT * FROM destination_kafka_metadata ORDER BY id ASC;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_tags"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_tags (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, tags json) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_tags' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_tags (id bigint, name text, short_name text, received_at text, size multi, tags json);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_tags TO destination_kafka_tags AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, tags FROM kafka_tags WHERE tags IN ('item1', 'item2');"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka_tags" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_tags;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT * FROM destination_kafka_tags ORDER BY id ASC;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_alter"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_alter (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, metadata json) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_alter' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_alter (id bigint, name text, short_name text, received_at text, size multi, views bigint);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_alter TO destination_kafka_alter AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, metadata.views as views FROM kafka_alter;"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m1 "REPLACE%20INTO%20destination_kafka_alter" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_table_alter suspended=1;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_alter;"
––– output –––
OK
––– input –––
sleep 10; docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_alter;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_table_alter suspended=0;"; echo $?
––– output –––
OK
––– input –––
timeout 120 bash -c 'while [[ $(docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_alter;" | grep -o "[0-9]*") -ne 57 ]]; do sleep 1; done && echo "Data processing completed."'
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_alter;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_ts"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_ts (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, timestamp_unix timestamp) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_ts' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_ts (id bigint, name text, short_name text, received_at text, size multi, timestamp_field timestamp);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_ts TO destination_kafka_ts AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size, timestamp_unix as timestamp_field FROM kafka_ts WHERE timestamp_field >= 1690761600;"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka_ts" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_ts;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT * FROM destination_kafka_ts ORDER BY id ASC;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_combined"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_combined (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, location json, metadata json, tags json, timestamp_unix timestamp) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_combined' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_combined (id bigint, name text, short_name text, received_at text, size multi, lat float, lon float, views int, info text, tags json, timestamp_combined timestamp);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_combined TO destination_kafka_combined AS SELECT id, term AS name, abbrev AS short_name, UTC_TIMESTAMP() AS received_at, GlossDef.size AS size, location.lat AS lat, location.lon AS lon, metadata.views AS views, metadata.info AS info, tags, timestamp_unix AS timestamp_combined FROM kafka_combined WHERE views > 1000 AND timestamp_combined >= 1690761600 AND tags IN ('item1', 'item2') AND lat > 50 AND lon > 5;"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_kafka_combined" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
- Data processing completed.
+ Data processing failed.
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_combined;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT * FROM destination_kafka_combined ORDER BY id ASC;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_stop"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_stop (id bigint, term text, abbrev '\$abbrev' text, GlossDef json, location json, metadata json, tags json, timestamp_unix timestamp) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_stop' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_kafka_stop (id bigint, name text, short_name text, received_at text, size multi, lat float, lon float, views int, info text, tags json, timestamp_combined timestamp);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_table_stop TO destination_kafka_stop AS SELECT id, term AS name, abbrev AS short_name, UTC_TIMESTAMP() AS received_at, GlossDef.size AS size, location.lat AS lat, location.lon AS lon, metadata.views AS views, metadata.info AS info, tags, timestamp_unix AS timestamp_combined FROM kafka_stop;"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m1 "REPLACE%20INTO%20destination_kafka_stop" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_stop;"
––– output –––
OK
––– input –––
sleep 1; docker exec manticore stdbuf -oL searchd --stopwait
––– output –––
OK
––– input –––
sleep 1; docker exec manticore stdbuf -oL searchd --logdebugvv > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
timeout 120 bash -c 'while [[ $(docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_stop;" | grep -o "[0-9]*") -ne 57 ]]; do sleep 1; done && echo "Data processing completed."'
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_kafka_stop;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE invalid-source (id bigint, term text) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_invalid' num_consumers='1' batch=50;"
––– output –––
OK
––– input –––
CONSUMER_GROUP="manticore_drop_source"
––– input –––
docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group ${CONSUMER_GROUP} --reset-offsets --to-latest --topic ${TOPIC_NAME:-my-data} --execute > /dev/null; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_drop_source (id bigint, term text) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_drop_source' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE TABLE destination_drop_source (id bigint, name text);"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE MATERIALIZED VIEW view_drop_source TO destination_drop_source AS SELECT id, term as name FROM kafka_drop_source;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW MV view_drop_source\G;" | grep suspended
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "DROP SOURCE kafka_drop_source;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW MV view_drop_source\G;" | grep suspended
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "CREATE SOURCE kafka_drop_source (id bigint, term text) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore_drop_source' num_consumers='1' batch=50;"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SHOW MV view_drop_source\G;" | grep suspended
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_drop_source suspended=0"; echo $?
––– output –––
OK
––– input –––
timeout 60 bash -c 'docker exec manticore bash -c "tail -f /var/log/manticore/searchd.log" | grep -m2 "REPLACE%20INTO%20destination_drop_source" > /dev/null' & GREP_PID=$!; sleep 2; docker exec kafka bash /import.sh; wait $GREP_PID && echo "Data processing completed." || echo "Data processing failed."
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "SELECT COUNT(*) FROM destination_drop_source;"
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_drop_source suspended=1"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_drop_source suspended=1"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_drop_source suspended=0"; echo $?
––– output –––
OK
––– input –––
docker exec manticore mysql -h0 -P9306 -e "ALTER MATERIALIZED VIEW view_drop_source suspended=0"; echo $?
––– output –––
OK

github-actions · 2025-10-29T19:24:20Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2118s
👉 Check Action Results for commit ec601e9

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-10-31T12:35:12Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2633s
👉 Check Action Results for commit 139ac1e

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

github-actions · 2025-11-03T11:16:41Z

test_support_logstash_versions

❌ CLT tests in test/clt-tests/integrations/test-integrations-support-logstash-versions
✅ OK: 0
❌ Failed: 1
⏳ Duration: 2357s
👉 Check Action Results for commit ecf48e1

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

test/clt-tests/integrations/test-integrations-support-logstash-versions.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd $SEARCHD_FLAGS > /dev/null; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
OK
––– input –––
set -b
––– output –––
OK
––– input –––
export PATH=/usr/bin:/usr/local/bin:/usr/sbin:/sbin:/bin
––– output –––
OK
––– input –––
apt-get update > /dev/null 2>&1 && apt-get install -y gnupg openjdk-11-jre-headless curl jq mysql-client > /dev/null 2>&1; echo $?
––– output –––
OK
––– input –––
# Static list of TESTED versions (never changes)
TESTED_VERSIONS="7.17
8.0
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
8.9
8.10
8.11
8.12
8.13
8.14
8.15
8.16
8.17
8.18
8.19
9.0
9.1"

# Check for NEW versions (after latest tested)
LATEST_TESTED_VERSION="9.1"
NEW_VERSIONS=$(curl -s "https://hub.docker.com/v2/repositories/library/logstash/tags/?page_size=100" \
| jq -r ".results[].name" \
| grep "^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$" \
| sed "s/\.[0-9]*$//" \
| awk '!/rc|beta|alpha/' \
| sort -t. -k1,1n -k2,2n | uniq \
| awk -v latest="$LATEST_TESTED_VERSION" '
function version_compare(v1, v2) {
    split(v1, a, ".")
    split(v2, b, ".")
    if (a[1] != b[1]) return a[1] - b[1]
    return a[2] - b[2]
}
version_compare($0, latest) > 0')

if [ -n "$NEW_VERSIONS" ]; then
    echo "🆕 NEW Logstash versions detected:"
    echo "$NEW_VERSIONS"
    echo "❌ Need to test new versions and update the list!"
    exit 1
else
    echo "✅ No new versions found after $LATEST_TESTED_VERSION"
fi

# Use static list for testing
echo "Using tested versions:"
echo "$TESTED_VERSIONS"
––– output –––
- ✅ No new versions found after 9.1
+ 🆕 NEW Logstash versions detected:
- Using tested versions:
+ 9.2
- 7.17
+ ❌ Need to test new versions and update the list!
- 8.0
+ bash: line 59: exit: command not found
- 8.1
+ Using tested versions:
- 8.2
+ 7.17
- 8.3
+ 8.0
- 8.4
+ 8.1
- 8.5
+ 8.2
- 8.6
+ 8.3
- 8.7
+ 8.4
- 8.8
+ 8.5
- 8.9
+ 8.6
- 8.10
+ 8.7
- 8.11
+ 8.8
- 8.12
+ 8.9
- 8.13
+ 8.10
- 8.14
+ 8.11
- 8.15
+ 8.12
- 8.16
+ 8.13
- 8.17
+ 8.14
- 8.18
+ 8.15
- 8.19
+ 8.16
- 9.0
+ 8.17
- 9.1
+ 8.18
+ 8.19
+ 9.0
+ 9.1
––– input –––
cat << 'EOF' > /tmp/logstash-single-test.sh
#!/bin/bash
set -euo pipefail
if [ $# -ne 1 ]; then
    echo "✗ Usage: $0 <logstash_version>" >&2
    echo "✗ Example: $0 8.14" >&2
    exit 1
fi
VERSION="$1"
LOG_DIR="/var/log"
LOGSTASH_LOG="$LOG_DIR/logstash.log"
TEMP_DATA_DIR="/tmp/logstash_test_${VERSION}"
TEST_LOG="/var/log/dpkg.log"
killall -9 java logstash ruby 2>/dev/null || true
sleep 2
rm -rf "$TEMP_DATA_DIR" 2>/dev/null || true
mkdir -p "$TEMP_DATA_DIR"
chmod 777 "$TEMP_DATA_DIR"
rm -f "$LOGSTASH_LOG"
touch "$LOGSTASH_LOG"
chmod 644 "$LOGSTASH_LOG"
echo ">>> Testing Logstash version: $VERSION"
echo -e "2023-05-31 10:42:51 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>\n2023-05-31 10:42:52 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>\n2023-05-31 10:42:53 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1\n2023-05-31 10:42:54 status installed libc-bin:amd64 2.31-0ubuntu9.9\n2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9" > "$TEST_LOG"
chmod 644 "$TEST_LOG"
log_lines=$(wc -l < "$TEST_LOG")
if [ "$log_lines" -eq 5 ]; then
    echo "✓ Log file has 5 lines"
else
    echo "✗ Error: Expected 5 lines, got $log_lines" >&2
    exit 1
fi
if curl -s localhost:9308/cli_json -d 'show status' > /dev/null; then
    echo "✓ Manticore Search available"
else
    echo "✗ Error: Manticore Search unavailable" >&2
    exit 1
fi
mysql -h0 -P9306 -e "DROP TABLE IF EXISTS dpkg_log" > /dev/null 2>&1
mysql -h0 -P9306 -e "CREATE TABLE dpkg_log (id bigint, message string attribute indexed, host string attribute indexed, path string attribute indexed, \`@version\` string attribute indexed, \`@timestamp\` timestamp)" > /dev/null 2>&1 || {
    echo "✗ Error: Failed to create table dpkg_log" >&2
    exit 1
}
if [ ! -d "/usr/share/logstash" ]; then
    mkdir -p /usr/share/logstash
fi
if [ ! -d "/usr/share/logstash/logstash-${VERSION}.0" ]; then
    if [ -f "logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    elif [ -f "/tmp/logstash-${VERSION}.0.tar.gz" ]; then
        tar -xzf "/tmp/logstash-${VERSION}.0.tar.gz" -C /usr/share/logstash/ > /dev/null 2>&1
    else
        echo "✗ Error: Logstash $VERSION archive not found" >&2
        exit 1
    fi
fi
if [ -f "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" ]; then
    ln -sf "/usr/share/logstash/logstash-${VERSION}.0/bin/logstash" /usr/bin/logstash > /dev/null 2>&1
else
    echo "✗ Error: Logstash $VERSION binary not found" >&2
    exit 1
fi
version_major=$(echo "$VERSION" | cut -d. -f1)
version_minor=$(echo "$VERSION" | cut -d. -f2)
ecs_args=""
if [ "$version_major" -gt 7 ] || ([ "$version_major" -eq 7 ] && [ "$version_minor" -ge 5 ]); then
    ecs_args="--pipeline.ecs_compatibility v1"
fi
ilm_settings=""
if [ "$version_major" -ge 8 ]; then
    ilm_settings="ilm_enabled => false
    manage_template => false"
fi
if [ "$version_major" -ge 9 ]; then
    export ALLOW_SUPERUSER=1
    sed -i '259s/^/# /' /usr/share/logstash/logstash-${VERSION}.0/logstash-core/lib/logstash/runner.rb 2>/dev/null || true
fi
CONFIG_FILE="/tmp/logstash_config_${VERSION}.conf"
if [ "$version_major" -gt 9 ] || ([ "$version_major" -eq 9 ] && [ "$version_minor" -ge 1 ]); then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "logstash-test" }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
elif [ "$version_major" -ge 9 ]; then
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "path" => "$TEST_LOG"
    }
    add_field => { "host" => "%{[host][name]}" }
    remove_field => ["event", "log", "[host][name]"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
else
    cat > "$CONFIG_FILE" << CONF
input {
  file {
    path => ["$TEST_LOG"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    mode => "read"
  }
}
filter {
  mutate {
    replace => {
      "@version" => "1"
      "host" => "logstash"
      "path" => "$TEST_LOG"
    }
    remove_field => ["event", "log"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9308"]
    index => "dpkg_log"
    action => "index"
    timeout => 30
    $ilm_settings
  }
}
CONF
fi
batch_size=10
export LS_JAVA_OPTS="-Xms256m -Xmx512m"
wait_time=60
timeout $wait_time logstash --path.data="$TEMP_DATA_DIR" \
  -f "$CONFIG_FILE" \
  --log.level=error \
  --pipeline.batch.size $batch_size \
  $ecs_args \
  > /dev/null 2>&1 || true
sleep 10
echo "✓ Logstash $VERSION started and processed logs"
row_count=$(mysql -h0 -P9306 -N -B -e "SELECT COUNT(*) FROM dpkg_log" | grep -o '[0-9]\+' || echo "0")
if [ "$row_count" -eq 5 ]; then
    echo "✓ Row count check for $VERSION: $row_count rows"
else
    echo "✗ Error: Expected 5 rows, got $row_count" >&2
    mysql -h0 -P9306 -e "SELECT * FROM dpkg_log\G" >&2
    exit 1
fi
structure_ok=true
for field in "id" "message" "host" "path" "@version" "@timestamp"; do
    if ! mysql -h0 -P9306 -N -B -e "SELECT $field FROM dpkg_log LIMIT 1" > /dev/null 2>&1; then
        structure_ok=false
        echo "✗ Error: Field '$field' check failed" >&2
        break
    fi
done
if $structure_ok; then
    echo "✓ Structure check for $VERSION: passed"
else
    echo "✗ Error: Structure check failed" >&2
    mysql -h0 -P9306 -e "DESCRIBE dpkg_log" >&2
    exit 1
fi
echo "✓ Logstash version $VERSION tested successfully"
rm -f "$CONFIG_FILE" "$TEST_LOG" > /dev/null 2>&1
EOF
––– output –––
OK
––– input –––
chmod +x /tmp/logstash-single-test.sh; echo $?
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz" -O logstash-7.17.0.tar.gz && gzip -t logstash-7.17.0.tar.gz 2>/dev/null && tar -tzf logstash-7.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 7.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 7.17.0 after 3 attempts" >&2 || rm -f logstash-7.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 7.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-7.17.0 logstash-7.17.0.tar.gz /tmp/logstash_test_7.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.0.0-linux-x86_64.tar.gz" -O logstash-8.0.0.tar.gz && gzip -t logstash-8.0.0.tar.gz 2>/dev/null && tar -tzf logstash-8.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.0.0 after 3 attempts" >&2 || rm -f logstash-8.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.0
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.0.0 logstash-8.0.0.tar.gz /tmp/logstash_test_8.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz" -O logstash-8.1.0.tar.gz && gzip -t logstash-8.1.0.tar.gz 2>/dev/null && tar -tzf logstash-8.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.1.0 after 3 attempts" >&2 || rm -f logstash-8.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.1
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.1.0 logstash-8.1.0.tar.gz /tmp/logstash_test_8.1
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.2.0-linux-x86_64.tar.gz" -O logstash-8.2.0.tar.gz && gzip -t logstash-8.2.0.tar.gz 2>/dev/null && tar -tzf logstash-8.2.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.2.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.2.0 after 3 attempts" >&2 || rm -f logstash-8.2.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.2
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.2.0 logstash-8.2.0.tar.gz /tmp/logstash_test_8.2
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.3.0-linux-x86_64.tar.gz" -O logstash-8.3.0.tar.gz && gzip -t logstash-8.3.0.tar.gz 2>/dev/null && tar -tzf logstash-8.3.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.3.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.3.0 after 3 attempts" >&2 || rm -f logstash-8.3.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.3
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.3.0 logstash-8.3.0.tar.gz /tmp/logstash_test_8.3
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.4.0-linux-x86_64.tar.gz" -O logstash-8.4.0.tar.gz && gzip -t logstash-8.4.0.tar.gz 2>/dev/null && tar -tzf logstash-8.4.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.4.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.4.0 after 3 attempts" >&2 || rm -f logstash-8.4.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.4
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.4.0 logstash-8.4.0.tar.gz /tmp/logstash_test_8.4
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz" -O logstash-8.5.0.tar.gz && gzip -t logstash-8.5.0.tar.gz 2>/dev/null && tar -tzf logstash-8.5.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.5.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.5.0 after 3 attempts" >&2 || rm -f logstash-8.5.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.5
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.5.0 logstash-8.5.0.tar.gz /tmp/logstash_test_8.5
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.6.0-linux-x86_64.tar.gz" -O logstash-8.6.0.tar.gz && gzip -t logstash-8.6.0.tar.gz 2>/dev/null && tar -tzf logstash-8.6.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.6.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.6.0 after 3 attempts" >&2 || rm -f logstash-8.6.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.6
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.6.0 logstash-8.6.0.tar.gz /tmp/logstash_test_8.6
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.7.0-linux-x86_64.tar.gz" -O logstash-8.7.0.tar.gz && gzip -t logstash-8.7.0.tar.gz 2>/dev/null && tar -tzf logstash-8.7.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.7.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.7.0 after 3 attempts" >&2 || rm -f logstash-8.7.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.7
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.7.0 logstash-8.7.0.tar.gz /tmp/logstash_test_8.7
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-linux-x86_64.tar.gz" -O logstash-8.8.0.tar.gz && gzip -t logstash-8.8.0.tar.gz 2>/dev/null && tar -tzf logstash-8.8.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.8.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.8.0 after 3 attempts" >&2 || rm -f logstash-8.8.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.8
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.8.0 logstash-8.8.0.tar.gz /tmp/logstash_test_8.8
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.9.0-linux-x86_64.tar.gz" -O logstash-8.9.0.tar.gz && gzip -t logstash-8.9.0.tar.gz 2>/dev/null && tar -tzf logstash-8.9.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.9.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.9.0 after 3 attempts" >&2 || rm -f logstash-8.9.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.9
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.9.0 logstash-8.9.0.tar.gz /tmp/logstash_test_8.9
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.10.0-linux-x86_64.tar.gz" -O logstash-8.10.0.tar.gz && gzip -t logstash-8.10.0.tar.gz 2>/dev/null && tar -tzf logstash-8.10.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.10.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.10.0 after 3 attempts" >&2 || rm -f logstash-8.10.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.10
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.10.0 logstash-8.10.0.tar.gz /tmp/logstash_test_8.10
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz" -O logstash-8.11.0.tar.gz && gzip -t logstash-8.11.0.tar.gz 2>/dev/null && tar -tzf logstash-8.11.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.11.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.11.0 after 3 attempts" >&2 || rm -f logstash-8.11.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.11
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.11.0 logstash-8.11.0.tar.gz /tmp/logstash_test_8.11
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-linux-x86_64.tar.gz" -O logstash-8.12.0.tar.gz && gzip -t logstash-8.12.0.tar.gz 2>/dev/null && tar -tzf logstash-8.12.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.12.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.12.0 after 3 attempts" >&2 || rm -f logstash-8.12.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.12
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.12.0 logstash-8.12.0.tar.gz /tmp/logstash_test_8.12
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-linux-x86_64.tar.gz" -O logstash-8.13.0.tar.gz && gzip -t logstash-8.13.0.tar.gz 2>/dev/null && tar -tzf logstash-8.13.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.13.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.13.0 after 3 attempts" >&2 || rm -f logstash-8.13.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.13
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.13.0 logstash-8.13.0.tar.gz /tmp/logstash_test_8.13
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.14.0-linux-x86_64.tar.gz" -O logstash-8.14.0.tar.gz && gzip -t logstash-8.14.0.tar.gz 2>/dev/null && tar -tzf logstash-8.14.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.14.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.14.0 after 3 attempts" >&2 || rm -f logstash-8.14.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.14
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.14.0 logstash-8.14.0.tar.gz /tmp/logstash_test_8.14
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz" -O logstash-8.15.0.tar.gz && gzip -t logstash-8.15.0.tar.gz 2>/dev/null && tar -tzf logstash-8.15.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.15.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.15.0 after 3 attempts" >&2 || rm -f logstash-8.15.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.15
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.15.0 logstash-8.15.0.tar.gz /tmp/logstash_test_8.15
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.16.0-linux-x86_64.tar.gz" -O logstash-8.16.0.tar.gz && gzip -t logstash-8.16.0.tar.gz 2>/dev/null && tar -tzf logstash-8.16.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.16.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.16.0 after 3 attempts" >&2 || rm -f logstash-8.16.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.16
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.16.0 logstash-8.16.0.tar.gz /tmp/logstash_test_8.16
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.17.0-linux-x86_64.tar.gz" -O logstash-8.17.0.tar.gz && gzip -t logstash-8.17.0.tar.gz 2>/dev/null && tar -tzf logstash-8.17.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.17.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.17.0 after 3 attempts" >&2 || rm -f logstash-8.17.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.17
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.17.0 logstash-8.17.0.tar.gz /tmp/logstash_test_8.17
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.18.0-linux-x86_64.tar.gz" -O logstash-8.18.0.tar.gz && gzip -t logstash-8.18.0.tar.gz 2>/dev/null && tar -tzf logstash-8.18.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.18.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.18.0 after 3 attempts" >&2 || rm -f logstash-8.18.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.18
––– output –––
OK
––– input –––
rm -rf /usr/share/logstash/logstash-8.18.0 logstash-8.18.0.tar.gz /tmp/logstash_test_8.18
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-8.19.0-linux-x86_64.tar.gz" -O logstash-8.19.0.tar.gz && gzip -t logstash-8.19.0.tar.gz 2>/dev/null && tar -tzf logstash-8.19.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 8.19.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 8.19.0 after 3 attempts" >&2 || rm -f logstash-8.19.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 8.19
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.0.0-linux-x86_64.tar.gz" -O logstash-9.0.0.tar.gz && gzip -t logstash-9.0.0.tar.gz 2>/dev/null && tar -tzf logstash-9.0.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.0.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.0.0 after 3 attempts" >&2 || rm -f logstash-9.0.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.0
––– output –––
OK
––– input –––
for i in {1..3}; do wget -q --timeout=60 --tries=1 "https://artifacts.elastic.co/downloads/logstash/logstash-9.1.0-linux-x86_64.tar.gz" -O logstash-9.1.0.tar.gz && gzip -t logstash-9.1.0.tar.gz 2>/dev/null && tar -tzf logstash-9.1.0.tar.gz | grep -q "bin/logstash" && { echo "✓ Logstash 9.1.0 downloaded successfully"; break; } || { [ $i -eq 3 ] && echo "✗ Failed to download Logstash 9.1.0 after 3 attempts" >&2 || rm -f logstash-9.1.0.tar.gz >/dev/null 2>&1; sleep 2; }; done
––– output –––
OK
––– input –––
timeout 300 bash /tmp/logstash-single-test.sh 9.1
––– output –––
OK
––– input –––
rm -f logstash-*.tar.gz
––– output –––
OK

PavelShilin89 · 2025-11-03T12:16:02Z

Merged with master

PavelShilin89 and others added 3 commits October 24, 2025 00:36

Added testing of filebeat version 9.2 in test-integrations-support-fi…

74b7516

…lebeat-versions.rec

Added configuration section for Filebeat 9.2+ with filestream input t…

b8f7f0a

…ype.

docs: Auto-translate documentation changes by PavelShilin89

14e1de0

PavelShilin89 added the done label Oct 24, 2025

sanikolaev requested changes Oct 24, 2025

View reviewed changes

PavelShilin89 and others added 3 commits October 27, 2025 23:20

Update docs Filebeat.md and test-integrations-support-filebeat-versio…

8050f28

…ns.rec

docs: Auto-translate documentation changes by PavelShilin89

8bd9df4

Merge branch 'master' into test/added-test-filebeat-9.2

05a2937

PavelShilin89 and others added 2 commits October 28, 2025 01:07

Unified configuration for Filebeat 9.0+ (9.0, 9.1, 9.2+)

9d55c08

docs: Auto-translate documentation changes by PavelShilin89

cfb1d80

PavelShilin89 added 2 commits October 28, 2025 08:03

Remove outdated Chinese & Rissuan Filebeat translation

c3f155e

Revert "Remove outdated Chinese & Rissuan Filebeat translation"

8278899

This reverts commit c3f155e.

PavelShilin89 requested a review from sanikolaev October 28, 2025 08:52

sanikolaev requested changes Oct 28, 2025

View reviewed changes

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec Outdated Show resolved Hide resolved

test/clt-tests/integrations/test-integrations-support-filebeat-versions.rec Outdated Show resolved Hide resolved

Update test/clt-tests/integrations/test-integrations-support-filebeat…

77a21ce

…-versions.rec Co-authored-by: Sergey Nikolaev <[email protected]>

PavelShilin89 added 2 commits October 28, 2025 11:48

For all 9.x versions, used filestream with fingerprint disabled

d23e5c4

Merge branch 'test/added-test-filebeat-9.2' of https://github.com/man…

051bf0e

…ticoresoftware/manticoresearch into test/added-test-filebeat-9.2

Final fix test-integrations-support-filebeat-versions.rec

6d1420f

PavelShilin89 requested a review from sanikolaev October 28, 2025 12:11

PavelShilin89 and others added 2 commits October 29, 2025 10:52

Fix for glibc 2.35+ compatibility (Ubuntu 22.04+)

5e9dbb5

docs: Auto-translate documentation changes by PavelShilin89

d325bdc

PavelShilin89 and others added 3 commits October 29, 2025 11:10

Updated output

d62436a

Merge branch 'test/added-test-filebeat-9.2' of https://github.com/man…

908ed6b

…ticoresoftware/manticoresearch into test/added-test-filebeat-9.2

Merge branch 'master' into test/added-test-filebeat-9.2

1c310c7

PavelShilin89 and others added 4 commits October 29, 2025 18:23

Updated conflict-handling-verification.rec

0b71d48

Deleted test-integrations-support-filebeat-versions.rec

c237bdf

Merge branch 'master' into test/added-test-filebeat-9.2

1499b19

docs: Auto-translate documentation changes by PavelShilin89

682e9e6

PavelShilin89 requested review from sanikolaev and removed request for sanikolaev October 30, 2025 07:17

PavelShilin89 added done and removed done labels Oct 30, 2025

Merge branch 'master' into test/added-test-filebeat-9.2

7b8a6d8

PavelShilin89 added 3 commits November 3, 2025 11:28

Merge branch 'master' into test/added-test-filebeat-9.2

82d024e

Update translator

fbf5b30

Documentation files for calling the auto-translator have been deleted.

7000ddb

Merge branch 'master' into test/added-test-filebeat-9.2

d3c705f

sanikolaev approved these changes Nov 3, 2025

View reviewed changes

PavelShilin89 merged commit ec681c8 into master Nov 3, 2025
127 checks passed

PavelShilin89 deleted the test/added-test-filebeat-9.2 branch November 3, 2025 12:15

Uh oh!

Add testing of Filebeat version 9.2 #3857

Add testing of Filebeat version 9.2 #3857

Conversation

PavelShilin89 commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 24, 2025

clt

Failed tests:

Uh oh!

sanikolaev left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 27, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 28, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

test_support_filebeat_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linux debug test results

Uh oh!

github-actions bot commented Oct 28, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 28, 2025

test_support_filebeat_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_support_filebeat_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_support_filebeat_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_kafka

Failed tests:

Uh oh!

github-actions bot commented Oct 29, 2025

test_support_logstash_versions

Failed tests:

Uh oh!

github-actions bot commented Oct 31, 2025

github-actions bot commented Oct 28, 2025 •

edited

Loading