System information
| Type |
Version/Name |
| Distribution Name |
Ubuntu |
| Distribution Version |
24.04.4 LTS |
| Kernel Version |
6.8.0-136-generic |
| Architecture |
x86_64 |
| OpenZFS Version |
master (4e00623), scripts/zloop.sh unmodified |
Describe the problem you're observing
On a system whose kernel.core_pattern is a pipe, zloop.sh counts a crash for
every iteration when it is not run as root, however well ztest does.
The pattern is read at startup:
coreglob="$(grep -E -o '^([^|%[:space:]]*)' /proc/sys/kernel/core_pattern)*"
if [[ $coreglob = "*" ]]; then
echo "Setting core file pattern..."
echo "core" > /proc/sys/kernel/core_pattern
coreglob="$(grep -E -o '^([^|%[:space:]]*)' \
/proc/sys/kernel/core_pattern)*"
fi
Ubuntu's pattern starts with |, so the extraction yields the empty string and
coreglob becomes *. Writing core_pattern needs privileges, so for an
ordinary user that write fails, the re-read returns * again, and core_file()
ls -tr1 $coreglob 2>/dev/null | head -1
returns the first file in the directory zloop was started from. store_core()
then treats it as a core:
if [[ $ztrc -ne 0 ]] || [[ -f "$core" ]]; then
The loop writes ztest.history and ztest.out into that directory before it
invokes ztest, so the check is already true on the first iteration. Every
iteration is archived as a crash and zloop.sh exits 1.
The zloop workflow invokes the script under sudo, so CI is unaffected.
Describe how to reproduce the problem
Using a stub in place of ztest, so that every iteration certainly succeeds:
$ cat > /tmp/fakeztest.sh <<'EOF'
#!/bin/sh
d=""
while [ $# -gt 0 ]; do
case "$1" in -f) d="$2" ;; esac
shift
done
[ -n "$d" ] && : > "$d/ztest.0a"
exit 0
EOF
$ chmod +x /tmp/fakeztest.sh
$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -F%F -- %E
$ rm -rf /tmp/zloop-run /tmp/zloop-cores
$ mkdir -p /tmp/zloop-work /tmp/zloop-run
$ cd /tmp/zloop-run # zloop bails at startup if it finds a "core" here
$ ZTEST=/tmp/fakeztest.sh ZDB=/bin/true \
/path/to/zfs/scripts/zloop.sh -I 5 -c /tmp/zloop-cores -f /tmp/zloop-work -- -T 1 -P 1
Include any warning/errors/backtraces from the system logs
Setting core file pattern...
/home/mike/zfs-mmp/scripts/zloop.sh: line 83: /proc/sys/kernel/core_pattern: Permission denied
core dump directory (/tmp/zloop-cores) does not exist, creating it.
08/21 01:24:19 /tmp/fakeztest.sh -G -VVVVV -K raidz -m 2 -r 0 -D 0 -S 0 -R 1 -v 2 -a 9 -C special=random -s 512m -f /tmp/zloop-work/zloop-run -T 1 -P 1
*** ztest crash found - moving logs to /tmp/zloop-cores/zloop-260821-012419
continuing...
08/21 01:24:19 /tmp/fakeztest.sh -G -VVVVV -K eraidz -m 0 -r 7 -D 0 -S 0 -R 1 -v 1 -a 9 -C special=random -s 512m -f /tmp/zloop-work/zloop-run -T 1 -P 1
*** ztest crash found - moving logs to /tmp/zloop-cores/zloop-260821-012419
continuing...
08/21 01:24:19 /tmp/fakeztest.sh -G -VVVVV -K eraidz -m 0 -r 9 -D 0 -S 0 -R 3 -v 1 -a 12 -C special=random -s 512m -f /tmp/zloop-work/zloop-run -T 1 -P 1
*** ztest crash found - moving logs to /tmp/zloop-cores/zloop-260821-012419
continuing...
08/21 01:24:19 /tmp/fakeztest.sh -G -VVVVV -K raidz -m 2 -r 0 -D 0 -S 0 -R 1 -v 2 -a 9 -C special=random -s 512m -f /tmp/zloop-work/zloop-run -T 1 -P 1
*** ztest crash found - moving logs to /tmp/zloop-cores/zloop-260821-012419
continuing...
08/21 01:24:19 /tmp/fakeztest.sh -G -VVVVV -K raidz -m 2 -r 0 -D 0 -S 0 -R 1 -v 5 -a 9 -C special=random -s 512m -f /tmp/zloop-work/zloop-run -T 1 -P 1
*** ztest crash found - moving logs to /tmp/zloop-cores/zloop-260821-012419
continuing...
zloop finished, 5 crashes found
/home/mike/zfs-mmp/scripts/zloop.sh: line 344: /proc/sys/kernel/core_pattern: Permission denied
I met this with a real ztest first, which is what makes it awkward to spot: six
iterations came back as crashes, and re-running those command lines by hand
showed each of them exiting 0.
Unprivileged operation came up before in #9613, which proposed switching to
coredumpctl so the script would not need to set core_pattern at all; it was
closed unmerged. Short of that, the script could stop when it cannot establish a
usable pattern, since it cannot collect cores in that state anyway, or
core_file() could be given a glob that will not match ordinary files, so a
missing core stays missing. Happy to put a patch together if there is a
preference.
System information
scripts/zloop.shunmodifiedDescribe the problem you're observing
On a system whose
kernel.core_patternis a pipe,zloop.shcounts a crash forevery iteration when it is not run as root, however well ztest does.
The pattern is read at startup:
Ubuntu's pattern starts with
|, so the extraction yields the empty string andcoreglobbecomes*. Writingcore_patternneeds privileges, so for anordinary user that write fails, the re-read returns
*again, andcore_file()returns the first file in the directory zloop was started from.
store_core()then treats it as a core:
The loop writes
ztest.historyandztest.outinto that directory before itinvokes ztest, so the check is already true on the first iteration. Every
iteration is archived as a crash and
zloop.shexits 1.The zloop workflow invokes the script under
sudo, so CI is unaffected.Describe how to reproduce the problem
Using a stub in place of ztest, so that every iteration certainly succeeds:
Include any warning/errors/backtraces from the system logs
I met this with a real ztest first, which is what makes it awkward to spot: six
iterations came back as crashes, and re-running those command lines by hand
showed each of them exiting 0.
Unprivileged operation came up before in #9613, which proposed switching to
coredumpctlso the script would not need to setcore_patternat all; it wasclosed unmerged. Short of that, the script could stop when it cannot establish a
usable pattern, since it cannot collect cores in that state anyway, or
core_file()could be given a glob that will not match ordinary files, so amissing core stays missing. Happy to put a patch together if there is a
preference.