B API for aux_list/OP_MULTICONCAT does not return the last segment when plain & utf8 representations are different

[p5pRT]

Migrated from rt.perl.org#133535 (status was 'open')

Searchable as RT133535$

[p5pRT]

From @atoomic

Created by @atoomic

I noticed this while using B API with op/substr.t to compile it using B​::C
with Perl 5.28.0

From the comment in pp_hot.c we can read that in some cases we can have two
sets of segment lengths

  * * If the string has different plain and utf8 representations
  * (e.g. "\x80"), then then aux[PERL_MULTICONCAT_IX_PLAIN_PV/LEN]]
  * holds the plain rep, while aux[PERL_MULTICONCAT_IX_UTF8_PV/LEN]
  * holds the utf8 rep, and there are 2 sets of segment lengths,
  * with the utf8 set following after the plain set.

I've the feeling that B API aux_list for multiconcat is missing to read the
last segment in that scenario

With this simplified version of op/substr.t, it's easier to debug as we
have one single multiconcat op.
________________________________________________________________________________
#!./perl

print "1..1\n";

use utf8;
my $refee = bless [], "\x{100}a";
my $string = $refee;
$string = "$string";
substr $refee, 0, 0, "\xff";
my $expect = "\xff$string"; # <---- multiconcat
print "$refee" eq $expect ? "ok 1\n" : "not ok 1\n";
________________________________________________________________________________

While running the program we are going through this code, where nargs=1,
so we are clearly using not the first but the second segment.

Perl_pp_multiconcat
  │676 const_lens = aux + PERL_MULTICONCAT_IX_LENGTHS; │
  │677
  │678 if (dst_utf8) { │
  │679 const_pv = aux[PERL_MULTICONCAT_IX_UTF8_PV].pv; │
  │680 if ( aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv │
  │681 && const_pv != aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv) │
  │682 /* separate sets of lengths for plain and utf8 */ │
  >│683 const_lens += nargs + 1;

Here is a look at aux

# ----- dump of aux from Perl_pp_multiconcat
# header
aux = aux[0] = 1
aux[1] = \377
aux[2] = 1
aux[3] = "ÿ",
aux[4] = 2

# first element
aux[5] 1 # <---- const_lens
aux[6] -1
# second segment which was not returned by B​::API
aux[7] 2
aux[8] -1

Not exactly sure if adding such a rule is good enough but this is fixing
the cases
where before that we would only read the first segment

# Suggested patch to B API for aux_list/OP_MULTICONCAT
if (
aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv
  && aux[PERL_MULTICONCAT_IX_UTF8_PV].pv
  && aux[PERL_MULTICONCAT_IX_UTF8_PV].pv !=
aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv ) {
# read the additional segment
nargs += 2;
}

Perl Info

Flags:
    category=library
    severity=low
    module=B

Site configuration information for perl 5.28.0:

Configured by nicolas at Wed Nov 29 10:26:27 MST 2017.

Summary of my perl5 (revision 5 version 26 subversion 1) configuration:

  Platform:
    osname=darwin
    osvers=15.6.0
    archname=darwin-2level
    uname='darwin nicolas-r.local 15.6.0 darwin kernel version 15.6.0: mon
oct 2 22:20:08 pdt 2017; root:xnu-3248.71.4~1release_x86_64 x86_64 '
    config_args='-de -Dprefix=/usr/local/perl/perls/perl-5.28.0
-Aeval:scriptdir=/usr/local/perl/perls/perl-5.28.0/bin'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    optimize='-O3'
    cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib
-fstack-protector-strong'



@INC for perl 5.28.0:
    /Users/nicolas/.dotfiles/perl-must-have/lib
    /Users/nicolas/perl5/lib/perl5/
    /usr/local/perl/perls/perl-5.28.0/lib/site_perl/5.28.0/darwin-2level
    /usr/local/perl/perls/perl-5.28.0/lib/site_perl/5.28.0
    /usr/local/perl/perls/perl-5.28.0/lib/5.28.0/darwin-2level
    /usr/local/perl/perls/perl-5.28.0/lib/5.28.0


Environment for perl 5.28.0:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/nicolas
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)

PATH=/usr/local/perl/bin:/usr/local/perl/perls/perl-5.28.0/bin:/usr/local/opt/ccache/libexec:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/usr/local/MacGPG2/bin:/Users/nicolas/.dotfiles/bin:/Users/nicolas/perl5/bin
    PERL5DB=use Devel::NYTProf

PERL5LIB=/Users/nicolas/.dotfiles/perl-must-have/lib:/Users/nicolas/perl5/lib/perl5/
    PERLBREW_BASHRC_VERSION=0.80
    PERLBREW_HOME=/Users/nicolas/.perlbrew
    PERLBREW_MANPATH=/usr/local/perl/perls/perl-5.28.0/man
    PERLBREW_PATH=/usr/local/perl/bin:/usr/local/perl/perls/perl-5.28.0/bin
    PERLBREW_PERL=perl-5.28.0
    PERLBREW_ROOT=/usr/local/perl
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    PERL_CPANM_OPT=--quiet
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @tonycoz

On Thu, 20 Sep 2018 09​:57​:52 -0700, atoomic wrote​:

I noticed this while using B API with op/substr.t to compile it using
B​::C
with Perl 5.28.0

From the comment in pp_hot.c we can read that in some cases we can
have two
sets of segment lengths

* * If the string has different plain and utf8 representations
* (e.g. "\x80"), then then aux[PERL_MULTICONCAT_IX_PLAIN_PV/LEN]]
* holds the plain rep, while aux[PERL_MULTICONCAT_IX_UTF8_PV/LEN]
* holds the utf8 rep, and there are 2 sets of segment lengths,
* with the utf8 set following after the plain set.

I've the feeling that B API aux_list for multiconcat is missing to
read the
last segment in that scenario

With this simplified version of op/substr.t, it's easier to debug as
we
have one single multiconcat op.
________________________________________________________________________________
#!./perl

print "1..1\n";

use utf8;
my $refee = bless [], "\x{100}a";
my $string = $refee;
$string = "$string";
substr $refee, 0, 0, "\xff";
my $expect = "\xff$string"; # <---- multiconcat
print "$refee" eq $expect ? "ok 1\n" : "not ok 1\n";
________________________________________________________________________________

While running the program we are going through this code, where
nargs=1,
so we are clearly using not the first but the second segment.

Perl_pp_multiconcat
│676 const_lens = aux + PERL_MULTICONCAT_IX_LENGTHS; │
│677
│678 if (dst_utf8) { │
│679 const_pv = aux[PERL_MULTICONCAT_IX_UTF8_PV].pv; │
│680 if ( aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv │
│681 && const_pv != aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv) │
│682 /* separate sets of lengths for plain and utf8 */ │

│683 const_lens += nargs + 1;

Here is a look at aux

# ----- dump of aux from Perl_pp_multiconcat
# header
aux = aux[0] = 1
aux[1] = \377
aux[2] = 1
aux[3] = "ÿ",
aux[4] = 2

# first element
aux[5] 1 # <---- const_lens
aux[6] -1
# second segment which was not returned by B​::API
aux[7] 2
aux[8] -1

Not exactly sure if adding such a rule is good enough but this is
fixing
the cases
where before that we would only read the first segment

# Suggested patch to B API for aux_list/OP_MULTICONCAT
if (
aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv
&& aux[PERL_MULTICONCAT_IX_UTF8_PV].pv
&& aux[PERL_MULTICONCAT_IX_UTF8_PV].pv !=
aux[PERL_MULTICONCAT_IX_PLAIN_PV].pv ) {
# read the additional segment
nargs += 2;
}

Considering aux_list() code for OP_MULTICONCAT turns the offsets into character rather than byte offsets, won't the 2 from​:

aux[7] 2
aux[8] -1

be converted into a 1, making it the same as the first segment?

I don't know what extra useful information you would get from this change.

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'