Autovivification should be ignored when testing for existence

[p5pRT]

Migrated from rt.perl.org#107528 (status was 'open')

Searchable as RT107528$

[p5pRT]

From rcolvin@people.net.au

Created by rcolvin@people.net.au

This is a bug report for perl from rcolvin@​people.net.au,
generated with the help of perlbug 1.33 running under perl v5.6.1.

-----------------------------------------------------------------
When testing the existence of a subkey in a hash, autovivification will create the top level key(s) which means subsequent tests against the top level key(s)
will give the incorrect result.

A simple code example​:

# perl -v
This is perl 5, version 12, subversion 3 (v5.12.3) built for sun4-solaris-thread-multi
# cat perl_bug.pl
#!/bin/perl
use strict;

our %list;

$list{A}{sub} = "VALUE(A)";
$list{B}{sub} = "VALUE(B)";

print "LIST\n";
foreach(sort(keys(%list)))
{
  print "\tKEY​: $_\t \$list{$_}{sub} contains '$list{$_}{sub}'\n";
}
print "\n";

print "TEST #1​: Output TRUE if \$list{A}{sub} present, otherwise output nothing​: EXPECT 'TRUE'\n";
print "\tTRUE\n" if($list{A}{sub});

print "TEST #2​: Ouput TRUE if \$list{C}{sub} present, otherwise output nothing​: EXPECT nothing\n";
print "\tTRUE\n" if($list{C}{sub});

print "TEST #3​: Output LIST again, \$list{C} should not be listed\n";
print "LIST\n";
foreach(sort(keys(%list)))
{
  print "\tKEY​: $_\t \$list{$_}{sub} contains '$list{$_}{sub}'\n";
}

print "TEST #4​: Output TRUE if \$list{D}{sub} defined, otherwise output nothing​: EXPECT nothing\n";
print "\tTRUE\n" if(defined $list{D}{sub});

print "TEST #5​: Output LIST again, \$list{D} should not be listed\n";
print "LIST\n";
foreach(sort(keys(%list)))
{
  print "\tKEY​: $_\t \$list{$_}{sub} contains '$list{$_}{sub}'\n";
}

print print "TEST #6​: Output TRUE if \$list{E}{sub} exists, otherwise output nothing​: EXPECT nothing\n";
print "\tTRUE\n" if(exists $list{E}{sub});

print "TEST #7​: Output LIST again, \$list{E} should not be listed\n";
print "LIST\n";
foreach(sort(keys(%list)))
{
  print "\tKEY​: $_\t \$list{$_}{sub} contains '$list{$_}{sub}'\n";
}

print print "TEST #8​: Output TRUE if \$list{F}{sub}{sub2} defined, otherwise output nothing​: EXPECT nothing\n";
print "\tTRUE\n" if(defined $list{F}{sub}{sub2});

print "TEST #9​: Output LIST again, \$list{E} should not be listed\n";
print "LIST\n";
foreach(sort(keys(%list)))
{
  print "\tKEY​: $_\t \$list{$_}{sub} contains '$list{$_}{sub}'\n";
}

produces the following output​:

LIST
  KEY​: A $list{A}{sub} contains 'VALUE(A)'
  KEY​: B $list{B}{sub} contains 'VALUE(B)'

TEST #1​: Output TRUE if $list{A}{sub} present, otherwise output nothing​: EXPECT 'TRUE'
  TRUE
TEST #2​: Ouput TRUE if $list{C}{sub} present, otherwise output nothing​: EXPECT nothing
TEST #3​: Output LIST again, $list{C} should not be listed
LIST
  KEY​: A $list{A}{sub} contains 'VALUE(A)'
  KEY​: B $list{B}{sub} contains 'VALUE(B)'
  KEY​: C $list{C}{sub} contains ''
TEST #4​: Output TRUE if $list{D}{sub} defined, otherwise output nothing​: EXPECT nothing
TEST #5​: Output LIST again, $list{D} should not be listed
LIST
  KEY​: A $list{A}{sub} contains 'VALUE(A)'
  KEY​: B $list{B}{sub} contains 'VALUE(B)'
  KEY​: C $list{C}{sub} contains ''
  KEY​: D $list{D}{sub} contains ''
TEST #6​: Output TRUE if $list{E}{sub} exists, otherwise output nothing​: EXPECT nothing
1TEST #7​: Output LIST again, $list{E} should not be listed
LIST
  KEY​: A $list{A}{sub} contains 'VALUE(A)'
  KEY​: B $list{B}{sub} contains 'VALUE(B)'
  KEY​: C $list{C}{sub} contains ''
  KEY​: D $list{D}{sub} contains ''
  KEY​: E $list{E}{sub} contains ''
TEST #8​: Output TRUE if $list{F}{sub}{sub2} defined, otherwise output nothing​: EXPECT nothing
1TEST #9​: Output LIST again, $list{E} should not be listed
LIST
  KEY​: A $list{A}{sub} contains 'VALUE(A)'
  KEY​: B $list{B}{sub} contains 'VALUE(B)'
  KEY​: C $list{C}{sub} contains ''
  KEY​: D $list{D}{sub} contains ''
  KEY​: E $list{E}{sub} contains ''
  KEY​: F $list{F}{sub} contains 'HASH(0x369dc)'

Thus, it would be useful if autovivification would ignore creating the top level key(s) when exit or defined is used. Or at least an easy way to disable autovivification in the language rather than having to change code flow or use
no autovivification;
with an external module which may not always be allowable in a production environment.

Perl Info

Flags:
    category=core
    severity=wishlist

Site configuration information for perl v5.6.1:

Configured by epkbjwi at Fri Jul 13 10:42:57 MET DST 2001.

Summary of my perl5 (revision 5.0 version 6 subversion 1) configuration:
  Platform:
    osname=solaris, osvers=2.6, archname=sun4-solaris
    uname='sunos s1 5.6 generic_105181-23 sun4u sparc sunw,ultra-1 '
    config_args='-de'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=undef d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
  Compiler:
    cc='gcc', ccflags ='-fno-strict-aliasing -I/usr/local/include -I/opt/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O',
    cppflags='-fno-strict-aliasing -I/usr/local/include -I/opt/local/include'
    ccversion='', gccversion='2.95.2 19991024 (release)', gccosandvers='solaris2.6'
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib -L/opt/local/lib '
    libpth=/usr/local/lib /opt/local/lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldb -ldl -lm -lc
    perllibs=-lsocket -lnsl -ldl -lm -lc
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib -L/opt/local/lib'

Locally applied patches:
    


@INC for perl v5.6.1:
    /usr/local/lib/perl5/5.6.1/sun4-solaris
    /usr/local/lib/perl5/5.6.1
    /usr/local/lib/perl5/site_perl/5.6.1/sun4-solaris
    /usr/local/lib/perl5/site_perl/5.6.1
    /usr/local/lib/perl5/site_perl/5.005/sun4-solaris
    /usr/local/lib/perl5/site_perl/5.005
    /usr/local/lib/perl5/site_perl
    .


Environment for perl v5.6.1:
    HOME=/home/erobcol
    LANG=C
    LANGUAGE (unset)
    LD_LIBRARY_PATH=
    LOGDIR (unset)
    PATH=/home/erobcol/bin:/proj/env/bin/:/usr/atria/bin:/usr/atria/etc:/usr/atria/etc/utils:/usr/bin:/usr/local/bin/:/opt/local/bin/:/usr/sbin:/etc:/usr/ccs/bin:/usr/dt/bin:/usr/bin/X11:/opt/local/X11/bin/:/opt/local/gnu/bin:/usr/local/gnu/bin:/usr/ucb/:/proj/vau/dev/luva_tools/:/opt/csw/bin/:/usr/bin:/usr/openwin/bin:/usr/atria/bin
    PERL_BADLANG (unset)
    SHELL=/bin/tcsh

[p5pRT]

From zefram@fysh.org

Robert Colvin wrote​:

When testing the existence of a subkey in a hash, autovivification
will create the top level key(s)

It's too late to change the default Perl behaviour in this
regard. However, a pragmatic module can change it. See
<http​://search.cpan.org/dist/autovivification/>.

-zefram

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From rcolvin@people.net.au

On Wed Jan 04 08​:48​:23 2012, zefram@​fysh.org wrote​:

Robert Colvin wrote​:

When testing the existence of a subkey in a hash, autovivification
will create the top level key(s)

It's too late to change the default Perl behaviour in this
regard. However, a pragmatic module can change it. See
<http​://search.cpan.org/dist/autovivification/>.

-zefram

Zefram,

agreed, and I threw the autovivification module in and it tested fine.
But in my opinion if the default behaviour is to create the variable
reference when I test for its existence, that is a bug.

I understand that it is too late to change, hence its severity is
"Wishlist".

The pragmatic module however is not always a solution, in some
controlled environments we have, we do not have the ability to add
modules - hence our only solution is a massive overhaul of the code to
add workarounds.

/Rob

[p5pRT]

From @ikegami

On Wed, Jan 4, 2012 at 8​:04 AM, Robert Colvin <perlbug-followup@​perl.org>wrote​:

When testing the existence of a subkey in a hash, autovivification will
create the top level key(s) which means subsequent tests against the top
level key(s)
will give the incorrect result.

The operand of C<< exists $list{E}{sub} >> is evaluated before C<< exists

is called, so the dereference vivifies C<< $list{E} >> before C<< exists
is called.

On Wed, Jan 4, 2012 at 11​:47 AM, Zefram <zefram@​fysh.org> wrote​:

It's too late to change the default Perl behaviour in this regard.

It seems to be this is a major complaint among Perl's users. Do we want to
fix this under C<< use 5.xxx; >>?

- Eric

[p5pRT]

From @Hugmeir

On Wed, Jan 4, 2012 at 6​:08 PM, Eric Brine <ikegami@​adaelis.com> wrote​:

On Wed, Jan 4, 2012 at 11​:47 AM, Zefram <zefram@​fysh.org> wrote​:

It's too late to change the default Perl behaviour in this regard.

It seems to be this is a major complaint among Perl's users. Do we want to
fix this under C<< use 5.xxx; >>?

Can autovivification or something similar get into the core instead?

[p5pRT]

From @rurban

On Jan 4, 2012 3​:08 PM, "Eric Brine" <ikegami@​adaelis.com> wrote​:

On Wed, Jan 4, 2012 at 8​:04 AM, Robert Colvin <perlbug-followup@​perl.org>
wrote​:

When testing the existence of a subkey in a hash, autovivification will
create the top level key(s) which means subsequent tests against the top
level key(s)
will give the incorrect result.

The operand of C<< exists $list{E}{sub} >> is evaluated before C<< exists

is called, so the dereference vivifies C<< $list{E} >> before C<< exists
is called.

On Wed, Jan 4, 2012 at 11​:47 AM, Zefram <zefram@​fysh.org> wrote​:

It's too late to change the default Perl behaviour in this regard.

It seems to be this is a major complaint among Perl's users. Do we want
to fix this under C<< use 5.xxx; >>?

Absolutely!

- Eric

[p5pRT]

From dcmertens.perl@gmail.com

On Wed, Jan 4, 2012 at 3​:08 PM, Eric Brine <ikegami@​adaelis.com> wrote​:

On Wed, Jan 4, 2012 at 8​:04 AM, Robert Colvin <perlbug-followup@​perl.org>wrote​:

When testing the existence of a subkey in a hash, autovivification will
create the top level key(s) which means subsequent tests against the top
level key(s)
will give the incorrect result.

The operand of C<< exists $list{E}{sub} >> is evaluated before C<< exists

is called, so the dereference vivifies C<< $list{E} >> before C<< exists
is called.

On Wed, Jan 4, 2012 at 11​:47 AM, Zefram <zefram@​fysh.org> wrote​:

It's too late to change the default Perl behaviour in this regard.

It seems to be this is a major complaint among Perl's users. Do we want to
fix this under C<< use 5.xxx; >>?

If it is such a major complaint, how might it go about getting changed? I'm
new, so forgive me if this is a faux pas, but it would appear that the
obvious solution would be to pull the autovivification module into core.
After all, the module addresses the OP's concern exactly, it does not break
normal autovivification, and it has a 100% pass rate.

So, what does it take to get something like this into core?

David

--
Sent via my carrier pigeon.

[p5pRT]

From zefram@fysh.org

Robert Colvin via RT wrote​:

But in my opinion if the default behaviour is to create the variable
reference when I test for its existence, that is a bug.

It's not a bug. The behaviour is perfectly consistent across operators.
"exists" does not create the hash element that you ask it about.
The autovivification applies to the *hash* in which you looked.

-zefram

[p5pRT]

From @ikegami

On Thu, Jan 5, 2012 at 9​:43 AM, David Mertens <dcmertens.perl@​gmail.com>wrote​:

If it is such a major complaint, how might it go about getting changed?
I'm new, so forgive me if this is a faux pas, but it would appear that the
obvious solution would be to pull the autovivification module into core.
After all, the module addresses the OP's concern exactly, it does not break
normal autovivification, and it has a 100% pass rate.

So, what does it take to get something like this into core?

The current (well reasoned) desire is to *remove* modules from the perl
distribution, or at least avoid adding any new modules.

You're saying this module works perfectly well without being bundled into
the perl distribution, so there's no need to include it.

- Eric

[p5pRT]

From rcolvin@people.net.au

On Thu Jan 05 06​:44​:30 2012,
dcmertens.perl@​gmail.com
wrote​:

On Wed, Jan 4, 2012 at 3​:08 PM,
Eric Brine
<ikegami@​adaelis.com>
wrote​:

On Wed, Jan 4, 2012 at 8​:04
AM, Robert Colvin
<perlbug-
followup@​perl.org>wrote​:

When testing the existence
of a subkey in a hash,
autovivification
will
create the top level key(s)
which means subsequent
tests against
the top
level key(s)
will give the incorrect
result.

The operand of C<< exists
$list{E}{sub} >> is
evaluated before C<<
exists

is called, so the
dereference vivifies C<< $list
{E} >> before C<<
exists
is called.

On Wed, Jan 4, 2012 at 11​:47
AM, Zefram
<zefram@​fysh.org> wrote​:

It's too late to change the
default Perl behaviour
in this regard.

It seems to be this is a
major complaint among
Perl's users. Do we
want to
fix this under C<< use 5.xxx;
?

If it is such a major
complaint, how might it go about
getting
changed? I'm
new, so forgive me if this is a
faux pas, but it would
appear that the
obvious solution would be to
pull the autovivification
module into
core.
After all, the module addresses
the OP's concern
exactly, it does not
break
normal autovivification, and it
has a 100% pass rate.

So, what does it take to get
something like this into
core?

David

this is certainly one solution,
and possibly the easiest, though,
in my opinion, not necessarily
the correct option as it doesn't
address, as I set it, the root
cause

[p5pRT]

From rcolvin@people.net.au

On Thu Jan 05 08​:18​:43 2012,
zefram@​fysh.org wrote​:

Robert Colvin via RT wrote​:

But in my opinion if the
default behaviour is to create
the variable
reference when I test for its
existence, that is a bug.

It's not a bug. The behaviour
is perfectly consistent across
operators.
"exists" does not create the
hash element that you ask it
about.
The autovivification applies to
the *hash* in which you looked.

-zefram

indeed it may be consistent,
though it is ultimately
incorrect! in testing existence
the state of variables are
modified. exists should be a safe
operation and it is therefore
reasonable to expect that it
won't make any modifications

[p5pRT]

From rcolvin@people.net.au

On Thu Jan 05 14​:20​:12 2012,
ikegami@​adaelis.com wrote​:

On Thu, Jan 5, 2012 at 9​:43 AM,
David Mertens
<dcmertens.perl@​gmail.com>wrote​:

If it is such a major
complaint, how might it go about
getting
changed?
I'm new, so forgive me if
this is a faux pas, but it would
appear
that the
obvious solution would be to
pull the autovivification module
into
core.
After all, the module
addresses the OP's concern
exactly, it does
not break
normal autovivification, and
it has a 100% pass rate.

So, what does it take to get
something like this into core?

The current (well reasoned)
desire is to *remove* modules
from the
perl
distribution, or at least avoid
adding any new modules.

You're saying this module works
perfectly well without being
bundled
into
the perl distribution, so
there's no need to include it.

- Eric

true, but pre-supposes that this
is not actually a bug, rather its
a particular operating mode which
is undesirable of has undesired
side effects.

as I stated earlier, I still
beleive this is a bug and so
disabling autovivification
whether by import or in the core
or other means is merely a work
around and not a fix.

I would love to be persuaded
otherwise, but I can see no
benefit to this effect in
autovivification, however I can
see plenty of negatives, and
consider it a logical error in
the expectations of how exists or
defined would operate.

finally whilst this module dies
have, to some degree the desired
effect, it is not always
practical or possible to
introduce external modules,
leaving the only sollution in
long and unnecessary work grounds

if the consensus is that it's not
a bug then so be it, however it
seems I'm not alone in being
tripped up by this side effect.

[p5pRT]

From @pjcj

On Thu, Jan 05, 2012 at 02​:58​:30PM -0800, Robert Colvin via RT wrote​:

On Thu Jan 05 08​:18​:43 2012,
zefram@​fysh.org wrote​:

Robert Colvin via RT wrote​:

But in my opinion if the
default behaviour is to create
the variable
reference when I test for its
existence, that is a bug.

It's not a bug. The behaviour
is perfectly consistent across
operators.
"exists" does not create the
hash element that you ask it
about.
The autovivification applies to
the *hash* in which you looked.

-zefram

indeed it may be consistent,
though it is ultimately
incorrect! in testing existence
the state of variables are
modified. exists should be a safe
operation and it is therefore
reasonable to expect that it
won't make any modifications

From perldoc -f exists​:

  This surprising autovivification in what does not at first--or even
  second--glance appear to be an lvalue context may be fixed in a future
  release.

--
Paul Johnson - paul@​pjcj.net
http​://www.pjcj.net

[p5pRT]

From @demerphq

On 5 January 2012 23​:19, Eric Brine <ikegami@​adaelis.com> wrote​:

On Thu, Jan 5, 2012 at 9​:43 AM, David Mertens <dcmertens.perl@​gmail.com>
wrote​:

If it is such a major complaint, how might it go about getting changed?
I'm new, so forgive me if this is a faux pas, but it would appear that the
obvious solution would be to pull the autovivification module into core.
After all, the module addresses the OP's concern exactly, it does not break
normal autovivification, and it has a 100% pass rate.

So, what does it take to get something like this into core?

The current (well reasoned) desire is to *remove* modules from the perl
distribution, or at least avoid adding any new modules.

If the (unthinking) position of "if we can remove it from the core
then it should be removed" is current policy I would appreciate a
pointer to where it says so.

The reason to do something should never be because we *can*, it should
be because it is *right*.

You're saying this module works perfectly well without being bundled into
the perl distribution, so there's no need to include it.

I consider this module to be in the same category as Scalar​::Util and
as such I personally feel it should be in the core.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"