Assignment as parameter value

[danielshahaf]

First case: (#670.1)

% foobar='x=$(ls)'
% $foobar

highlights the ba in green from ls, as foreseen in #572 (comment).

Second case: (#670.2)

Exactly the same with a shorter parameter name:

% y='x=$(ls)'
% $y
zsh-syntax-highlighting: BUG: _zsh_highlight_highlighter_main_paint: start(2) >= end(2)

[danielshahaf]

Any relation to the following? —

zsh-syntax-highlighting/highlighters/main/test-data/alias-command-substitution.zsh

Lines 30 to 37 in 521fedf

	# Alias must be at least 4 characters to test the regression
	# cf. 139ea2b189819c43cc251825981c116959b15cc3
	alias foobar='echo "$(echo foobar)"'
	BUFFER='foobar'
	expected_region_highlight=(
	"1 6 alias" # foobar
	)

[danielshahaf]

Added to changelog as a known issue.

Analysis:

$arg is assigned to but start_pos/end_pos are unchanged, so highlighting is usually correct. The highlighting would be incorrect when $arg expands to something that equals a token, a reserved word, etc.. That's what happens with assignments: the = is taken to be syntactically significant even though it's not. That causes the code that recognizes assignments in command position to kick in, and that code assumes that $arg[i] is okay to be highlighted.

[danielshahaf]

@phy1729 Have you made progress on reviewing #675, master...danielshahaf:parameters-multi-word-machinery-v2, or the kludge I proposed on IRC (also pasted below)?

From 7c04246ec8e94e3e340dd4c0c063695e873593af Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <danielsh@apache.org>
Date: Sat, 25 Jan 2020 21:43:18 +0000
Subject: [PATCH 1/2] 'main': Add a kludge for issue #670, concerning command
 position parameter expansions that expand to things that look like
 assignments, other tokens, or reseved words.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

If the tests added in the next (child) commit are run on the previous
(parent) commit, they fail as follows:

    % make quiet-test
    make test QUIET=y
    make[1]: Entering directory '/home/daniel/in/zsh-syntax-highlighting'
    ZSH_PATCHLEVEL=debian/5.7.1-1
    Running test brackets
    Running test main
    # parameter-value-contains-command-position1
    1..2
    ok 1 - [1,7] «$foobar» - # TODO "issue #670"
    not ok 2 - have 1 expectations and 6 region_highlight entries: «expected_region_highlight=( '1 7 assign "issue ♯670"' )» «region_highlight=( '0 7 assign' '2 7 default' '2 7 command-substitution-unquoted' '2 4 command-substitution-delimiter-unquoted' '4 6 command' '6 7 command-substitution-delimiter-unquoted' )»
    zsh-syntax-highlighting: BUG: _zsh_highlight_highlighter_main_paint: start(2) >= end(2)
    Bail out! On './highlighters/main/test-data/parameter-value-contains-command-position2.zsh': output on stderr
    # parameter-value-contains-command-position2
    1..2
    ok 1 - [1,2] «$y» - # TODO "issue #670"
    ok 2 - cardinality check
    Running test pattern
    Running test regexp
    make[1]: *** [Makefile:40: test] Error 2
    make[1]: Leaving directory '/home/daniel/in/zsh-syntax-highlighting'
    make: *** [Makefile:51: quiet-test] Error 2
    zsh: exit 2     make quiet-test
    %

Since they fail with "Bail out!", I've committed the fix (this commit)
fix and the tests afterwards.
---
 highlighters/main/main-highlighter.zsh | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/highlighters/main/main-highlighter.zsh b/highlighters/main/main-highlighter.zsh
index 6af3f9d..6edbfc3 100644
--- a/highlighters/main/main-highlighter.zsh
+++ b/highlighters/main/main-highlighter.zsh
@@ -638,6 +638,11 @@ _zsh_highlight_main_highlighter_highlight_list()
             arg=${(P)MATCH}
             ;;
         esac
+        # Kludge to prevent tokens and reserved words from being matched in the
+        # expansion of the modified arg:
+        arg=${arg//[][#\$^\*()=|{}\`<>\?~,\!\'\";\\-]/_}
+        # Based on ztokens[] in Src/lex.c.
+
         _zsh_highlight_main__type "$arg" 0
         res=$REPLY
       fi
@@ -814,6 +819,17 @@ _zsh_highlight_main_highlighter_highlight_list()
                           local i=$(( arg[(i)=] + 1 ))
                           if [[ $arg[i] == '(' ]]; then
                             in_array_assignment=true
+                          elif (( i > end_pos )); then
+                            # Kludge/hotfix: The parameter expansion assigns to
+                            # $arg, and the assigned value might be longer than
+                            # the original value.  In that case, the 'else' block's
+                            # assumption that $arg[i] is between $start_pos and $end_pos
+                            # would not be true, and a "BUG" message would result (see
+                            # issue #670 case (2)).
+                            # 
+                            # As a hotfix in order to get 0.7.0 released, we
+                            # add this 'elif' block to prevent the 'else' block
+                            # from running when its precondition doesn't hold.
                           else
                             # assignment to a scalar parameter.
                             # (For array assignments, the command doesn't start until the ")" token.)

From 57df14c2631ce768e1b466b522f343227873993d Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <danielsh@apache.org>
Date: Thu, 16 Jan 2020 19:02:39 +0000
Subject: [PATCH 2/2] tests: Add tests for issue #670.

---
 ...meter-value-contains-command-position1.zsh | 38 +++++++++++++++++++
 ...meter-value-contains-command-position2.zsh | 38 +++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100644 highlighters/main/test-data/parameter-value-contains-command-position1.zsh
 create mode 100644 highlighters/main/test-data/parameter-value-contains-command-position2.zsh

diff --git a/highlighters/main/test-data/parameter-value-contains-command-position1.zsh b/highlighters/main/test-data/parameter-value-contains-command-position1.zsh
new file mode 100644
index 0000000..c2029e9
--- /dev/null
+++ b/highlighters/main/test-data/parameter-value-contains-command-position1.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local foobar='x=$(ls)'
+
+BUFFER=$'$foobar'
+
+expected_region_highlight=(
+  # Used to highlight the "ba" as 'command' because the 'ls' showed through
+  '1 7 assign "issue #670"' # $foobar
+)
diff --git a/highlighters/main/test-data/parameter-value-contains-command-position2.zsh b/highlighters/main/test-data/parameter-value-contains-command-position2.zsh
new file mode 100644
index 0000000..024df8c
--- /dev/null
+++ b/highlighters/main/test-data/parameter-value-contains-command-position2.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local y='x=$(ls)'
+
+BUFFER=$'$y'
+
+expected_region_highlight=(
+  # Used to trigger a "BUG" message on stderr
+  '1 2 assign "issue #670"' # $y
+)