Syntax highlighting - eyes needed

[finanalyst]

@Raku/core
We have changed the syntax highlighter from one developed for the atom editor (atom-perl6-highlighter) to a Raku based one (Rainbow).

• colouration request. Rainbow offers more categories that can be colourised, eg three separate regex categories, separate delimiter categories.
• The colours chosen for the moment are the same as before, but the same colour is used for duplicate categories. Ideally each category should be a different colour (more generally styling).
• Although the styling of each category is done via CSS, categories can be styled using colour, font, font decoration etc. However, that would need a bit more refactoring of the SCSS, eg adding variables for font as well as colour.
• Please offer suggestions. (Values are needed for the light and dark themes separately)
• concrete suggestions can be made using a PR changing the following files:
  • <repo root>/Website/plugins/rainbow/scss/rainbow-light.scss
  • .../scss/rainbow-dark.scss
  • the relevant colour values for the highlighting are assigned to SCSS variables $code-..... and start at about line 15.

Known problems:

• when the RakuDoc source has =for code :allow<C> or other value in 'allow', the :allow option is not respected.
  • I think that only the Pod.rakudoc file still retains ':allow'
  • in the discussions about RakuDoc v2, the allow option is difficult to implement in combination with syntax highlighting, and needs to be handled separately. This will be developed in due course. The problem existed with the previous highlighter too.
• Rainbow does not handle metadata options in RakuDoc blocks correctly as yet. The priority for addressing this flaw is considered low at present.
  • It is not known yet whether this affects any code snippets in the main sources. Help to determine this would be appreciated.

[patrickbkr]

• in the discussions about RakuDoc v2, the allow option is difficult to implement in combination with syntax highlighting, and needs to be handled separately. This will be developed in due course. The problem existed with the previous highlighter too.

Actually I've already done something similar in Rainbow with quote adverbs already (see the Adverbs section here). This shouldn't be too hard to get right.
Can you provide a quick primer how this option is meant to work exactly? (Ideally create a ticket.)

[finanalyst]

Quick smartphone response to Patrick. ``` =for code :allow<B I> my $var = 'filename.conf'.B<IO>.I<slurp>(:close); my $strng = 'This is B<bold> but this is NOT U<underlined>'; Normal para in Rakudoc source. ```

…

On Mon, 11 Nov 2024, 13:10 Patrick Böker, ***@***.***> wrote: - in the discussions about RakuDoc v2, the allow option is difficult to implement in combination with syntax highlighting, and needs to be handled separately. This will be developed in due course. The problem existed with the previous highlighter too. Actually I've already done something similar in Rainbow with quote adverbs already (see the Adverbs section here <https://git.sr.ht/~patrickb/Rainbow/tree/main/item/t/test-data/string.rakumod>). This shouldn't be too hard to get right. Can you provide a quick primer how this option is meant to work exactly? (Ideally create a ticket.) — Reply to this email directly, view it on GitHub <#419 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYZHGOIAJ63GZC7CBKDDT2ACUDJAVCNFSM6AAAAABRO7STZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYGE2DGOJWG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[habere-et-dispertire]

The comment in the last example of the leftward feed operator ( docs.raku.org ) seems peculiar :

my @result
    <== sort()
    <== grep({ /<[PE]>/ })
    <== my @caps            # unlike ==>, there's no need for additional statement
    <== map({ .tc })
    <== <people of earth>;

[habere-et-dispertire]

The last example in Literal Strings: Q ( docs.raku.org ) seems unaware of a unicode quoting character｟ -- FULLWIDTH LEFT WHITE PARENTHESIS :

Q｢｢Delimiters can be repeated/nested if they are adjacent.｣｣
Q｟Quoting with fancy unicode pairs｠

[finanalyst]

@habere-et-dispertire Thanks for the issues.
@patrickbkr Why is a comment contents being treated as Raku code? I thought it was just a string.

[finanalyst]

@habere-et-dispertire Your second question may be due to interpolation??
@lizmat : When an open punctuation is repeated, does this trigger interpolation, in the same way as << triggers interpolation, while < does not?

[habere-et-dispertire]

@habere-et-dispertire Your second question may be due to interpolation??

If I understand correctly, then it is being interpolated when it shouldn't. Note that the character is not being repeated -- it is a single character :

say '｟'.chars
# 1

say '｟'.uniname.say
# FULLWIDTH LEFT WHITE PARENTHESIS

say '｟'.ord.base(16)
# FF5F

Rainbow appears aware of this to the degree that it includes it as an open-close-punct under hex value FF5F.

I'd hazard a guess that we need a case for capital Q ( git.sr.ht ) here :

given $<q-letter> {
    when "q" { $*quote-mode.q = True; }
    when "qq" | "qqw" | "qqww" | "qqx" {
        $*quote-mode.s = True;
        $*quote-mode.a = True;
        $*quote-mode.h = True;
        $*quote-mode.f = True;
        $*quote-mode.c = True;
        $*quote-mode.b = True;
    }
#    ...

[habere-et-dispertire]

At the end of the interpolation basics page, there is a final colon suggestive of a missing example to follow ?

[finanalyst]

@habere-et-dispertire @coke The : appears to be a mistake in the content. I think it is a .

[patrickbkr]

@finanalyst I've now released Rainbow 0.3.0. That improves the RakuDoc support tremendously, including =begin code :allow<B>. There are now also additional tokenizer subs to tokenize other languages (regex, string, rakudoc).

[timo]

Here, a screenshot from this page, near the end of the linked section: https://docs.raku.org/language/containers#Flattening,_items_and_containers

the highlighting of the decont operator used on $x looks odd, this one: $x<>

i think the two angle brackets should at least be the same color. not sure what exactly they should be highlighted as.

[patrickbkr]

On December 25, 2024 10:50:19 PM GMT+01:00, timo ***@***.***> wrote: Here, a screenshot from this page, near the end of the linked section: https://docs.raku.org/language/containers#Flattening,_items_and_containers  the highlighting of the decont operator used on $x looks odd, this one: `$x<>` i think the two angle brackets should at least be the same color. not sure what exactly they should be highlighted as.

That's a Rainbow issue. I'll take it. Cross posted here: <https://todo.sr.ht/~patrickb/Rainbow/4>

[timo]

another post "full of" Rainbow issues i've stumbled upon:

the code a little bit below https://docs.raku.org/language/variables#The_$__variable

for <ab:c d$e fgh ij*> {
    .say if m/<-alpha>/;
}

looks like this:

and generates HTML like this:

<span class="highlite-ROUTINE">say</span> <span class="highlite-STRING_DELIMITER">"</span><span class="highlite-STRING">Looking for strings with non-alphabetic characters...</span><span class="highlite-STRING_DELIMITER">"</span>;
<span class="highlite-KEYWORD">for</span> &lt;ab:c <span class="highlite-ROUTINE">d</span><span class="highlite-NAME_SCALAR">$e</span> fgh ij*<span class="highlite-OPERATOR">&gt;</span> {
    .<span class="highlite-ROUTINE">say</span> <span class="highlite-KEYWORD">if</span> <span class="highlite-REGEX_DELIMITER">m/</span><span class="highlite-REGEX_SPECIAL">&lt;-alpha&gt;/;
}</span>

i see three oddities here:

• the < > array of string syntax is a bit of text, then d$e is a routine call followed by a scalar, and fgh ij* are also text again. It should be some kind of literal instead. the > is also highlighted as an operator.
• the { as well as the . in the next line probably should be something rather than regular text, but maybe that's just because the stuff before it was mislexed
• the highlighting of <-alpha> as "REGEX_SPECIAL" extends to the closing / of the regex, the ; of the line and also the next line's }.

Here's a little bit of code from further down the page after https://docs.raku.org/language/variables#The_$/_variable

'abc 12' ~~ /\w+/;  # sets $/ to a Match object
say $/.Str;         # OUTPUT: «abc␤»

is highlighted like this:

<span class="highlite-STRING_DELIMITER">'</span><span class="highlite-STRING">abc 12</span><span class="highlite-STRING_DELIMITER">'</span> <span class="highlite-OPERATOR">~~</span> <span class="highlite-OPERATOR">/</span>\<span class="highlite-ROUTINE">w</span>+<span class="highlite-OPERATOR">/</span>;  <span class="highlite-COMMENT"># sets $/ to a Match object
</span><span class="highlite-ROUTINE">say</span> $<span class="highlite-OPERATOR">/</span>.<span class="highlite-TYPE">Str</span>;         <span class="highlite-COMMENT"># OUTPUT: «abc␤»</span>

And looks like this:

The oddities are:

• the opening / of the regex is an OPERATOR, the \ after it is unhighlighted text, the w is ROUTINE, the + is text, the / after it is OPERATOR again, and then the ; that ends the line is text.
• In the following line, $/ is not recognized as a variable. instead, the $ is just text and the / is an operator. Probably because of that, the . of the method call is just text, the Str is "TYPE" instead of something like METHOD or ROUTINE, and the ; is just text.

Here in the code after https://docs.raku.org/language/variables#Named_attributes

'I... see?' ~~ / \w+ $<punctuation>=[ <-[\w\s]>+ ] \s* $<final-word> = [ \w+ . ] /;
say $/<punctuation>; # OUTPUT: «｢....｣␤»
say $/<final-word>;  # OUTPUT: «｢see?｣␤»

it looks like the <-[\w\s]> is not interpreted as a negative character class. instead, the s] is highlighted as REGEX_DELIMITER, as if it was starting an s[...] = ... or s/.../.../, and the ] a few characters later is treated as REGEX_DELIMITER again as if it was closing the same. The rest of the code block is then treated as regex insides.

In the code

say $*RAKU.compiler.version; # OUTPUT: «v2020.01␤»

for some reason the compiler in the middle is interpreted as TYPE. Not sure what that's about? Is it case insensitive and matching the type Compiler, as in Perl6::Compiler, HLL::Compiler, or Raku::Compiler?

In the code after https://docs.raku.org/language/variables#The_$__variable

$_++ for @numbers;  # increment all elements of @numbers

we get the $_ as NAME_SCALAR, the first + as unhighlighted text, the second + as OPERATOR

On the rest of the page there's a few recurring issues:

• $/ is often (always?) misrecognized / ignored
• the $<blah> syntax seems not supported?
• %h<oranges> and %h.finalize have %h as NAME_HASH, but the postcircumfix and method calls are not highlite-d?

And finally I think I have an issue that's more related to the original idea of this github issue:

REGEX_LITERAL and COMMENT have the same color, at least in rainbow-dark.css

That's all I've seen on this page so far :)