Idea: power lambdas

[jgonggrijp]

2023-10-31 update: the newest version of my proposal is here.

2022-10-21 update: the following excerpt from #138 (comment) summarizes how I currently envision power lambdas.

The way I currently think of it, lambdas receive a second argument which somehow (i.e., in an implementation-defined way) makes the following things possible:

• retrieve any context frame;
• retrieve a list of all keys visible in the current (full) context;
• resolve any key against the current (full) context;
• identify the context frame in which a key is resolved;
• render a template of choice against the current context.

Whereby lambdas must not modify the pre-existing contents of the context, and implementations are welcome to actively prevent this if the programming language can enforce it. However, lambdas can (already) push a new frame on the stack, which still has the net effect of changing what's available in the context.

2022-10-16 update: added playground links and savestates.

This is a feature I've meant to implement for a while, and which I was planning to propose to the spec after implementing it. However, it seems relevant to the discussion about dynamic partials that is currently taking place with @anomal00us, @gasche and @bobthecow in #134 and #54, so I decided to describe the idea now already.

The context name resolution algorithm described in the current interpolation, section and inverted section specs states that if any name in a dotted path happens to refer to a function (method), that function is called in order to obtain the next frame from which to resolve the next name in the path (or the final result). In case of a section or inverted section, the function is also passed the contents of the section as a string argument, if it is capable of accepting an argument. Illustration with JavaScript lambda code:

{{#key.lambda.otherKey}}Hello {{name}}{{/key.lambda.otherKey}}

{
    key: {
        lambda(section) {
            // section will be the string 'Hello {{name}}'
            return {
                otherKey: { name: 'John' }
            };
        }
    }
}

Hello John

Try the above example in the playground by pasting the following code:

{"data":{"text":"{\n    key: {\n        lambda(section) {\n            // section will be the string 'Hello {{name}}'\n            return {\n                otherKey: { name: 'John' }\n            };\n        }\n    }\n}"},"templates":[{"name":"","text":"{{#key.lambda.otherKey}}Hello {{name}}{{/key.lambda.otherKey}}"}]}

The lambdas spec adds to this that if the last name in the path is a function and it returns a string, then that string is rendered as a template before finally being substituted for the interpolation or section tag.

{{#key.lambda}}Hello {{name}}{{/key.lambda}}

{
    key: {
        lambda(section) {
            // section will be the string 'Hello {{name}}'
            return 'I changed my mind about the template, {{name}}';
        }
    },
    name: 'John'
}

I changed my mind about the template, John

Try the above example in the playground by pasting the following code:

{"data":{"text":"{\n    key: {\n        lambda(section) {\n            // section will be the string 'Hello {{name}}'\n            return 'I changed my mind about the template, {{name}}';\n        }\n    },\n    name: 'John'\n}\n"},"templates":[{"name":"","text":"{{#key.lambda}}Hello {{name}}{{/key.lambda}}"}]}

I would like to suggest another optional extension spec on top of this, which I'll dub "power lambdas" for now. It amounts to passing an additional argument to functions that can accept it, containing a (read-only) view of the full context stack. I'll illustrate a couple of advanced features that could be implemented using power lambdas below. For now I'll assume that the context stack is passed as the most conventional sequence type of the programming language, with the top of the context stack as the last element. It could also be left to the implementation whether the stack frames are ordered top or bottom first.

Access to properties of lower context stack frames that have been hidden by higher frames:

{{#nested1}}
    {{#nested2}}
        {{name}}
        {{parentScope.name}}
        {{rootScope.name}}
    {{/nested2}}
{{/nested1}}

{
    parentScope(section, stack = section) {
        return stack[stack.length - 2];
    },
    rootScope(section, stack = section) {
        return stack[0];
    },
    name: 'John',
    nested1: {
        name: 'Lizzy',
        nested2: {
            name: 'Deborah'
        }
    }
}

        Deborah
        Lizzy
        John

Hash iteration by key-value pairs like in Handlebars's #each helper:

{{#nested}}{{#byKey}}
    The {{key}} is called {{value}}.
{{/byKey}}{{/nested}}

{
    byKey(section, stack) {
        var context = stack[stack.length - 1];
        var pairs = [];
        for (var key in context) {
            pairs.push({key, value: context[key]});
        }
        return pairs;
    },
    nested: {
        'police officer': 'Jenny',
        'doctor': 'Hildegard',
        'hairdresser': 'Tim'
    }
}

    The police officer is called Jenny.
    The doctor is called Hildegard.
    The hairdresser is called Tim.

Dynamic partials:

{{!template.mustache}}
{{animal}} goes:
    {{#dereference.>}}animal{{/dereference.>}}

{{!cow.mustache}}
Moo!

{{!dog.mustache}}
Woof!

{
    dereference(section, stack) {
        // This example presumes that there is a way to render a template with a prepared
        // template stack instead of just a plain view that will become the root of a new
        // context stack. Offering such an interface will be attractive for implementations
        // that support power lambdas.
        var name = renderMustache('{{' + section + '}}', stack);
        return {
            '>': function() { return '{{>' + name + '}}'; }
            // I'm oversimplifying here, you could do this for all sigils without code
            // duplication.
        };
    },
    animal: 'cow'
}

cow goes:
    Moo!

[bobthecow]

First, I don't think addition additional optional params to your lambda implementation goes against the spec, as long as the base case is compatible. You should just be able to implement this.

Second, specifying exactly what happens inside lambdas, down to the params, feels like it goes against the principles of Mustache. The Mustache way, to me, kind of defers on these sorts of matters to what feels "right" for each implementation's language, if that makes sense?

So mustache.php implements lambdas with a second "lambda helper" parameter, which exposes a method for rendering a template with the current stack as context. I went back and forth with a really similar idea to this proposal, and on what felt like too "logic-y", and came to the conclusion that exposing the context stack just felt wrong. I've considered adding a context lookup method to the helper, though, as that feels cleaner and more Mustachey.

[jgonggrijp]

Thanks for sharing your insight! Yes, makes sense.

Just to be clear, I wasn't worried that my implementation might be incompatible with the spec because of the additional argument. I just wanted to have an implementation to back my proposal.

I can see why enforcing a particular form for the second argument would be too restrictive for implementers, so I agree a hypothetical spec shouldn't do that. On the other hand, two of the three examples I gave above are only possible if the lambda can access each individual stack frame. So in order for an implementation to qualify as supporting power lambdas, I believe it should offer some way to do that. I also like your idea of offering a context lookup helper, and your existing support for rendering against the current stack.

[agentgt]

TLDR; Maybe block parameters could be used as stack resolved parameters?

I'll throw my 2 cents on this one given some experience based on the Java world of mustache and handlebars.

Most people with no experience of mustache expect or want section lambdas (currently 1 arity) to work very similar to handlebars helpers or just normal function calls.

e.g.

{{#some arg1 arg2 ...}}some body{/some}}

The above syntax is way too much logic and obviously breaks regular mustache but in general
what they want is to some part of the context (aka {{.}} and not parents) plus some optional parameters that are usually string as well as finally possibly some body contained within the lambda.

In the mustache spec world and mustache.java you only get the unprocessed body of the lambda. In mustache.java this is Function<String,String>. On the other hand JMustache passes the immediate context object as well as the body which I find far more useful.

The classic use case for this is date formatting and or escaping or possibly creating nontrivial href links. In JMustache you would do it like:

{{#someDate}}
{{dateFormat}}someFormatStringThatCannotBeDynamicInterpolation{{/dateFormat}}
{{/someDate}}

The someDate object gets passed to the dateFormat lambda (as well the format string). However because of how hideous laborious notation wise the above is as well as nonstandard we just ended up building richer models (ie generating the formatted date apriori) or using handlebars.

Consequently in my current implementation of mustache (which has many challenges because it is statically checked and thus the templates/partials/lambdas cannot be generating dynamic templating) I have been toying with the following:

{{dateFormat}}{{$date}}.{{/date}{{$fmt}}"iso"{{/fmt}} rest of the body {{/dateFormat}}

Basically block tags inside lambdas do dotted named based resolution unless they are in quotes. The above is also currently technically legal mustache as allowed block tag location is nebulous. The other option is to allow the block parameters to expand eagerly and leave the body untouched with the expectation that the lambda just deals with a string name value pair.

See in my situation unlike @bobthecow I am generating static code w/o special context stack types. The goal of my mustache implementation is to be typesafe. Thus I don't want to just pass the whole stack as some generic stack object but instead resolve to an actual type.

To make this clear my implementation would at compile time convert the above to a method call:

// the template has to be analyzed at compile time so the return object is the context
@TemplateLambda("The time is {{.}}")  
public String dateFormat(LocalDateTime date, String fmt) {
// return formatted string
}

[jgonggrijp]

Hey @agentgt, nice suggestion. I invited you to join here in #131, but you already found it.

I have been thinking about something very similar to what you're describing, but would suggest different syntax:

{{#dateFormat}}{{#date}}*someDate{{/date}{{#fmt}}iso{{/fmt}} rest of the body {{/dateFormat}}

(Note that the {{dateformat}}{{/dateformat}} outer pair in your original example is not allowed by current syntax, but maybe that was just a typo on your end.)

The trick here is that dateFormat is a lambda that returns a new object with date and fmt methods. That object goes on the top of the stack, so those methods are invoked as lambdas, which allows you to apply the arguments in a curried fashion. The date method would store the value and return nothing, while the fmt method would take the previously stored value and return the formatted result, so it ends up interpolated inside the section.

The *dateFormat names a key in the context, analogous to the recently added dynamic names spec, while iso is taken as a literal string. The interpretation of the *dateFormat notation would be the responsibility of the date method, but that can only be successfully implemented if lambdas can somehow access the context stack. Otherwise, all ingredients are already supported by the current spec.

The above is also currently technically legal mustache as allowed block tag location is nebulous.

Just to clarify, blocks are allowed everywhere. The meaning depends on whether the block is directly nested with a parent tag pair or not. If not directly nested within a parent pair, it marks a part of the template that can be configured from the outside. If it is directly nested within a parent pair, it serves to provide that outside configuration (but is itself also still configurable).

Your challenge of implementing safely typed templates is very interesting. I would love to see your code.

[agentgt]

I have been playing around with jstachio's lambdas which for a variety of reasons has to be a little different than javascript or other dynamic implementations and found some issues with lists.

How jstachio does lambdas is that the contents of the lambda block are used as a template and the lambda returns an object that is effectively push on to the context stack (current context for the inputted template). I think this is vastly superior to the current spec of the lambda returning a template as there are many more things that can go wrong.

If the above idea were put into the spec an easy backward compatibility is if the lambda returns an object and takes two parameters like:

function(input, currentContext){ 
   return {"item": currentContext}; 
}

input is the contents of the lambda block and currentContext is the top of the context stack.

Here is how that looks in JStachio:

    /*
     * { "items" : [ 1, 2, 3], "lambda" : function(input, item) {...no analog in current spec..} }
     */
    static final String template = """
            {{#items}}{{#lambda}}{{item}} is {{stripe}}{{/lambda}}{{/items}}""";

    @JStache(template=template)
    public record Items(List<Integer> items) {
        @JStacheLambda
        public LambdaModel lambda(Integer item) {
            return new LambdaModel(item, item % 2 == 0 ? "even" : "odd");
        }
    }
    /*
     * In jstachio if you return an object it is then pushed on the stack
     * and the contents of the of the lambda block are used as a sort of inline partial.
     * 
     * This is in large part because we cannot handle dynamic templates and also
     * because I think it is the correct usage is as the caller knows how it wants to render
     * things and avoids the whole delimeter nightmare.
     */
    public record LambdaModel(Integer item, String stripe) {}
    
    @Test
    public void testName() throws Exception {
        String expected = "5 is odd";
        String actual = JStachio.render(new Items(List.of(5)));
        assertEquals(expected, actual);
    }

Anyway this works great with one giant problem.... there is now way to get the entire list. That is there is no spec way to pass the entire list.

The above is important because one of the biggest complaints when using mustache is the need to know index information while in a list. Such as first, last, and numeric index albeit that last one is less common. Of course you could redecorate the model and depending on implementation you might even be able to use lambdas as an accumulator to figure out first and maybe index but in general it is a huge annoyance.

So ideally a lambda would provide you with that list information but in my implementation it is not possible as the lambda cannot get the entire list. I will probably add fake fields like:

{{#items.this}}{{#lambda}}{{/lambda}}{{/items.this}} 

This would force the list to be treated as an object instead of a list.

So my question is if you do do: *items using the asterisk notation for context lookup will it get the entire list?

[jgonggrijp]

Hey @agentgt, nice to discuss these matters with you. Indeed, JStachio is quite different from the JavaScript-oriented implementations I have seen. Interesting to see how JStachio.render operates.

As a general remark: please feel absolutely free to not implement parts of the spec that you feel are infeasible or unfitting for your situation, and also to implement behavior that is not (yet) in the spec. @bobthecow put it elegantly a few posts up:

(...) The Mustache way, to me, kind of defers on these sorts of matters to what feels "right" for each implementation's language, if that makes sense?

Some detail remarks:

How jstachio does lambdas is that the contents of the lambda block are used as a template and the lambda returns an object that is effectively push on to the context stack (current context for the inputted template).

Most people do not realize that there are two variants of the specification for lambdas: the obligatory variant in the interpolation and sections modules, and the optional variant in the ~lambdas module. The first, obligatory variant prescribes exactly what you are describing here. So in that sense, JStachio already behaves as specified!

The dynamic template replacement thing is what is specified in the optional ~lambdas module. Wontache implements both behaviors, as illustrated in the following example.

Data

{
    /* lambda pushes a new stack frame */
    today: function(ignored) {
        return {
            day: 20,
            month: 'October',
        };
    },
    /* lambda returns a new template */
    emphasize: function(section) {
        return '<em>' + section + '</em>';
    },
}

Template

It is {{#emphasize}}{{#today}}{{month}} {{day}}{{/today}}{{/emphasize}}.

Output

It is <em>October 20</em>.

Copy the following code to the playground in order to try the above example, as well as a variant with today nested outside instead of inside emphasize (which gives the same result):

{"data":{"text":"{\n    /* lambda pushes a new stack frame */\n    today: function(ignored) {\n        return {\n            day: 20,\n            month: 'October',\n        };\n    },\n    /* lambda returns a new template */\n    emphasize: function(section) {\n        return '<em>' + section + '</em>';\n    },\n}"},"templates":[{"name":"today-inside","text":"It is {{#emphasize}}{{#today}}{{month}} {{day}}{{/today}}{{/emphasize}}."},{"name":"today-outside","text":"It is {{#today}}{{#emphasize}}{{month}} {{day}}{{/emphasize}}{{/today}}."}]}

So these interpretations of lambdas are not necessarily in conflict with each other. However, when they are, you should probably prioritize the stack-pushing variant over the template-replacing variant. You are doing the right thing!

Anyway this works great with one giant problem.... there is now way to get the entire list. That is there is no spec way to pass the entire list.

Right, that's why I'm proposing to enable this, somehow.

(...)

So ideally a lambda would provide you with that list information but in my implementation it is not possible as the lambda cannot get the entire list.

To be clear, is it not possible in your implementation because of the spec, or would it still not be possible if power lambdas made it into the spec?

So my question is if you do do: *items using the asterisk notation for context lookup will it get the entire list?

Could you illustrate this question with a template in which you demonstrate this usage of the *items notation?

[agentgt]

Sorry for the late reply on this @jgonggrijp . I must have missed your questions.

First some update.

JStachio is pretty much 1.0.0 ready. Its documentation is here: https://jstach.io/jstachio/

Here is a glorified example:

 @JStache(template = """
     {{#people}}
     {{message}} {{name}}! You are {{#ageInfo}}{{age}}{{/ageInfo}} years old!
     {{#-last}}
     That is all for now!
     {{/-last}}
     {{/people}}
     """)
 public record HelloWorld(String message, List<Person> people) implements AgeLambdaSupport {}

 public record Person(String name, LocalDate birthday) {}

 public record AgeInfo(long age, String date) {}

 public interface AgeLambdaSupport {
   @JStacheLambda
   default AgeInfo ageInfo(
       Person person) {
     long age = ChronoUnit.YEARS.between(person.birthday(), LocalDate.now());
     String date = person.birthday().format(DateTimeFormatter.ISO_DATE);
     return new AgeInfo(age, date);
   }
 }

The above might be fairly confusing if you are not familiar with Java but the things prefixed with @ are annotations and essentially immutable and static but are visible to compiler plugins. JStachio is essentially a compiler plugin.

So hopefully you can now see how in JStachio unlike almost all other implementations templates cannot be dynamically created. Consequently the lambda either returns an object which will be pushed on to the stack and then the lambda section body is used kind of like a partial or it returns a raw string. Either way it does have access to the head of the context stack (as well as the raw body).

A current real world limitation that JStachio does have is that it cannot render the section body and then decorate like say wrapping it with <em> tags. I'm exploring some ways to deal with that issue.

To be clear, is it not possible in your implementation because of the spec, or would it still not be possible if power lambdas made it into the spec?

It might be accessible if you get access to the entire stack. e.g. Currently my implementation does not allow lambdas to have whole stack like parameters but I plan on adding that. However it still does not fix the list problem.

Using the above Persons list and let us assume I have a lambda that accepts a list of persons how do I reference the list?

{{#people}}
{{#peopleLambda}}
{{/peopleLambda}}
{{/people}}

Two major problems arise:

1. peopleLambda cannot access people (list of person) in any way if the list is empty.
2. Assuming the list (people) is not empty the top of the stack is now a single person for peopleLambda and peopleLambda will be executed however many people there are.

So in short I cannot see how a lambda can get access to a list of anything without being executed multiple times or not at all. That is probably OK for the most part ignoring the most desired feature request of mustache: index information (which JStachio does support).

Could you illustrate this question with a template in which you demonstrate this usage of the *items notation?

I can't recall how I got that notation but I think it was the idea that lambdas could take parameters in prefix style (ie normal function call) instead of the current stack postfix style.

e.g. postfix currently in regular mustache and I think in your current power lambda in wontache: {{#parameter}}{{#lambda}}body{{/lambda}}{{/parameter}}

what was talked about I think at some point: {{#lambda}}{{*parameter}}{{/lambda}}. I think that is where the asterisk came from.

That is mustache and current lambdas are "cake eat" where as most function call in languages (ignoring fortran) are "eat cake". With the prefix notation we can access the whole list:

{{#peopleLambda}}{{*people}}{{/peopleLambda}}

But is probably not in the spirt of Mustache.

Anyway I'm fine with your current recommendations for power lambda. I think really what is needed is a standard way to access index information and if power lambdas can do that across the board that would be nice but I think it is easier of the spec add some "virtual" keys as optional similar to handlebars (e.g. @index).

[jgonggrijp]

Right, so I was suggesting you don't do

{{#peopleLambda}}{{*people}}{{/peopleLambda}}

but (subtle difference)

{{#peopleLambda}}*people{{/peopleLambda}}

which would enable peopleLambda to access the entire list (people), even if it is empty. I imagine the implementation of peopleLambda to go something like this in JavaScript:

function peopleLambda(sectionText, contextMagic) {
    var contextKey = sectionText.slice(1); // substring 'people' in the above example
    var theList = contextMagic.lookup(contextKey);
    return 'The length of this list is ' + theList.length + '.';
}

In my suggested version, the *people part of the template has no special meaning to the Mustache engine; the lambda itself is doing the interpretation. I'm using the literal part of the template text to pass arguments to the lambda, with an asterisk signifying that the argument should be looked up in the context. For a lambda that should always look up its argument in the contenxt, you could also leave off the asterisk and simplify the implementation further:

{{#peopleLambda}}people{{/peopleLambda}}

function peopleLambda(contextKey, contextMagic) {
    var theList = contextMagic.lookup(contextKey);
    return 'The length of this list is ' + theList.length + '.';
}

However, I imagine either approach might be difficult of even impossible for JStachio. If the notation {{*people}} improves matters for you, then by all means go for it.

I have seen notations like {{*x}} and {{>*x}} elsewhere, but with a different meaning. Especially in #54. The similarity is still close enough that you might be able to justify its reuse here, though.

Edit to add: my compliments on your readable JavaDoc!

[agentgt]

{{#peopleLambda}}*people{{/peopleLambda}}

Ok now I remember. I think this actually would be possible with special hints that the section body has parameters and the parameters have to be in a certain format. The compiler then parses that format. Otherwise runtime reflection and parser could be used but that is against the spirit of jstachio. Let us ignore runtime and dynamically constructing templates for now because obviously if we are given the stack and dynamic renderer we can almost do anything.

I believe that is why I was proposing reusing block notation (or another new sigil) in the lambda sections as parameters (which reminds me I need to test parents calling lambdas with blocks as I'm not sure what now happens in JStachio). The idea being a new notation to pass a template (section body) and parameters by using block notation.

Using the person example let us assume our lambda function looks like (using typescript-like to show types):

function peopleLambda(people: Person[]): DecoratedPerson[] {
}

interface DecoratedPerson {
  person: Person
  index: number;
}

If we choose to support multiple context parameters:

{{#peopleLambda}}
{{$people}}*people{{/people}}
{{#.}}
<em>{{index}} {{person.name}}
{{/.}}
{{/peopleLambda}}

If you can only pick one binding on the stack I suppose the notation could be:

{{#peopleLambda}}
{{*people}}
{{#.}}
<em>{{index}} {{person.name}}
{{/.}}
{{/peopleLambda}}

The idea being the parameters are not rendered and removed from the section body but are looked up in the context stack and passed to the lambda. That is the compiler does the context lookup instead of the lambda author.

That is I rather avoid passing special things like "contextStack" and "renderer" to the lambda.

Anyway the above is rather complicated so I have avoided implementing it. Also because there is a spec format for passing parameters around: handlebars.

I will play around more to see what works and doesn't and let you know my findings. Thanks for the response!