Free variables and binding

[fwaaade]

Issue type:

• ➕ Feature request

---

Short description:

Free variables and binding.

A new operator called "Free variable" that takes no parameters (type "Any"). It can be used as the input of other operators. Then there is another operator called "Bind" which takes a free variable and anything (type "Any -> Any -> (Any -> Any)"). This will return an operator that will replace all occurences of the free variable in the second input to the value given.

Currently, to make predictates like Item->Boolean, you have to use Operator and Pipe. But with a free variable, you just use the operators directly with a free variable and bind it to the input at the end. It's also much easier to work with flipped parameters since the free variable can go on any side of an operator.

[rubensworks]

Thanks for the suggestion!

[Archengius]

Hello, new user here, first time playing with Integrated Dynamics on FTB Oceanblock 2 and extensively using it for automation and as a logistic network. From the new user perspective, in my opinion this feature is very important and, if possible, should be prioritized, because ID is one of the most interesting and flexible mods I have ever played with.

I think using Operators and Pipe is fine to make predicates, and it is more or less expected given the core design of the mod, and is very straightforward to understand and work with for people familiar with functional programming. However, what is not fine, and is in fact extremely frustrating and confusing to work with right now, is writing generic ID programs that can be reused.

Suppose I have a following ID graph:

Export Item Slot(Integer) = Get or Default(Filter(Pipe(Apply(Operator(List Get), Aspect(Items, 395)), Apply(Operator(Relational Equals), Item(None))), List(Integer(0), Integer(1) ... Integer(11))), Integer(0), Integer(-2))

And I want to make it reusable by rewriting it so that Aspect is passed as an argument in some way as opposed to being hard-coded in the middle of the graph.

Right now the only 2 ways to do this is to either manually copy the graph and replace the Aspect reference to another aspect of your choice (which is tedious and is extremely poor user experience), or to rewrite the graph in such a way that Aspect can be passed as the argument to it with Apply to construct the final program.

The latter requires taking existing functional patterns, like Pipe or Operator/Apply, and adding one more level of indirection to them. This significantly increases the complexity and verbosity of the graph, and also makes it extremely frustrating to debug any issues (and when working with two or more levels of indirection, mistakes are bound to happen), because you will also get extremely unhelpful error messages such as "Expected Item, got Operator", rather than an actual problem with the graph. Additionally, working with multiple levels of indirection is frustrating because the mod does not provide tools that are flexible enough for this (so you have to resort to verbose and extremely error prone constructs like Apply(Flip(Apply N), List(...)), which are very frustrating to work with by themselves, but even more frustrating to work with when additional indirection is involved.

Compare the graph above with the following rewritten graph to carry the Aspect out (it took me quite some time to come up with this version, compared to the original graph which took less than a couple of minutes, and I was unable to do this in-game because the amount of indirection is too much to keep in your head without writing it down):

Generic Program(List -> Integer) = Pipe(Pipe(Pipe(Apply(Operator(Apply), Operator(List Get)), Apply(Flip(Operator(Pipe)), Apply(Operator(Relational Equals), Item(None)))), Apply(Flip(Operator(Filter)), List(Integer(0), Integer(1) ... Integer(11)))), Apply(Flip(Apply(Flip(Operator(Get Or Default)), Integer(0))), Integer(-2)))

Export Item Slot(Integer) = Apply(Generic Program, Aspect(Items, 395)

I do not think this level of indirection is fun to work with, and like I said troubleshooting graphs like this is extremely frustrating due to the unhelpful error messages and the inability to view the entire program as a single expression tree (which is another big problem with the mod in my opinion).

There are two possible solutions to this problem in my opinion:

1. Allow copying variable graphs: This solves the problem by taking the original graph without indirection, copying it to an isolated graph, and then swapping out the parts that should be replaced in the new graph. This way original program stays the same but can be easily "forked" and modified to be used in another context.

2. What this issue suggests: Introduce Free Variables and Binding, which allows the original graph to remain indirection-free, but allows reusability in a controlled manner by allowing specific parts of the program to be swapped out and replaced. The original graph can then be rewritten like this:

Generic Program(Integer) = Get or Default(Filter(Pipe(Apply(Operator(List Get), Free Variable), Apply(Operator(Relational Equals), Item(None))), List(Integer(0), Integer(1) ... Integer(11))), Integer(0), Integer(-2))

Export Item Slot(Integer) = Bind(Generic Program, Aspect(Items, 395))

Which is much simpler to write, troubleshoot and understand than the graph with explicit redirection from above.

I personally think free variable solution is a better solution long term, but the ability to easily copy variable graphs is also something that should be looked into because it is usable in more contexts than just filling the gap left by the absence of free variables/binding.

However, there is definitely an argument to be made about this feature technically exposing and/or limiting the underlying implementation of variable evaluation (e.g. right now evaluating variable X always results in the same value regardless of the evaluation context, but with the introduction of free variables evaluating variable X can result in different values depending on whenever it's being evaluated in the context of variable Y or Z).

But because free variables are technically a syntactic sugar and do not let you do anything that is not possible to be done with indirection, any variable graph utilizing free variables/bind can technically be rewritten as an indirect variable graph resulting in an operator that once applied returns the original graphs evaluation result. So it should be possible for this feature to be implemented without violating the invariance of variable values over current evaluation context (possibly via introduction of a separate "Template" value type, which is a transitive type given to any variable that directly or indirectly references free variable(s). Then Bind Operator would take a variable of Template value type and instantiate the template using the provided values of free variable(s), resulting in a value of the original templated graph with the free variables substituted with their actual values)

[kirjorjos]

Reading through, 2 things already existing that are similar to (but not quite) your proposed solutions do exist that I would recomend checking out if you haven't.

There are two possible solutions to this problem in my opinion:

1. Allow copying variable graphs: This solves the problem by taking the original graph without indirection, copying it to an isolated graph, and then swapping out the parts that should be replaced in the new graph. This way original program stays the same but can be easily "forked" and modified to be used in another context.

You can materalize a graph to a static independent copy, just note that values from readers will become frozen.

2. What this issue suggests: Introduce Free Variables and Binding, which allows the original graph to remain indirection-free, but allows reusability in a controlled manner by allowing specific parts of the program to be swapped out and replaced. The original graph can then be rewritten like this:

Generic Program(Integer) = Get or Default(Filter(Pipe(Apply(Operator(List Get), Free Variable), Apply(Operator(Relational Equals), Item(None))), List(Integer(0), Integer(1) ... Integer(11))), Integer(0), Integer(-2))

Export Item Slot(Integer) = Bind(Generic Program, Aspect(Items, 395))

Depending on the exact use case, a proxy may be useful here. It allows you to to reference the card in it's inventory which can be eaisly swapped out rather than refercening a card directly.

[Archengius]

Thanks for the feedback! Both features are indeed useful, and when combined could possibly solve the problem, but unfortunately due to the fact that Materializer performs "deep copy" (Proxy variable values are also materialized) rather than a shallow copy (proxy variables are replaced with their values, but their values are not materialized), proxies cannot be used to solve the example I have provided. Maybe a third proposed solution would be to introduce Materialization modes: Shallow and Deep. Default would stay Deep and work like it currently does, but Shallow would only Materialize immediate values and will not recurse into Proxy Variables.