Syntax for raw identifiers

[L0r3m1p5um]

There's a problem that sometimes comes up with recursive references to record fields, which is that a field name can be defined to a value that is not a valid identifier in other contexts. That can make it impossible to reference some record fields recursively. For example, a record field with the same name as a keyword can't be referenced:

nickel> { priority = "high", x = priority }
error: unexpected token
  ┌─ <repl-input-3>:1:26
  │
1 │ { priority = "high", x = priority }
  │                          ^^^^^^^^

Another example is field names with a period:

nickel> { "some.property" = 1, x = some.property }
error: unbound identifier `some`
  ┌─ <repl-input-4>:1:28
  │
1 │ { "some.property" = 1, x = some.property }
  │                            ^^^^ this identifier is unbound

One solution that I think was discussed in the discord a while back is to have a way to create raw identifiers, like id%"some.property"%.

[L0r3m1p5um]

A few questions I've been thinking about with respect to this:

1. Should there be any limits to what an identifier created in this way can be, or should any arbitrary string be allowed?
2. Is there any context where an identifier is normally allowed where we would not want to allow a raw identifier?
3. Can/should string interpolation be allowed in a raw identifier?

My answers would be

1. I think it's fine to allow any string. As best as I can tell there's not any limit on what sorts of strings can be used as record field names, so I think we'd want to allow any string as a raw identifier too.
2. I can't think of a reason to limit the contexts in which raw identifier is used. Implementation-wise, it would be easier not to limit it, because then I think this can just be implemented as a lexer change.
3. On one hand, there's a use case for allowing string interpolation, since string interpolation is allowed in record field names. So it would make sense to want to do something like { "%{x}" = 1, foo = id%"%{x}"% }. On the other hand it would certainly complicate the types of analyses that can be done statically, since it wouldn't in possible to know in general if an identifier is even bound until runtime. I'm not sure if it'd even be possible to allow string interpolation, and if it is then it's probably more complicated than it's worth. If we did have it then it would also affect question 2 since there are contexts where we definitely wouldn't want to allow string interpolation, like in type annotations.

[L0r3m1p5um]

I guess if we allow any string we'd have to be careful about what sort of output nickel eval and similar thing put out since it could be pretty confusing if rendered badly. And I guess another idea for solving the problem would be a self type keyword that's an identifier that's always bound to the top level record. That would have the nice property that it would handle dynamic record fields just fine, since { "%{x}" = 1, foo = self."%{x}" } is already a valid record path syntax. It would have the downside that adding a keyword is going to be a breaking change though. I haven't thought through it much so there might be other problems with it but I think it's worth considering.

[yannham]

There's a hack currently in the stdlib that makes use of the fact that some identifiers aren't representable: the entries of the internals module are made available in the environment with leading $ like $dyn for example, which can't be referred to or overridden with simple variables. Although this isn't true with recursive record, so in fact I wonder if you could do strange things in a record by re-defining $dyn... In any case, this is a hack, and shouldn't be a blocker; on the contrary, maybe it's a good opportunity to get rid of that hack and use proper generated identifiers or something like that.

On one hand, there's a use case for allowing string interpolation, since string interpolation is allowed in record field names. So it would make sense to want to do something like { "%{x}" = 1, foo = id%"%{x}"% }

Note that currently you cannot refer to a dynamically defined identifier recursively in a record. That is, let x = "foo" ++ "bar" in {"%{x}" = 1, baz = foobar + 1} will give you an unbound identifier. Identifiers are statically, lexically scoped. {"%{x}" = <val>, ..} is actually a shorthand for record.insert x val {..} more than a proper identifier definition.

I would ban any dynamic interpolation here. Raw identifiers should just be surface syntax detail.

I guess if we allow any string we'd have to be careful about what sort of output nickel eval and similar thing put out since it could be pretty confusing if rendered badly

Sure, but this is fine. We already need to be careful when printing strings, enum tags that need to be quoted, multiline strings with indentation, etc.

And I guess another idea for solving the problem would be a self type keyword that's an identifier that's always bound to the top level record. That would have the nice property that it would handle dynamic record fields just fine, since { "%{x}" = 1, foo = self."%{x}" } is already a valid record path syntax. It would have the downside that adding a keyword is going to be a breaking change though. I haven't thought through it much so there might be other problems with it but I think it's worth considering.

This is an idea that's been floating around since long ago (it is mentioned in RFC001), and which is used in Jsonnet for example. In the case of Nickel, it would require non trivial changes to the runtime if we were to make self and actual record/object that can be passed around; this is just not how recursive records are currently implemented. We could make it limited so that only the form self.foo is allowed. It would simply be a different, less ambiguous syntax for recursive occurrences of fields. But I'm personally against those kind of things, because it's surprising and frustrating when you try to pass self to a function to just get some weird errors "yeah, actually self looks like an identifier, but for reasons you don't really care about, it's not, and you can't do that". Another concern with self is: what do you call the "top-level record" if you have nested records? Usually the answer is rather that self is bound to the inner-most record (the closest to the current scope). But then you might want a super, or super.super, etc. One stance could be to rather just provide self, and let people use an explicit let binding if they need to make it go through inner records, but this can prove annoying.

All in all, I think the self part and raw identifiers are somehow orthogonal (not entirely, because with self arguably you shouldn't really need raw identifiers). At least they're not exclusive, so it's possible to have raw identifiers first and think about self in a longer term effort.

[L0r3m1p5um]

We could make it limited so that only the form self.foo is allowed. It would simply be a different, less ambiguous syntax for recursive occurrences of fields. But I'm personally against those kind of things, because it's surprising and frustrating when you try to pass self to a function to just get some weird errors "yeah, actually self looks like an identifier, but for reasons you don't really care about, it's not, and you can't do that".

I agree with this. I think having self would only be an advantage over raw identifiers if it allowed you do things that wouldn't be possible with raw identifiers alone, like handling optional fields or dynamic field names. I think doing anything that raw identifiers can't do would require it to be valid as a function argument though, for example being able to do std.record.has_field "something" self.

All in all, I think the self part and raw identifiers are somehow orthogonal (not entirely, because with self arguably you shouldn't really need raw identifiers). At least they're not exclusive, so it's possible to have raw identifiers first and think about self in a longer term effort.

Yeah, I was bringing it up here because I can't think of any use cases for raw identifiers apart from the ones also solved by self. So if self were able to solve more problems without too much additional work, it's better to avoid implementing both and having redundant features in the language. Since it sounds like implementing self would be much more involved though, I think it makes sense to go ahead with adding raw identifiers. I think it would solve most of the problems I have in practice.

[yannham]

To be a bit more specific, the advantage of the current model is that it gives more information about dependencies. An easy syntactical analysis gives you an idea of which fields depend on which; for example in {foo = 1, bar = 2, baz = some_fun foo}, you know that baz doesn't depend on bar. This makes recursive overriding potentially do less work (we don't do it currently actually, but could determine that if only bar is overriden by a merge, you don't need to update baz), and also allow to free more memory: if you do let x = {foo = 1, bar = 2, baz = some_fun foo}.baz, you don't need to keep bar around. Now, self doesn't have this kind of information: a field holding onto a bare self says basically "I might be using any of the field at any point".

That being said, we could be at least be smart enough to detect fields that only use self.foo form, which would have the same information as before. At runtime, I don't think the change are very complicated: basically we would just need a specific register (a field in the VM state) for self that is properly affected during record operations. But don't mark my words.