[RFC] Drop `|raw` in favor of a more secure approach

[Toflar]

I would like to propose deprecating (and in future drop) the use of |raw which is quite a radical change of the current concept, I'm very much aware of that 😊 But bear with me for a second here - I named it RFC for a reason 😉

Here's why I think the concept of |raw in templates is fundamentally flawed.
If you take a step back and think of what {{ variable|raw }} does, the first thought that comes to your mind because of how the filter is named is probably:

Hello Twig, output variable exactly as it is, I know it is safe here.

And this thought is exactly what I would like to challenge.
Because in projects where you write the controllers and you control the templates. you know exactly what input you have, where you pass the variables, in what context etc. - you are in full control.
Hence, this makes you think (!) that {{ variable|raw }} is safe and you can ouput it |raw.
But can you?

Here's the thing: I think, this assumption is wrong.
Because very often, the ones that write the templates and the ones that use them (aka write the controllers), are not the same people.

Imagine you have template designer Alice that designs a table.html.twig:

<table>
    {% for row in rows %}
        <tr>
            {% for cell in row %}
                <td>{{ cell }}</td>
            {% endfor %}
        </tr>
    {% endfor %}
</table>

That's great, developer Bob can now use this template when writing his controller:

$rows = [
    ['Cell 1', 'Cell 2', 'Cell 3'],
    ['Cell 4', 'Cell 5', 'Cell 6'],
];

echo $twig->render('table.html.twig', [
    'rows' => $rows,
]);

Nice! Now two weeks later, developer Eve also needs to output content formatted as a table, in a completely different context. She needs to output user input in one of the cells, though:

$rows = [
    ['Cell 1', 'Cell 2', 'Cell 3'],
    ['Cell 4', 'Cell 5', $request->query->get('user-input')],
];

echo $twig->render('table.html.twig', [
    'rows' => $rows,
]);

Well, no problem. She checks the template, sees that it contains <td>{{ cell }}</td> and knows that Twig's output encoding is going to take care of that properly. All good!

Now another two weeks later, developer Bob comes back to his controller and his task is to wrap Cell 1 into a <strong>Cell 1</strong>.
So he asks template designer Alice to adjust the template to allow for that.
Now what does template designer Alice do?
If we are lucky, she knows (or can see from the source code, if the project lives in the same git repository) that developer Eve passes untrusted input and so she'll go for

<table>
    {% for row in rows %}
        <tr>
            {% for cell in row %}
                <td>{{ cell|sanitize_html }}</td>
            {% endfor %}
        </tr>
    {% endfor %}
</table>

Or she implements some special logic where one can say which cells should be wrapped in <strong></strong> (but then again, what if tomorrow someone needs another formatting option?).

Phew! That was close! But what if she did not know this template was used elsewhere? What if there's a general repository for some company CI/CD and re-usable Twig templates used in all sorts of projects?
Would she go for

<table>
    {% for row in rows %}
        <tr>
            {% for cell in row %}
                <td>{{ cell|raw }}</td>
            {% endfor %}
        </tr>
    {% endfor %}
</table>

and thus introduce a security vulnerability without knowing it? 💥

And here's why I think that

Hello Twig, output variable exactly as it is, I know it is safe here.

is always the wrong thing to say. You never want to allow |raw values which essentially translates to "whatever format".
What you actually want to say in this case is

Hello Twig, if variable is safe for the current escaping strategy (so html because derived from table.html.twig), you can output it raw. If not, please apply output encoding for current escaping strategy.

The template designer never has this information. So in my opinion, it's inherently wrong to ask them to add |raw.
As the template designer, what you want is: {{ variable|rawIfEscapingStrategiesMatch }} 😊
But you always want that, so might as well drop the filter alltogether 😉

Imho, we should invert the logic and move this responsibility to the one using the template (aka the controller).

Imagine something along the lines of this (I know a lot of it is missing, it's just for illustration purposes):

class TemplateVariable
{
    public function setContentForEscapingStrategy(string $content, string $strategy): self
    public function getContentForEscapingStrategy(string $strategy): mixed
}

Now there's no more |raw so our template designer Alice doesn't have to worry. We're back to

<table>
    {% for row in rows %}
        <tr>
            {% for cell in row %}
                <td>{{ cell }}</td>
            {% endfor %}
        </tr>
    {% endfor %}
</table>

Developer Bob can still do this in the simplest case:

$rows = [
    ['Cell 1', 'Cell 2', 'Cell 3'],
    ['Cell 4', 'Cell 5', 'Cell 6'],
];

echo $twig->render('table.html.twig', [
    'rows' => $rows,
]);

For his <strong> case, he now needs to do this:

$cell1Content = new TemplateVariable();
$cell1Content->setContentForEscapingStrategy('<strong>Cell 1</strong>', 'html'); // Bob knows this is safe, it's HIS responsibility!

$rows = [
    [$cell1Content, 'Cell 2', 'Cell 3'],
    ['Cell 4', 'Cell 5', 'Cell 6'],
];

echo $twig->render('table.html.twig', [
    'rows' => $rows,
]);

Twig would now check if {{ cell }} represents an instance of TemplateVariable and fetch the content for the current escaping strategy which is html. If it's configured like in this case, it can be output as is, because the strategies match. If it's not, then the escaping strategy needs to be applied just as usual.

And what would Eve do to use that same template? Well:

$cell6Content = new TemplateVariable();
$cell6Content->setContentForEscapingStrategy($this->sanitizer->sanitize($request->query->get('user-input')), 'html'); // Eve knows this is safe now

$rows = [
    ['Cell 1', 'Cell 2', 'Cell 3'],
    ['Cell 4', 'Cell 5', $cell6Content]
];

echo $twig->render('table.html.twig', [
    'rows' => $rows
]);

You see, the responsibility to tell Twig which variables are safe for what strategy, must not be the template designer's. Using |raw effectively disables all output encoding which is never what you want as a template designer. You don't want to allow "everything". You want to allow "everything that is safe".
So all you should care about as a template designer is to write your templates with the correct escaping strategies in mind:

{# Ensure this template is called `button_with_js.html.twig` because we want `html` to be the default escaping strategy for this template #}
<div>
    <p>{{ value }}</p>
    <button title="{{ value|e('html_attr') }}">Click</button>
    <script>
        const msg = "{{ value|e('js') }}";
        console.log(msg);
    </script>
</div>

And here we go. This template can now be used by any developer in that project safely.
In its simplest form, you can just use it like you do it today. Pass the value as a string:

echo $twig->render('button_with_js.html.twig', [
    'value' => 'I am not dangerous',
]);

And if you want value to contain Bob "The Builder" <script>doSomethingSafe()</script>, because you know that it's safe for e.g. html, you have to take care of things yourself:

$value = <<<'TXT'
Bob "The Builder" <script>doSomethingSafe()</script>
TXT;
$variable = new TemplateVariable();
$variable->setContentForEscapingStrategy($value, 'html'); // I know this is safe for "html", I want the script to be executed - for all the other strategies, escape normally.

echo $twig->render('button_with_js.html.twig', [
    'value' => $variable,
]);

Not sure if this was presented logically and the examples make sense 😄
But I think it's good enough to make my point.

• Currently |raw outputs the contents always raw no matter the escaping strategy.
• I think this is never what you want. You want to always output things safely no matter the strategy.
• Template designers should not think about the contents of variables. They should think about the context they are outputting those and apply the correct escaping filters there (so nothing new for them actually).
• Developers that make use of those templates need a way to mark a variable as being safe for a certain strategy
• This shifts responsibilities to where they belong

[stof]

This proposal is similar to \Twig\Markup, except that it associates an info about which strategy should be considering the value as safe (currently, \Twig\Markup is considered safe for any strategy).

[nicolas-grekas]

No need to deprecate raw: we can very well say that both sides should express that rawness: only vars labelled as safe could be used with raw.
About deprecating eventually, dunno if that's required. We could also make this opt-in. We could even add a flag to trigger a deprecation when a template or a controller don't agree on the safety so help template/controller authors to move to the stricter mode.

[Toflar]

This proposal is similar to \Twig\Markup, except that it associates an info about which strategy should be considering the value as safe (currently, \Twig\Markup is considered safe for any strategy).

That's correct. Essentially |raw currently equals \Twig\Markup. You can also already mark certain classes as safe using the addSafeClass() on the EscaperRuntime - but not for certain strategies dynamically only.

No need to deprecate raw: we can very well say that both sides should express that rawness: only vars labelled as safe could be used with raw.

In other words: |raw would not mean "raw" but "raw if marked safe for the current strategy".
We could but then again, I think from a template designer's point of view it should just never be needed, imho.
Also, it would be super easy for tools like the Twig CS fixer to prohibit the use of |raw if it did not have 2 different behaviors depending on some configuration value.

About deprecating eventually, dunno if that's required. We could also make this opt-in. We could even add a flag to trigger a deprecation when a template or a controller don't agree on the safety so help template/controller authors to move to the stricter mode.

I totally agree. I think this is an implementation detail. How the classes are named/how you pass variables marked as safe for certian strategies and whether or not to deprecate |raw or make it opt-in or what not is secondary.

I think what should be done in the first place is discussing whether that idea/concept/approach even makes sense or if there are conceptual issues with my thoughts (which I'm definitely not ruling out) 😉

[stof]

raw if marked safe for the current strategy

you don't need a dedicated filter for that. If it is marked safe, the auto-escaping will not escape the value again, without any filter.

The raw filter has only 1 effect: marking a value as safe for auto-escaping (without actually changing the value).
If you don't want to allow marking arbitrary values as safe, you can forbid that filter in your linter.

[stof]

Also, it would be super easy for tools like the Twig CS fixer to prohibit the use of |raw if it did not have 2 different behaviors depending on some configuration value.

it does not have 2 different behaviors.

[Toflar]

I know that I can do that. I'm questioning today's state of things and whether it should be changed for better security/usability for everyone.

it does not have 2 different behaviors

I didn't say it has. It would have if a configuration that influences its behavior was introduced as suggested by Nicolas.

[stof]

There is no configuration to be introduced.
Making raw usable only on values that are already safe is totally useless, as that's the same than removing the filter.

[nicolas-grekas]

Making raw usable only on values that are already safe is totally useless, as that's the same than removing the filter.

Well that's a bit direct, and wrong, if I may 😅
That's not the same because it is visible when reading a template: this var contains markup. That is important to me.

[aschempp]

That's not the same because it is visible when reading a template: this var contains markup. That is important to me.

But isn't the point here that you – as a template developer – might not know whether a variable contains markup or not?

[nicolas-grekas]

As a template dev, you have to know which var exists for what, and which ones will output markups, otherwise you won't be able to know what to do with them, that's not new and we can build on that IMHO.

[Toflar]

I appreciate the passionate comments so far. After all, it's Twig. It's been around for years and we've all come to love it for what it is so questioning established practices always meets resistance to some degree. But hey, I did an RFC, I requested those comments, so all good! 😊

I do think, that we can all agree on one thing, though: Twig is meant to make things safe(r) out of the box by applying output encoding by default compared to using just plain old <?= $variable => in PHP directly.

What I am proposing is indeed to drop the raw filter because I'm just convinced that a template designer never wants to accept "raw" values but instead "safe" values. I just don't see a use case where I, as the template designer, ever want to output anything just "raw". What I want is not "raw". What I want is "safe"!

I think my button_with_js.html.twig example is the perfect illustration for that:

<div>
    <p>{{ value }}</p>
    <button title="{{ value|e('html_attr') }}">Click</button>
    <script>
        const msg = "{{ value|e('js') }}";
        console.log(msg);
    </script>
</div>

Template designers already have to think about how and where to encode the variables correctly. It's what we've been doing forever. It's the current status quo.
When you want to use a variable within JavaScript, you already had to encode it properly using |e('js'). And I think this is perfect. Because only the template designer knows in what context the variable is used. So they have to give Twig the information on how to encode this variable properly.

Which I guess is exactly what @nicolas-grekas meant with

As a template dev, you have to know which var exists for what, and which ones will output markups, otherwise you won't be able to know what to do with them, that's not new and we can build on that IMHO.

If you did, then I think we're on the same page! As the template dev, I have to give Twig the information - and now I am asking myself, when would I ever use raw then?

<p>{{ value|raw }}</p>

To me, raw is not an encoding hint to Twig. It's "hello Twig, just don't touch this variable at all!".
And I'm just not sure there's a use case for that. Because - at least in this example - what I want is not raw content. I want to output the contents of value safely encoded for html. Always.
So in fact, I would have to write

<p>{{ value|e('html') }}</p>

because I know this is within the context of html and whatever value contains - I don't want to output it raw but properly escaped for html. Just like I do for js, css, html_attr etc.
The only reason we're not having to write <p>{{ value|e('html') }}</p> over and over again is because every template comes with a default output encoding strategy for convenience. Which is - always has been - and always will be a brilliant decision. It makes writing templates easy and safe by default.

My humble assumption as to why |raw exists and we're using it, is not because we need that filter to exist. But because we're lacking a way to inform Twig about the fact that a given variable is already safe for a given context.
Today, we mainly pass on scalars. We just do

echo $twig->render('button_with_js.html.twig', [
    'value' => 'string or integer or float',
]);

If we had the option to do

$variable = new TemplateVariable('**this is my default variable representation**');
$variable->setContentForEscapingStrategy('<strong>this is my variable representation for html which is save</strong>', 'html');
// etc.

echo $twig->render('button_with_js.html.twig', [
    'value' => $variable,
]);

then I don't see why we would need the raw filter anymore. By default, things are just always encoded using either the default strategy or if hinted using |e('whatever-strategy'), the whatever-strategy strategy. And if I want to prevent that Twig encodes my variable (aka double encoding), I have to tell Twig "hey, this variable is already safe for whatever-strategy".

Just one more thing: My intention is not to break anybody's workflow that's been established over the course of the last decade or more! I don't want to take away your precious raw filter!
If there are use cases and thus good reasons to keep |raw then we should do that. No doubt.
The reason why I created this issue and am now investing a lot of time in trying to explain my thoughts is simple: My intention is to make Twig even more secure by default if we can. And I think we can - it just requires questioning the status quo sometimes 😊

[stof]

To me, what we need first is to introduce the ability to provide a markup object (either directly in Twig\Markup or in a separate class depending on BC concerns) that knows for which strategy that markup was produced (instead of considering it safe for all of them).
This would solve your use case (currently, using a Twig\Markup would not escape it in value|e('js') due to the fact that it is considered safe for all strategies).
I don't think we should build an object that contains multiple representation of the markup, as this adds lots of complexity (if you pass the value to some functions or filters, which representation should be used to cast it to string ?). Your example seems to use some markdown string with an HTML representation. A better solution for that IMO is to use the markdown filter in the template to produce the HTML representation (and that filter is already integrated with auto-escaping rules)

Removing raw entirely from Twig is a separate topic IMO. Projects wanting the safer usage of Twig can already ban the usage of this filter in a linter.

[stof]

That's not the same because it is visible when reading a template: this var contains markup. That is important to me.

|raw does not indicate "this variable contains markup". It indicates "I want to allow markup in this variable without any safety check" (could also be described in the more scary way "disable safety checks on that variable"). You can have variables that contain safe markup that will be rendered fine without raw.

[aschempp]

[…] introduce the ability to provide a markup object (either directly in Twig\Markup or in a separate class depending on BC concerns) that knows for which strategy that markup was produced (instead of considering it safe for all of them). This would solve your use case (currently, using a Twig\Markup would not escape it in value|e('js') due to the fact that it is considered safe for all strategies).

I think that's exactly what is being proposed here. And if that "object" exists, and everyone uses it correctly (that's a very wide assumption 😆) then raw would no longer be necessary. Because |raw is only necessary if you basically want to bypass the auto encoding (html in most cases). Keeping |raw in the first place certainly wouldn't harm though (and obviously would be necessary for backwards compatibility for a long time).