T-REPLACE-RX doesn't work as expected when escaping curly braces

[maddes-b]

Have you searched for an existing issue?

• Yes, I tried searching and reviewed the pinned issues

Brief Summary

Update: Refined in this comment

When using T-REPLACE-RX for empty placeholders and/or in a nested way, then KeePassXC fails to process the functions.

Empty placeholder: {T-REPLACE-RX:/&digits={S:TimeOtp-Length}/&digits=\{S:TimeOtp-Length\}//}
Nested fucntion: {T-REPLACE-RX:#{T-REPLACE-RX:!{S:TimeOtp-Algorithm}!HMAC-!!}#SHA-#SHA#}

In my case I run into the pitfall of different OTP definitions between KeePassXC and KeePass2, due to switching to KeePassXC for private use, while having to use KeePass2 at work.
So I convert the TOTP settings when needed. I started on KeePass2 by converting the KP2 fields into an otpauth URI for KPXC - folllowing IETF draft [1].
Worked well in KP2 for the first batch I converted, but failed in KPXC when converting further for otpauth parameters that are build with T-REPLACE-RX.
This is either due to escaping the curly braces {} or due to having nested T-REPLACE-RX statements.

[1] https://datatracker.ietf.org/doc/draft-linuxgemini-otpauth-uri/

Steps to Reproduce

Create a test entry that has the fields for TOTP as stored by KeePass2:

• field "TimeOtp-Algorithm", value "HMAC-SHA-256"
• field "TimeOtp-Length", value "7"
• field "TimeOtp-Period", value "15"
• field "TimeOtp-Secret-Base32", value "AAAQEAYEAUDAOCAJBIFQYDIOB4IBCEQTCQKRMFYYDENBWHA5DYPSAIJCEMSCKJRHFAUSUKZMFUXC6MBRGIZTINJWG44DSOR3HQ6T4PY"

Add additional field to convert KeePass2 TOTP fields into an otpauth string (IETF draft) for KeePassXC

• field "kp2-otpauth", value "otpauth://totp/{T-CONV:/{USERNAME}/Uri/}?secret={S:TimeOtp-Secret-Base32}&issuer={T-CONV:/{TITLE}/Uri/}{T-REPLACE-RX:/&digits={S:TimeOtp-Length}/&digits={S:TimeOtp-Length}//}{T-REPLACE-RX:/&period={S:TimeOtp-Period}/&period={S:TimeOtp-Period}//}{T-REPLACE-RX:/&algorithm={T-REPLACE-RX:#{T-REPLACE-RX:!{S:TimeOtp-Algorithm}!HMAC-!!}#SHA-#SHA#}/&period={S:TimeOtp-Algorithm}//}"

Copy attribute "kp2-otpath" and check resulting value.

Expected Versus Actual Behavior

Expected result (as returned by KeePass 2.58 [2]):
otpauth://totp/std_usr?secret=AAAQEAYEAUDAOCAJBIFQYDIOB4IBCEQTCQKRMFYYDENBWHA5DYPSAIJCEMSCKJRHFAUSUKZMFUXC6MBRGIZTINJWG44DSOR3HQ6T4PY&issuer=Test%20Convert%20KeePass%20to%20XC&digits=7&period=15&algorithm=SHA256

Actual result of KeePassXC:
otpauth://totp/std_usr?secret=AAAQEAYEAUDAOCAJBIFQYDIOB4IBCEQTCQKRMFYYDENBWHA5DYPSAIJCEMSCKJRHFAUSUKZMFUXC6MBRGIZTINJWG44DSOR3HQ6T4PY&issuer=Test%20Convert%20KeePass%20to%20XC

[2] https://keepass.info/help/base/placeholders.html#texttrf

KeePassXC Debug Information

KeePassXC - Version 2.7.10
Revision: b342be4

Qt 5.15.11
Debugging mode is disabled.

Operating system: Windows 10 Version 2009
CPU architecture: x86_64
Kernel: winnt 10.0.19045

Enabled extensions:
- Auto-Type
- Browser Integration
- Passkeys
- SSH Agent
- KeeShare
- YubiKey
- Quick Unlock

Cryptographic libraries:
- Botan 3.1.1

Operating System

Windows

Linux Desktop Environment

None

Linux Windowing System

None

[droidmonkey]

We support reading KeePass2 TOTP attributes natively in 2.7.10, you do not need to do any conversions.

#11229

[maddes-b]

@droidmonkey Sorry for the confusion with TOTP stuff, this bug report is about T-REPLACE-RX functionality.
I use T-REPLACE-RX for optional fields, or when I need a part of an field. I can re-write the bug report without any TOTP relationship.

Field some_optional_field = cut1-value-cut2

Missing field: {T-REPLACE-RX:/{S:some_optional_field}/\{S:some_optional_field\}//}
Remove multiple parts from a field: {T-REPLACE-RX:#{T-REPLACE-RX:!{S:some_optional_field}!cut1-!!}#-cut2##}
Combined: {T-REPLACE-RX:#{T-REPLACE-RX:!{T-REPLACE-RX:/{S:some_optional_field}/\{S:some_optional_field\}//}!cut1-!!}#-cut2##}

Rename some_optional_field to make it missing.

P.S.: Somehow I missed that TOTP change in 2.7.10. Thanks.

[droidmonkey]

I don't understand why you need such complex processing. Just store the values in separate advanced attributes.

[maddes-b]

For example I use these to build attributes that hold command lines.
I have one cmd line pattern (e.g. scp/rsync/ssh) and maintain attributes as needed.
Not needed attributes are missing/deleted to keep a dense view on the attributes.
Much more comfortable than maintaining a long string itself.

The "remove multiple parts from a field" patten is used only for 2 cases.
Indeed can be workarounded with separated attributes.
But still want to mention it, because it does not work as expected. And to provide some "weird" test case.

[droidmonkey]

Yah I guess "as expected" might be a stretch hah, we did not intend to support nested replacement structures, I am surprised that KeePass does. To get past the nesting limitation you can use multiple attributes that each contain the inner T-REPLACE-RX then you just include the final result in your cmd url. That also cleans up your cmd url.

I took a dive into your example, I am not sure what you are trying to do here:

Missing field: {T-REPLACE-RX:/{S:some_optional_field}/{S:some_optional_field}//}

Are you trying to remove a literal {S:some_optional_field} from the resolved {S:some_optional_field}??
Basically this read as:

1. Resolve {S:some_optional_field} to cut1-value-cut2
2. Find (literally) {S:some_optional_field}
3. Replace with nothing

So this should be a no-op in your example, because that string won't be found and cut1-value-cut2 should be returned. However I can see that we do not honor the use of \ as an escaping character.

[maddes-b]

In KeePass a reference like "{S:some_optional_field}" becomes a literal, when the field is missing.
Checked "{S:some_optional_field}" in KPXC and the result is empty, when the field is missing.

IMHO it should be as following:

• Reference should be empty when the field is missing (KeePass bug / KeePassXC correct)
• Nested T-REPLACE-RX should be supported (KeePass correct / KeePassXC bug)

[droidmonkey]

We actually do support nested commands, I was wrong

[maddes-b]

Ok, then something else is an issue, so I did some more tests, starting with a simple one and then extending more and more.

Test Cases A: Attribute "some_optional_field" is available with value "HMAC-SHA-256"

1. Attribute "test-kpxc-00-normal": {S:some_optional_field}
   result: HMAC-SHA-256 [OK]
2. Attribute "test-kpxc-01-empty-kp": {T-REPLACE-RX:!{S:some_optional_field}!\{S:some_optional_field\}!!}
   result: empty [WRONG]
3. Attribute "test-kpxc-01-empty-kpxc": {T-REPLACE-RX:!{S:some_optional_field}!(.+)!&value=$1!}
   result: &value=HMAC-SHA-256 [OK]
4. Attribute "test-kpxc-02-groups": {T-REPLACE-RX:!{S:some_optional_field}!(.*)HMAC-([^-]*)-(.*)!$1$2$3!}
   result: SHA256 [OK]
5. Attribute "test-kpxc-03-nested": {T-REPLACE-RX:/{T-REPLACE-RX:!{S:some_optional_field}!(.*)HMAC-([^-]*)-(.*)!$1$2$3!}/(.+)/&value=$1/}
   I could reduce the T-REPLACE-RX nesting by one level via using regex groups.
   result: &value=SHA256 [OK]

Test Cases B: Attribute "some_optional_field" is missing

1. result: empty [OK]
2. skipped as not working with KPXC
3. result: empty [OK]
4. result: empty [OK]
5. result: empty [OK]

Note:
In KeePass regex 3, 4 and 5 are not working as wanted, due to placeholder literal when attribute is missing.
Dominik said in my KeePass bug report that putting out the reference/substitution as a literal is intended when the attribute/field is missing.

Conclusion

• Nesting T-REPLACE-RX works in KPX
• Different separator in T-REPLACE-RX works in KPX
• Regex groups in T-REPLACE-RX work in KPX
• Escaping {S: in search regex of T-REPLACE-RX is not working

This missing/failing feature of escaping the placeholder in KPXC causes the incompatibility with the KeePass solution.
For me this could be categorized as a bug as escaping chars to avoid their functionality is regex standard.

Idea for solution:

• {S:some_optional_field} = put content of placeholder into regex
• \\{S:some_optional_field} = put literal \ and content of placeholder into regex, this should be the case if an even count of backslashes is before the curly brace (0,2,4,6,etc.).
• \{S:some_optional_field} = put literal {S:some_optional_field} into regex, this should be the case if an odd count of backslashes is before the curly brace (1,3,5,7,etc.).

Maybe checking out the related KeePass source code is also helpful (special cases? different handling?).
SprEngine.cs/PerformTextTransforms() -> SprEngine.cs/ParseAndRemovePlhWithParams + SprEngine.cs/TransformContent()

Note to everyone:
Putting the content of a placeholder into a regex can cause unexpected behavior due to regex special chars (like "+.?/", etc.).
Can only be handled by user, not KPXC.
A possible solution (if supported) would be to put a T-REPLACE-RX into the regex search (more nesting).
Don't do this!