Nondeterministic output with garbled strings

[jgm]

I suspect this is a memory handling bug related to PR #13; see the discussion there.

@andersk can you have a look? I'm almost positive it's an issue with your PR, rather than with cmark.

@kivikakk this affects cmark-gfm too.

If a quick fix is not forthcoming, we should roll back PR #13 and make a new release.

Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^Z\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^\\*%\n"

Also reproducible with other strings:

Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT " ") []]]
" p@R\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT " ") []]]
" \n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"hiC\SOHB\n"

Affects other writers as well:

Prelude CMarkGFM> nodeToMan [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
".PP\nhiz\v\SOH\n"
Prelude CMarkGFM> nodeToLaTeX [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"hiaR\n"
Prelude CMarkGFM> nodeToXml []  $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"*** Exception: Cannot decode byte '\xd2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
Prelude CMarkGFM> nodeToHtml [] [] $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"<p>hi[\aB</p>\n"
Prelude CMarkGFM> nodeToHtml [] [] $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"<p>hio\EOTB</p>\n"

[jgm]

We need that null terminator!

[jgm]

@kivikakk can you cherry-pick this change for cmark-gfm-hs and make a new release?
It seems to fix the issue.
EDIT: You will also need 15041a8

[andersk]

Good catch, that was silly of me. I’ve proposed the addition of withCString to Data.Text in haskell/text#254.

[jgm]

Anders Kaseorg <notifications@github.com> writes:

Good catch, that was silly of me. I’ve proposed the addition of `withCString` to Data.Text in haskell/text#254.

That would be a good addition indeed!

[kivikakk]

Thanks for the update again! ❤️