Adding the Sweet transpiler #60

maju-degrandi · 2025-06-10T17:52:54Z

This pull request adds the complete implementation of the transpiler responsible for converting documents written in the sweet flavor of the Vinum language into the dry flavor.

Key features included in this implementation:

Full sweet-to-dry transformation: The transpiler parses the sweet syntax, including nested and chained function calls, and generates the corresponding dry-style blocks.
Accurate lexical analysis: The tokenizer distinguishes between content and function names correctly, even in edge cases involving dots, parentheses, and arbitrary characters.
Right-associative parsing: Supports nested call chains with proper block insertion, preserving syntactic structure.
Output handling: Transpiled results are written to a new .vin file derived from the input filename, using the _dry suffix.
Extensibility: Designed with extensible features in mind; formatting rules like paragraph detection (e.g., via double newlines) can be added later.

This version prioritizes syntactic correctness and faithful interpretation of the sweet input, ensuring that all elements — including text, parentheses, and function markers — are processed as intended.

…into sweet-transpiler

Mudanças no lexer para não diferenciar regras na tokenização -> passando tudo para o bison

Cases like (content).name and .name are working in flex and bison

There is still a lack of facilities for sweet, the "markdown" defined by Monaco

The error that occurred when blocks of different types with contents of more than one word were linked together has been resolved added start states on Flex. Now there is just one thread error still unresolved: blocks of type .block.global need to be transpiled to [global [block]], but are transpiling to [block][global]. (I think this could be resolved fixing the parser, but I don't know how yet) Furthermore, we still don't accept '(', ')' and '.' as content.

This error was resolved with changes to the parser. A new "call" rule has been created, which takes care of function calls in the ".function" format. Furthermore, the construction of "program" is given by a more complex string manipulation instead of just concatenations, given that the "call" rule returns to "program" with the prepared block, simply inserting its content in the correct position. NOTE: the characters '(', ')' and '.' are not yet allowed as block content.

Some developers are using Ubuntu 22.04. This distribution packages a _very_ old version of meson. Add support for it by checking the meson version to add code that would break older versions. Also, use the old file name for meson options.

…d output handling This update significantly improves the tokenizer and parser to correctly interpret the syntax of the "sweet" flavor of the Vinum language. Changes and improvements include: - **Improved Tokenization**: - The previous implementation incorrectly classified most words as `NAME`, unnecessarily increasing the number of parse tree nodes. This is now fixed: names are only recognized after a dot (`.`) followed by an identifier, as intended by the language design. - Introduced a custom lexer state (`AFTER_DOT`) to properly tokenize the sequence `.<name>`. If a dot is not followed by a valid identifier, it is now treated as part of `CONTENT`. - The lexer now correctly accepts all characters, including `(`, `)`, and `.`, as valid parts of `CONTENT` when they do not form a function call. This ensures that full text content is preserved and interpreted correctly. - **Parser Functionality**: - Blocks with or without parentheses are now properly parsed, including arbitrarily nested function calls, right-associative chains, and inline content. - Function calls without arguments (`.name`) are correctly recognized and translated into dry-style `[name]` blocks. - Chained calls like `(a (b) (c).d).e` are parsed into right-associative dry blocks: `[e a (b) [d c]]`. - **Output File Generation**: - The transpiler now writes the dry-style output to a `.vin` file derived from the input filename by appending the `_dry.vin` suffix. - Existing dry blocks (already in `[]` form) are preserved as-is. - **Design Prioritization**: - The current focus is on syntactic correctness and faithful transformation from sweet to dry syntax. - Features such as double line breaks marking new paragraphs will be implemented later as extensions. Overall, this refactor brings the transpiler much closer to the intended behavior of the Vinum language by improving syntax recognition, handling edge cases in content, and producing clean and structured output files.

…-child

By ensuring calls to external functions always have an ARGS node, we can prevent them from being skipped during evaluation

fix: prevent external calls from being skipped

Grillo-0

Please rebase your changes to the current develop branch

Grillo-0 · 2025-06-29T04:50:54Z

src/ylwrap10791/y.tab.c

@@ -0,0 +1,1475 @@
+/* A Bison parser, made by GNU Bison 3.8.2.  */


Please remove this file

Grillo-0 · 2025-06-29T04:52:24Z

src/test_sweet.vin

@@ -0,0 +1,14 @@
+(foo).bar


Now that we have the vunit test library, we could use it in here for testing.

Grillo-0 · 2025-06-29T04:53:45Z

src/Makefile.am

@@ -1,9 +1,9 @@
 AM_CFLAGS = -Wall -Wextra -Werror -g


We don't use autotools anymore, please use the meson build system

Grillo-0 · 2025-06-29T04:54:54Z

src/sweet_bison.h

@@ -0,0 +1,93 @@
+/* A Bison parser, made by GNU Bison 3.8.2.  */


We don't want the file generated by lex/bison on the repo. Please add the to the .gitignore

Grillo-0 · 2025-06-29T05:01:56Z

About the commit's.

For now, just keep adding commits addressing the reviews. In the end - as it's just one "thing" being added to the project - we want just one commit, but it's faster doing this way and squashing all of them after.

Mudanças no lexer para não diferenciar regras na tokenização -> passando tudo para o bison

Cases like (content).name and .name are working in flex and bison

There is still a lack of facilities for sweet, the "markdown" defined by Monaco

The error that occurred when blocks of different types with contents of more than one word were linked together has been resolved added start states on Flex. Now there is just one thread error still unresolved: blocks of type .block.global need to be transpiled to [global [block]], but are transpiling to [block][global]. (I think this could be resolved fixing the parser, but I don't know how yet) Furthermore, we still don't accept '(', ')' and '.' as content.

This error was resolved with changes to the parser. A new "call" rule has been created, which takes care of function calls in the ".function" format. Furthermore, the construction of "program" is given by a more complex string manipulation instead of just concatenations, given that the "call" rule returns to "program" with the prepared block, simply inserting its content in the correct position. NOTE: the characters '(', ')' and '.' are not yet allowed as block content.

…d output handling This update significantly improves the tokenizer and parser to correctly interpret the syntax of the "sweet" flavor of the Vinum language. Changes and improvements include: - **Improved Tokenization**: - The previous implementation incorrectly classified most words as `NAME`, unnecessarily increasing the number of parse tree nodes. This is now fixed: names are only recognized after a dot (`.`) followed by an identifier, as intended by the language design. - Introduced a custom lexer state (`AFTER_DOT`) to properly tokenize the sequence `.<name>`. If a dot is not followed by a valid identifier, it is now treated as part of `CONTENT`. - The lexer now correctly accepts all characters, including `(`, `)`, and `.`, as valid parts of `CONTENT` when they do not form a function call. This ensures that full text content is preserved and interpreted correctly. - **Parser Functionality**: - Blocks with or without parentheses are now properly parsed, including arbitrarily nested function calls, right-associative chains, and inline content. - Function calls without arguments (`.name`) are correctly recognized and translated into dry-style `[name]` blocks. - Chained calls like `(a (b) (c).d).e` are parsed into right-associative dry blocks: `[e a (b) [d c]]`. - **Output File Generation**: - The transpiler now writes the dry-style output to a `.vin` file derived from the input filename by appending the `_dry.vin` suffix. - Existing dry blocks (already in `[]` form) are preserved as-is. - **Design Prioritization**: - The current focus is on syntactic correctness and faithful transformation from sweet to dry syntax. - Features such as double line breaks marking new paragraphs will be implemented later as extensions. Overall, this refactor brings the transpiler much closer to the intended behavior of the Vinum language by improving syntax recognition, handling edge cases in content, and producing clean and structured output files.

…into sweet-transpiler

artP2 · 2025-07-02T14:16:46Z

Create the vinumt subproject, and include it in vinumc as a shared library using meson.

arthurvergacas and others added 18 commits December 4, 2024 22:41

alterações com a maju

5a54e2b

alterações com a maju

a02ddbc

Merge branch 'sweet-transpiler' of https://github.com/vinumdoc/vinum …

1fb6349

…into sweet-transpiler

Fazendo transpilação a partir do bison

60eadbf

Mudanças no lexer para não diferenciar regras na tokenização -> passando tudo para o bison

Fixing compilation error

90445fd

Simple cases are working

7679df9

Cases like (content).name and .name are working in flex and bison

All block types are working now - including nested blocks

5942fff

There is still a lack of facilities for sweet, the "markdown" defined by Monaco

Merge pull request #58 from vinumdoc/old-meson

3881362

Revert "vinumc: reuse args rule to define a program's children"

f5c7baa

Merge pull request #61 from vinumdoc/revert-55-fix/48/word-as-program…

8cb6e22

…-child

fix: prevent external calls from being skipped

3e55ea4

By ensuring calls to external functions always have an ARGS node, we can prevent them from being skipped during evaluation

Merge pull request #68 from vinumdoc/ensure-external-call-args-node

1378c5e

fix: prevent external calls from being skipped

vinumc/tests: add test_call_no_args to test suite

c6f6fd9

Merge pull request #69 from vinumdoc/test-no-args-extern-calls

b597095

Grillo-0 assigned maju-degrandi Jun 25, 2025

Grillo-0 self-requested a review June 25, 2025 13:57

Grillo-0 requested changes Jun 29, 2025

View reviewed changes

arthurvergacas and others added 7 commits June 30, 2025 16:47

alterações com a maju

7088d81

Fazendo transpilação a partir do bison

ea16042

Mudanças no lexer para não diferenciar regras na tokenização -> passando tudo para o bison

Fixing compilation error

e89cd59

Simple cases are working

0bc57b3

Cases like (content).name and .name are working in flex and bison

All block types are working now - including nested blocks

04a6b9e

There is still a lack of facilities for sweet, the "markdown" defined by Monaco

maju-degrandi added 2 commits June 30, 2025 16:47

Merge branch 'sweet-transpiler' of https://github.com/vinumdoc/vinum …

f720400

…into sweet-transpiler

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding the Sweet transpiler #60

Adding the Sweet transpiler #60

Uh oh!

maju-degrandi commented Jun 10, 2025

Uh oh!

Grillo-0 left a comment

Uh oh!

Grillo-0 Jun 29, 2025

Uh oh!

Grillo-0 Jun 29, 2025

Uh oh!

Grillo-0 Jun 29, 2025

Uh oh!

Grillo-0 Jun 29, 2025

Uh oh!

Grillo-0 commented Jun 29, 2025

Uh oh!

artP2 commented Jul 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		@@ -0,0 +1,1475 @@
		/* A Bison parser, made by GNU Bison 3.8.2. */

		@@ -0,0 +1,93 @@
		/* A Bison parser, made by GNU Bison 3.8.2. */

Adding the Sweet transpiler #60

Are you sure you want to change the base?

Adding the Sweet transpiler #60

Uh oh!

Conversation

maju-degrandi commented Jun 10, 2025

Uh oh!

Grillo-0 left a comment

Choose a reason for hiding this comment

Uh oh!

Grillo-0 Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

Grillo-0 Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

Grillo-0 Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

Grillo-0 Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

Grillo-0 commented Jun 29, 2025

Uh oh!

artP2 commented Jul 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants