ArkScript v4.0.0-14

[SuperFola]

Added

• more tests for the io builtins
• added lines and code coloration in the error context
• new dependency: fmtlib
• added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
• $repr macro to get a string representation of a given node
• added boost-ext/ut to write unit tests in C++
• basic ArkScript code formatter, available through the CLI: arkscript -f|--format
• comments are now tracked in the AST and attached to the nearest node below them
• VM::forceReloadPlugins, to be used by the REPL to force reload the plugins and be sure that their symbols are all define
• added help, save (save history to disk), history (print history), reset (reset vm and code) commands to the REPL
• REPL can now show when a code block isn't terminated (prompt changes from > to :)
• more controls available inside the REPL
• fuzzing step in the CI
• better error reporting on unknown import
• check on number of arguments passed to type
• warning when the formatter deletes comment(s) by mistake
• check on arguments passed to list, concat, append and friends to only push valid nodes (that produces a value)
• introduced Ark::internal::Pass to describe compiler passes: they all output an AST (parser, import solver, macro processor, and optimizer for now)
• add -f(no-)importsolver, -f(no-)macroprocessor and -f(no-)optimizer to toggle on and off those compiler passes
• added resolving empty? as a macro when possible
• added short-circuiting to and and or implementation
• added --check to the formatter as an option: returns 0 if the code is correctly formatted, 1 otherwise
• the name & scope resolution pass now checks for mutability errors
• compile time checks for mutability errors with append!, concat! and pop!
• new MAKE_CLOSURE <page addr> instruction, generated in place of a LOAD_CONST when a closure is made
• added -fdump-ir to dump the IR entities to a file named {file}.ark.ir
• added 11 super instructions and their implementation to the VM
• support for the glob import syntax and symbol import syntax
• modify list and return a copy (string:setAt string index char) (bound checked)
• added in place list mutation: (@= list|string index new_value), (@@= list|list<string> index1 index2 new_value|char) (bound checked)
• compile time argument count check for and and or
• basic dead code elimination in the AST optimizer
• new operator @@ to get elements in list of lists / list of strings
• new builtin random, returning a random number between INT_MIN and INT_MAX, or in a custom range
• $as-is to paste a node inside a maro without evaluating it further ; useful to stop recursive evaluation of nodes inside function macros
• LOAD_SYMBOL_BY_INDEX instruction, loading a local from the current scope by an index (0 being the last element added to the scope)
• STORE_FROM_INDEX and SET_VAL_FROM_INDEX instructions for parity with the super instructions not using load by index
• INCREMENT_BY_INDEX and DECREMENT_BY_INDEX instructions for parity with the super instructions not using load by index
• STORE_TAIL_BY_INDEX, STORE_HEAD_BY_INDEX, SET_VAL_TAIL_BY_INDEX, SET_VAL_HEAD_BY_INDEX super instructions added for parity with the super instructions not using load by index
• RESET_SCOPE instruction emitted at the end of a while loop to reset a scope so that we can create multiple variables and use LOAD_SYMBOL_BY_INDEX
• instruction source location ; two new bytecode tables were added: one for filenames, another for (page pointer, instruction pointer, file id, line), allowing the VM to display better error messages when the source is available
• show source location when a runtime error is thrown in the VM

Changed

• instructions are on 4 bytes: 1 byte for the instruction, 1 byte of padding, 2 bytes for an immediate argument
• enhanced the bytecode reader and its command line interface
• added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
• fixed underline bug in the error context
• the str:format functions now expects strings following this syntax: https://fmt.dev/latest/syntax.html
• more documentation about the compiler implementation
• more documentation about the virtual machine
• closures can be now be compared field per field: (= closure1 closure2) will work only if they have the same fields (name) and if the values match
• macros are now defined like ($ name value) / ($ name (args args args) body) / ($if cond then else)
• upgraded from C++17 to C++20
• new parser, new syntax for imports: (import package.sub.file)
• allow nodes to be empty when dumping the AST to JSON
• macros can be declared inside a begin block within a cond macro and used in the scope surrounding the cond macro
• arkscript --version and arkscript --help now output ArkScript version with the commit hash
• void Value::toString(std::ostream&, VM&) now becomes std::string Value::toString(VM&)
• removed Node::operator<< to replace it with Node::debugPrint
• fixed a bug in the compiler where one could pass a non symbol to let, mut or set, resulting in a compiler crash
• fixed a bug in the macro processor where one could pass an unknown symbol to argcount and crash the processor
• fixed a bug in the compiler where one could pass something other than a list to (fun) as the argument block, resulting in a crash
• fixed a bug in the compiler generating not callable functions
• fixed a bug in the macro processor generating invalid let / mut / set nodes
• fixed a bug in the macro processor allowing out of bounds access with ($ test (@ [1 2 3] -5))
• fixed a bug in the vm which wrongfully allowed self concat in place: (concat! lst lst)
• fixed a bug in the compiler where one could "use" operators without calling them: (print nil?)
• fixed a bug in the compiler allowing the use of operators without any argument: (+)
• fixed a bug in the vm during error reporting when a non-function was used as a function
• refactored code inside the bytecode reader to promote code reuse
• fixed a bug in the compiler generating invalid while nodes
• fixed a bug when passing the wrong number of arguments to a function inside an async call was crashing the VM because the function couldn't be named
• fixed a bug in the compiler generating invalid fun nodes
• fixed a bug when generating let, mut or set nodes inside macros with an invalid node type
• fixed a bug when reading invalid UTF8 codepoints in the parser caused out of bounds reads
• fixed a bug with recursive macro, exhausting the stack space due to recursive evaluation
• futures can be awaited again, they will return nil on all the tries
• checking for reused argument name in macros during parsing
• enhanced comment after node handling in macros
• adding a hard limit on package names length (255 characters, to comply with posix limits)
• disallow passing invalid nodes as arguments to functions and operators
• checking for unevaluated spread inside macros
• checking for invalid symbols when defining a function through a macro
• added a max macro unification depth (256)
• added a max macro evaluation depth (256)
• introduced internal::listInstructions with the different instructions, to be used by the compiler and name resolution pass
• checking for forbidden variable/constant name in the name & scope resolution pass, to give errors to the user before compiling some weird code
• repl completion and colors are now generated automatically from the builtins, keywords & operators
• fixed formating of comments inside function declarations
• renamed the macros symcat and argcount to $symcat and $argcount for uniformity
• the Ark::VM class is now final
• the STORE instruction has been renamed SET_VAL
• the STORE instruction is emitted in place of the LET and MUT instructions, without any mutability checking now
• io:writeFile no longer takes a mode and has been split into io:writeFile and io:appendToFile
• instructions are now positioned like this: inst byte1 byte2 byte3
  • byte1 is 0 if the instruction takes a single argument on 16 bits, split on byte2 and byte3
  • if the instruction takes two arguments, they each have 12 bits ; the second one is on byte1 and upper half of byte2, the first on lower half of byte2 and then byte3
• ast-to-json dump now supports macros
• the parser can detect ill-formed macros (that are seen as function macros while being value macros)
• adding a CALL_BUILTIN <builtin> <arg count> super instruction
• fixed formatting of comments after the last symbol in an import node
• renamed str:xyz builtins to string:xyz for uniformity with the standard library
• string:find takes an optional third argument, startIndex (where to start the lookup from, default 0
• list:setAt can work with negative indexes, and is now bound checked
• re-enabled the AST optimizer, only used for the main arkscript executable (not enabled when embedding arkscript, so that one can grab variables from the VM)
• loops have their own scope: variables created inside a loop won't leak outside it
• upgraded fmtlib to 11.1.3-13
• allow capture in nested scope (before it was targeting only the current scope)
• -bcr option can be given a source file, it will then be compiled before its bytecode is shown
• magic numbers for tables start in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xA1, 0xA2, 0xA3 (symbols, values, code) to make them stand out in hex editors
• magic numbers for value types in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xF1, 0xF2, 0xF3 (number, string, function)
• numbers in the values table in bytecode files are no longer turned to string, but their IEEE754 representation is now encoded on 12 bytes (4 for the exponent, 8 for the mantissa)
• changed how scopes are stored inside the VM to enhance performances. All scope data are now contiguous!
• when possible, accessing variables from the current scope is compiled to a new instruction LOAD_SYMBOL_BY_INDEX, to avoid the sometimes expansive lookup by id
  • this works inside normal scopes (introduced by while loops) and functions scopes, but not for closures
• VM stack size is now 4096 instead of 8192
• Ark::CodeError now takes a CodeErrorContext to store the source (filename, line, column, expression) of an error

Removed

• removed unused NodeType::Closure
• removing the custom string, replacing it with std::string (the format engine of the custom string had a lot of memory leaks)
• Utils::digPlaces and Utils::decPlaces got removed as they were no longer needed
• removed deprecated code (list:removeAt, ark executable now replaced by arkscript)
• removed VM::getUserPointer and VM::setUserPointer
• removed ARK_PROFILER_COUNT define
• removed useless \0 escape in strings
• removed termcolor dependency to rely on fmt for coloring outputs
• removed and and or instructions in favor of a better implementation to support short-circuiting
• removed LET and MUT instructions in favor of a single new STORE instruction
• removed SAVE_ENV instruction
• removed Value VM::resolve(const Value* val, Args&&... args), which has been deprecated in ArkScript v3.4.0

---

This discussion was created from the release ArkScript v4.0.0-14.