forked from erkyrath/Inform7-IDE-Mac
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathTechMan.txt
8617 lines (6772 loc) · 373 KB
/
TechMan.txt
1
------------------------------------------------------------------------------ Inform 6.21 Technical Manual Graham Nelson 27th April 1996 revised on the following dates: 5th May, 10th May, 6th September, 23rd September, 16th December 1996, 27th January, 26th March, 5th April, 8th September 1997, 22nd March 1998, 10th December, 30th January 1999, 27th April------------------------------------------------------------------------------ 1 The source code 1.1 Inventory 1.2 Map 1.3 Naming conventions 1.4 Typedef-named types 2 Porting Inform to a new environment 2.1 Dependence on the OS 2.2 Portability issues: the types int32 and uchar 2.3 The character set and the format of text files 2.4 The OS definitions block in "header.h" 2.5 Running Inform in a multi-tasking OS 3 Front end 3.1 The ICL interpreter 3.2 Fundamental method 4 Lexical analysis 4.1 Aim and structure of the lexer 4.2 Level 1: lexical blocks and buffered input 4.3 Level 2: breaking text into lexemes 4.4 Representation of tokens 4.5 Level 3: identifying identifiers 4.6 Hash-coding and comparing names 4.7 The symbols table 4.8 Token lookahead: the circle 4.9 Summary of the token output 5 Syntax analysis 1: the top-down structural parser 5.1 Aim and structure of the syntax analyser 5.2 Predictive parsing 5.3 The context-free grammar 5.4 Assigning values to symbols 5.5 Outputs other than assembly language 5.6 Assembly operands 5.7 Translation to assembly language 5.8 Summary of assembly language instructions output 6 Syntax analysis 2: the bottom-up expression parser 6.1 Map and structure 6.2 The operator precedence grammar 6.3 Lexical level 4: tokens to etokens 6.4 Resolution of ambiguous tokens 6.5 Constant folding 6.6 Checking lvalues and simplifying the parse tree 6.7 Summary of parse tree output 7 Code generation from parse trees 7.1 Aim and structure of the code generator 7.2 Annotation for conditions 7.3 Main traversal 7.4 Void context 7.5 Results of subexpressions 7.6 Conditional and logical operators 7.7 System functions 7.8 Strict mode assertions 8 Assembly of code, text and data structures 8.1 Assembly of initial code 8.2 Branch offset backpatching and optimisation 8.3 Text translation: ISO and Unicode resolution 8.4 Dictionary management 8.5 Format of tables unspecified by the Z-machine 8.6 Grammar version numbers GV1 and GV2 8.7 Value backpatching 8.8 Packed address decoding 9 Run-time veneer 9.1 Services provided by the veneer 9.2 Specification of the veneer routines 9.3 Properties 2 and 3, "ofclass" and "metaclass" 9.4 Class numbering and class objects 9.5 Individual property identifiers 9.6 Individual property tables 9.7 Availability of symbol names at run-time 10 Module compilation and linking 10.1 Model 10.2 Linking a module 10.3 Imports and exports 10.4 Backpatch backpatching 10.5 How modules differ from story files 10.6 Restrictions on what modules may contain 11 Service levels 11.1 Variables and arrays 11.2 Memory allocation and deallocation 11.3 Error messages 11.4 File input/output 12 Low-level language features 12.1 Using the "Trace" directive 12.2 System constants and other secret syntaxes 12.3 The "Zcharacter" directive 12.4 Sequence points 12.5 Format of debugging information files 12.6 How to syntax-colour Inform code 13 What little I remember 13.1 Publication history 13.2 Design history 13.3 Implementation history 13.4 Modification history------------------------------------------------------------------------------ 1 The source code --------------- 1.1 Inventory ---------Inform is written in portable ANSI C and the source code is divided into 21files of code, called "sections", plus 1 #include file of linkage, constantand type definitions. These files are: arrays.c asm.c bpatch.c chars.c directs.c errors.c expressc.c expressp.c files.c inform.c lexer.c linker.c memory.c objects.c states.c symbols.c syntax.c tables.c text.c veneer.c verbs.c header.hNote that all their names fit into the 8.3 filenaming convention. Thesubdivision into 21 sections is intended to ensure that each .c file can becompiled into one linkable object file: under some C compilers objectcode files cannot exceed 64K in length. On my machine, expressc.o (theobject code derived from expressc.c) is the largest, at 40K (about a third ofwhich is the static table of operator data).A concise tree giving the structure of the source code follows. A sectionname is given brackets when it has already been given on a previous line: so"(inform.c)" is intended to be read as "now back to inform.c again". A namewritten as a function, like "compile()", is indeed a function, whosearguments are omitted here. Text in square brackets indicates the presenceof interesting tables of static data. Finally, note that this structureis not absolutely rigorous: the error-reporting routines in "errors.c", forexample, are called from all over Inform, not just from the lexical analyser. inform.c ICL parser: switches, filename translation, path variables memory.c ICL memory command parser: sets memory settings. (inform.c) compile(): polling all sections to manage variables, allocate and free arrays syntax.c syntax analyser, top level lexer.c lexical analyser: converts source text to a stream of tokens [Tables of all Inform keywords] chars.c performs character set translations files.c reads source code files into buffers for the lexer; performs miscellaneous file I/O symbols.c keeps table of symbol names found in the source; recognises from and adds to this table errors.c issues error, fatal error and warning messages (syntax.c) parse_program(): top level routine in syntax analyser parse_directive() directs.c parse_given_directive(): parses and obeys the easier directives; manages conditional compilation; delegates directive parsing down to other sections for harder cases: text.c make_abbreviation(): Abbreviate arrays.c make_global(): Array, Global objects.c make_attribute(): Attribute make_property(): Property make_class(): Class make_object(): Object verbs.c make_fake_action(): Fake_Action make_verb(): Verb extend_verb(): Extend linker.c link_module(): Link (symbols.c) assign_symbol(): giving value and type to symbols (syntax.c) parse_routine() parse_code_block() states.c parse_statement(): assigns ".Label" labels, parses and generates code for statements parse_action(): handles <Action ...> statements parse_print(): handles print and print_ret asm.c parse_assembly(): handles assembly language source code (preceded by "@") expressp.c parse_expression(): parses all expressions (including constants), conditions and assignments. expressc.c [Table of operators] code_generate(): generates code from parse trees previously found by parse_expression() (asm.c) assemble_2_to() (and many other similarly named routines) [Database of all Z-machine opcodes] assemble_instruction(): assembles a single Z-code instruction assemble_label_no(): puts label N here assemble_routine_end(): finishes routine, backpatches and optimises branches (text.c) compile_string(): translates ASCII text to Z-encoded text the dictionary manager the abbreviations optimiser (chars.c) performs character set translations veneer.c compile_veneer(): compiles in any whole routines needed by the rest of the compiled code [Table of Inform source code for veneer routines] tables.c construct_storyfile(): glues together all the code, dictionary, object tree, etc. into a story file bpatch.c backpatch the code in the light of recent knowledge 1.2 Map ---Here follows a map of the Inform archipelago, marking the inhabited islands,their shipping lanes and chief imports and exports: command line and/or ICL files in | | ICL commands \|/ +----------+ FRONT | inform.c | END | memory.c | +----------+ | | filenames | \|/ +------------+ LEXICAL ------> files.c -----> | lexer.c | ANALYSER source chars | symbols.c | code in +------------+ /|\ | symbol | | values | | tokens | \|/ +------------+ SYNTAX | syntax.c | -----+------> asm.c --->---\ ANALYSER: | states.c | / assembly /|\ initial| STATEMENTS | . | / language | code| | ---------- | / | | @ ASSEMBLY | asm.c | -/ | | | ---------- | | | | . | parse trees | \|/ EXPRESSIONS | expressp.c | --+-------> expressc.c asm.c | (map 6.1) | \ | | . | \ | | . | \ TEXT | | ---------- | strings +-------+Z-text | DIRECTIVES | directs.c | \---->|text.c |-->---)|(--\ | . | |chars.c| | | | . | +-------+ | | | . | dictionary| | | | . | alphabets \|/ \|/ | | . | | | | | arrays.c | ------->------\ | | | | . | array area | | raw| | | . | | | Z-code| | | objects.c | ----->-----\ | | | | | verbs.c | objects | | | | | +------------+ | | | | | | \|/\|/\|/ \|/ | | grammar +----------+----------+ | \---------------> | tables.c bpatch.c | | +----------+----------+ | | OUTPUT | | | | | Z-machine \|/ Z-code| | up to | | | code area | \|/ \|/ \-----------> files.c | | \|/ story file out(For clarity, the linker and a few smaller tables are missed out; and the"service" sections of Inform, such as "errors.c" and the allocation code in"memory.c", are missed out since they are, so to speak, pipes and sewerswhich lie beneath the surface of the ocean.) 1.3 Naming conventions ------------------The "header.h" makes over 700 constants using #define.These are mainly in capital letters and are followed by _ and then some shortcode indicating what kind of constant is being defined: for instance,NUMBER_TT means "the token type <number>". We write *_TT for the set ofconstants ending in _TT. Similarly, though to a lesser extent, groups ofrelated variables and routines have grouped names. ------------------------------------------------------------------------- Set of constants Used for ------------------------------------------------------------------------- *_Extension File extensions, such as ".z5", used if the host OS supports them *_Directory Initial values for the ICL path variables (e.g., default pathname where story files are written to) *_TT Token types *_CODE Token values for statement and directive names *_COND Token values for textual condition names *_SEGMENT Token values for object definition segment names *_MK Token values for misc statement keywords *_TK Token values for "trace" directive keywords *_DK Token values for misc directive keywords *_SC Token values for system constant names (* is written in lower case) *_SYSF Token values for system function names [In all of the above eight cases, * is the name of the statement, keyword, etc. referred to, written in upper case except as specified above] *_SEP Token values for separators (the name sometimes reflects the text, e.g., DARROW_SEP for the double-length arrow "-->"; sometimes its use, e.g. NOTEQUAL_SEP for "~=") *_OP Token values for operators *_A Associativity values for operators *_U "Usage" (infix, prefix or postfix) values for operators *_T Symbol types *_SFLAG Symbol flags (bitmasks containing one bit set, so that (sflags[i] & *_SFLAG) is true if flag is set) *_CONTEXT Contexts in which the expression evaluator can run (e.g., "void", "condition") *_zc Internal numbers referring to Z-machine opcodes (* is the Standard 0.2 name for the opcode, written in lower case) *_OT Assembly operand types *_STYLE Two constants used to set whether the Z-machine has "time" or "score/moves" on its status line *_ZA Z-machine areas: e.g. PROP_ZA refers to the property values table area *_MV Marker values *_RTE Run-time error numbers *_VR Veneer routines (* is the name of the routine, usually in mixed case) *_DBR Record types in debugging information files ------------------------------------------------------------------------- Set of variables Used for ------------------------------------------------------------------------- *_switch Flag indicating whether a command-line switch such as -s is on or off *_setting Numerical value set by a command-line switch such as -t3 MAX_* A limit on something: note that a few of these are #define'd but most are memory setting variables no_* Number of things of this type made so far max_* Maximum number of things of this type made token_* Three variables used to hold the value, type and lexeme text for the last token read *_trace_level 0 if tracing information is not being printed out about *; otherwise, the larger this is, the more output is produced *_offset Byte offset in the Z-machine, either from the start of this Z-machine area or from the start of Z-machine memory *_top Pointer marking the current end in some uchar array (usually holding a Z-machine area being put together) ------------------------------------------------------------------------- Set of routines Used for ------------------------------------------------------------------------- init_*_vars() Routine in which section * of Inform initialises its variables to begin compilation *_begin_pass() ...and in which it initialises its variables at the start of the source code pass *_allocate_arrays() ...and in which is allocates any memory or arrays it needs to begin compilation *_free_arrays() ...and in which it deallocates any memory or arrays it has allocated, after compilation parse_*() Routine in the syntax analyser to parse the source code construct * assemble_*() Instructing the assembler to generate an instruction: assemble_#() (where # is a number from 0 to 4) an instruction with # operands assemble_#_to() an instruction with #operands which stores a result assemble_#_branch() an instruction with #operands which makes a conditional branch *_linenum() Keeping and writing line references to the debugging information file 1.4 Typedef-named types ------------------- ------------------------------------------------------------------------- Typedef name Where defined Used for ------------------------------------------------------------------------- int32 H signed 32-bit integer uint32 H unsigned 32-bit integer uchar H unsigned char assembly_operand H holding Z-machine numbers in the form used in Z-code, together with linkage information about how they were calculated assembly_instruction H a convenient representation of an instruction of Z-code to assemble opcode asm.c everything about a Z-machine opcode: how many operands it has, whether branch or store, etc. verbl H grammar line of 8 token values verbt H grammar table of grammar lines prop H list of values for a property propt H property values table fpropt H the same, but with attributes too objectt H object tree-position and attributes dict_word H Z-text of a dictionary word dbgl H source code reference used for the debugging file (to file, line, char) keyword_group H the plain text of a group of keywords (such as: all the statement names) token_data H lexical tokens expression_tree_node H node in a parse tree produced by the section "expressp.c" operator H everything about an operator: its name, how to recognise it, its usage and associativity, etc. memory_block H an extensible area of memory (allocated in 8K chunks as required) tlb_s text.c used in abbreviations optimiser optab_s text.c used in abbreviations optimiser FileId H filename and handle for a source file ErrorPosition H filename and line reference for error message printing purposes LexicalBlock lexer.c name, line count, etc. within a block of text being lexed Sourcefile lexer.c buffer, pipeline and lexical block for a source code file being lexed ImportExport linker.c holds import/export records Marker linker.c holds marker records VeneerRoutine veneer.c holds low-level Inform source code for a veneer routine ------------------------------------------------------------------------- "H" is an abbreviation here for "header.h" 2 Porting Inform to a new environment ----------------------------------- 2.1 Dependence on the OS --------------------Strenuous efforts have been made over the last three years to make the sourceas operating-system independent as possible. As a general principle, mostlyadhered to, all operating-system differences should be in the "header.h"file, and not in the 20 sections.As a general rule, for each target OS which Inform is being ported to, a new#define name is invented. For example, the name LINUX is used for the Linuxport. When Inform is being compiled, only that one symbol will be defined,and none of the others: thus#ifdef LINUX ...code...#endifcompiles the given code only for the Linux port.There are some very poor "ANSI C" compilers out there, and many more mediocreones (which almost obey the standard, but don't quite): in any case the ANSIstandard is very broadly defined. For example, the codeint x;x = 45*1007;printf("%d\n", x);is entirely ANSI compliant, but results in different numbers being printedon different machines, due to the fact that ANSI does not specify the rangeof numbers which a variable of type int can safely hold. Since C is so highlyunportable a language, and since some of the compilers used to produce Informare poor, the whole Inform code has to be written with the worst possiblecompiler in mind.An illustration of this is that all preprocessor commands, such as #define,must begin on column 1 of the source code: even when they occur in code whichis #ifdef'd out. VAX C (a particularly bad compiler) will reject#ifndef VAX #define FROG 2#endiffor example, even when VAX is defined. This makes the declarations in"header.h" annoyingly illegible for everybody. 2.2 Portability issues: the types int32 and uchar ---------------------------------------------The main issues when porting Inform have been found to be: (a) the size of "int", (b) whether "char" is unsigned or signed by default, (c) what conventions apply to filenames, (d) assumptions about sizeof() when casting pointers, (e) how parameters (switches, filenames, etc.) are passed to Inform.(a) ANSI requires that "int" be at least 16 bit, though advances in CPU technology mean that under most of today's environments it will in fact be 32 bit. ANSI further requires that "long int" be at least as big as "int", but not that it has to be any bigger. Inform needs at least one integer type to be able to hold 32 bit signed values, and "header.h" contains code which attempts to typedef the name "int32" to such a type. This should happen automatically. Under ANSI rules, as the above says, a compiler need not have any integer type larger than 16 bits: if so, that compiler will not be able to compile Inform. An annoying issue here is that compilers vary widely in the extent to which they give errors or warnings when they detect silent "promotions" from one integer type to another. This makes it very hard to sort out the types of every object in the program between int and int32 so that everybody is happy: in practice, every time a new release of the source code has been made, a few dozen types have had to be fiddled with until everybody can compile it.(b) Compilers seem to divide about fifty-fifty on this. Again, the original standard is vague: the issue is really about how the value (char) 253, say, should be interpreted when it is cast to (int). Should the answer be 253, or -3? ANSI did not specify this because compiler writers wanted to be able to choose whichever could be done instantly on the CPUs they were working with. (If you store a char in the bottom 8 bits of a 32 bit register, then casting the value (char) 253 to (int) -3 means setting all 24 of the upper bits, which requires code to be compiled.) Inform uses a typedef'd type called "uchar" when it needs an unsigned char type: it uses plain "char" when it doesn't mind. It never needs a signed char type. In theory ANSI C compilers must recognise the keywords "signed" and "unsigned", but some don't: typedef unsigned char uchar; actually produces an error on some compilers. So the typedef can only be made with your help. (On other compilers, "unsigned" is legal but "signed" is illegal.) (c) Many people think that the minimal 8.3 convention will work on any operating system, but this is not true (it won't work under Acorn's RISC OS). Much of each OS specification in "header.h" is therefore to do with filenaming.(d) For instance, sizeof(char *) sizeof(int *) sizeof(int32 *) sizeof(int) may all be different numbers on machines with segmented memory maps. This being so, casting between pointer types may lose information, and a few arrays in the source have surprising types to ensure safety. One thing Inform does need to be able to do is to subtract one pointer (of the same type) from another: it defines the macro subtract_pointers(X, Y) to do this. X and Y are normally of type uchar; there seems to have been no problem with this in practice.(e) The ANSI standard is quite good on the command line, and Inform expects to read parameters by the standard argc, argv mechanism. Unfortunately the Macintosh, for instance, has no orthodox command line. Such a port probably wants to have an "outer shell" which displays a window, allows options to be set and then calls the Inform 6 core as needed. The section "inform.c" normally compiles a "main" routine which makes a few machine-dependent changes and then passes its arguments straight on to "sub_main". For instance, here's the v6.10 source: int main(int argc, char **argv) { int rcode; #ifdef MAC_MPW InitCursorCtl((acurHandle)NULL); Show_Cursor(WATCH_CURSOR); #endif rcode = sub_main(argc, argv); #ifdef ARC_THROWBACK throwback_end(); #endif return rcode; } The Macintosh Programmer's Workshop port is making multi-tasking work before embarking on compilation; the Acorn Desktop Debugging Environment port is tidying up after any error throwbacks, at the end of compilation. The point is that here is the place for such minor machine quirks. However, if you want an entirely new front end (such as Robert Pelak's Macintosh port of Inform has), then you need to define #define EXTERNAL_SHELL in your machine definition block (see later). This will mean that no "main" routine is compiled at all from "inform.c" (so you can simply link the Inform source into your own code, which will contain its own "main.c"): Inform should be run by calling extern int sub_main(int argc, char **argv); having set up argc and argv suitably. For instance, the outer shell might take names typed into dialogue boxes, and various ticked options on a window, and make these into a series of ICL commands, which are then handed over textually to sub_main. I suggest that the most efficient way to do this is to write them as an ICL file somewhere and to pass sub_main a single parameter telling it to run this ICL file. 2.3 The character set and the format of text files ----------------------------------------------The Inform source code assumes that the compiler is running on a machinewhose character set agrees with ASCII in the range $20 to $7e. (Thisallows both plain ASCII and any of the ISO 8859 extensions to ASCII.)ASCII is now universal, but there is no common format for plain text files,and in particular how lines of text are ended. For example: MS-DOS, Windows, etc.: $0d $0a Mac OS: $0d RISC OS: $0aInform 6 can read source code files in all these formats, and which furtheruse any of the character sets above: plain ASCII or ISO 8859-1 to -9.(This is configurable using the -C switch.) 2.4 The OS definitions block in "header.h" --------------------------------------Each Inform port makes a block of definitions in the header file. Theseblocks take a standard format.Firstly, the block is put in #ifdef's so that it will only be processedin this one port. The block is divided into 6 sections, as follows. /* 1 */ MACHINE_STRING should be set to the name of the machine or OS. /* 2 */ Section 2 contains some miscellanous options, all of which are on/off: they are by default off unless defined. The possibilities are: USE_TEMPORARY_FILES - use scratch files for workspace, not memory, by default EXTERNAL_SHELL - this port is providing an entire external front end, with its own "main" routine: see above PROMPT_INPUT - prompt input: ignore argc and argv, instead asking for parameters at the keyboard. (I hope people will write front-ends rather than resort to this, but it may be a useful staging post.) TIME_UNAVAILABLE - if the ANSI library routines for working out today's date are not available CHAR_IS_SIGNED - if on your compiler the type "char" is signed by default Note that defining USE_TEMPORARY_FILES does not make a mandatory choice (as it did under Inform 5): whether to use allocated memory or temporary files is selectable with -F0 (files off) or -F1 (files on) in ICL. All that this option does is to define the default setting for this -F switch. Running -F0 is faster (possibly, depending on whether your C library provides buffering or not, much faster) but consumes 100 to 300K more memory (it does so flexibly, allocating only what it needs, unlike the Inform 5 option). Most users will not want to understand the issues involved here, so please make a sensible default choice for them. Once again, note that CHAR_IS_SIGNED must be defined if "char" is signed: otherwise "uchar" will be typedef'd wrongly. /* 3 */ An estimate of the typical amount of memory likely to be free should be given in DEFAULT_MEMORY_SIZE. (This is only a default setting.) There are three settings: HUGE_SIZE, LARGE_SIZE and SMALL_SIZE. (I think it was Andrew Plotkin, though, who remarked that HUGE_SIZE might sensibly be renamed "not-bad-by-1980s-standards-size": these all allocate quite small amounts of memory compared to, say, the 8M of workspace that Windows appears to need just to keep breathing.) For most modern machines, LARGE_SIZE is the appropriate setting, but some older micros may benefit from SMALL_SIZE. /* 4 */ This section specifies the filenaming conventions used by the host OS. It's assumed that the host OS has the concept of subdirectories and has "pathnames", that is, filenames giving a chain of subdirectories divided by the FN_SEP (filename separator) character: e.g. for Unix FN_SEP is defined below as '/' and a typical name is users/graham/jigsaw.z5 Normally the comma ',' character is used to separate pathnames in a list of pathnames, but this can be overridden by defining FN_ALT as some other character. Obviously it should be a character which never occurs in normal pathnames. If FILE_EXTENSIONS is defined then the OS allows "file extensions" of 1 to 3 alphanumeric characters like ".txt" (for text files), ".z5" (for game files), etc., to indicate the file's type (and, crucially, regards the same filename but with different extensions -- e.g., "frog.amp" and "frog.lil" -- as being different names). If FILE_EXTENSIONS is defined, then Inform uses the following standard set of extensions unless they are overridden by other definitions at this point. (Please don't override these definitions without reason.) Source_Extension ".inf" Source code file Include_Extension ".h" Include file (e.g. library file) Code_Extension ".z3" Version 3 story file V4Code_Extension ".z4" 4 V5Code_Extension ".z5" 5 V6Code_Extension ".z6" 6 V7Code_Extension ".z7" 7 V8Code_Extension ".z8" 8 Module_Extension ".m5" Linkable module file (version 5, which is all that Inform 6 supports yet) ICL_Extension ".icl" ICL file The debugging information file and the transcript file also have defined default-names which can be over-ridden in this section if desired: Transcript_File "gametext.txt" or "gametext" Debugging_File "gameinfo.dbg" or "gamedebug" If you do not define FILE_EXTENSIONS, then it is essential to define STANDARD_DIRECTORIES instead. (You can also define both, if you so choose.) The STANDARD_DIRECTORIES option causes Inform to put all files of a particular kind into a standard directory for them: e.g., a "games" directory might hold the story files compiled, etc. All that happens when a standard directory is defined is that Inform sets the default value of the relevant pathname variable to that standard directory: otherwise, its pathname variable starts out as "". The standard directories are, once again, defined by default as follows: once again you can define these settings yourself, but please don't do so without a good reason. Source_Directory "source" Include_Directory "library" Code_Directory "games" Module_Directory "modules" Temporary_Directory "" ICL_Directory "" Note that the actual user of Inform can still override anything you choose by setting the pathname with an ICL command. A good way to test all this is to run inform -h1, which does some experimental filename translations and prints the outcome. /* 5 */ Section 5 contains information on how to choose the filenames for the three temporary files. (Note that this needs to be done even if USE_TEMPORARY_FILES is not defined.) On many machines, you only need to give a suitable name. (As usual, if you don't bother, something fairly sensible happens.) Temporary_Name is the body of a filename to use (if you don't set this, it becomes "Inftemp") and Temporary_Directory is the directory path for the files to go in (which can be altered with an ICL command). However, under some multi-tasking OSs it is desirable for multiple Inform tasks to work simultaneously without clashes, and this means giving the temporary files filenames which include some number uniquely identifying the task which is running. If you want to provide this, define INCLUDE_TASK_ID and provide some code... #define INCLUDE_TASK_ID #ifdef INFORM_FILE static int32 unique_task_id(void) { ...some code returning your task ID... } #endif /* 6 */ Finally, section 6 is "anything else". In particular this is where DEFAULT_ERROR_FORMAT should be set. This switches between different styles of error message. (This is not a matter of aesthetics: some error-throwback debugging tools are very fussy about what format error messages are printed out in.)For example, here is a typical OS definition block:#ifdef UNIX/* 1 */#define MACHINE_STRING "Unix"/* 2 */#define CHAR_IS_SIGNED/* 3 */#define DEFAULT_MEMORY_SIZE LARGE_SIZE/* 4 */#define FN_SEP '/'#define FILE_EXTENSIONS/* 5 */#define Temporary_Directory "/tmp"#define INCLUDE_TASK_ID#ifdef INFORM_FILEstatic int32 unique_task_id(void){ return (int32)getpid();}#endif#endif 2.5 Running Inform in a multi-tasking OS ------------------------------------As mentioned above, if Inform is being used in a multi-tasking environmentthen temporary file-naming will need a little attention.Another issue is that under some systems the other tasks may all freezeup while Inform is working, because tasks only voluntarily hand controlback to the OS (allowing it to poll the other tasks and share out theprocessor time). This means that some call to an OS primitive routinemay have to be inserted into Inform somewhere: a good place to do thisis in the routine reached_new_line() of the section "lexer.c". 3 The front end ------------- 3.1 The ICL interpreter -------------------Inform's front end is quite simple and there is little to say about it.Its task is to translate the user's compilation command into five kinds ofICL variable: switches, which are on/off flags controlling compilation options; switch settings, which are numerical values (between about 1 and 8) controlling more finely-controllable compilation options; path variables, which hold the text of some directory pathname; memory settings, which hold positive numbers (the extent of certain memory allocations within Inform); the filenames of the source code to read, and the object code to write.See "inform.c" and the memory-setting-parser in "memory.c" for the details. 3.2 Fundamental method ------------------It is not possible, in almost any compiled language, to make a directtranslation from source to object code "a line at a time": the first timea line is reached, it often cannot be finally translated until informationis known which cannot yet be known. (For example, Inform translates anobject's name into a number indicating its position in the final game'stree of objects. If the name comes up before the definition of the objecthas been seen, Inform cannot know what number to translate the name into.)Inform makes only one pass through the entire source code. The translationit makes is (roughly speaking) left blank in places, and these blanks arefilled in at the very end, once the necessary information is available.This process is called "backpatching": Inform is patching things up behinditself. 4 Lexical analysis ---------------- 4.1 Aim and structure of the lexer ------------------------------The task of the lexical analyser, or "lexer" for short, is to translate thetext it reads into a sequence of "tokens". The aim is that the rest of thecompiler should work with the tokens and need never look at the original textagain; Inform mainly achieves this aim.Each token represents a sequence of characters, called a "lexeme", which isindivisible from a syntactic point of view. For instance, the text $12+duck.&featherscontains five atomic pieces of syntax: Token Lexeme <the number 18> $12 <the separator +> + <symbol number 1> duck <the separator .&> .& <symbol number 2> feathersThe Inform lexer has three, or perhaps four levels: Level 1: reading in text, filtering out control characters, etc. Level 2: a context-free tokeniser Level 3: a context-dependent process of deciding whether any identifier names found are keywords or names made up by the programmerTo make a fairly revolting analogy: when parsing dinner, lexical analysisis done with the teeth, and syntax analysis with the stomach. But withprogramming languages it is difficult to give a very precise point atwhich one ends and the other begins: and in Inform, the "perhaps level 4"of the lexer occurs when tokens are sorted out more finely in the expressionparser (is this an operator? is it a constant value? is it a variable name?),which will be documented as part of the syntax analyser in this manual.At any rate, "lexer.c" and "symbols.c" contain levels 1 to 3 only.For details of some standard algorithms in compiler theory, the account belowand in section 5 gives page references to "ASU": this is Aho, Sethi andUllman (1986), "Compilers: Principles, Techniques and Tools", the so-called"Dragon book".It's called this because the front cover features a knight armoured invarious algorithms confronting a dragon labelled "Complexity of compilerdesign": in the case of Inform, though, the dragon might reasonably betwice as big and labelled "Ignorance of designer who originally chose thesyntax for Inform". Compiler-making tools like "lex", "yacc" and "bison"are difficult to apply to this language, since Inform doesn't have aconvenient LALR(1) grammar (more like LALR(4), in fact). In any case thelexical and syntax analysers are hand-coded for speed, which is generallyconsidered to produce code twice as fast as that generated by mechanicaltools like "yacc". 4.2 Level 1: lexical blocks and buffered input ------------------------------------------The lexer can take input from two sources: from the source code files whichare being compiled, or from a null-terminated string somewhere in thecompiler. (The latter is used when compiling the "veneer" of run-timeroutines.)A "lexical block" is a piece of text which is individually named andline-counted (so that error messages can indicate where an error has occurredwith reference to the original source). Each single file of source code(the original file and each Include file) is a lexical block in its ownright. Likewise, if the lexer is reading from an internal string, thenthat string is a lexical block.The LexicalBlock structure holds data about the position inside such a block.Note that several may be needed at once: if one source file includes another,which includes another, then three different LexicalBlocks will containuseful information.However, information about the files currently being worked from is storedin a different structure way, and not in LexicalBlock structures. Forefficiency reasons, we can't read characters out of the files one at a time:instead we read them in 4096-byte chunks into a buffer. Note that thismeans that the read-position of a file will be well ahead of theread-position of the lexer within it. It may have been closed before thelexer is halfway through its buffer.A further complication is that it's natural to use a stack structure to holdthe files being read from: include file 2 <-- reading from this include file 1 <-- still open but not being read from main source code file <-- still open but not being read fromand it is also natural to use a stack structure to hold the LexicalBlocksbeing worked through: LB for include 2 LB for include 1 LB for main source code fileDespite the apparent similarity here, we can't combine this into one stack,since "include file 2" will have been pushed off the files stack before itsLB is pushed off the LB stack. Since further Include directives may occurat any point in the current LB, the stacks can be very different.Otherwise level 1 is simple and fast. Note that characters are fed up intolevel 2 through a "pipeline" (see the next section for what this means andwhy), and are also filtered. Inform can read source files in any plainASCII or any of the ISO 8859 standard ASCII extensions (five variants withaccented Latin characters, plus Cyrillic, Arabic, Hebrew, Greek). Thefiltering process removes any character codes undefined in the currentISO setting, and normalises new-line and spacing characters: 00 remains 0 (meaning "end of file") TAB becomes SPACE 0d becomes 0a, i.e., '\n' other control characters become '?' 7f becomes '?' 80 to 9f become '?' a0 (ISO "non-breaking space") becomes SPACE ad (ISO "soft hyphen") becomes '-' any character undefined in ISO between a0 and ff is mapped to '?'(Here "ISO" means "the current ISO setting", which can be 0, meaningplain ASCII only: in this mode any character value of 80 to ff willbe filtered to '?'.)The filtering ensures that (i) no error message can contain an unprintablecharacter and (ii) the user cannot type a character outside a standardISO set and which might work on one platform but not on another.Note that the 0 character is only fed upwards into level 2 when the entirelexical source has run out: that is, all the source files are exhausted,or else the string being analysed has run out. 4.3 Level 2: breaking text into lexemes -----------------------------------The input to level 2, then, is a stream of characters: and its output is astream of tokens, each of which represents a sequence of characters called a"lexeme".Some definitions of terms used in the Inform source code. There is a fixedset of "separators", all of them sequences of 1 to 3 mainly non-alphabeticcharacters: -> --> -- - ++ + * / % || | && & ~~ ~= ~ == = >= > <= < ( ) , .& .# ..& ..# .. . :: : @ ; [ ] { } $ ?~ ? #a$ #n$ #r$ #w$ ## #An "identifier" is a sequence of one or more characters from the set: A B C D ... Z a b c ... z 0 1 2 3 4 5 6 7 8 9 _which does not begin with 0 to 9. The lexer makes the longest lexeme itpossibly can for any particular token, so the next character after anyidentifier will not be one of the set above.The lexemes representing numbers are: (a) one or more chars from the set 0 1 2 3 4 5 6 7 8 9 (b) $ followed by one or more chars from 0 1 2 3 4 5 6 7 8 9 A B C D E F (hexadecimal numbers) (b) $$ followed by one or more chars from 0 1 (binary numbers)Note that "-67" is not a lexeme for a single token: it is broken into twolexemes, "-" and "67".A ", followed by any number of characters and then another ", is a singlelexeme. Similarly for '.Finally, as a special case, the six separators #a$ #n$ #r$ #w$ ## #are expected to be followed by an identifier. For example, "##Take" isa single lexeme, and is not divided into "##" and "Take". (Except that#n$ can be followed by any character, so that e.g. #n$1 is a single token,representing "the new dictionary word '1'" in the language.)To sum up, the lexical analyser expects legal source code to contain: "comments": sequences beginning with a ! character and ending with a new-line or the end of the file or string containing it any of the lexemes above "white space", that is, characters in the set space tab new-lineWhen breaking down text into lexemes (and throwing away the white spaceand comments), the lexer needs 3 characters of look-ahead. That is, it candecide definitely what lexeme character N belongs to provided that it knowswhat characters N+1, N+2 and N+3 are. (The figure 3 occurs because that'sthe worst case arising: deciding whether character N is the start of aseparator in the form "#a$" followed by an identifier character.)Because of this, level 1 maintains a "pipeline" of four variables to hold thecurrent character and the next three on the way. By looking ahead at thepipeline as needed, level 2 decides what to do with the current character andthen advances one character: the three waiting in the pipeline shuffle up anda new character is read into the last place in the pipeline. The advantageof maintaining a pipeline is that the lexer never needs to undo any decisionsand go backward through the text.One token is output for each lexeme found, and when the lexer runs out ofinput altogether (when all the source files are finished, or when the stringbeing analysed has run out) a special token, called "end of file", is output. Thus the lexer's output is a sequence of tokens from the list: numbers strings in " quotes strings in ' quotes separators identifiers end-of-fileAs will be seen in the next section, identifiers are analysed further beforebeing passed out of the lexer.Except for the problem of deciding what an identifier means, the lexer is"context-free": it needs to know nothing about the syntax of the Informlanguage, what kind of code it is currently analysing, etc. There is astandard state-machine algorithm for writing such a lexer: see ASU, p. 98.The tricky part is to efficiently store a transition table for the statesinvolved, which will be neither so small that slow code is required toread it, nor so large that it takes up an unacceptable amount of memory.Here the "tokeniser_grid" stores most of a transition table: this is analgorithm originally suggested to me by Dilip Sequeira, similar to that usedby S. C. Johnson's "yacc" lexical analyser. 4.4 Representation of tokens ------------------------Tokens are internally stored as quadruples: typedef struct token_data_s { char *text; int value; int type; int marker; } token_data;though the lexer is not responsible for writing "marker" values, which arethe concern of the syntax analyser. The "type" identifies which kind oftoken it is (this value must be one of the *_TT set #define'd in "header.h"):for example, DQ_TT means "a double-quoted string". The "value" is anumber whose meaning depends on the type. For example, in the case oftype DQ_TT, the number has no meaning and is not used; in the case of typeSEP_TT (for "separator"), the number gives the index in the above tableof possible separators, thus identifying which separator it is."text" is a pointer to a null-terminated string containing the lexeme. Thereare two reasons this is needed: firstly, so that comprehensible errormessages can be printed out from higher up in the compiler, and secondlybecause in the case of DQ_TT and SQ_TT the text may need to be compiledinto Z-text format and added to the static strings area, to the Z-code areaor to the dictionary. The decision on whether the string should be compiledand if so, what to, cannot be taken until code generation time.Unfortunately code generation time may not be for a while, and this raises aproblem: we clearly need to keep lexeme text in memory for a while, but itwould be very costly on memory to keep the text of every token in memorypermanently. When is it safe for the lexer to throw a lexeme away?One solution would be for a formal system by which the code generatorexplicitly deallocated the space, but this is too bureaucratic and too risky(suppose we miss 0.1% of these? Memory will slowly clog up and begin to causeodd errors on large compilation runs). Instead, the lexer simply writes itstext cyclically into a large buffer. Code generation almost always takesplace before the lexer has got any more than 100 or so bytes of text ahead;since the buffer is a good 10K long, there is a generous safety margin. 4.5 Level 3: identifying identifiers --------------------------------A "keyword" is an identifier which has a meaning understood by the Informcompiler: for example the lexeme whileis understood (in some contexts) to refer to the "while" statement, ratherthan being a name coined by the programmer for some variable or object.In most small languages such a lexeme would always be a keyword and never aname. (This is the concept of "reserved word": a word reserved for the useof the compiler, forbidden as a name.) For example, the fairly-large languageC++ has 48 such keywords.Inform is unable to reserve keywords (that is, forbid their use as names) fortwo reasons: firstly, because there are 281 of the damn things (and 116 ofthem are only significant in lines of assembly language: why forbid the useof these to non-assembly-language programmers?), and secondly because it isimportant to the language that some keywords definitely should be the same assome object names. "Class" is both a keyword meaning "the class-makingdirective" and a name meaning "the object representing the class of allclasses".So the decision "is this identifier intended to be a keyword or intendedto be the name of something?" relies on the context. What the lexer doesis to divide the 281 keywords into twelve groups. The syntax analyserswitches these groups on or off as it finds its way through the program.For example, the group "opcode_names" is only enabled (i.e., switched on)after an @ character has been read at the beginning of a statement.A complication is that case is sensitive for some of these groups and notothers. For example, "IFDEF and "Ifdef" both match the directive name"ifdef", but "WHILE" and "While" do not match the statement name "while".The rules in fact are: When matching... Example Sensitive? Language keywords: directive names Class No keywords used within directives with Yes statement names while Yes keywords used within statements else Yes condition names hasnt Yes built-in function names random Yes built-in constant names #code_offset Yes Assembly language keywords: opcode names put_prop Yes customised opcode names @"1OP:4S" Yes the stack pointer name sp Yes Names for variables, objects, etc. lantern No Names for actions Take NoMoreover, in contexts where local variables are allowed, they takeprecedence over the rest. (For example, you could define a local variablecalled "random", and for the duration of that routine the word "random"would never be recognised as the keyword for the system function random().)Each keyword group has its own token type. The token value for a keywordtoken is the index within the group: see "header.h" for #define's forall of these indices.Finally, there are times when the syntax analyser just wants the raw textof an identifier, and doesn't want it identified (for efficiency reasons:this prevents it from being entered into the symbols table). If thedont_enter_into_symbols_table flag is set, the lexer simply returns atoken of type DQ_TT as though the name had been found in double-quotes. 4.6 Hash-coding and comparing names -------------------------------It would be computationally very expensive indeed to decide whether string Xis a keyword or not by carrying out direct string comparisons with all 281keywords. Instead, hash-coding is used. The idea is to perform a fastoperation on X whose result is some number from, say, 1 to 100: this is thehash code of X. If X is the same string as K, then K must have the same hashcode. So we organise the keywords into 100 different groups, the group withhash code 1, with hash code 2, etc.: then if h is the hash code of X, we onlyneed to compare X with the keywords in group h.For this to be any use, the operation performed has to be chosen so thatthe keywords are sorted out into groups with about equal numbers in each.Many sensible "hash functions" are known (see ASU, p. 437 for experimentalperformance data). Inform's choice is simple but effective: it usesmultiplication rather than bit-shifting, but then on modern CPUsmultiplication is not very expensive. The algorithm is what ASU wouldcall "X 30011": extern int hash_code_from_string(char *p) { unsigned int32 hashcode=0; for (; *p; p++) hashcode=hashcode*30011 + case_conversion_grid[*p]; return (int) (hashcode % HASH_TAB_SIZE); } where "hashcode" is meant to overflow repeatedly. Note that 30011 is a primenumber. (case_conversion_grid[*p] is just a fast way to convert all lowercase letters to upper case ones, since we want case to be insignificant whencalculating hash codes.) It doesn't matter if this algorithm behavesdifferently on different machines (depending on how the CPU copes withoverflows): all that matters is it delivers a reasonable spread of hashvalues.The hash code range is 0 to HASH_TAB_SIZE-1: this is a memory settingwhich by default is 512. Reducing this by any substantial amount causes asignificant slowing down of Inform. 4.7 The symbols table -----------------If an identifier, in context, isn't a keyword then it is called a "symbol":this basically means "a name for something created by the programmer"(except that Inform creates a few symbols automatically as well).The section "symbols.c" keeps a table of all the symbols which have turnedup so far. (It often picks up a few extraneous names of keywords, too,when the lexer has been looking ahead in the wrong context: but this doesn'tmatter and it's faster to tolerate the slight waste of memory.)The symbols table is organised into HASH_TAB_SIZE-1 linked lists, each ofwhich is kept in alphabetical order. A single operation is provided forsearching it: symbol_index(), which works out the hash code of the string Ssupplied (unless it's already known, in which case it need not work it outagain) and looks through the linked list for that hash code until thefirst letter matches. (Thus, the only full string comparisons made arebetween S and those strings in the table which have the same hash codeand the same initial letter.) If S is not present, it is inserted intothe table.In addition to its text, each symbol in the table has four pieces of dataattached: its type, its value, the line on which it was assigned and itsflags. The symbol with index i has: symbs[i] text stypes[i] type char svals[i] value int32 sflags[i] flags word int slines[i] line number int32(It would be more elegant to store this as an array of structures, butthe symbols table is a major consumer of memory, so we can't afford to letlazy compilers waste it by aligning the fields of such a structureat 4 byte intervals. Hence, five separate arrays.)Types and values are assigned by the syntax analyser: the type indicateswhat kind of thing the symbol is a name of (a routine, a label, an object,etc.), and must always be one of the #define names *_T (see "header.h"). Thevalue holds some number indicating which particular thing the symbol is aname of (a routine address, an object number, etc.).The line number is set when the symbol is first used, and then reset when itis first assigned a value (if ever). The information is stored as file-number*0x10000 + line-numberwhere file-number is an index into the InputFiles array maintained by"files.c".The flags word holds (presently) 15 different flags, for different propertiesthat the symbol name can have. These flags have #define names in the form*_SFLAG.The token for a symbol has type SYMBOL_TT and value equal to the index i ofthe symbol within the symbols table. 4.8 Token lookahead: the circle ---------------------------The output of the lexer is sent, one token at a time, when the syntaxanalyser calls get_next_token().The syntax analyser is a predictive parser which makes mistakes and has tobacktrack (that is, it reads in token X, guesses that it might be looking atXY and reads the next token to see... but reads a Z instead, and has to goback to X and try a different theory). So the syntax analyser needs to beable to go backwards, as well as forwards, in the stream of tokens.This means that the lexer needs to remember its last N tokens, where N isthe maximum number of backward steps the syntax analyser ever takes in onego. The number N is the amount of "lookahead", since one could also thinkof it as the maximum tokens one needs to look ahead of the current positionto be sure of what is happening.These tokens are therefore stored in a "circle" of N+1 tokens: whenget_next_token() is called, the lexer works out the next token, writesit into the circle and moves on one place clockwise; when put_token_back() iscalled, the lexer moves back one place anticlockwise, so that the nextget_next_token() will send back the token in that position rather than workout a new one.However, the interpretation of identifiers as keyword or symbol depends onthe current context, and this may have changed since the token was firstcalculated. Therefore a numerical representation of the context is storedin the circle too, so that if this has changed and if the token is anidentifier then it is re-identified under the context which now prevails. 4.9 Summary of the lexer's output -----------------------------To summarise, the lexer converts source text to a stream of tokens of typesas follows: -------------------------------------------------------------------------- Type Lexeme Value -------------------------------------------------------------------------- NUMBER_TT literal number in the number decimal, hex or binary DQ_TT string extracted from (none) double-quotes (or identifier name: see above for when) SQ_TT string extracted from (none) single-quotes SEP_TT separator a #define'd *_SEP value EOF_TT (end of source) (none) SYMBOL_TT identifier assumed index in the symbols to be a name table LOCAL_VARIABLE_TT local variable name 1 to 15, its Z-machine variable number STATEMENT_TT statement name a #defined *_CODE value SEGMENT_MARKER_TT with/has/class etc. a #defined *_SEGMENT value DIRECTIVE_TT directive name a #defined *_CODE value CND_TT named conditional a #defined *_COND value SYSFUN_TT built-in function a #defined *_SYSFUN value OPCODE_NAME_TT opcode name a #defined *_zc value (i.e., an internal opcode number) MISC_KEYWORD_TT keyword like "char" used a #defined *_MK value in statement syntax DIR_KEYWORD_TT keyword like "meta" used a #defined *_DK value in directive syntax TRACE_KEYWORD_TT keyword used in syntax a #defined *_TK value of "trace" directive SYSTEM_CONSTANT_TT system #constant name a #defined *_SC value --------------------------------------------------------------------------Note that other token types do exist, but are set in "level 4": i.e., withinthe expression parser. 5 Syntax analysis --------------- 5.1 Aim and structure of the syntax analyser ----------------------------------------The syntax analyser is the heart of Inform, and contains within it thespecification of the language.Inform code fundamentally consists of directives, which are instructions tothe compiler to do something, and routines containing statements, which arelines of code to be executed at run-time.The syntax analyser takes as input the stream of tokens from the lexer. Someof these are interpreted as directives and acted on immediately. Othersare realised to be statements and compiled; we can regard the output of thecompiling part of the analyser as being a stream of Z-machine assemblylanguage, passed down into "asm.c".In most modern compilers, the syntax analyser would convert the tokenstream into a "parse tree" expressing the structure of the input: somethingalong the lines of routine / \ statement statement | | "if" ...etc., etc. / \ condition statement | | "==" "print" / \ | "x" "0" "Good heavens!"This is aesthetic, and makes possible many optimisations (such as eliminationof common sub-expressions), since the compiler is able to see the wholestructure before beginning to compile it. But it uses up a fair amount ofmemory (admittedly, most compilers only keep the tree for the currentroutine, not the tree for the entire program); and a lot of speed, as thetree is trawled through again and again.Characteristically, Inform instead uses an algorithm which is about 40 yearsold (see ASU, p.181 et seq). It doesn't generate a parse tree for wholeroutines, though it does do so for expressions, assignments and conditions:for example, it would have built the little tree: "==" / \ "x" "0"For higher level-parsing, it works "top-down", making recursive functioncalls which mimic a depth-first traversal of the tree drawn above. That is, routine() calls statement() which reads the token "if" and calls condition() which works out the expression "x == 0" and compiles some code to perform this test and then to skip the next instruction if it's false and calls statement() which reads the token "print" and then the token "Good heavens!" and compiles some code to print this and routine() next calls statement() which reads... etc., etc.Although we are only able to go through the tree once, it's a quick andefficient trip, effectively using the C run-time stack to hold the parsetree structure. The result is also a rather legible piece of sourcecode (as compared, for example, with the output from "yacc"). (The mainreason we can escape from having to make parse trees is that our targetmachine, the Z-machine, has an architecture making register allocationunnecessary: the variables are the registers, and there's no space ortime penalty for use of the stack.)The syntax analyser begins with parse_program() in the section "syntax.c".Because its structure is embodied in the source code, there is relativelylittle to say about it except to refer the reader to that: and to thethe table in section 1.1 above. 5.2 Predictive parsing ------------------Note that tokens can be read from many different places, and code can bepassed out from many different places: whereas the lexer and the assemblereach have a single channel of input and output.As a typical example of the syntax analyser working as a "predictive parser"and having to "backtrack", consider how parse_action() begins to parse throughaction statements like <Take fishknife>; tokenised as < symbol "Take" symbol "fishknife" >When it's called, the first token, "<" has already been read (this is howparse_statement() knew to call parse_action() in the first place).What it does is: "Predict" that it's going to be a << ... >> statement Ask the lexer for a token, expecting it to be another "<" Discover that, unfortunately, it's a symbol, so the prediction was wrong Backtrack to the point where it made the wrong prediction (this actually only means giving the token "Take" back into the lexer again) Now predict the next possibility, that it's a < ... > statement Ask the lexer for a token, expecting it to be a symbol name Discover that this time it is ("Take"), so the prediction was correct ... and so on.The code to do this is more like: get_next_token(); if (token == "<") return parse_double_angle_action(); put_token_back(); return parse_single_angle_action();(The actual code doesn't make these inefficient function calls, and tokencomparison is a bit fiddlier, but this is the idea.)Clearly, the longer such a prediction goes on before it is found to bewrong, the more tokens which the lexer has to take back again. The syntaxof the language determines this number, N: for many languages N = 1 (becausethey've been designed that way) but for Inform N = 5 (because it was definedby an ignorant oaf). (Though, as will appear in the next section, it wouldbe possible to reduce this by implementing the grammar differently: andthere are compensations, such as the relative conciseness of Inform code.) 5.3 The context-free grammar ------------------------Here are the productions which the top-down syntax analyser implements.(Most of the major ones are handled by routines.)"void_expression", "constant", "condition" and "quantity" are left undefined:these are handled by the expression parser according to an operator-precedencegrammar. "vcondition" is a condition whose first token is a VARIABLENAME.Symbols are represented in capitals; NEWNAME means one not so far assigneda value, while CLASSNAME, etc., mean an existing symbol of the given type.Obsolete features which are still supported are omitted.Details of conditional compilation are abbreviated heavily: the notationPROGRAM means "anything at all until the right directive keyword comes upat the same #if... level". ========================================================================= program -> directive ; program directive -> [ NEWNAME routine "Abbreviate" strings "Array" NEWNAME arraytype array "Attribute" NEWNAME "Attribute" NEWNAME "alias" ATTRIBUTENAME "Class" NEWNAME objbody "Class" NEWNAME ( constant ) objbody "Constant" NEWNAME "Constant" NEWNAME constant "Constant" NEWNAME = constant "Default" NEWNAME constant "End" "Extend" extension "Fake_action" NEWNAME "Global" NEWNAME "Global" NEWNAME = constant "Ifdef" NAME condcomp "Ifndef" NAME condcomp "Iftrue" constant condcomp "Iffalse" constant condcomp "Ifv3" constant condcomp "Ifv5" constant condcomp "Import" manifest "Include" STRING "Link" STRING "Lowstring" NEWNAME STRING "Message" diagnostic "Nearby" objhead objbody "Object" arrows objhead objbody "Property" NEWNAME propdef "Release" quantity "Replace" ROUTINENAME "Serial" STRING "Statusline" "score" "Statusline" "time" "Stub" NAME quantity "Switches" TEXT <---- An unquoted string "System_file" "Trace" trace "Verb" verb "Zcharacter" zchar CLASSNAME arrows objhead objbody condcomp -> ; PROGRAM ; "Endif" ; PROGRAM ; "Ifnot" ; PROGRAM ; "Endif" manifest -> import "," manifest import import -> "global" NEWNAME diagnostic -> STRING "error" STRING "fatalerror" STRING "warning" STRING propdef -> propdefault "additive" propdefault propdefault -> <empty> quantity ========================================================================= trace -> t_section t_level Tracing t_section -> <empty> "objects" "symbols" "verbs" "assembly" "expressions" "lines" t_level -> <empty> "on" "off" quantity ========================================================================= zchar -> char_spec Character set STRING STRING STRING "table" char_specs "table" "+" char_specs char_specs -> char_spec char_specs char_spec char_spec -> LITERAL_CHARACTER ========================================================================= arrows -> "->" arrows Object definition <empty> objhead -> <empty> NEWNAME STRING NEWNAME STRING NEWNAME OBJECTNAME STRING OBJECTNAME NEWNAME STRING OBJECTNAME objbody -> segment objbody segment "," objbody segment -> "class" class_s "with" with_s "private" with_s "has" has_s class_s -> CLASSNAME class_s with_s -> PROPERTYNAME property PROPERTYNAME property "," with_s has_s -> attributes property -> <empty> [ routine arrayvals ========================================================================= arraytype -> -> Arrays --> "string" "table" array -> constant STRING arrayvals [ manyvals ] manyvals -> constant manyvals constant ; manyvals arrayvals -> constant arrayvals ========================================================================= extension -> "only" strings priority grammar Grammar STRING priority grammar priority -> <empty> "replace" "first" "last" verb -> strings v_setting "meta" strings v_setting v_setting -> = STRING grammar grammar -> g_line grammar g_line -> * tokens -> g_action g_action -> ACTIONNAME ACTIONNAME "reverse" tokens -> g_token tokens g_token -> preposition "noun" "held" "multi" "multiheld" "multiexcept" "multiinside" "creature" "special" "number" "topic" "noun" = ROUTINENAME "scope" = ROUTINENAME ATTRIBUTENAME ROUTINENAME preposition -> DICTIONARYWORD DICTIONARYWORD / preposition strings -> STRING strings attributes -> attribute attributes attribute -> ATTRIBUTENAME ~ ATTRIBUTENAME ========================================================================= routine -> * locals ; body ] Everything below locals ; body ] here is code to locals -> NAME locals compile <empty> body -> action_case : body "default" : body statement body action_case -> ACTIONNAME ACTIONNAME , action_case block -> ; statement ; { states } { states . NEWNAME ; } states -> statement states; s_block -> { s_states } { s_states . NEWNAME ; } s_states -> case : s_states "default" : s_states statement s_states case -> range : range , case range -> constant constant "to" constant ========================================================================= statement -> . NEWNAME ; statement "@" assembly ; "<" action ">" ; "<<" action ">>" ; indiv_state STRING void_expression action -> ACTIONNAME ACTIONNAME quantity ACTIONNAME quantity quantity assembly -> opcode operands options opcode -> OPCODENAME STRING <--- customised opcode operands -> operand operands descriptions are operand -> constant not parsed by the VARIABLENAME syntax analyser sp options -> ? branch -> store -> store ? branch store -> VARIABLENAME sp branch -> LABELNAME ~ LABELNAME ========================================================================= indiv_state -> "box" strings ; "break" ; "continue" ; "do" eblock "until" ( condition ) ; "font" "on" ; "font" "off" ; "for" ( for1 : for2 : for3 ) block ; "give" quantity attributes ; "if" ( condition ) block "if" ( condition ) block "else" block "inversion" ; "jump" LABELNAME ; "move" quantity "to" quantity ; "new_line" ; "objectloop" ( ospec ) block ; "print" printlist ; "print_ret" printlist ; "quit" ; "read" quantity quantity ; "read" quantity quantity ROUTINENAME ; "remove" quantity ; "restore" LABELNAME ; "return" ; "return" quantity ; "rtrue" ; "rfalse" ; "save" LABELNAME ; "spaces" quantity ; "string" quantity STRING ; "style" textstyle ; "switch" ( quantity ) s_block "while" ( condition ) block for1 -> void_expression <empty> for2 -> condition <empty> for3 -> void_expression <empty> ospec -> VARIABLENAME VARIABLENAME "in" quantity VARIABLENAME "near" quantity VARIABLENAME "from" quantity vcondition printlist -> printitem , printlist printitem printitem -> STRING quantity ( form ) quantity form -> ROUTINENAME "number" "char" "address" "string" "The" "the" "a" "an" "name" "object" "identifier" textstyle -> "roman" "reverse" "bold" "underline" "fixed" =========================================================================A few critical comments on the above.From this grammar it's possible to work out N, the maximum look-ahead neededto distinguish which production is being used in the source code. The worstcases are: (a) distinguishing productions (1) and (2) in ospec -> VARIABLENAME VARIABLENAME "in" quantity (1) VARIABLENAME "near" quantity VARIABLENAME "from" quantity vcondition (2) which requires N = 5; these two productions need to be distinguished between because objectloop ( a in b ) objectloop ( a in b ... ) (the second containing a compound condition) compile quite different code: one loops through the siblings in the object tree, the other through the tree in numerical order. (b) distinguishing productions (1) and (2) in printitem -> STRING quantity (1) ( form ) quantity (2) i.e., between print (routine-name) expression print (expression which happens to begin with a bracket) which requires N = 3.The grammar contains one serious ambiguity: the innocent-looking production arrayvals -> constant arrayvalsmeans that array initialisations list constants in sequence without commasor other separating characters. This makes it impossible to distinguishbetween unary and binary minus in a line like: Array X 2 - 1 ; The answer is "binary", since the grammar makes the longest match possible;but a "this is ambiguous" warning is issued.A further inconvenience in the grammar, though not much of an ambiguity,occurs in the initialisation part of "for" statements: there is a danger of for (p=Class::)being mis-parsed due to "::" being recognised as a binary operator (withouta right operand, which would cause an error) and not as two consecutive":" delimiters.If I were free to redesign the Inform grammar in the light of the lastthree years' experience (which I am loath to do, since so much Inform sourcecode now exists), here are the changes I think I would make: introduce commas as separators between array values and <<Action ...>> parameters; remove the statements quit, restore, save, style, font, spaces, box, inversion and read: the first three ought to be used via the assembler anyway, and the last six ought to be system-provided functions; use single quotes to refer to dictionary words used as values of "name", thus removing an anomalous rule going back to Inform 1, and to refer to dictionary words in grammar; find some notation for literal characters which does not look like the notation for dictionary words: e.g., ''z'' rather than 'z'; abolish the distinction between actions and fake actions; rename the Global directive "Variable"; require an = sign to be used in "Constant X 24" directives. 5.4 Assigning values to symbols ---------------------------Assigning values to symbols is the main way by which the syntax analyserchanges the way it will behave further on in the program. From a stricttheoretical point of view one could insist the symbols table contains theonly information remembered by the syntax analyser about the program so far,which otherwise churns out code which it forgets as soon as it has written:however, Inform's analyser also keeps a number of other data structures,such as the game dictionary and the object tree.When the lexer creates a new symbol (that is, when the table is searchedfor a string which isn't there, so that it is added to the table as a newentry), it has: value 256 type CONSTANT_T flags only UNKNOWN_SFLAG line the current source line"Unknown" means that the syntax analyser doesn't recognise it as meaninganything yet. The flag will only be cleared when the syntax analyserassigns some value to the symbol (using the assign_symbol() routine), if ever.The line is reset to the source line then applying if the symbol is assigneda value, but is otherwise never changed.The type and value of a symbol are only altered via assign_symbol(). Withone exception (see CHANGE_SFLAG below), the value once assigned is neversubsequently altered by reassignment. Each symbol always has one of thefollowing 11 types: Type Meaning Value ------------------------------------------------------------------------ CONSTANT_T Defined constant or Value of constant value not known yet LABEL_T Name of label in source Label number GLOBAL_VARIABLE_T Name of global variable Z-machine variable number ARRAY_T Name of array Z-machine byte offset in dynamic variables area ROUTINE_T Name of routine Scaled offset in Z-code ATTRIBUTE_T Name of attribute Z-machine attribute number PROPERTY_T Name of common property Z-machine property number between 1 and 63 INDIVIDUAL_PROPERTY_T Name of indiv property Property identifier >= 64 OBJECT_T Name of object Z-machine object number - 1 CLASS_T Name of class Z-machine object number - 1 of its class-object FAKE_ACTION_T Name of fake action Action number >= 256 ------------------------------------------------------------------------The full list of symbol flags, and their meanings, is as follows: ------------------------------------------------------------------------ UNKNOWN_SFLAG no value has been assigned to this symbol USED_SFLAG the value of this has been used in an expression REPLACE_SFLAG the programmer has asked to Replace a routine with this name DEFCON_SFLAG this constant was defined by the "Default" directive STUB_SFLAG this routine was defined by the "Stub" directive (similarly) INSF_SFLAG this symbol was originally assigned a value in a system_file SYSTEM_SFLAG this symbol was assigned a value by Inform itself, as one of the standard stock (such as "true" or "recreate") provided to all programs UERROR_SFLAG a "No such constant as this" error has already been issued for this name ALIASED_SFLAG this is an attribute or property name whose value is shared with another attribute or property name CHANGE_SFLAG this is a defined Constant whose value needs later backpatching, or is a Label whose label number has been allocated before its declaration in the code ACTION_SFLAG this is a ## name (of an action or a fake action) REDEFINABLE_SFLAG the symbol can be defined more than once using the "Constant" directive, without errors being produced. (Used for the special constants DEBUG, USE_MODULES, MODULE_MODE and Grammar__Version.) ------------------------------------------------------------------------ IMPORT_SFLAG this name is "imported" (used in module compilation) EXPORT_SFLAG this name was "exported" from some module (used in story file compilation) ------------------------------------------------------------------------USED_SFLAG is used only to decide whether or not to issue "declared but notused" warnings. A symbol has been "used" if one of the following is true: (i) its value occurred in an expression; (ii) it has been assigned, and tested with IFDEF or IFNDEF; (iii) it's an action routine name used in grammar or via ## (e.g. use of ##Take causes TakeSub to be "used"); (iv) it's a fake action name used via ##; (v) it's a property, attribute or class name used in an object or class definition; (vi) it's a label name branched to in assembly language or the destination of a "jump" statement; (vii) it's the routine name "Main"; (viii) it's a routine name referred to by the obsolete #r$ construct; (ix) it's a routine name of a veneer routine whose definition is being over-ridden in the source code, and to which a call has been compiled; (x) it's a routine name used in a grammar token; (xi) it's referred to in a module which has been linked into the story file being compiled.Note that such warnings are not issued for object names, since in muchInform 5 code objects are given irrelevant object symbol-names (the syntaxrequired it); nor for symbols defined by Inform, or in the veneer, or ina system file. Warnings are never issued (except for labels and localvariables, which have local scope) when compiling modules, since there isno way of knowing at compile time which exported symbols will be used andwhich will not.The warnings are issued at the end of the source code pass, but before theveneer is compiled. The slines[] array is used to work out which sourcelines to report from.CHANGE_SFLAG is used when definitions such as: Constant frog_word = 'frog';are reached, where the value (the dictionary address for 'frog') cannotimmediately be known. Such symbol values are backpatched later as needed.All symbols in the table have "global scope" (that is, their definitions arevalid everywhere in the source program) except for label names, whose scopeis restricted to the current routine. Thus, the same label name can be usedin many different routines, referring to a different label in each.To achieve this, the routine-end routine in "asm.c" cancels the assignmentof label names, returning them to type CONSTANT_T and flag UNKNOWN_SFLAG.Local variables also have local scope, but for efficiency reasons these arenot stored in the symbols table but in a special hash table in level 3 ofthe lexer.Note that action names have a different "name-space" from other symbols:this is why the library's action name "Score" is never confused with itsvariable "score". Rather than use a different symbols table altogether,actions are stored as integer constants in the form Score__A(thus the value of this symbol is the value ##Score). Similarly, fakeactions are stored this way (but with type FAKE_ACTION_T rather thanINTEGER_CONSTANT_T); both forms of symbol are flagged ACTION_SFLAG. 5.5 Outputs other than assembly language ------------------------------------Below the parse_routine() part of the syntax analyser, the only output isassembly language (together with dictionary words and encoded text as neededwithin it).However, parse_directive() has several other outputs, as shown on the sourcecode map (section 1.2 above). Directives create objects to be remembered andwritten into the final program: arrays, verb definitions, actions, objectsand so on. These will be called "program elements".Directives also "create" constants, attributes, properties and fake actions,but in these cases creation only consists of assigning a suitable value to asymbol. So these do not count as program elements.The data structures to hold "program elements" are all created and maintainedwithin "directs.c" and its subsidiaries (such as "objects.c", theobject-maker); they are then translated into a usable Z-machine form in"tables.c". (With only trivial exceptions, the data structures are notaccessed anywhere else in Inform.) --------------------------------------------------------------- Program element Section Name of (main) data structure --------------------------------------------------------------- Global variable arrays.c global_initial_value[] Array arrays.c dynamic_array_area[] Release number directs.c release_number Serial code directs.c serial_code_buffer[] Statusline flag directs.c statusline_flag Object tree objects.c objects[] Common property objects.c prop_default_value[] default value Class-to-object- objects.c class_object_numbers[] numbers table Common property objects.c *properties_table values for objects Individual prop objects.c *individuals_table values for objects Table of static symbols.c individual_name_strings[] string values for property names And attribute names symbols.c attribute_name_strings[] And action names symbols.c action_name_strings[] Abbreviation text.c abbreviations_at[] table entry Grammar table verbs.c Inform_verbs[] Token-routine verbs.c grammar_token_routine[] addresses List of dict verbs.c adjectives[] addresses for "adjective" words Action routine verbs.c action_byte_offset[] addresses ---------------------------------------------------------------A rather unequal gathering: the storage required to hold the common propertyvalues table may be as much as 32K, whereas the statusline flag can be heldin 1 bit.There are three other program elements: Z-code, kept by "asm.c"; the staticstrings of Z-encoded text, kept by "text.c"; and the dictionary, also kept by"text.c". 5.6 Assembly operands -----------------The type "assembly_operand" is used for all numerical values (and local orglobal variable references) which are destined one day to be written into theZ-machine. Many of these will indeed be operands in Z-code instructions,hence the name. Others will be property values, array entries and so on.The definition is: { int type; int32 value; int marker; }which is a pretty generous memory allocation for holding a signed 16-bitnumber. (However, there are no large arrays of this type; type and markercould easily each have type char, but I suspect this would be slower becauseof alignment problems on some compilers; and speed does matter here.)The possible types are: SHORT_CONSTANT_OT number with value between 0 and 255 LONG_CONSTANT_OT number with any 16 bit value VARIABLE_OT reference to the stack pointer if value is 0 local variable if value 1 to 15 global variable if value 16 to 255In addition, two special types are in use by the expression parser: OMITTED_OT (the operand holds no information) EXPRESSION_OT reference to a parse treeThe "marker" value is used to record the origin of the data, which isessential to make backpatching work. For example, in the line v = "Jackdaws love my big sphinx of quartz";the right operand is marked with STRING_MV, because the value which needsto be stored in the Z-machine is a packed string address and this cannot beknown until after the compilation pass (when the size of the tables andthe code area are known). Wherever the operand is written, in code or ina table, "bpatch.c" will later find and correct it. 5.7 Translation to assembly language --------------------------------The Z-machine has a very easy architecture to generate code for. Mostcompilers' syntax analysers generate a "three-address code" as anintermediate stage towards final object code; this is a sequence ofinstructions in the form x = y <operator> ztogether with conditional branches and labelled points to branch to. (SeeASU, p.466.) Translating this code into assembly language for some CPUis a further compilation phase: the tricky part is not translating theoperators into instructions, but deciding where to locate the valuesx, y and z. On most CPUs, a limited number of registers can hold values, andarithmetic operations can only be performed on these registers; moreover,holding data in a register is much more efficient than holding it elsewhere.The problem is therefore to allocate registers to quantities, and theperformance of the compiled code depends very heavily on how well this isdone. (Register allocation is far from being a solved problem in the waythat lexical analysis, say, is considered to be.)What makes the Z-machine particularly easy is that its instruction set ismore or less three-address code already. Arithmetic can be performed withconstants as operands as well as with "registers"; and not only is everylocal or global variable automatically allocated to a register of its own,but a stack is available to hold intermediate values (and with no performancepenalty for its use, since it is accessible as though it were a register).The key point to remember when looking at Z-code is that writing a value tothe "stack-pointer variable" pushes it onto the Z-machine stack; using the"stack-pointer variable" as the operand for an instruction pulls the valuebeing read off the stack.Despite the availability of the stack, it's still convenient to have a fewspare variables to use as "registers" holding temporary values. Informreserves the variables 249, 250, 251, ..., 255 for its own use: though thisslightly exaggerates the position, since two of these are used for thevariables "self" and "sender". One more is used to hold the switch-valueof the current switch statement (if there is one); the remaining four inorder to implement various system functions (such as "children") withinline-code rather than routine calls to the veneer. (The one inconveniencewith the Z-machine's instruction set is that there's no good way to read thetop of the stack non-destructively, or to duplicate the top value, or toreorder the top few values.)The syntax analyser produces code by making function calls to the assembler.There are four types of function call: assemble_routine_header(...) assemble_routine_end(...)at the start and end of every routine; assemble_label_no(N)to indicate that the Nth label belongs here (i.e., at the point where thenext instruction will be put); and then a fair variety of function callsto generate actual instructions. For example, assemble_jump(N)assembles an unconditional jump to label N. A typical "three-address code"is assembled by a routine like assemble_2_to(mul_zc, AO1, AO2, stack_pointer) meaning "the instruction mul_zc, which has 2 operands and has a result whichit writes to the variable indicated by the third operand". AO1 and AO2are assembly_operands (the abbreviation is often used in the source), andso is stack_pointer (it has type VARIABLE_OT and value 0).A typical example of how the top-down syntax analyser generates code isgiven by the code for the "while" statement: case WHILE_CODE: assemble_label_no(ln = next_label++); match_open_bracket(); code_generate(parse_expression(CONDITION_CONTEXT), CONDITION_CONTEXT, ln2 = next_label++); match_close_bracket(); parse_code_block(ln2, ln, 0); assemble_jump(ln); assemble_label_no(ln2); return;Note that this expects to match while ( condition ) statement or { statements }The expression parser is called to turn the condition into a parse tree,and its output is fed straight into the code generator for parse trees.The label numbers ln2 and ln are supplied to the routine for parsing codeblocks because they indicate which labels the statements "break" and"continue" should generate jumps to.For example, while (i <= 10) print i++;generates the assembly language .L0; @jg i 10 ?L1; @inc i; @print_num i; @jump L0; .L1; In terms of function calls to "asm.c": assemble_label_no(0); assemble_2_branch(jg_zc, AO_for_i, AO_for_10, 1, TRUE); assemble_inc(AO_for_i); assemble_1(print_num_zc, AO_for_i); assemble_jump(0); assemble_label_no(1);(Note that incrementing is needed so often that assemble_inc() is providedas a shorthand: actually also because of a form of indirect addressing usedby the Z-machine for store_zc and inc_zc, dec_zc. assemble_store() andassemble_dec() are also provided.) 5.8 Summary of assembly language instructions output ------------------------------------------------The list of Z-machine instructions which Inform actually generates code forby itself is quite short (about half of the 115 possible instructions areever used). The assembly-language "@ statement" parser can, of course,generate every Z-machine instruction.The instructions used for three-address-style code (that is, for programcontrol, arithmetic, etc.) are as follows. For function calls and returns,unconditional jumps and for moving values between variables, memory locationsand the stack: call_* rfalse rtrue ret_popped ret inc dec store push pull storeb storew loadb loadw jump set_attr(There are various forms of call instruction to do with the number ofoperands, whether a return value is wanted or not, etc.). Six conditionalbranch instructions are used: je jz jl jg jin test_attr check_no_args(the first four are numerical branches, the next two related to the objecttree, and the last one is used only in V5 or later games, and then onlyfor printing out tracing code): note that each can also be used in a"negate this condition" form. Finally, signed 16-bit arithmetic andunsigned 16-bit bitwise operations: add sub mul div mod and or notA further batch of instructions is used, so to speak, in lieu of calls to astandard library of input/output and utility routines (like C's "stdio"): print print_ret print_char print_num print_addr print_paddr print_obj new_line insert_obj remove_obj get_parent get_child get_sibling get_prop get_prop_addr get_prop_len put_prop random aread/sread quit save restore output_streamwith five more exotic screen control instructions used only if a "box"statement is ever compiled: split_window set_window style set_cursor buffer_mode 6 Syntax analysis 2: the bottom-up expression parser -------------------------------------------------- 6.1 Map and structure -----------------It would be possible to continue the top-down parser down into the level ofexpressions: for instance, to have a routine parse_plus() which wouldparse_expression(), then match a plus sign, then parse_expression() again,and so on. This would however be highly inefficient in terms of functioncalls, difficult to recover from when an error occurs, and require eithermuch greater lookahead or rather complicated mechanisms for storing parsetrees.Expressions (including assignments, conditions and constant values) aretherefore parsed by a different syntax analyser, which is "bottom up" (itbegins by reading in the tokens, and only works out what they mean afterward)rather than "top down" (starting from an assumption about the meaning andthen trying to find the evidence to justify it).The algorithm used is an operator precedence parser, a simple form ofshift-reduce parser. There is alas no space to describe this beautifulalgorithm here: see ASU, pp.195-215. In this account we treat it as a"black box".The expression parser "expressp.c" has the structure: --------> "Level 4" of --------> Operator tokens lexical analysis etokens precedence in parser | | etokens re-ordered into RPN \|/ Emitter | | plain parse tree \|/ Lvalue and condition checking | | more detailed parse tree \|/ out(This is effectively a magnification of the part of the source map insection 1.2 above which reads just "expressp.c".) Note that there is asingle input and a single output channel.RPN is "reverse Polish notation": for example (2 + 4)*45 + 34 becomes 2 4 + 45 * 34 +the idea being that one reads it left to right: each number is added toa stack of numbers, and each operation takes some numbers off the stackand puts an answer back. (E.g., + takes 2 and 4 off and replaces themwith 6. * takes 6 and 45 off, replacing them with 270. + takes 34 and 270off, replacing them with 304, which is the answer.) The advantage ofthis is that is unambiguous which operator acts on which operands, and thebrackets have gone.The emitter builds a (plain) parse tree out of this sequence, as follows.It stacks 2 and then 4: Stack: 2 4It reads +, which it knows has two operands, and takes the top two itemsoff the stack, joining them into a tree segment like so: + / \ 2 4It then makes a special value, say E1, meaning "the sub-expression in thistree", and stacks that: Stack: E1It then stacks up 45, and reaches *: it takes off the top two values, whichare E1 and 45, and makes * / \ E1 45which it calls E2, and so on. When the process is finished, we have thetree: + <--- this is E3 / \ this is E2 ---> * 34 / \ this is E1 ---> + 45 / \ 2 4and the stack contains just E3, which is the "answer". 6.2 The operator precedence grammar -------------------------------A list of productions for this grammar would be tiresomely long. It canbe roughly summarised as: expression -> ( expression ) a single operator acting on some operands in some way operand -> token representing a constant ( expression ) expression ( arguments ) arguments -> <empty> expression , argumentsThe tokens representing a constant are given in the next section. Theoperators have tokens as follows: -------------------------------------------------------------------------- Level Lexeme Usage Associativity Purpose -------------------------------------------------------------------------- 0 , binary left separating values to work out 1 = binary right set equal to 2 && binary left logical AND 2 || binary left logical OR 2 ~~ unary (prefix) logical NOT 3 == binary none equal to? 3 ~= binary none not equal to? 3 > binary none greater than? 3 >= binary none greater than or equal to? 3 < binary none less than? 3 <= binary none less than or equal to? 3 has binary none object has this attribute? 3 hasnt binary none object hasn't this attribute? 3 in binary none first obj a child of second? 3 notin binary none first obj not a child of second? 3 ofclass binary none obj inherits from class? 3 provides binary none obj provides this property? 4 or binary left separating alternative values 5 + binary left 16-bit signed addition 5 - binary left 16-bit signed subtraction 6 * binary left 16-bit signed multiplication 6 / binary left 16-bit signed integer division 6 % binary left 16-bit signed remainder 6 & binary left bitwise AND 6 | binary left bitwise OR 6 ~ unary (prefix) bitwise NOT 7 -> binary left byte array entry 7 --> binary left word array entry 8 - unary (prefix) 16-bit (signed!) negation 9 ++ unary (pre/postfix) read/increment or increment/read 9 -- unary (pre/postfix) read/decrement or decrement/read 10 .& binary left property address 10 .# binary left property length 10 ..& (**) binary left individual property address 10 ..# (**) binary left individual property length 11/14 ( ) binary left function call/message send 12 . binary left property value 12 .. (**) binary left individual property value 13 :: binary left "superclass" operator -------------------------------------------------------------------------- (**): Illegal except internally, in Inform's own source for the run-time veneer --------------------------------------------------------------------------Note that, unlike C, Inform does not provide unary plus.A binary infix operator is used in the form "X op Y"; a unary prefix operatorin the form "op X"; a unary postfix operator in the form "X op".The "level" is the precedence level: the higher this number, the more tightlyan operator is glued to the operands adjacent to it.Cases where two binary infix operators have equal precedence are resolvedaccording to whether that level is left or right associative. For instance, a / b / c means (a/b)/csince /, on level 6, is left associative. But a = b = c means a = (b = c)since =, on level 1, is right associative.Cases in the above table where the associativity is "none" mean that anattempt to silently associate the operator will cause an error. For example, a == b == cgenerates an error as being ambiguous, but (a == b) == c and a == (b == c)are both permitted.The function-call-brackets have an asymmetric precedence level accordingto whether they're being compared with something on the left or on the right:the point is that object.message(...) function_returning_an_object(...).propertymust be recognised as (object.message)(...) . 12 > (...) 11 (function_returning_an_object(...)).property (...) 14 > . 12respectively. 6.3 Lexical level 4: tokens to etokens ----------------------------------The shift-reduce parser expects tokens sorted out into operators, such as"+" or "ofclass", operands such as "34" or "score" (assuming that's avariable) and three tokens of special significance: ( ) end-of-expression"Level 4 of the lexer" does just this. It converts the stream output from"lexer.c" (i.e., from level 3) into a stream of tokens from the followingtable (and it uses a good deal of context information to do so). Token type Meaning ------------------------------------------------------------------------- OP_TT Operator: value is a *_OP constant LARGE_NUMBER_TT Operand (number): value is the number SMALL_NUMBER_TT Operand (number guaranteed in the range 0 to 255): value is the number VARIABLE_TT Operand: value is Z-machine variable number DQ_TT Operand (text used as static string): text in text DICTWORD_TT Operand (text used as dictionary word): text in text ACTION_TT Operand (##action number): name of action in text SYSFUN_TT Operand (system fn name): value is a *_SYSF constant SYSTEM_CONSTANT_TT Operand (#system_constant): value is a *_SC constant SUBOPEN_TT Open bracket used to open a sub-expression SUBCLOSE_TT Close bracket used to close a sub-expression ENDEXP_TT End of expression -------------------------------------------------------------------------The tokens in this output stream are called "etokens", short for "expressiontokens": they are the raw material from which parse trees are built up.(Although formally different from ordinary tokens, they have the same storagetype, "token_data", in the Inform source code.)At first sight this seems an unnecessarily large stock: why not, for example,compile static strings and dictionary words and convert them to theiraddresses here and now? Why not evaluate system constants and action numbersnow? Because: (i) these tokens may not be accepted by the parser, so wecan't compile text on the strength of them (and indeed, the parse tree whichis being parsed may not ever be code-generated from); and (ii) we can'tallow ourselves to lose sight of where numbers are coming from so soon,or backpatching will be impossible.The presence of two different tokens for numbers - one for numbers guaranteedto be unsigned 8 bit, one for numbers with no such guarantee and whose valuemay be any signed 16 bit quantity - is due to the architecture of the targetZ-machine. In Z-code considerable space savings are made by storing smallnumbers in 1, rather than 2, bytes. We are anxious to take advantage ofthis.Unfortunately, what seems to be a small number now may not be a small numberlater (after backpatching). Dictionary word values (such as 'word') areinternally stored as the accession number of that word into the dictionary(say 23, if it is the 23rd different word to be entered) during thecompilation pass, and then backpatched later to the byte address of thisword's dictionary entry: but this will be a large number.Most operands with non-NULL backpatch markers are forced into long form, to beon the safe side. (This includes all operands which are forward referencesto constants not yet defined.) The exceptions are that variable and actionnumbers (though not fake action numbers) are guaranteed always to fit intoshort form.There are two tasks involved in translating tokens to etokens: evaluatingtokens representing quantities into one of the "operand" etokens, and sortingother tokens into the right operator and special etokens. The second processis not as easy as it looks because of ambiguities, and is discussed in thenext section. This section discusses the evaluation of quantity tokens.Tokens in expressions are read in a context where the keywords are localvariables, textual operators, system function names and system constantnames. Other identifiers are symbol names. So the token types which mustrepresent quantities are: DQ_TT, SQ_TT, LOCAL_VARIABLE_TT, NUMBER_TT, SYMBOL_TT, SYSFUN_TT and SYSTEM_CONSTANT_TT.(Actually, this omits a detail: the largely obsolete constant forms #a$...,#r$..., #n$... are SEP_TT tokens converted into the above; and the very muchstill with us constant form ##... is a SEP_TT token convertedstraightforwardly into the ACTION_TT etoken.)These all translate directly into etokens except for SYMBOL_TT and SQ_TT.SQ_TT requires a little work because of the Inform syntax for single-quotedstrings: 'x' means the character code (i.e. ASCII value) for "x" '@ss' means the character code for the German letter "sz" (i.e. the code specified in the accented set in the Z-Machine standard document) 'xyzzy' means the byte address of the dictionary word "xyzzy"The first two become SMALL_NUMBER_TT etokens. The third becomes aDICTWORD_TT etoken.This leaves SYMBOL_TT tokens: names for constants. There are twopossibilities: the symbol is so far unknown, or else it has been assigneda value in earlier syntax analysis.In most languages it would be an error to use an unknown symbol name asa value: in C, for example, a function name cannot be used in an expressionunless it has been declared beforehand.Inform is more tolerant: as far as expressions are concerned, any symbol canbe used earlier than the point in the source code where its value isassigned, excepting only a local or global variable name. (This exceptionis mainly for Z-machine architecture reasons, but it also makes Inform codemore legible.)When an unknown symbol is reached, it is converted to a LARGE_NUMBER_TT andmarked as SYMBOL_MV, with the value being its index in the symbol table.(This allows the backpatcher to know where to find the true value later.)When a known symbol is reached which is flagged as CHANGE_SFLAG, this istreated in the same way. (CHANGE_SFLAG is given to symbols defined byConstant definitions whose values themselves require backpatching: forinstance, symbols defined equal to dictionary words or routine addresses.)Otherwise, the symbol is not only known, but its current value is known tobe correct, and so: if it's a variable, it's translated to etoken VARIABLE_TT; otherwise the LARGE_NUMBER_TT/SMALL_NUMBER_TT decision is now made: if it is marked with a marker other than VARIABLE_MV, it becomes "long"; otherwise the value is compared with the range 0 to 255.Any symbol name fed into the shift-reduce parser is flagged with USED_SFLAGto indicate that its value has at some point been used in an expression.(This information enables "This ... was declared but not used" warnings to bemade at the end of compilation.) 6.4 Resolution of ambiguous tokens ------------------------------It remains to translate to the etokens OP_TT, SUBOPEN_TT, SUBCLOSE_TT andENDEXP_TT. The token types which represent operators are all from SEP_TT andCOND_TT; open and close bracket are also SEP_TT tokens.There are two main difficulties here. Firstly, shift-reduce parsers areunsuitable for resolving ambiguities, and so it is not possible to map tokensdirectly into operators. The ambiguities in the expression grammar forInform are:(a) ++ and -- each refer to either a prefix or a postfix operator(b) - refers to either a unary prefix operator or a binary infix one(c) ( and ) are used both to indicate function calls and subexpressions(d) , is used both to indicate a sequence of values, each to be evaluated and thrown away with only the last one kept, and to separate function arguments(e) -> does not represent an operator when parsing constants as operand values in a line of "@" assembly language (because it needs to mean "the store value goes in")These are resolved as follows:(a), (b) The "level 4 lexer" watches for context: for instance, MINUS_SEP translates to UNARY_MINUS_OP if there was no previous token, or it was an open bracket, or an infix or prefix operator, or a comma; otherwise it translates to MINUS_OP. (c) Similarly. If an open bracket token is read, and the previous token was an operand of some kind, then it must represent a function call (unless we are in constant context); an extra etoken is generated for an imaginary infix operator called FCALL_OP. The open bracket token is then translated to SUBOPEN_TT as usual; a close bracket is always SUBCLOSE_TT. Thus, the stream Eroica ( Beethoven , 3 ) is translated into the etokens Eroica <fcall> ( Beethoven <comma> 3 ) so that a tree will come out of the emitter looking like <fcall> / \ Eroica <comma> / \ Beethoven 3 although in fact the emitter recognises what is going on and automatically simplifies this to: <fcall> / | \ / | \ Eroica Beethoven 3 (d) A comma at the top level is either disallowed (in constant context) or treated as the operator <evaluate from left to right but throw away the result on the left and keep the one on the right>. (Just as in C.) Otherwise, comma is a separator of arguments. "Top level" means "top bracket level", which the "level 4 lexer" keeps track of by counting the bracket tokens going through it. (e) There is a special expression context called "assembly language", in which "->" is translated to ENDEXP_TT.The second main difficulty is entirely of the language designer's own making. How is the end of an expression to be detected? In C, it always possiblewhen parsing an expression to say "the end will have been reached whensuch-and-such a token is reached": for example, ";" or "}". This is notpossible in Inform.Even if one knows in advance what the terminating token ought to be, it mightbe impossible to recognise for context reasons. For example, the statement move <first expression> to <second expression>;appears to be no problem: the first expression is terminated by the keyword"to", the second by ";". But suppose "to" has another meaning, as the nameof a constant or variable? In this case, move 2 + to to to;is a legal statement.For all these reasons, Inform actually detects the end of an expression atlexical level 4 by watching the context carefully: for instance, in move 4 * banana to ...when "4 * banana" has been passed, the next token is expected to be an openbracket, an infix or postfix operator, or (perhaps) a comma. Since the nexttoken is none of these, the expression has ended. Note that this decisioncan be taken without knowing whether "to" is a keyword or a symbol.From a grammatical point of view the worst design flaw in the Inform languageis that array initialisations (either in Array directives or object propertyvalues) consist of sequences of expressions given without any separator inbetween. For example, Array X --> 2 4 6 * MAX_FROGS 45 ;This is arguably a convenient shorthand. Unfortunately, it falls prey tothat predator of all computing grammars, unary minus. The directive Array X --> 10 -5;is ambiguous. Is it an array of 10-5 = 5 entries, or an array of two entries,10 and -5?In Inform 5 there was no problem since constants were not allowed to containoperators anyway (and negative numbers were represented by single tokens,which is not true in Inform 6): the directive can only mean that there aretwo entries, 10 and -5.In Inform 6 the other answer is true, because the parser always makes thelongest match it can. Since this may mean that some Inform 5 code needschanging in a few places, a warning that "this is ambiguous" is issuedfor unbracketed minus signs used in array initialisations.(Illustrating the problem further, if the user tries to clarify things with Array A --> 10 (-5);then this is parsed under the normal rules as the function call 10(-5);therefore another constant-context rule is used to interpret brackets asalways referring to subexpressions, not function calls.)Similar difficulties at first seem to occur with < and >, and with << and >>. Does <Action 3 -4>refer to the pair (Action, -1) or the triple (Action, 3, -4)? (It is nowobvious that the correct syntax design would be to insist on a comma inbetween 3 and -4.) Fortunately, though, both Informs 5 and 6 apply a "makethe longest expression possible" principle to this case, so they agree thatthe answer is (Action, -1). On the same grounds, the statement <Action x ++ y>is understood as applying ++ to x, not to y, because the "longest expressionpossible" when parsing the first expression is "x ++". A similar slightexception is made to the rules parsing open-brackets in action statements,so that <Action firstobj (4*counter)>is not parsed as having one argument, the result of the function call firstobj(4*counter). 6.5 Constant folding ----------------Recall that the emitter takes as input a stream of etokens from theshift-reduce parser, consisting of the expression re-ordered into RPN. Whenthe whole stream has been received (i.e., when it has received ENDEXP_TT),the emitter has to produce an operand as "answer".For instance, if the emitter receives just the operand 56, then the answeris 56. More often, the answer may be a special value (written above as E1,E2, ...) meaning "the numerical value has to be worked out by working throughthis tree". (E1 is actually a new type of etoken, of type TREE_NODE_TT.)If this were done strictly, then the original text 4 + 7 would result in theemitter producing the answer E1, where E1 refers to the expression tree + / \ 4 7and this is not a helpful answer, for two reasons: it means the valuecouldn't be used as a constant (since it seems to need Z-code to compiled towork it out), and even if it's used in code, there's not much point incompiling the Z-code instruction @add 4 7 -> sp;just to put the value 11 on the stack.Therefore, if the emitter finds an operator that it knows how to calculate(such as +), whose operands are all constants (and which are not unknownfor linking reasons), it eliminates the sub-expression in favour of theresult. This is called "constant folding": the constant value 4+7 isfolded up into 11.Two issues arise: firstly, arithmetic must be done exactly as it would bedone in the Z-machine, that is, as signed 16-bit arithmetic. (Otherwiseconstants may fold differently in one port of Inform or another.)Secondly, constant folding cannot safely be performed on a marked value.(For otherwise: consider what might happen to the expression SODS - LAW,at a time when neither SODS nor LAW have been defined. Both are evaluatedas constants with value their symbol numbers and marker SYMBOL_MV: the resultis folded to the difference between their symbol numbers, and the markeris lost: i.e., the result is nonsense.)The emitter is able to fold the following operators: + - (unary or binary) * / % & | ~ == ~= > < >= <= && || ~~The inclusion of the conditionals here is quite important: it enablesdirectives like Iftrue #version_number == 3;to work, since the expression "#version_number == 3" can be parsed in aconstant context.The inclusion of unary minus is also important, as it is only by foldingthis that the text "-45" can be parsed in constant context. 6.6 Checking lvalues and simplifying the parse tree -----------------------------------------------If the emitter produces a parse tree, then it's now modified according toseveral rules. The aim is to remove any ambiguities in the tree (forinstance, "=" does several different things according to what its leftoperand is) and make a final correctness check (for instance, this is where"45 = 12" results in an error message).Firstly, the tree is traversed to find instances where a value is used wherea condition was expected. (If the expression is in a condition context, thetop node is expected to be a condition; the children of a logical operator&&, || or ~~ are expected to be conditions.) The configuration <value or tree>is replaced by <~= 0 condition operator> | <value of tree>If the <value or tree> has root node the "=" operator, a warning is printedout, since it seems likely that if (x = 4) ...(which would invariably be found true, since the value of "x=4" is 4 which isnon-zero) is a mistype for if (x == 4) ...A second, depth-first, traversal is then made to remove usages of ~~, usingde Morgan's laws: ~~(x && y) = ~~x || ~~y ~~(x || y) = ~~x && ~~y ~~(x == y) = x ~= y ~~(x >= y) = x < yand so on. (Note that, internally, every operator etoken has a correspondingnegated operator etoken.)One ambiguity remaining in the tree is that the operators ".", ".#" and ".&"act on both common and individual properties (that is, properties handled bythe Z-machine's architecture directly, and properties handled by the run-timeveneer routines). The decision about which is intended is made here, bylooking at the right operands of the operators. (If it's a common propertynumber, or an individual property number, the decision is obvious; if it's avariable or some compound expression, then we decide in favour of individualproperties (the run-time veneer routines can patch things up if this is awrong decision).)So a traversal is made to carry out the following transformation: . <common .> <indiv .> / \ becomes / \ or / \ a b a b a b(Since the veneer routines are themselves written in Inform code, a slightlydifferent translation rule applies to them: "." always means <common .>and so on, while three "secret" operators are provided, "..", "..&" and"..#", to mean <indiv .> and so on. Outside of the veneer, use of theseoperators is illegal.)In the same traversal, function calls are distinguished from sent messages: <fcall> <send message> / \ \ ... / | | \ ... <common .> c d becomes a b c d or <indiv .> / \ a bThe final traversal performs "lvalue checking". Five operators act directlyon storage objects (such as variables) rather than values: these are <lvalue> = <ordinary value> <lvalue> ++ <lvalue> -- ++ <lvalue> -- <lvalue>where <lvalue> has to be a reference to a storage object. There are up tofive kinds of storage object in the tree: local/global variable VARIABLE_TT common property value of an object <common .> subexpression individual property value of an object <indiv .> subexpression entry in byte -> array -> subexpression entry in word --> array --> subexpressionThis makes 25 combinations. As well as checking that what ought to be anlvalue is indeed one of these five possibilities, the traversal alsorewrites the tree explicitly with one of 25 operators. For instance: = <set byte array entry => / \ / | \ -> v becomes a b v / \ a band so on.Thus, the end of this process is a tree representing a syntactically correctexpression (if necessary having mended any errors in the source code'sdescription of the expression), which has a minimal number of logicaloperators and no negation operators, and in which the action to be taken byany node does not depend on what its children are. 6.7 Summary of parse tree output ----------------------------It is time to formally specify the input and output of the main routine in"expressp.c": assembly_operand parse_expression(int context)takes, as argument, one of the seven *_CONTEXT constants: QUANTITY_CONTEXT the default: the result may be any quantity VOID_CONTEXT the expression is used as a statement, so that its value will be thrown away and it only needs to exist for any resulting side-effects (used in function calls and assignments) CONDITION_CONTEXT the result must be a condition CONSTANT_CONTEXT the result must be known now (at compile time) ASSEMBLY_CONTEXT like QUANTITY_CONTEXT, but with the '->' operator forbidden (this is needed for assembly language to indicate store destinations) FORINIT_CONTEXT like VOID_CONTEXT, but with the '::' operator forbidden at the top level (needed to resolve ambiguity in grammar) ARRAY_CONTEXT like CONSTANT_CONTEXT, but with different rules to resolve whether a minus sign represents unary or binary minus (needed to resolve ambiguity in grammar)The return value is the result of the expression. This is an assemblyoperand, the type of which indicates what has happened: LONG_CONSTANT_OT, SHORT_CONSTANT_OT the result is this number VARIABLE_OT the result is in this global/local variable OMITTED_OT there is no resulting value (for instance, this happens if the expression was successfully parsed in void context) EXPRESSION_OT the result can only be determined by running the code generated from this parse treeIf an error has occurred in the expression, from which recovery was notpossible, then the return is (short constant) 0. This should minimise thechance of a cascade of further error messages.The type EXPRESSION_OT is not a valid Z-machine assembly operand type, andit's only used here and in the code generator "expressc.c". The value ofsuch an operand is an index N, indicating that the root node of the tree isin Nth position in the expression-tree-array ET.The expression-tree-array ET[0], ET[1], ... is an array of "nodes", each ofwhich is a structure as follows: { /* Data used in tree construction */ int up, down, right; int operator_number; /* Only meaningful for non-leaves */ assembly_operand value; /* Only meaningful for leaves */ /* Attributes synthesised during code generation */ ... }The numbers ET[n].up, ET[n].down and ET[n].right are node numbers for the nodeabove, the leftmost node below, and the next child to the right of the sameparent: they hold -1 if there is no node in that position.A "leaf" is a node for which ET[n].down == -1. Leaves hold operands, andother nodes hold operators. (Note that leaf operands must be of operand typeSHORT_CONSTANT_OT, LONG_CONSTANT_OT or VARIABLE_OT: they must be honestZ-machine operands.)Nothing distinguishes the root node except that its number happens to be heldin an EXPRESSION_OT operand value somewhere.A brief word on deallocation: clearly these parse trees need to surviveintact until code generation occurs for them. No elaborate system is providedhere, since the syntax analyser always generates code for any expressions itreads while it's still parsing the same syntax line as the expression wasparsed on. Each time a new syntax line is reached, "expressp.c" is notifiedand it then wipes the ET[n] array empty again. 7 Code generation from parse trees -------------------------------- 7.1 Aim and structure of the code generator ---------------------------------------The section "expressc.c" takes as input a parse tree and generates suitableZ-code to calculate the result of the expression which the parse treeembodies. As a result, its main output is a stream of function calls to"asm.c" (see chapter 5 for details).The syntax analyser calls only one routine: assembly_operand code_generate(assembly_operand AO, int context, int label)where AO is expected to be the output from a previous run through "expressp.c".The contexts are the same as those for expression parsing, except that onlyVOID_CONTEXT, CONDITION_CONTEXT and QUANTITY_CONTEXT are allowed.The "label" is only meaningful in CONDITION_CONTEXT: if label >= 0, branch to this label if the expression is false as a condition; if label == -3, rfalse if the expression is true; if label == -4, rtrue if the expression is true.(Note that the presence of rfalse/rtrue here allows better code generationfor statements in the form if ( ... ) return;owing to a feature of the Z-machine's architecture for branch instructions.)The return AO is only meaningful in QUANTITY_CONTEXT; it holds the resultof the expression, often the stack pointer variable (meaning, the answeris on the top of the stack) but not always: e.g., from j++, 2the result would be SHORT_CONSTANT_OT 2. Similarly, from the expression 2the code generator returns 2 without actually generating code. 7.2 Annotation for conditions -------------------------The first thing the code generator does is to "annotate" the tree forconditions. Annotation is the process of assigning useful values toevery node on the tree, usually in such a way that either the value atone node determines the values for its children, or vice versa. (Suchvalues are called "attributes" and this inductive definition of themis called "synthesis": see ASU, p.280.)The explanation of this algorithm is about five times longer than itssource code!We wish to assign a pair of values written (on paper) in the form a | bto every conditional node (that is, every node which is a conditionaloperator such as "~=" or "ofclass", or is a logical operator such as "&&").This is to be read as "branch to label a if true, or label b if false".Inform has four special label numbers: -1 means "carry on rather than branch" -2 means "branch to the immediately following instruction" -3 means "return false rather than branch" -4 means "return true rather than branch"(-2 is used in the assembly of instructions like get_child_zc, which isa conditional branch in Z-machine terms but which Inform wants to ignorethe branch result from. We can ignore -2 here: it behaves exactly like-1.)One of the two numbers a and b must always be other than -1 (otherwise wewould have rendered a branch meaningless, which must be a mistake). Ideallyexactly one of them is -1, so that the node can generate a single branchinstruction.There are two points where these a | b attributes start to appear in thetree. Firstly, in condition context they'll appear at the root node: forexample, in CONDITION_CONTEXT with label 40 the top node would be labelled -1 | 40("jump to L40 if false"). Secondly, at condition subexpressions used asvalues, e.g. in the tree: = / \ v == / \ t 1This is trickier: the result of == is not a branch somewhere, but a numericalvalue (which must be 0 or 1). To cope with such conversions, two moreattributes are provided: to_expression, label_afterA node flagged as "to_expression" is one where a condition is being convertedto a numerical value; the "==" node above would have this flag set."label_after" is either -1 or the number of a label which the code generatoris to assemble after code generation for the node is complete.So here are the rules for generating these attributes for node N. Each nodeis sent a pair of values a | b from its parent (in the case of the rootnode, from the code_generate() routine). annotate N with a | b if N is &&, || or a conditional operator, and a | b is -1 | -1, then this must be a condition used as a value: so re-annotate N with to_expression = TRUE label_after = a new label L L | -1 if N is an && operator: a label for "go here if false" is going to be needed to provide a destination for the branch generated by the lefthand operand. So, if b = -1, re-annotate N with: label_after = a new label L a | L if N is an || operator: a label for "go here if true" is going to be needed to provide a destination for the branch generated by the lefthand operand. So, if a = -1, re-annotate N with: label_after = a new label L L | b Finally, we need to pass a pair of values down to each child of the node. In the case of all nodes except && and ||, the values passed down are -1 | -1 (because the operator expects numerical operands). In the case of && and ||, let x | y be the annotation which was finally made for this node. We pass values down as follows: && || / \ / \ -1 | y x | y x | -1 x | y(Here "condition node" means a conditional or logical operator. Recall thatthe ~~ operator was eliminated earlier.)For example, consider the condition in if (x == 1 || y >= 2) ...The syntax analyser decides to generate code which will branch to L23 if thecondition is false. The parse tree || || -1 | 23, l_a = 50 / \ / \ == >= is annotated to == 50 | -1 >= 50 | 23 / \ / \ / \ / \ x 1 y 2 x 1 y 2where the annotation routine created label 50 as a point to branch to if thefirst condition turned out to be true.Note that it's actually a little wasteful to annotate this way: the node >=is annotated with 50 | 23, but label L50 will appear immediately after thecode generated for this node anyway, so it might as well be annotated -1 | 23.The annotation routine does indeed make this simplification. Lemma: One of the values a, b sent down to any node is always -1. Proof: By induction down through the tree. The values sent down to the root node are either -1 | L, -3 | -1 or -4 | -1, so the property holds for the root node. Now suppose such a pair a | b is sent down to node N. Unless N is && or ||, it sends -1 | -1 to its children, so the property holds for the children of N. If N is &&, then it has two children, left and right. The left child is always sent -1 | y, so it has the property. The right child is sent x | y, unless y is a new label appearing after N, in which case it is sent x | -1. Note, though, that N itself was sent a pair in which one number was -1 (by inductive hypothesis) and so x | y can only have both numbers other than -1 if a new label was created by N. Since N is an && operator this label must be y; and therefore the child is sent x | -1, which means the right child also has the property. The argument for || is symmetrical to this. Corollary: In the annotation "a | b" of every node other than a && or || operator, either a is -1 and b is not, or vice versa.The annotation values a | b are only used to generate code for conditionoperators, so we now have the desired result. 7.3 Main traversal --------------The main algorithm is simple. We make a depth-first (i.e., bottom-up)traversal of the parse tree, pruning off sub-expressions as code isgenerated for them. In the tree * / \ a + / \ b cwe first descend to the +, and convert that into @add b c -> spand prune the tree to * / \ a spso that the next instruction generated is @mul a sp -> spand the tree is now just spand the result is therefore on the stack. A complication, however, is whatorder in which to remove the subexpressions. Faced with the tree - / \ * + / \ / \ a b c ddo we go left first from the top -, or right first? This decision is takenfor us by the Z-machine's architecture. When the Z-machine executes theinstruction @sub sp sp -> x;it takes the two operands off the stack in the order left to right, i.e.,it pulls the first operand first and then the second. (Most stack machineswork the other way, to make RPN expressions easy to evaluate. The Z-machinewas designed for forward Polish notation, though, because Infocom's ownlanguage for it was a Lisp-variant with syntax like (PLUS 2 3).)This means that we need to push the second operand first, then the firstoperand. In other words, we have to code generate the + operation first,and then the * operation.Therefore we need to traverse the tree depth-first and right-to-left.But there is a complication: the && and || nodes need to have their childrencode-generated left-to-right. (Although "and" is logically commutative,so that it shouldn't matter, "&&" is not computationally commutative, becausethe language specification requires that the left condition is never evaluatedif the right condition was false.)There is one other left-to-right node: the comma operator. For example,if the original value of x is 10, then , / \ ++ x / xmust evaluate to 11, not 10, because "x++, x" must evaluate "x++" first,throw the result away and then evaluate "x". 7.4 Void context ------------As the above example demonstrates, in some cases sub-expressions need toproduce a result and in other cases they must not do so. If the "++" operatorhad left a value on the stack, then it would still be there long after theexpression had been evaluated.Therefore, when code generating each node it is essential to know where theresult of the expression is supposed to go. A flag called void_flag ispassed down to indicate this: if set, then the operator is being evaluatedin "void context", meaning that the value should be thrown away. The rulesare: if the expression is in VOID_CONTEXT, then the root node is; the left operand of a comma operator is in void context; if a comma operator is in void context, its right operand also is.Not every subexpression is permitted to be in void context. For instance, 3*x, y++;generates the error Result of expression is thrown awaybecause the * calculation was pointless. Numerical values or variables arenot allowed in void context, and nor are any operators which have no "sideeffect": i.e., whose evaluation causes nothing in the Z-machine to change.The operators with side effects are: = in its various forms (5 of them) ++ in its various forms (10 of them) -- in its various forms (10 of them) function call (including message sending)It is only now, at code generation time, that the error message above isgenerated. (In C this would only cause a warning, but I didn't want towaste effort making the code generator assemble code to explicitly throwresults away.) 7.5 Results of subexpressions -------------------------By the rules above, then, either an operator is in void context and itsresult should be thrown away, or it isn't and the result should go onto thestack. However, this would be wasteful, as an expression like "x = 4 + y"(where x is a local variable, say) would then generate @add 4 y -> sp @pull xand give result "x". In fact we want to generate @add 4 y -> xand give the result "x". Therefore, when code-generating an operator witha result (such as @add), we look at the operator above (which is guaranteedstill to exist): if it is <set-variable-equal> then we get its left operand,write the result of the operator (such as @add) to that, and replace the<set-variable-equal> operator with the variable. For instance, set-var-equal x / \ x + becomes just / \ 4 yOne method used to throw away the results of expressions in void contextis to write the result to temporary variable 255 rather than the stack.This is how unwanted function call return values are disposed of (well,unless the Z-machine version number is 5 or more, in which case instructionsfor call-but-throw-away-result are used directly).A few other optimisations are used in void context: for example, ++ used as postfix on variable | vargenerates the code @push var @inc varwith result "sp" in a quantity context. In void context, it generatessimply @inc varThis is quite an important optimisation since postfix-++ is used very oftenin void context (in "for" statements and just as an assignment). 7.6 Conditional and logical operators ---------------------------------The above account covers code generation for arithmetic and assignmentoperators.In generating && and ||, nothing needs to be done except possibly toassemble a label (if the attribute label_after isn't -1).Generating for conditions like == is in principle easy but in practicedifficult. Recall that we have three attributes to go on: a | b labels to branch to on true | false, one which is -1 meaning "carry on" to_expression flag indicating "convert to numerical value"One problem is that, because of the "or" operator, a condition like ==may have an arbitrary number of children: if (x == a or b or c or d or e) ...for instance, must be assembled into the instructions: @je x a b c ?L2 @je x d e ?~L1 .L2 ... .L1(and if x is a use of the stack pointer, it must be pulled into some temporaryvariable so that it can be non-destructively read more than once). The codeto achieve this is baroque but uninteresting.The second question is how to convert a condition to a numerical value:either 0 (false) or 1 (true). At present, this is not very optimised, andthe code looks like: <if condition is true jump to L1> @push 0 @jump L2 .L1 @push 1 @jump L2with the result being "sp". 7.7 System functions ----------------When generating the <function call> operator, Inform looks at the leftmostoperand (the function to call) to see if it's the name of a system function.(Note that if the system function has been Replaced, then it will have beenfiltered out long since and the name will have been treated as the symbolname for a function in the usual way.)These are all implemented as short inline functions (sometimes only oneinstruction long) except for "metaclass", implemented as a veneer routine.Note that random / | | a b c ....nodes are generated by creating a word array whose contents are a, b, c, ...and then assembling code to read a random entry from this array. (Anerror is issued if a, b, c, etc. are found not to be constant values.) 7.8 Strict mode assertions ----------------------When the compiler runs in "Strict" mode, with the -S switch set, itcompiles otherwise wasteful code to protect the programmer from crashingthe Z-machine through programming errors. For instance, the statement x = child(nothing);would otherwise compile to Z-code which is technically illegal: @get_child nothing -> x ?L0; .L0;because the "get_child" opcode should have a legal Z-machine objectnumber as first opcode. Most interpreters would overlook this technicalbreach of the Z-machine specification. A more serious example might be my_array --> 34000 = 1;which would compile code trying to use @storew to write a value wayoutside the Z-machine's dynamic memory area: this would typically crashan interpreter. The generic term for mistakes like these, coinedby Mr David Glasser, is "vile zero errors from hell".In strict mode, Inform aims to compile code such that no program(which does not contain @ assembly language) can possibly violatethe Z-machine specification in any respect. In effect it compileswhat other compilers have called "assertions" before any use of adangerous opcode whose operands have been supplied by the user'sprogram. For example "storew" will not be executed until its operandvalue has been checked to lie in range. That is, the statement z = x-->y;compiles in non-strict mode to @storew x y -> zbut in strict mode to @call_vs RT__ChSTW x y -> zwhere RT__ChSTW is a veneer routine ("run-time check store word")which verifies only that x+2*y lies between 0 and the top of dynamicmemory: if so, the routine performs the required opcode; if not, itcauses the error message [** Programming error: tried to read outside memory using --> **]to be printed up.Not all the protected opcodes are replaced simply by function calls,as this could be very slow: some are replaced by in-line code, whichis quicker. For example, if the source code contained z = my_array-->y;where "my_array" had been declared as an array, then Inform wouldinstead compile this: jl y short_0 to L1 if TRUE jl y <n> to L0 if TRUE .L1 call_vn2 RT__Err short_28 y short_6 short_5 short_1 jump L2 .L0 loadb long_680 y -> z .L2This has wasted some 23 bytes, but is faster than calling a function,and directly checks that 0 <= y < <n>where <n> is the 1 more than the highest legal entry of my_array, whichInform knows thanks to having already compiled my_array. The longrun of numbers fed to the veneer's run-time error system specify theerror number (28), the index tried (y) and so forth, enabling it toprint an error like so: [** Programming error: tried to write to -->14 in the array my_array, which has entries 0 up to 9 **]The current list of assertions is as follows: Inform syntax Assertion ============= ========= child(x) metaclass(x) == Object, children(x) i.e., is the number of a Z-machine object elder(x) but isn't a class-object eldest(x) parent(x) sibling(x) younger(x) youngest(x) x in y metaclass(x) == Object or Class x notin y i.e. x is the number of a Z-machine object (y, however, is unrestricted: the second operand of @jin does not have to be a valid object number) x has y metaclass(x) == Object or Class, x hasnt y y is a valid attribute number give x y metaclass(x) == Object remove x move x to y metaclass(x) == metaclass(y) == Object, and the object tree remains a tree if x is moved to y x/y y ~= 0 x%y x.y = z metaclass(x) == Object, y is a valid property, ++x.y and x provides y x.y++ --x.y x.y-- x.y (used as value) metaclass(x) == Object, y is a valid property, and x provides y or else y is a common property x.&y metaclass(x) == Object, y is a valid property x.#y (not necessarily provided by x) x-->y = z x+2*y,x+2*y+1 lie within "alterable memory" x-->y++ (see below) x-->y-- Moreover, if x is the name of a declared array, ++x-->y x+2*y,x+2*y+1 lie within the bounds of the array --x-->y (and not the length entry 0 if a table or string) x-->y (used as value) x+2*y,x+2*y+1 lie within byte-accessible memory Similarly for declared arrays, though entry 0 can be read x->y = z x+y lies within "alterable memory" (see below) x->y++ x->y-- Moreover, if x is the name of a declared array, ++x->y x+y lies within the bounds of the array --x->y (and not the length entry 0 if a table or string) x->y (used as value) x+y lies within byte-accessible memory Similarly for declared arrays, though entry 0 can be read print (char) x x is a valid ZSCII character for output, as defined in section 3.8 of the Z-Machine Standards Document print (address) x x lies within Z-machine byte-accessible memory print (string) x metaclass(x) == String print (object) x metaclass(x) == Object or ClassThe definition of "alterable memory" is:either (a) within the array space area; or (b) within the object common property values table; or (c) within the object individual property values table; or (d) being the word "Flags 2" in the Z-machine header, at address $0010, $0011.All other areas in byte-accessible memory can be read but are"write-protected": in particular the global variable values table,which needs to be defended as corrupting the contents during a loopcan cause the Z-machine to hang.A further assertion is compiled into loops of the form objectloop (x in something) { ... }where "something" is any simple term (but not a calculated expression):at the end of the loop, the assertion checks that x is still in "something"and has not been removed or moved elsewhere in the object tree, causingthe iteration to go awry. (This is a popular Inform coding mistakeand often arises from code like "objectloop(x in player) remove x;".)Note that the veneer code is always compiled in non-strict-mode, whichis necessary to prevent some circular problems (e.g. can't do X withoutcalling the veneer to ask permission, but then the veneer has to callitself to ask permission, but then...); and also enables the veneer todo horrid things to class objects in ways which the outside program isn'tallowed to. 8 Assembly of code, text and data structures ------------------------------------------ 8.1 Assembly of initial code ------------------------The front end of "asm.c" consists of three routines to assemble routinestart and finish, and to place labels; plus many routines for assemblingindividual instructions. These all fill out data in theassembly_instruction AI and then call a single assemble-instruction routine.Thus, the back end of "asm.c" contains four operations to place data intothe Z-machine's code area: Routine start assemble_routine_header() Routine end assemble_routine_end() Label assemble_label_no() Instruction assemble_instruction()For the format of Z-code instructions, see the Z-Machine Standard Document,which goes into the Z-machine's format with exhaustive thoroughness. Notethat Inform 6 is much more compliant to the standard (so far as I know,perfectly compliant) than earlier releases: for example, the Flags 2 bitsfor "UNDO is needed" and so on are set correctly for the first time.One of the few pieces of information fed back to Inform's higher levels by"asm.c" is a flag indicating whether execution can possibly reach the nextinstruction or not. The assembly of certain instructions (returning froma routine, terminating the Z-machine, unconditionally jumping) sets theflag execution_never_reaches_herewhile the placing of a label removes it again (because by branching to thatlabel it may be possible to reach this point after all). This is used toissue dead code warnings (and also to remove some unnecessary brancheswhen code-generating "if"..."else"...).Assembling a routine header is straightforward: note that an option existshere to assemble some code along with it to print out tracing information.For this reason, and also to print out assembly trace at compile time,"asm.c" accesses the variable names in the symbols table.All labels are referred to in Inform's higher layers by number, rather thanby address; only "asm.c" has any access to what their addresses are. Labelsare numbered from 0 upwards, but the assembler also recognises -2 branch only to the next instruction (this converts a branch instruction such as @get_child to a non-branch one, since it causes the instruction to behave identically whether the branch is made or not) -3 return false rather than branch -4 return true rather than branchThe code -1, used higher up for "no branch intended", should never descendthis far.Note that the Z-machine has no formal concept of "routine end". Therefore,assemble_routine_end() only generates code to cause a return (and only thenif the present instruction position can be reached). By long-standing(if unfortunate) Inform convention, the value returned in this case willbe 0 for a routine in an object definition, or 1 for any other routine.The routine-start/end routines also keep track of the usage of localvariables and labels, and issue warnings about those which were declaredbut not used. Since the scope of these symbols is restricted to the routinein which they are defined, label names are unassigned (i.e. reset toUNKNOWN_SFLAG CONSTANT_T's). Local variable names are not held in thesymbols table and are automatically cleared when the next routine starts.Finally, we can now detect the use of a label name not defined in the routine,so we issue an error message if this has happened.What happens in this first phase of code assembly is that a form of Z-codecalled "initial code" is stored in a temporary holding area of memory untila routine has been completed. It is then treated by the branch optimiser(see the next section)."Initial code" is the same as Z-code, except that: (a) all branches to labels have long form, and instead of an offset the branch value is the label number within this routine (0, 1, 2, ...); (b) LABEL operands (to "jump" instructions) are, similarly, label numbers of the label to branch to; (c) parallel to the holding area is a uchar[] buffer of the same size, holding a "marker value" for each byte.(Note that branches to -2, -3 and -4 are assembled as short-form branchesand not marked as BRANCH_MV.) The marker values are as follows: BRANCH_MV This byte is the first of two in a branch operand LABEL_MV Ditto but in a LABEL operand DELETED_MV (Inserted by the optimiser, rather than at initial code assembly.) This byte has been deleted and should be skipped over when the code is transferred to long-term storage. a "module value" marker value This byte is the first of two in a LONG_CONSTANT_OT operand which should be marked with this value in module compilation ditto + 128 Ditto but a one-byte SHORT_CONSTANT_OT or VARIABLE_OT operand 8.2 Branch offset backpatching and optimisation -------------------------------------------There are three phases of this process: (i) Deciding whether each branch should or should not be optimised to the 1-byte short form, using the label positions in the initial code; (ii) Recalculating the label positions to take account of this (which will tend to move the labels closer together as second bytes of branch operands are deleted); (iii) Transferring the initial code to long-term storage, replacing branch and LABEL operand values with address offsets and issuing module markers as indicated (if in module mode).A branch can be "short form" if it is a forward branch of between 0 and 29bytes from the PC position of the next instruction. To record a branch as "short form" during (i), the second byte of the branchoperand is marked as DELETED_MV.The long-term storage alluded to is either temporary file 2 or else thezcode_area memory block. Note that the routines moved there are still notfinally correct, but are "raw Z-code": Z-code which has incorrect operandsbecause value backpatching has not yet occurred. 8.3 Text translation: ISO and Unicode resolution --------------------------------------------The algorithm encoding ZSCII text to a compressed sequence of 5-bit"Z-characters" has been documented many times: see the Z-machine standardsdocument. The section of Inform responsible, "text.c", keeps text for three Z-machineareas: the static strings area, the dictionary and the "low strings pool".(In addition, as requested by "asm.c", it encodes text into the Z-code areaas part of the assembly instructions @print and @print_ret.)Static strings are written to a temporary holding area of memory, manipulateduntil correct and then transferred to long-term storage: either temporaryfile 1 or the static_strings_area memory block.The low strings pool holds a table of strings in low Z-machine memory usedfor abbreviations. Such strings need to be accessible in an unusual way(they are addressed by halved byte addresses, not packed addresses) owingto an anomaly in the Z-machine architecture, and therefore must reside inthe bottom 64K of memory: they cannot be written to the static strings area.Source text arrives at the text translation routines still encoded in anISO extension to the ASCII character set (depending on the current valueof the -C switch). A character value of $ca might mean Latin E-circumflex,E-ogonek, Cyrillic capital hard-sign, Arabic Teh or Greek capital Kappaaccording to this context. It would be illegal in plain ASCII or ISOHebrew, but since the Inform lexer has already filtered out any illegalISO characters, this possibility no longer arises.Furthermore, characters can be specified by string escapes beginning withthe @ character. The sequence @^E would specify Latin E-circumflex, andthe sequence @{39A} would specify Greek capital Kappa (by its Unicodevalue, $039A) regardless of the current -C setting.The work of sorting out character codes is handled by "chars.c". It maybe useful to summarise the different meanings which character codes canhold within Inform: (1) "Source". Raw source files can contain any character values between $00 and $ff. The lexer filters these characters to reduce them to: (2) "ISO". An unsigned 8-bit character code within which the values $20 to $7e always have their usual ASCII meanings, and $7f to $9f are always undefined, and some values in the range $a0 to $ff may have meanings as specified in ISO 8859-1 to -9 if the current Inform -C setting is 1 to 9. If -C is set to 0, "ISO" means the same as plain ASCII. (3) "Unicode". Unicode, or ISO 10646-1, is an unsigned 16-bit character code which includes essentially every human script. Inform uses it as an intermediate state between the diverse ISO possibilities and the final ZSCII result. (4) "Text". A string of ISO characters to specify a sequence of one or more Unicode characters. Usually, one ISO character is translated to one Unicode character: the exception is for string escapes. For instance, the text "a@{39a}c" specifies a sequence of 3 Unicode characters. (5) "ZSCII". An unsigned 10-bit character code used within the Z-machine. See the Standards document for its definition: note that between $20 and $7e, it agrees with ASCII. There are several translation routines in "chars.c" to move between theseforms: for instance, "zscii_to_text" is used when printing out thecontents of the dictionary into a transcript file. (Transcript filesuse the same ISO setting as the original source code.) However, themain sequence of translation is: Source -------> Text -------> Unicode -------> ZSCII lexer ISO-to- Unicode-to- filtration Unicode; ZSCII string escape parsingNote that the first two stages (lexer filtration and ISO to Unicode)depend only on the current -C setting, and are always possible. Thethird stage (Unicode to ZSCII) is more complex because it is programmableand may fail, depending on what the text is needed for.Unicode already specifies over 30000 different characters andcompromise is inevitable in trying to squash this into the 1024 possibleZSCII values. The Z-machine's stock of "extra characters"(see Standard Document 1.0) is configured by the story file tocorrespond to a block of up to 97 Unicode characters, and theseare the only ones usable in Inform strings, dictionary words oras quoted values.Inform sets up this block to a sensible default set given thecurrent ISO setting. For example, if Inform reads in source code under-C6 (ISO Arabic) then it will choose all the Arabic letters fromISO Arabic. (This default setting can be overridden using the"Zcharacter" directive.) Note that if the designer wants, say,Unicode $274b ("heavy eight teardrop-spoked propeller asterisk")as well as plain old $0621 (Arabic letter Hamza), this can simplybe added to the stock accumulated so far.The stock of extra characters is defined by a list of Unicode valueswritten out as the "Unicode translation table" in "tables.c". 8.4 Dictionary management ---------------------As dictionary words are found in the source code, for instance in the valueof "name" properties, or in grammar, or in constants like 'this', they areadded to the dictionary by a routine called dictionary_add(), which numberseach different word in "accession order": word 0 is the first which wasentered, and so on.References to dictionary words are actually stored in the Z-machine asbyte addresses within the dictionary table, which cannot be known until afterthe compilation pass (when the full alphabetical ordering is known). Suchreferences are therefore always marked (with DICT_MV) for backpatching.During compilation a dictionary table similar to the Z-machine format iskept: there is a 7-byte header (left blank here to be filled in at theconstruct_storyfile() stage in "tables.c") and then a sequence of records,one per word, in the form <Z-coded text> <dict_par1> <dict_par2> <dict_par3> 4 or 6 bytes byte byte byteThe difference between this and the final Z-machine dictionary table isthat the records occur in accession order, not alphabetical order. (Theconstruct_storyfile() routine in "tables.c" rearranges them.)The Z-machine does not specify what use is made of the three bytes of"dictionary parameters" (it doesn't even specify that there have to be onlythree): for this, see the next section.The type "dict_word" is used for a structure containing the (4 or) 6 bytes ofZ-coded text of a word. Usefully, because the PAD character 5 is < allalphabetic characters, alphabetic order corresponds to numeric order ofZ-encoding (regarding the Z-encoded bytes as a 32 or 48 bit big-end-firstnumber; sign is unimportant as the initial bit is never set). For thisreason, the dict_word is called the "sort code" of the original text word.A special text-translation routine is used to "prepare" such a sort code:as dictionary words do not use every feature possible in ordinary texttranslation (abbreviations, for instance, or two of the string escapeforms) it's worth having a separate routine, optimised for speed. Notethat this routine makes use of "text_to_unicode" and "unicode_to_zscii"in "chars.c" when, but only when, it hits a string escape character @.(For ordinary ISO characters, when no string escape is hit, these routineswould be too slow.)Since dictionaries typically contain 500 to 2000 words, speed is extremelyimportant when searching and sorting: a simple O(n^2) algorithm to search thedictionary, for example, greatly increases Inform's total compilation time onmedium-size games. (Here "n" is the number of dictionary words.)Inform 1 to 3 used a crude O(n^2) mass comparison algorithm, which wasunacceptably slow. Inform 4 to 6.05 used a form of hash table, with thefirst character of a word as hash function (there were 27 hash values, forA to Z plus "other"): this behaved very acceptably most of the time, but wasultimately still O(n^2) and the uneven distribution of initial letters inEnglish meant that the hashing function was not ideal.Inform 6.1 instead stores the dictionary as a "red-black tree". (See, e.g.,Robert Sedgewick, "Algorithms" (1983, second ed. 1988).) The tree is binaryand always has the property that at node X, anything hanging from the leftbranch of X is alphabetically before what's at X, and anything hanging fromthe right is alphabetically after. For instance, a legal configuration is: fieldmouse / \ abacus patrician / \ a constantinopleThis is simple enough to build and to search through. The problem is thatone usually ends up with a haphazardly arranged or "unbalanced" tree in whichsome branches are very much longer than others, so that searching times can beunpredictable, and in worst case the construction process is still O(n^2).Thus one wants a method of building the tree which attempts to restore thebalance as it does so. The algorithm here involves "colouring" each brancheither red or black (that is, storing one bit of information for each branch).Roughly speaking, suppose X is added as a branch of Y. Then the branchleading down from Y to X -- the new branch -- is coloured black. But thebranch leading down to Y -- which was already there -- is recoloured red.We do not permit the tree to contain two consecutive red branches: wheneverwe notice this happening, we perform a "rotation". A typical rotation is: a a \ \ bag cat / \(red) ---> (red)/ \(red) baa cat bag dog / \(red) / \ bun dog baa bunIt's like lifting the "bag" subtree and re-hanging it from the word "cat".(There is another case, when the red branches slope in opposite directions.)We also try to percolate redness upwards (redness being the possibility forchange in the tree structure). So, when searching, if we ever find \(black) node (red)/ \(red)then we replace it with \(red) node (black)/ \(black)and check the "rotate if you get two consecutive red branches" rule again.It is known that:(i) if there are N words in the tree, then any search takes less than 2 log-to-base-2(N) + 2 comparisons (i.e., this is the maximum depth of the tree);(ii) at worst you need to perform only one rotation for every four comparisons.In practice, the observed values are even better: the average seems tobe log-to-base-2(N) comparisons and 1 rotation (though this is unproven).Trees are almost optimally balanced most of the time.For an Inform game with a dictionary of 2000 words, then, we expect to performabout 12 comparisons per search. This compares with a typical figure of 75using the Inform 6.05 hashing algorithm. Comparisons are not terriblyexpensive, but searches are quite frequent. 8.5 Format of tables unspecified by the Z-machine ---------------------------------------------The Z-machine standards document requires many tables to have particularformats and locations: but Inform makes several other tables, and thissection is to document their formats. (See the construct_storyfile()routine in "tables.c" for exactly how the byte-addressable Z-machine memoryis put together.)Details of differences in modules as compared with story files are givenlater. (i) HeaderFollowing standard conventions, Inform: leaves bits 2 and 3 of "Flags 1"clear; places the release number in the word at bytes 2 and 3; the grammartable address at bytes 14 and 15; and the serial number in bytes 18 to 23. The default release number is 1 and the default serial number is either970000 (if no access to the current date is available) or the compilationdate in the form YYMMDD.In addition, Inform writes its version number in ASCII form into bytes60 to 63 (the last four bytes of the header) in the form "6.01". Thismakes story files produced by Inform 6 distinguishable from all otherstory files, something interpreter writers have long wanted. (ii) Low strings pool and abbreviations table By default, the 96 entries are all " " (this Z-encodes to the $80 $00,which makes it a convenient default value). The first two banks of 32are used for strings declared by Abbreviate; the third is set at run timeby the "string" statement, to values which have previously been declaredby Low_string.The pool of low strings is always placed at $0040, immediately after theheader and before the abbreviations table. (iii) Header extension tableThe Z-machine standard allows games not to provide a header extensiontable (formally called the "mouse data table" after the only informationwhich Infocom ever stored in it) and Inform 6.01 to 6.11 never do.Inform 6.12 and later always generate this extension, which alwayscontains at least 3 word entries. (iv) Character set tableThis is a table of 78 = 3*26 bytes, giving the Z-character numbers ofthe characters to appear in the three translation alphabets. The tableis only written if the "Zcharacter" directive has been used to alterthe standard settings. (See section 12.2.) (v) Unicode translation tableThis is generated (by Inform 6.12 and later) only when needed, i.e., onlywhen a game has asked to use a stock of "extended characters" whichdiffers from the usual ISO Latin1 set. Games compiled under the ISOsetting -C0, -C1 (the default setting) and -C9 will usually neverhave a Unicode translation table, though they can have. (vi) Property default values tableNote that Inform defines properties 1, 2 and 3 automatically: 1 is "name",while 2 and 3 are used to support the veneer routines' implementation ofobject orientation. (vii) Class-number-to-object-number table (viii) Individual property values tableSee the notes on object orientation; these tables are used by the veneerand are unspecified by the Z-machine. (ix) Grammar table (x) Actions table (xi) "Preactions" table (xii) "Adjectives" tableSee the next section. The Z-machine specifies none of these tables. (xiii) Dictionary tableThe format of this table is largely specified by the Z-machine. Althoughthe Z-machine allows some flexibility in setting up the dictionary, Informsticks to the usual Infocom configuration: the separating characters are full stop, comma and space; the word-entry length is 7 (in version 3) or 9 (otherwise).Thus the dictionary begins with seven header bytes: 3 '.' ',' ' ' 7-or-9 <word containing the number of entries>Each word is encoded with an entry giving 4 (in version 3) or 6 (otherwise)bytes of textual representation, fully specified by the Z-machine, andthen three bytes of data, which Inform can do with as it pleases.These are the so-called "dictionary parameters" dict_par1, 2 and 3. Inform'spleasure is to write into them as follows: dict_par1: flags dict_par2: verb number (counting downwards from 255) dict_par3: preposition number (counting downwards from 255) in grammar version 1; not used in grammar version 2The flags are given as follows: bit: 7 6 5 4 3 2 1 0 <noun> <adj> <plural> <meta> <verb>The bits <verb>, <noun> and <adj> are set if the word can be used in thecontext of a verb, noun and/or preposition (all three can be simultaneouslyset). The <meta> bit indicates that the English verb is "meta", that is,is a command to the program and not a request in the game.The <plural> bit is set using the '...//p' notation, like so: 'egg' 'eggs//p'Library 6/3's parser uses this to automatically detect commands referringto multiple game objects. (xiv) Module mapThis is an extension to the header, only used in module files (see later). (xv) Linking data tableA table placed after the end of static strings, but only in module files(see later).To summarise, then, an Inform game has the following memory map: ----------------------------------------------------------- Dynamic Header memory Low strings pool Abbreviations table Header extension Character set table (if differs from normal) Unicode table (if differs from normal) Common property default values (*) Object tree Common property value tables (*) Class number to object number conversion table Property names table (+) Attribute names table (+) Array names table (+) Individual property value tables (*) Global variable values, part of... (*) ...Dynamic array space (*) ----------------------------------------------------------- Static Table of grammar table addresses byte The grammar tables (one for each Inform verb) (-) addressed Table of action routine addresses (+) memory Table of parsing routine addresses (+) Table of "adjectives", i.e., prepositions (+) Dictionary (Module map) Synchronising gap of up to 7 null bytes ----------------------------------------------------------- Static Z-code area (*) "virtual" Static strings area memory (Link data table) ----------------------------------------------------------- (*) This area requires full value backpatching (+) This area requires a simple automatic correction (e.g. adding on the static strings or code offsets), which is done without the full backpatching machinery (-) In grammar version 1, this area requires no backpatching, but in GV2 each grammar token's data is checked to see if a dictionary word or routine address, and backpatched if so: so in GV2 it comes under category (+) 8.6 Grammar version numbers GV1 and GV2 -----------------------------------The "grammar tables", used by the parser in an adventure game, are notspecified by the Z-machine at all (contrary to popular opinion) and musttherefore be fully specified here. There are four such tables: Grammar table Actions table "Preactions" table "Adjectives" tableInform can generate two different formats for these tables, known asgrammar version 1 (henceforth GV1) and grammar version 2 (GV2).Inform makes the GV1 or GV2 decision according to the value of the symbol Grammar__Versionwhich Inform usually defines as 1, but allows programs to redefine. Library6/3 and later, in particular, contain the line Constant Grammar__Version = 2;Note that it is essential for the library's parser to understand thegrammar tables being generated by Inform, so attempting to use GV2 withlibrary 6/2 or earlier would fail.The designs of GV1 and GV2 have some points in common. The Grammar tableis a --> array containing addresses of grammars, one for each Inform verb.The Actions table is a --> array containing addresses of action routines(such as "TakeSub"), one for each action. (Fake actions have no actionroutines and aren't in the table.)(The term "Inform verb" means essentially a common parsing grammar, such asthat shared by "take" and "hold", and is used to distinguish this from an"English verb", such as the word "take".)GV1 is at heart an imitation of middle-period Infocom table formats, andwas used in order that common debugging tools would still work on Informgames (an important consideration in the early days of debugging Inform 1).In GV1, an individual grammar table has the format: <number of grammar lines> 1 bytefollowed by that many 8-byte lines in the form: <parameter count> <token1> ... <token6> <action number> -- byte --------- --byte-- --byte-- -- byte -------The parameter count is the number of tokens which are not prepositions.This is needed because the run of tokens is terminated by null bytes (00),which is ambiguous: "noun" tokens are also encoded as 00.The numerical token values are as follows: "noun" 0 "held" 1 "multi" 2 "multiheld" 3 "multiexcept" 4 "multiinside" 5 "creature" 6 "special" 7 "number" 8 noun=Routine 16 + parsing-routine-number Routine 48 + parsing-routine-number scope=Routine 80 + parsing-routine-number attribute 128 + attribute number 'preposition' adjective number illegal values: 9-15, 112-127This one-byte value has to identify particular prepositions and routines,which is only possible using a numbering system for each. GV1 numbersparsing-routines upwards from 0 to 31, in order of first use. A separatetable translates these into routine packed addresses: the "preactions"table. (As usual, the term is traditional: Inform has no concept ofpreaction, but the Infocom games from which it inherits GV1 do have such aconcept.) The preactions table is a simple --> array.(Note that in Infocom's games the preactions table always has the samelength as the actions table: this is not true in either GV1 or GV2 Informgames.)Prepositions are also identified by their "adjective number". (An earlymisunderstanding in Z-machine decipherment led to the traditional use ofthe word "adjective" for dictionary words explicitly written into grammarlines, which are mainly prepositions like 'into' or 'against'.) Adjectivenumbers count downwards from 255 in order of first use. They are translatedback into dictionary words using the "adjectives table".The adjectives table contains 4-byte entries: <dictionary address of word> 00 <adjective number> ----2 bytes----------------- ----2 bytes----------- To make life more interesting, these entries are stored in reverse order(i.e., lowest adjective number first). The address of this table israther difficult to deduce from the file header information, so the constant#adjectives_table is set up by Inform to refer to it.GV2 is a much more suitable data structure, easier to read and write,less limiting and marginally faster and more economical of Z-machine andInform memory. In GV2 an individual grammar table has the format: <number of grammar lines> 1 bytefollowed by that many grammar lines. Individual lines are no longer always8 bytes long, as in GV1. Instead they have the form: <action number> <token 1> ... <token N> <ENDIT> ----2 bytes---- -3 bytes- -3 bytes- -byte--The action number is actually contained in the bottom 10 bits of the wordgiven first: the top five are reserved for future use, which leaves action_number & $400as a bit meaning "reverse the order of the first and second parametersif this action is to be chosen".The ENDIT marker is the number 15. There can be anything from 0 to 30tokens, and each occupies three bytes, arranged as: <token type> <token data> -- byte ---- --2 bytes---Token type bytes are divided into the top two bits, the next two and thebottom four.The "next two bits" are used to indicate alternatives. In a sequence oftokens T1 / T2 / T3 / ... / Tnthen T1 will have $$10 in its "next two bits", and each of T2 to Tn willhave $$01. Tokens not inside lists of alternatives always have $00. (Notethat at present only prepositions are allowed as alternatives, but theformat is designed to open the possibility of extending this to all tokens.)The bottom four are the "type" of the token. The top two indicate what kindof data is contained in the token data. Strictly speaking this could bededuced from the bottom six bits, but it's convenient for making backpatchingGV2 tables a simple matter within the compiler. Type Means Data contains Top bits 0 illegal (never compiled) 1 elementary token 0 "noun" 00 1 "held" 2 "multi" 3 "multiheld" 4 "multiexcept" 5 "multiinside" 6 "creature" 7 "special" 8 "number" 9 "topic" 2 'preposition' dictionary address 01 3 noun = Routine routine packed address 10 4 attribute attribute number 00 5 scope = Routine routine packed address 10 6 Routine routine packed address 10GV2 makes no use of "adjective numbers" (so that dict_par3 is always zeroin GV2's dictionary words) and leaves both the adjectives table and thepreactions table empty.There is one further difference between GV1 and GV2: in GV1, fake actionsare numbered from 256 upwards; in GV2, from 4096 upwards.Note that although GV2 takes three bytes for a token as compared with GV1'sone byte, omission of the redundant null tokens and adjective table meansthat when compiling a small "shell" library game, GV2 actually producesmore economical tables: 1920 bytes as opposed to 2337.The first two entries in the following table are the real reason for GV2: Limit in GV1 Limit in GV2 Prepositions per game 76 unlimited Parsing routines (general ones, noun= filters, scope= routines all put together) per game 32 unlimited Tokens per grammar line 6 unlimited Actions per game 256 4096 Inform verbs per game 256 256In practice the Inform compiler restrains the number of verbs (but that'san adjustable memory setting) and lines per verb: in Inform 6.05 and earlier,the maximum number of lines per verb is 20. Inform 6.10 internally storesgrammar roughly as GV2 even when it's going to compile GV1 at the end, andthis allows a more economical use of Inform's memory: as a bonus, then, themaximum number of lines per verb is raised to 32 in Inform 6.10. 8.7 Value backpatching ------------------Value backpatching is the final translation phase in Inform. It is theprocess of correcting temporary values which were written into the Z-machineat a time when final values could not be known.In addition to the Z-machine regions marked in the memory map above, thesymbols table also undergoes value backpatching: defined Constant symbolsflagged as CHANGE_SFLAG are backpatched as needed, before first use of theirvalues in backpatching something else.The positions of such values have been "marked" with a "marker value"indicating the type of value needed. The backpatchable marker values are: Marker value Significance of temporary value STRING_MV Scaled address within static strings area ARRAY_MV Byte address within dynamic array area IROUTINE_MV Scaled address within Z-code area VROUTINE_MV Veneer routine number (a *_VR value) NO_OBJS_MV None INCON_MV "Inform constant": the index in the #constants keyword group (a *_SC value) DWORD_MV Accession number of dictionary word INHERIT_MV Byte address within common property values table of the value which is being inherited here INHERIT_INDIV_MV Ditto, but for individual property values table INDIVPT_MV Offset in individual property values table MAIN_MV None SYMBOL_MV Index within symbols tableand these are backpatched as follows: Marker value Final value STRING_MV Packed address of static string ARRAY_MV Byte address IROUTINE_MV Packed address of routine VROUTINE_MV Packed address of veneer routine NO_OBJS_MV Number of objects in object tree INCON_MV Value of constant (typically an address of some Z-machine table) DWORD_MV Byte address of word's entry in dictionary table INHERIT_MV The value to inherit (after it has been backpatched) INHERIT_INDIV_MV The value to inherit (after it has been backpatched) INDIVPT_MV The byte address of this point in the individual property valyes address MAIN_MV Packed address of "Main" routine SYMBOL_MV Value of symbol (after it has been backpatched)The code NULL_MV is sometimes used to indicate "no marker", i.e., that avalue is correct as written. Additional marker values also exist foroperands held within modules: IDENT_MV Property identifier number ACTION_MV Action number OBJECT_MV Object number VARIABLE_MV Variable number(see chapter 10). Note that modules are not value-backpatched at compilationtime, but at Link time.Two different memory blocks are used to store backpatch markers: one for the"Z-machine image", the byte-addressable memory at the bottom of the memorymap; and another for the Z-code area. In the interests of economy, these usedifferent formats to hold marker data: see "bpatch.c". 8.8 Packed address decoding -----------------------Recall that the formula relating a packed address P to its byte address B is: B = 2P version 3 4P versions 4 and 5 4P + 8R versions 6 and 7: routines 4P + 8S versions 6 and 7: static strings 8P version 8In versions 6 and 7, R and S can be chosen fairly freely in the range 0 toM/8, where M is the minimum address of a routine or static string asappropriate. The point is to expand the address space by making higher Bvalues available (whereas P has to lie within 0 and 65535): so one wants tomake R, S larger rather than smaller. On the other hand, it's necessary tothe Inform language that the following are all different: (a) the number 0; (b) valid object numbers; (c) packed addresses of routines; (d) packed addresses of strings.To be sure that (c) and (d) differ, Inform sets S = R. To keep (c) from(a) and (b), we must ensure that the packed address of every routine is atleast X, where X is slightly more than the largest object number. Informalso knows the byte address M of the lowest routine in memory ("Main__").Note that M = 4X + 8RInform therefore nudges up M and X to ensure that M and 4X are both divisibleby 8, and sets R = (M - 4X)/8 S = R.To compare this with the code: Write_Code_At is M; extend_offset is X;scale_factor and length_scale_factor hold the magic constants 4 and 8.code_offset and strings_offset are the scaled addresses of the firstroutine and the first static string respectively, worked out by code_offset = 4X strings_offset = 4X + size of code areaThe reason for having these is that when compilation was actually happening,Inform did not know enough to calculate packed addresses, and insteadwrote scaled addresses starting from 0 for each area. In backpatching,then, Inform adds the above offsets to each packed address, and then theycome right. 9 The run-time veneer ------------------- "I think you now have a fully-stocked veneer shop, with the rest of the compiler being a pencil sharpener bolted to the wall in the back room." -- Andrew Plotkin, letter to the author, 10 January 1999 9.1 Services provided by the veneer -------------------------------The "veneer" is a thin section of code generated by Inform as an intermediatelayer between the code compiled from the source, and the Z-machine itself:like the veneer on a table, it gilds the surface of the Z-machine, or sothe term is supposed to mean.It consists of 38 routines, 4 class-objects, 2 properties and 2 globalvariables, together with several tables in the Z-machine's dynamic memoryand a number of strings, notably the source-code names of arrays andproperties, which help in printing out good error messages. The veneeradds a major data structure not present in the Z-machine architecture:the concept of "individual property", a mechanism to allow games to havemore or less unlimited numbers of properties which don't need to bedeclared before use.The "veneer.c" section of Inform organises the compilation of the routines;each one is compiled only if actually needed, and if no over-riding definitionhas already been given in the source code. (For instance, "PrintShortName"is a veneer routine but the Inform library provides a much fuller versionthan the one in "veneer.c".)Note that the full Inform source code for these routines is present instatic strings in "veneer.c" (which are compiled using the lexer's "readfrom a string" mode).The routines come in five groups. Note that names with double underscoresin are so chosen not to clash with identifiers used by the programmer. Main__ This is not really a routine at all, but is the piece of code which the Z-machine's PC is initially set to. It simply makes a function call to Main(), and then quits. Box__Routine Code to display an array of static strings in a black "quotations" box, as required by the "box" statement. Printing routines PrintShortName, DefArt, CDefArt, etc.; all normally over-ridden within the Inform library. Also RT__Err, "print an error message at run-time". Object orientation routines Routines to implement message sending, access to individual properties, "metaclass", "ofclass" and "provides". Assertion routines Routines needed by "strict mode" to check that proposed object moves, memory read/writes, etc. will not crash the Z-machine if executed.Since veneer routines are compiled at the end of the pass (when it's knownwhich will be needed), the code area looks like this: start of code area --> 00 (routine header for Main__) initial PC value --> @call_1n Main @quit ... routines as defined in source code... veneer routines as required, in order of appearance in the "veneer.c" table(Note that Main__ is assembled as a routine. In the Version 6 Z-machine,this is called to begin execution. In other versions, the initial PC valueis set to the call_1n instruction -- which is why Main__ must be first: theinitial PC value has to lie in the bottom 64K of Z-machine memory. Notealso: for the Versions 3 and 4 Z-machine more primitive call opcodes areused to get into Main.)The four objects in the veneer are the class-objects for the fourmetaclasses defined in all Inform programs: Class, Object, Routine andString. The object tree therefore looks like this: 1 Class 2 Object 3 Routine 4 String ... objects and class-objects as defined in source code ...The two global variables are "self" and "sender", used in message-passing,and which are two of the 7 highest-numbered global variables which Informreserves to its own use as "registers". 9.2 Specification of the veneer routines ------------------------------------Please note that some of the specifications given here may change in futurecompiler releases without warning or apology -- so it is unwise to writecode which accesses the veneer directly.Box__Routine(maxw, table) Display a "Trinity"-style quotation box, whose text is given as a table array of packed addresses of strings giving individual lines, and whose maximum text width is "maxw".R_Process(action, noun, second) Implement <action noun second>. (The Inform library does this properly by defining its own "R_Process" routine; the ordinary veneer would just print text like "<23 2 3>".)DefArt(object)IndefArt(object)CDefArt(object) Print the object's name, prefixed by definite/indefinite/capitalised definite article. (Again, the veneer's plain version is very plain.)PrintShortName(object) Print just the object's short name: but give a sensible response, and in particular don't crash the Z-machine, even if "object" is any illegal value.EnglishNumber(n) Print out "n" in words. (The Inform library does this properly: the plain veneer just prints n as a number.)Print__PName(property) Print a textual description of the property, e.g. "before" or "Coin::before".WV__Pr(object, property, value) Implement object.property = value, printing suitable run-time errors if either object or property are invalid or if the object doesn't provide property.RV__Pr(object, property) Likewise but simply read object.property.CA__Pr(object, property, a, b, c, d, e, f) Implement object.property(a, b, c, d, e, f) with a to f being optional. Note that object might be a class, routine or string and this is where the messages provided by these pseudo-objects are implemented.IB__Pr(object, property) Implement ++object.property.IA__Pr(object, property) Implement object.property++.DB__Pr(object, property) Implement --object.property.DA__Pr(object, property) Implement object.property++.RA__Pr(object, property) Implement object.&property (much more difficult than it sounds: this is where individual property tables are effectively implemented).RL__Pr(object, property) Implement object.#property.RA__SC(class, property) Implement class::property, returning this property number.OP__Pr(object, property) Implement the condition "object provides property".OC__Cl(object, class) Implement the condition "object ofclass class".Copy__Primitive(o1, o2) Make o1 a copy of o2, in the sense that: the attributes of o1 become exactly those of o2; and for every property provided by o2, that property of o1 has the o2 value copied over.RT__Err(error_number, ...parameters...) Print the appropriate run-time error message, which always takes the form "[** Programming error: ... **]". <string>, <class>, <property> "class <class> (object number ...) has no property <property> to <string> [and nor does any other object" <string>, <object>, <property> "<object> (object number ...) has no property <property> to <string> [and nor does any other object" <string>, <class>, -<class2> "class <class> (object number ...) is not of class <class>" <string>, <object>, -<class> "<object> (object number ...) is not of class <class>" 1, <class> "class <class>: 'create' can have 0 to 3 parameters only" 2, <illegal-object> "tried to test "in" or "notin" of object <illegal-object>" 3, <illegal-object> "tried to test "has" or "hasnt" of object <illegal-object>" 4, <illegal-object> "tried to find the "parent" of object <illegal-object>" 5, <illegal-object> "tried to find the "eldest" of object <illegal-object>" 6, <illegal-object> "tried to find the "child" of object <illegal-object>" 7, <illegal-object> "tried to find the "younger" of object <illegal-object>" 8, <illegal-object> "tried to find the "sibling" of object <illegal-object>" 9, <illegal-object> "tried to find the "children" of object <illegal-object>" 10, <illegal-object> "tried to find the "youngest" of object <illegal-object>" 11, <illegal-object> "tried to find the "elder" of object <illegal-object>" 12, <illegal-object> "tried to use "objectloop" of object <illegal-object>" 13, <illegal-object> "tried to use "}" at end of "objectloop" of object <illegal-object>" 14, <illegal-object> "tried to "give" an attribute to object <illegal-object>" 15, <illegal-object> "tried to "remove" object <illegal-object>" 16, <object1> <object2> "tried to "move" <object1> to <object2>" where <object1> is illegal 17, <object1> <object2> "tried to "move" <object1> to <object2>" where <object2> is illegal 18, <object1> <object2> "tried to "move" <object1> to <object2>, which would make a loop: <object2> in ... in <object 1> in <object2>" 19, <object> "tried to "give" or test "has" or "hasnt" for a non-attribute of <object>" 20, "tried to divide by zero" 21, <illegal-object> "tried to find the ".&" of object <illegal-object>" 22, <illegal-object> "tried to find the ".#" of object <illegal-object>" 23, <illegal-object> "tried to find the "." of object <illegal-object>" 24, "tried to read outside memory using ->" 25, "tried to read outside memory using -->" 26, "tried to write outside memory using ->" 27, "tried to write outside memory using -->" 28, <index>, <max-index>, <array-type>, <array-number> "tried to read from -><index> in the <array-type> <array-name>, which has entries <min-index> to <max-index>" The array types are: 0 = byte -> array, 1 = word --> array, 2 = string array, 3 = table array and the array name is the packed string address at #array_names_table--><array-number>. 29, <index>, <max-index>, <array-type>, <array-number> "tried to read from --><index> in the <array-type> <array-name>, which has entries <min-index> to <max-index>" 30, <index>, <max-index>, <array-type>, <array-number> "tried to write to -><index> in the <array-type> <array-name>, which has entries <min-index> to <max-index>" 31, <index>, <max-index>, <array-type>, <array-number> "tried to write to --><index> in the <array-type> <array-name>, which has entries <min-index> to <max-index>" 32, <object> "objectloop was broken because the object <object> was moved while the loop passed through it" 33, <code> "tried to print (char) <code> which is not a valid ZSCII character code for output" 34, "tried to print (address) on something not the byte address of a string" 35, "tried to print (string) on something not a string" 36, "tried to print (object) on something not an object or class"Z__Region(value) Returns 1 if value is a Z-machine object number, 2 if a packed address within the code area, 3 if a packed address within the strings area and 0 otherwise.Unsigned__Compare(x, y) Returns 1 if x>y, 0 if x=y, -1 if x<y, regarding x and y as positive numbers in the range 0 to 65535.Meta__class(value) Implements metaclass(value).CP__Tab(property_table, property) If property is -1, return the address of the first byte after the property table. Otherwise: If the property table contains the property, return the address of the first byte of the property value. If the property table doesn't contain the property, return 0.Cl__Ms(class_object, property, arity, a, b, c, d) Here property must be one of: create, recreate, destroy, remaining and copy. Arity must be the number of parameters supplied, 0 to 4, in the optional variables a, b, c and d. This function implements class_object.property(a, b, c, d). It's really a part of CA__Pr but the code has been moved here to prevent CA__Pr from becoming inconveniently large for the veneer-compilation code.RT__ChT(object1, object2) If it is legal to move object1 to object2, do so; otherwise print an error message. If the library is present and the appropriate debugging flag is set, trace this by printing something like "[Moving torch to Conservatory]".RT__ChR(object) If it is legal to remove object, do so; otherwise print an error message. If the library is present and the appropriate debugging flag is set, trace this by printing something like "[Removing coin]".RT__ChG(object, attribute) If it is legal to give attribute to object, do so; otherwise print an error message. If the library is present and the appropriate debugging flag is set, trace this by printing something like "[Giving torch moved]".RT__ChPS(object, property, value) If it is legal to set object.property = value, where property is known to a common (i.e. declared with Property) property, then do so. Otherwise print an error message.RT__TrPS(object, property, value) Traces an object property setting, printing text such as "[Setting Conservatory.time_left to 7]". This routine is called only when the library is present and the appropriate debugging flag is set.RT__ChLDB(base, offset) If it is legal to read base->offset, do so; otherwise print an error message.RT__ChLDW(base, offset) If it is legal to read base-->offset, do so; otherwise print an error message.RT__ChLDB(base, offset, value) If it is legal to set base->offset = value, do so; otherwise print an error message.RT__ChLDW(base, offset) If it is legal to set base-->offset = value, do so; otherwise print an error message.Symb__Tab(code, num) Only present in games compiled with -X for Infix. In effect this holds some static data in the code area, rather than using up the limited readable memory area with further tables. If code is 1, returns data on the array numbered "num" and stores further in temp_var2 and temp_var3; if code is 2, ditto for routines; if 3, for constants. This routine is subject to change without notice, to put it mildly. 9.3 Properties 2 and 3, "ofclass" and "metaclass" ---------------------------------------------Inform reserves common properties 2 and 3 to itself, and uses them as follows: property 2 (additive): a word array containing the class-object numbers of all classes of which this object is a member; the property is absent if the object belongs to no classes. (Note that metaclasses do not appear here: tree objects are all members of either Object or Class but neither is listed here.) property 3: a byte address pointing to the individual properties table for this object; property 3 is absent if it has no individual properties.The veneer tests "X ofclass Y" according to the rules: if X is a valid Z-machine object number: if X is a class-object (tested by: if X is a child of Class) then only if Y is Class otherwise only if Y appears in the property 2 list for X if X is the packed address of a routine: only if Y is Routine if X is the packed address of a string: only if Y is Stringgiving a run-time error if Y isn't a class-object. Note that determiningwhich of these ranges contains X needs code essentially the same as thatin the Inform 5/12 library routine "ZRegion": indeed, this is how theveneer routine implementing the "metaclass" function works. 9.4 Class numbering and class objects ---------------------------------In Inform 6 each class definition results in the creation of a relatedobject, the class-object for that class.Class objects have: their internal symbol-name as short name; no attributes (*); no (common) properties except possibly for property 3.All except for Class are placed in the object tree as children of Class.Immediately following the property values table for a class (which is boundto be short, since it can only contain the short name and perhaps property 3)is a six-byte block of attributes which would be inherited from the class (*),and then a second property values table, specifying the properties whichmembers of the class would inherit from it. (These values are accessed atrun time to implement the superclass access operator :: when applied to acommon property.)Property 3 is used to hold the byte address of the individual properties table of properties which members would inherit. It is absent if thereare no such properties. (These are accessed at run-time to implement thesuperclass access operator :: when applied to an individual property.)Note that there are two numbering systems used for classes: the "classnumber" counts 0, 1, 2, 3, ... in order of declaration, with Class numbered0. (This is used in superclass message number coding.) Classes are alsoreferred to by the object numbers of their class objects. It's necessaryto be able to convert between the two at run time; the "class numbers table"is a word array whose Nth entry is the class-object number corresponding toclass number N.[(*) This is a change made in Inform 6.12. Previously, class objectshad the attributes which would be inherited from the class. 6.12 movesthis information to a harmless block of six bytes elsewhere, essentiallyso that loops like "objectloop (x has animate)" do not pick up classesas well as objects.] 9.5 Individual property identifiers -------------------------------Just as the common properties are identified by property numbers 1 to 63,so the individual properties are identified by numbers 64 upward. Thegroup 64 to 71 is used to identify the properties provided by inheritancefrom the metaclasses: Id Name Provided by 64 create class-objects other than metaclass-objects 65 recreate ditto 66 destroy ditto 67 remaining ditto 68 copy ditto 69 call objects of metaclass Routine 70 print objects of metaclass String 71 print_to_array dittoHowever, in addition to this a property ID number with its top bit set hasa special meaning: Bit 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 set -------------------------- ----------------------------- entry number class number within class's i.p. table (counting from 0)which means "the given class's inheritable value of the property with thisID number", and is used to implement the "superclass" operator as somethingwhich acts on ID numbers: Class :: identifieris implemented by writing the class number into the bottom bits and searchingas above. Further, a property ID number in the form: Bit 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 00 set --------------------- ----------------------------- common prop number class numberrefers to "the given class's inheritable value of this common property".As a special rule, in the case of class Object this accesses the Z-machinedefault property value. Thus, since the library defines Property cant_go "You can't go that way.";then X.Object::cant_go will always evaluate to "You can't go that way."for any Object X.Note that the need to be able to pack such descriptions into 16 bits is onereason classes need to have a 0, 1, 2, ... numbering system in additionto the class-object system of numbering (which has many lacunae). Even asit is, the present veneer implementation constrains us to a maximum of 256class definitions, each able to provide at most 128 individual properties(in addition to the potentially 63 declared as common using Property);more work may be needed to alleviate this at a later date. 9.6 Individual property tables --------------------------For each object that provides any individual properties at all, property3 holds the byte address of its individual property table. The entriesin this table are pairs of identifier number and current value: thus, raw identifier number (2 bytes) but with top bit set if "private" to the object in question property length (1 byte) current value (as many bytes as specified in the property length, though this will always be an even number >= 2)terminated by a zero word, 00 00. (Note that there is no property 0,common or individual.)The properties can occur in any order and need not be in numerical orderof their IDs (which is helpful to assist linking, though it may have speedimplications). The following Inform code prints out an object's table: [ IndivTable obj x n; x = obj.3; if (x == 0) rfalse; print "Table for ", (object) obj, " is at ", (hex) x, "^"; while (x-->0 ~= 0) { print (hex) x-->0, " (", (property) x-->0, ") length ", x->2, ": "; for (n = 0: n< (x->2)/2: n++) print (hex) (x+3)-->n, " "; new_line; x = x + x->2 + 3; } ]; [ hex x y; y = (x & $ff00) / $100; x = x & $ff; print (hdigit) y/$10, (hdigit) y, (hdigit) x/$10, (hdigit) x; ]; [ hdigit x; x = x % $10; if (x<10) print x; else print (char) 'a'+x-10; ];From an inheritance point of view, individual properties are alwaysnon-additive: a specified value overrides and replaces an inherited value.(The idea is that access to superclass values is a more elegant alternativeto using additive properties to make class values remain available toinheritors which have altered them.)The veneer contains routines to implement "provides", ".", ".&" and ".#",together with routines to apply ++ and -- as pre- or postfix operators,and a routine to implement = in the context of individual properties.These all lapse back into ordinary Z-machine instructions for commonproperties if they find a property number in the range 1 to 63.Note that the veneer's implementation of "provides" contains a specialrule meaning that class-objects do not provide any properties except thefive inherited from the metaclass Object, even though they have a tablein the above format (which contains the inheritable values).The veneer's highest-level routine sends a message to an individualproperty. (This same routine contains implementations of the eight messagesnumbered 64 to 71.) Note that the trickiest part is keeping "self" and"sender" right: push the current "self" value push the current "sender" value set "sender" to "self" set "self" to the destination object now make the function call pull to "sender" pull to "self"Note that the object creation/deletion system works in a quite simple way:the pool of uncreated duplicate members of a class are exactly the childrenof its class-object. Objects are re-initialised when they are returned tothis pool, leaving them "fresh" and ready for recreation. 9.7 Availability of symbol names at run-time ----------------------------------------It is not possible to determine at compile-time if a message is wronglyaddressed, i.e., if a message is sent to property X of object O but object Odoes not provide X. For one thing, availability depends on who is calling(if the property is private to O); for another, message identifiers do notneed to be constants.Therefore a good error-reporting mechanism is needed at run-time; errors like"Object 32 has no property 57" would make debugging hopelessly difficult.The only way to achieve this is for the property names to be available atrun time. Similarly, it's helpful for debugging purposes if attributenames and action names can also be available.To this end, Inform compiles a table of identifier names into everystory file (though not into modules) and provides a system constant called#identifiers_table pointing to the start of the table. The table contains: --> 0 = Number of property names plus 1 = P --> 1 = Packed address of string containing name of property 1 to --> (P-1) = Packed address of string containing name of property P-1 (properties 1 to 63 will be common, and subsequent ones individual: this table includes both kinds; note that there is no property numbered 0) --> P = Packed address of string containing name of attribute 0 to --> (P+47) = Packed address of string containing name of attribute 47 --> (P+48) = Packed address of string containing name of action 0 and so on, followed by #array_names_table-->0 = Packed address of string containing name of array 0 and so on. (Array numbers are allocated from 0 upwards as arrays are declared in the source code: they're used by the veneer only for printing array-bound-violation error messages, and only in strict mode.)In the case of (common) property and attribute names, the ability ofprogrammers to "alias" names together means that these names will sometimesbe given as a list of possibilities with slashes in between.Note that printing property names so as to properly handle usage of ::is harder than simply looking up in this table (for instance, there aretimes when the right output should look like "Coin::value"). Informprovides the construct print (property) X;to handle this. But the corresponding cases for attributes and actionsare easier. Simply define: [ DebugAction a anames; if (a>=256) print "<fake action ", a-256, ">"; anames = #identifiers_table; anames = anames + 2*(anames-->0) + 2*48; print (string) anames-->a; ]; [ DebugAttribute a anames; if (a<0 || a>=48) print "<invalid attribute ", a, ">"; else { anames = #identifiers_table; anames = anames + 2*(anames-->0); print (string) anames-->a; } ];(These are in fact both part of Library 6/3 anyway: a different DebugActionroutine is present in Library 6/1 and 6/2 if DEBUG is set.) You can then: print (DebugAction) X; print (DebugAttribute) X;It is also for error reporting that the names of the classes are needed atrun-time, which is why the short name of a class-object is the name of theclass. 10 Compiling and linking modules ----------------------------- 10.1 Model -----The model for linking is that a game being compiled (called the "external"program) may Link one or more pre-compiled sections of code called "modules". Suppose the game Jekyll has a subsection called Hyde. Then these twomethods of making Jekyll are (almost) equivalent:(i) Putting Include "Hyde"; in the source code for "Jekyll", and compiling "Jekyll".(ii) Compiling "Hyde" with the -M ("module") switch set, putting Link "Hyde"; into the same point in the source code for "Jekyll", and compiling "Jekyll".Option (ii) is of course much faster if "Hyde" does not change very often,since its ready-compiled module can be left lying around while "Jekyll"is being developed.Option (ii) also slightly increases story file size, as some codeoptimisations are impossible this way: e.g., linking rather than includingthe library costs about 160 bytes on final story file size.In the linking process, two incarnations of Inform need to communicate witheach other: A. one in the middle of compiling a story file and wanting to link a module into it; B. one compiling a module for future linking into some unknowable story file.To be more precise, B needs to send information to A. It does so by writinga modified form of story file with additional tables attached, and this iswhat a module is.We discuss how Inform copes with situation A first, as this indicates theinformation which B needs to send it; we then discuss how B gathers andsends this information.Note that A and B can never apply in the same run-through of Inform, sincea module cannot link another module into it.Finally, note that A and B may be different ports of Inform running ondifferent machines at different times, since the module format ismachine-independent. 10.2 Linking a module ----------------There are two tasks: (a) merging the module's data structures into those ofthe external story file, and (b) mending all the constants used in these datastructures which couldn't be known at module compilation time.(a) Merging data structuresThis is formidably hard, since it means extracting tables in Z-machine formatand putting them back into Inform's own (usually higher-level) datastructures: in effect, an inverse is needed for each translation operation.The following table briefly lists the data structures which are merged, andhow this is done: Structure Method Initial global variable values Directly copy in Initial array entry states Append Code area Append. Z-code is relocatable except that function calls are to absolute addresses: these are handled as though they were references to constants unknown at module compilation time Static strings area Append Dictionary Extract words from the module, one at a time, and insert into the story file's dictionary. (The dictionary is just too complex to merge the internal structures.) Object tree Append, and also fix all the object numbers used in parent, sibling and child fields: a particular concern is with classes defined in the module, which have to be added as children of the main program's Class object Property values area Append Individual property values Append "Flags 2" header bits Bitwise OR Class to object numbers table Effectively append, moving the object numbers up appropriately. The module's class numbers table contains offsets of the class inheritance property blocks as well, and these are appended too(b) Mending referencesThe problem here is to absorb numerous references within the module intothe backpatch table for the story file. It is not possible just to appendthe module's backpatch tables onto the story file's backpatch tables:all the offsets need adjusting, and certain kinds of backpatching need totake place immediately (to fix the four marker values not allowable instory file backpatch tables, ACTION_MV, IDENT_MV, VARIABLE_MV and OBJECT_MV). 10.3 Imports and exports -------------------The main "extra" ingredients in a module (compared with an ordinary storyfile) are: two backpatching tables (see the next section), and animport/export table giving the names of symbols to be shared with the mainstory file which will link in with it.The language of "imports and exports" considers the module to be the homecountry, and the external program to be foreign. (An exported symbol's valueis assigned in the module and used in the external program; an importedsymbol's value is assigned in the external program and used in the module.)An export is a symbol defined in the module which may be referred toin the external program. All variables, routines and general constantsdefined in the module except for (a) system-defined symbols always present(such as "Class", "true", etc.) and (b) attributes, common properties andlabels are exported.An import is a symbol name used in compiling the module which isn't definedwithin it. During module compilation, just as in story file compilation, ifan unknown name is hit then an operand is constructed which is markedSYMBOL_MV and has the symbol index as its value. In the case of a storyfile, at value backpatching time the correct value is found out and writtenin. In the case of a module, there is no value backpatching phase.Instead, all symbols are flagged with IMPORT_SFLAG if ever used when undefined(i.e., if any SYMBOL_MV marker is ever issued). Any which remain undefinedat the end of module compilation (in fact, at the same point where exportsare made) are "imported": that is, their names are written down so that theirexistence can be checked on at Link time. It is an error for any importedsymbol not to exist in the external program performing the Link.Note, therefore, that backpatch markers in the module with marker valueSYMBOL_MV may refer either to symbols which were assigned values later onduring module compilation (and are thus exported) or to symbols which werenever assigned values (and are thus imported). 10.4 Backpatch backpatching ----------------------Although modules undergo branch backpatching and optimisation just as storyfiles do, they do not undergo value backpatching.Instead, the two value-backpatching tables (zcode_backpatch_table andzmachine_backpatch_table) are appended to the module file. The basic ideais that when the external file merges the module's data structures into itsown, it also adds the module's backpatch markers into its own collection. Thus, the module's code and data are finally value-backpatched when theexternal program is value backpatched.Unfortunately, these backpatch markers have addresses which are correct forthe module, but wrong for the external program: they require correctionbefore they can be transferred, and this is called "backpatch backpatching".In addition to this, a few special types of backpatch markers (only evergenerated inside modules, never in story files) are dealt with immediately.A further complication is that the module may have exported a symbol whichitself has a value needing backpatching. For example, if the module contains Constant Swine 'hog';then, in its symbol table, the symbol Swine will have the marker DWORD_MVattached to its value. When this is exported to the external program, thesymbol value will need backpatch-backpatching.The sequence of events is roughly: 1. Load the module into an allocated block of memory. 2. Merge the dictionary, finding out which dictionary accession numbers in the external program correspond to which in the module. 3. Go through the import/export table, until we know which symbol numbers in the module correspond to which symbol numbers in the external program; and which variable numbers to which; and which action numbers to which; and which property identifiers to which. (Arrays called "maps" are created to encode all these.) 4. Backpatch, or deal with the backpatch markers attached to, any exported symbols from the module. 5. Go through the Z-code backpatch table: deal with IDENT_MV, ACTION_MV, VARIABLE_MV and OBJECT_MV markers immediately, by backpatching the copy of the module held in memory; backpatch the other markers and then transfer them into the external program's Z-code backpatch table. 6. Do the same for the Z-machine backpatch table. 7. Now merge the memory copy of the module into the external program.(There are actually 17 stages, but most of the latter ones are mergers ofdifferent tables which can happen in any order.)As an example of "backpatch backpatching", the backpatch marker for thequantity 'hog' will be DWORD_MV, with value the accession number of the word"hog" in the module's dictionary. This accession number needs to be changedto the accession number of "hog" in the story file's dictionary.The four "module only" backpatch marker values are: VARIABLE_MV global variable number in module's set OBJECT_MV object number in the module's tree IDENT_MV identifier number in the module's set ACTION_MV action number (always between 0 and 255, though always in a "long" constant so that a fake action number can be backpatched over it)Note that within the module, there is no way to tell an externally definedaction from a fake action. That is, the reference ##Duckling might beto an external action or fake action called "Duckling". Within the module,it is created as an action name in the usual way; at Link time, the actionmap corresponds this module action number to a fake action number or a realone as required.Variable numbers in the module are not marked if they are local (or thestack pointer), or if they are from the system set ("self", "sender" and soon), whose numbers are guaranteed to be correct already. In a module, theavailable variable numbers are 16 to N-1, where N is the number of the lowestsystem-used variable (at present, 249). Imported variables are numberedfrom 16 upwards, and variables defined newly in the module from N-1downwards. If these allocations collide in the middle, then the Z-machinehas run out of variables. 10.5 How modules differ from story files -----------------------------------Module files are identical to story files, except that they have notundergone value backpatching, and do not contain any part of the veneer; andexcept as detailed below.(i) Size---------A module is at most 64K in size.(ii) Header entries (where different from story file header entries)-------------------------------------------------------------------- Byte Contents 0 64 + V (where V is the version number) 1 module version number 6,7 byte address of the module map table (note that this is the "initial PC value" slot in the story file header format, but no such value is meaningful for modules)V is used to make sure that we aren't trying to link, e.g., a Version 5module into a Version 8 story file: this would cause heaps of backpatcherrors, as the packed addresses would all be wrong. Although we couldin principle automatically fix them all up, it isn't worth the trouble:the user only needs to compile off suitable versions of the modules.The module version number is a version number for the module formatbeing used: this document describes module version 1. (iii) Class numbers table-------------------------In a module, the class numbers table contains additional information,and has the form: <word containing object number for Class> <word containing class 1 inheritance-properties block offset> <word containing object number for class 1> <word containing class 1 inheritance-properties block offset> <word containing object number for class 2> <word containing class 2 inheritance-properties block offset> ... <word containing object number for class N> <word containing class N inheritance-properties block offset> 00 00 (In a story file, the inheritance-properties block offsets are absent.)(iv) Identifier names table---------------------------This is missing altogether in a module.(v) Dictionary--------------A module dictionary is in accession order, not alphabetical order. (This isnecessary so that backpatch markers in the module, which refer to dictionarywords by accession number, can be connected with their original words atlink time.)(vi) Module map---------------The module map is (currently) 15 words long and contains the following: Word 0 byte address of object tree 1 byte address of common property values table 2 scaled address of static strings table 3 byte address of class numbers table 4 byte address of individual property values table 5 number of bytes in indiv prop values table 6 number of symbols in module's symbols table 7 number of property identifier numbers used in module 8 number of objects present in module 9 byte address of import/export table 10 its size in bytes 11 byte address of Z-code backpatch table 12 its size in bytes 13 byte address of Z-machine image backpatch table 14 its size in bytesThe map serves as an extension to the header, which has more or less run outof available slots for new pieces of information.(vii) Parents of class objects------------------------------Objects representing classes are given in the object tree as havingparent $7fff (and having next-object 0, as if each were an only child).These are to be understood as being children of the Class object (whichis not present in a module).(viii) Import/export table--------------------------This is a sequence of symbols (mixing up imports and exports in no particularorder), occupying records as follows: Record type (1 byte) Symbol number in module symbols table (2 bytes) Symbol type (1 byte) Symbol backpatch marker (1 byte) Symbol value (2 bytes) Symbol name (null terminated)where the possible record types are: IMPORT_MV import a symbol (in which case the symbol backpatch marker is omitted) EXPORT_MV export a symbol defined in a non-system file EXPORTSF_MV export a symbol defined in a system file EXPORTAC_MV export an action name (in the form "Name__A")Note: we need to distinguish between EXPORT_MV and EXPORTSF_MV in order tomake Replacement of routines work properly. Suppose the user asks to Replacethe routine DrawStatusLine, normally found in the parser, and then links theparser module and also a module of his own containing his new definition ofDrawStatusLine. The linker knows which of these to accept because the firstwas compiled from a system module, and thus exported with EXPORTSF_MV, whilethe second was not, and was exported with EXPORT_MV. Other than for thispurpose, the two values are treated equivalently.(ix) Z-code backpatch table---------------------------This is exactly the module's Z-code backpatch table, no different in formatfrom that used for backpatching story files (except that four additionalmarker values are permitted).(x) Z-machine image backpatch table-----------------------------------This is exactly the module's Z-machine backpatch table, no different informat from that used for backpatching story files (except that fouradditional marker values are permitted). 10.6 Restrictions on what modules may contain ----------------------------------------[1] It is impractical to allow the module and the external program use of each others' attribute and property names, because there is no rapid way of repairing the module's object tables. Instead, both program and module must use the same Include file to make the same attribute and property definitions as each other.[2] An object defined in a module cannot: (a) have an externally-defined parent object, or (b) inherit from an externally-defined class.[3] A module can only use externally-defined global variables if they have been explicitly "imported" using the Import directive.[4] A module cannot use Verb or Extend: that is, it cannot define grammar.[5] A module cannot use Stub or Default (for obvious reasons).[6] A module cannot use "unknown at compile time" constants in every context. (Unknown in that, e.g., 'frog' and MAX_FROGS might be unknown: the former because the dictionary address of 'frog' in the final program can't be known yet, the latter (say) because MAX_FROGS is a constant defined only in the external program.) In the following list of contexts in which constants occur in Inform source code, those marked with a (*) are not permitted to be "unknown at compile time": 1. In high-level source code (including as a switch value and a static string in a "box" or "string" statement). 2. In assembly language source code. 3. As the initial entries in an array. 4. As the initial value of a variable. 5. In a CONSTANT definition. 6. As an object's (common or individual) property value. * 7. As a release number. (Defining the module release number only.) * 8. As a version number. (Modules are always version 5.) * 9. In grammar. (You can't define grammar in a module.) * 10. As the size of an array. * 11. In a DEFAULT constant definition. (You can't use Default.) * 12. In an IFTRUE or IFFALSE condition. * 13. As a property default value (though this is unnecessary since the definitions in the outside program take precedence). * 14. As the number of duplicate objects provided for a class. Only (10) and (14) are restrictive. Combining a module with a short Include file to make such definitions will get around them. 11 Service levels -------------- 11.1 Variables and arrays --------------------Each of the sections is responsible for the initialisation of its ownvariables, and for the allocation and deallocation of its own arrays. Tothis end, every section except "inform.c" (which is considered to be incharge of the others) provides a set of four routines to manage itsdata structures: init_*_vars() called when the compiler is beginning a run *_begin_pass() called at the beginning of the compilation pass through the source code: this usually sets some of the variables *_allocate_arrays() called when the compiler is beginning a run *_free_arrays() called when the compiler is ending its runNote that *_free_arrays() must exactly undo what *_allocate_arrays() hasdone.The 19 routines of each kind are polled in no particular order (actually,in alphabetical order).Note that there are also lexer_begin_prepass() files_begin_prepass()which need to happen even earlier than the general *_begin_pass() poll,and also lexer_endpass() linker_endpass()but since the other sections generally don't need to do anything at endof pass, it wasn't worth making a formal poll of it.Static initialisation of variables is only allowed when they are certainnever to change (for example, the string containing the accent escapecharacters never changes). 11.2 Memory allocation and deallocation ----------------------------------All allocation and deallocation is routed through routines given in"memory.c": specifically, my_malloc() and my_free()Note that these are called with a string as an extra parameter (as comparedwith malloc() and free()), which is used to print helpful error messages incase of collapse. This is where the -m ("print the memory allocation") switchis implemented, and it's also a good place to add any OS-specific code whichmay be needed.This may be needed because, under ANSI rules, malloc() is entitled to refuseto allocate blocks of memory any larger than 32K minus a few bytes. On a fewimplementations malloc() actually does this, even on very large machines,because it's convenient to avoid memory segmentation problems. (For instance,on a few versions of C for the Macintosh, it's possible to have 16M freeand still be refused a request for 33K of it.) Consequently it may benecessary to use some compiler-specific routine instead.Inform allocates many arrays (50 or so) and in a normal compilation run,the largest of these occupies about 26K. However, the implementation ofLink requires the whole of a module file to be held in one contiguous blockof memory, potentially 64K long; and the same applies to the array holdingthe unpaged memory area of the Z-machine during construct_storyfile().Failure of my_malloc() causes a fatal error.Inform also provides a structure (memory_block) for allocating extensibleregions of memory (at a slight access-speed penalty): these allocate8K chunks as needed. Three of these are used as alternatives to thetemporary files when -F0 is operating, and two more are used to holdbackpatch markers. 11.3 Error messages --------------All diagnostic and error messages are routed through the "errors.c" section(except for errors in parsing ICL, as this takes place before the compileritself is active).There are three kinds of diagnostic: warning, error and fatal error.Fatal errors cause Inform to halt with an exit(1), which is about as fatalas anything can be to a C program. (Fatal errors are mainly issued when thememory or file-handling environment seems to have gone wrong.) Errorsallow compilation to carry on, but suppress the production of any outputat the end: in some ways this is a pity, but there are not many errors whichone can be confident of safely recovering from.Note that a sequence of MAX_ERRORS (normally 100) errors causes a fatal error.Error messages which are generated during the compilation pass try to quotethe source code line apparently responsible (unless the "concise" switchis set): but Inform may be unable to do this if the text was read and losttoo long ago, for example in the case of a "declared but not used" warning.Note that informational messages (such as the banner line, the statisticsprintout, etc.) are not routed through "errors.c", or indeed anywhere else.Error recovery in Inform is mainly a matter of going into "panic mode"(amusingly, this is a term of art in compiler theory): tearing through thetokens until a semicolon is reached, then starting afresh with a newstatement or directive. This pretty often goes wrong (especially if aroutine start/end is not noticed), and more work might be needed here. 11.4 File input/output -----------------The "files.c" section handles almost all of the file input/output in Inform:the two exceptions are in cli_interpret() in "inform.c", which loads in ICLfiles; and in link_module() in "linker.c", which loads in modules (a simplejob which there was no point abstracting).The biggest shake-up to this section since Inform 5 was the realisation that,far from being a grubby low-level activity, working out good filenames wasin fact a task which had to be done much higher up in Inform: so all ofthe code which used to do this is now located in the ICL interpreter in"inform.c".What remains in "files.c" is: opening source code and reading it into buffers; opening and closing the temporary files; calculating the checksum and length of, and then outputting, the story file or module being compiled; opening, writing to and closing the text transcript file; opening, writing to and closing the debugging information file.For each different source file that has ever been open, "files.c" maintainsa structure called FileId, containing the full filename and a file handle.The files are never read from except via the routine file_load_chars(),which fills up the lexical analyser's buffers.The routine output_file() writes the story/module file: see its commentsfor details. It takes up where construct_storyfile() leaves off, andcontributes the last two pieces of header data to be worked out, thechecksum and the length fields. Note that the checksum calculation isonly known after the whole file has been written (mainly because Z-codebackpatching alters the checksum, but can't be done until output time withoutconsuming a good deal of extra memory); so that an "fseek" is made to skipthe write position back into the header and overwrite the correct checksumjust before closing the file. This call is made in a strictly ANSI waybut, like everything on the fringes of the C library-to-operating-systeminterface, may cause trouble on some compilers.The other routines are largely self-explanatory.There are three temporary files (although they won't be used if -F0 applies),as follows: Temporary file Contents 1 The static strings area 2 The Z-code area 3 The link data area (only opened and used in -M, module mode)For the format of the debugging information file, see the utility "infact.c",which prints from it. This format is likely to change in the near futurein any case, as it is already inadequate for some of the new features inInform 6. 12 Low-level language features --------------------------- 12.1 Using the "Trace" directive ---------------------------The trace directive is primarily intended for compiler maintenance purposes.It can cause Inform to print tracing information on nine different aspectsof what it does, in most cases at several levels of detail. ------------------------------------------------------------------------- Tracing option Level Output ------------------------------------------------------------------------- "assembly" 1 all assembly language produced, in an imitation of "@..." syntax 2 and also the actual bytes produced, in hexadecimal (Note that because trace printing occurs before label optimisation, addresses of instructions cannot be relied on: nor can the values of operands which are marked for later backpatching.) "tokens" 1 lexemes of all tokens output from the lexer to the syntax analyser, and indication when a token is put back 2 full token descriptions 3 and also the lexical context in which they were identified "expressions" 1 annotated parse trees for all expressions being code-generated 2 and the behaviour of the shift-reduce parser when parsing it, and the parse tree received by the code generator (not yet annotated) 3 and the token-to-etoken translations made, and the parse tree produced by the emitter before lvalue checking "linker" used in 1 list all import/export information sent compiling module 2 and all marker information sent "linker" used in 1 list all modules linked in compiling story file 2 and all import/export information received 3 and all marker information received 4 and how each marker was dealt with "lines" --- (currently inoperable) ------------------------------------------------------------------------- "dictionary" 1 print current dictionary contents (including verb/preposition nos.) in alphabetical order "objects" 1 print current object tree "verbs" 1 print current grammar "symbols" 1 print out those entries in the symbols table which are not: unknown, generated by the veneer or in a system file 2 print entire symbols table ------------------------------------------------------------------------- 12.2 System constants and other secret syntax ----------------------------------------The addresses of many important tables in the Z-machine are not recordedin the header, or anywhere else: but they are known to the compiler, andneeded by the run-time code. The system constants are provided mainlyas a way of passing this information into run-time code, usually codewithin the veneer. Constant Evaluates to -------------------------------------------------------------------------- #version_number Version number of Z-machine format being compiled to #dict_par1 Byte offset in a dictionary table entry of the the first ("flags") parameter byte #dict_par2 And the second ("verb") byte #dict_par3 And the third ("adjective") byte (note that these three depend only on the version number) #largest_object The largest object number constructed, plus 256 (the "+256" is due to a quirk in the implementation used by Inform 3 or so; since this constant is not a very public feature, I don't mind leaving it in) #actual_largest_object Ditto, but without the "+256" #adjectives_table Byte addresses of adjectives table, #actions_table actions table, #preactions_table "preactions table", #classes_table class number to object number table, #identifiers_table property ID names table #array_names_offset array names table #readable_memory_offset The byte address of the first byte which isn't accessible using readb and readw: i.e., the first byte of the first Z-code routine #code_offset Packed address of Z-code area #strings_offset Packed address of static strings area #array__start Start of array space (byte address) #array__end End of array space + 1 #cpv__start Start of common property values space (byte address) #cpv__end End + 1 #ipv__start Start of individual property values space (byte addr.) #ipv__end End + 1 --------------------------------------------------------------------------Two more secret syntaxes were introduced in Inform 6.10. The first changesthe behaviour of the "Include" directive: Include "Language__";(and no other string) includes the current language definition file, whosename is an ICL variable.The second controls which grammar table format is generated: normally GV1,but this can be set to GV2 by Constant Grammar__Version = 2;The "Grammar__Version" symbol is redefinable; if no such Constant directiveis made, then it will have the value 1. It needs to be changed to itsfinal value before the first Verb, Extend or Fake_Action directive isreached. 12.3 The "Zcharacter" directive --------------------------Finally, the "Zcharacter" directive is provided mostly for the benefitof language definition files, for configuring Inform games to use anon-English alphabet or character set. (See the Inform Translator'sManual.) Different forms of "Zcharacter" allow both the Z-machinealphabet table and the Unicode translation table to be specified.(i) The Z-machine alphabetThe Z-machine's text encryption system is optimised to make it especiallycheap on memory to use letters in alphabet 1, then cheapish to use lettersin alphabets 2 and 3 but rather expensive to use letters which aren't inany of the three. We aren't much concerned about lack of memory in thegame as a whole, but the economy is very useful in dictionary words,because dictionary words are only stored to a "resolution" of nineZ-characters. Thus, in a dictionary word: something from alphabet 1 costs 1 Z-character 2 or 3 2 Z-characters outside the alphabets costs 4 Z-charactersThe standard arrangement of these alphabets (A1 lower case a to z, A2 uppercase A to Z, A3 numerals and punctuation marks) includes no accentedcharacters. In a language with frequent accented or non-English letters,such as Finnish or French, this means that 4 of the 9 Z-characters ina dictionary word may be wasted on just one letter. For instance, 't@'el@'ecarte' is stored as 't@'el' 't@'el@'ephone' is stored as 't@'el'(there are not even enough of the 9 Z-characters left to encode the seconde-acute, let alone the "c" or the "p" which would distinguish the two words).On the other hand if e-acute could be moved into Alphabet 3, say in place ofone of the punctuation marks which is never needed in dictionary words,the two e-acutes would take just 2 Z-characters each and then 't@'el@'ecarte' would be stored as 't@'el@'ecar' 't@'el@'ephone' would be stored as 't@'el@'epho'which is far more acceptable.The Z-machine has a mechanism (at least in Version 5 or better) for changingthe standard alphabet tables, invented to make the German translation ofInfocom's "Zork I" work. The "Trace dictionary" will print the currentcontents of the alphabet table (as well as the dictionary contents).(i).1 Moving a single character inThere are two ways to change the standard English alphabet table.One way, which is probably good enough for a language definition filefor a mostly Latin language (where only up to around 10 accented ornon-English letters are commonly needed) is to move characters intothe least-used positions in Alphabet 2. For this, use the directive: Zcharacter <char>;It will only be possible if there's a letter in A2 which hasn't yetbeen used (otherwise, changing that entry in A2 will make some of thetext already compiled wrong). The directive is thus only practicableearly in compilation, such as at the start of the library definition file.For instance the code Trace dictionary; Zcharacter '@'e'; Zcharacter '@`a'; Zcharacter '@^a'; Trace dictionary;might produce output including...Z-machine alphabet entries: a b c (d) e (f) g (h) i j k l m n o (p)(q) r s t u v (w)(x)(y)(z) A (B) C (D) E (F)(G) H I (J)(K) L (M)(N) O (P)(Q) R S (T)(U)(V)(W)(X)(Y)(Z)( ) ^ 0 1 (2) 3 (4)(5) 6 (7)(8) 9 (.) , (!)(?)(_)(#)(')(~) / (\) - (:)(()())Z-machine alphabet entries: a b c (d) e (f) g (h) i j k l m n o (p)(q) r s t u v (w)(x)(y)(z) A (B) C (D) E (F)(G) H I (J)(K) L (M)(N) O (P)(Q) R S (T)(U)(V)(W)(X)(Y)(Z)( ) ^ 0 1 @'e 3 @`a@^a 6 (7)(8) 9 (.) , (!)(?)(_)(#)(')(~) / (\) - (:)(()())...in which note that bracketed letters are ones which have not been encodedyet. The three Zcharacter directives have inserted e-acute, a-grave anda-circumflex into the positions previously occupied by the numerals 2, 4 and5. It is reasonable to make up to about 10 such insertions, after which anyfurther attempts will only be successful if the game being compiled doesn't(let us say) have a title like "123456789: An Interactive Lesson In Counting",which would have used 9 of the numerals and forced them to stay in the finalalphabet table.(i).2 Changing the entire alphabetThis has to be done very early in compilation, before any strings aretranslated, so that it can't be done by a language definition file.One might put such directives into a file called "Alphabet.inf" andthen begin the main game with Include "Alphabet";to achieve this.The form required is to give three strings after "Zcharacter", containing26, 26 and 23 characters respectively. For instance: Zcharacter "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "0123456789!$&*():;.,<>@{386}";(Note that "@{386}" specifies only one character: Unicode $0386.) Space,new-line and quotation marks " are automatically included, while~, @ and ^ have special meanings in Inform and should not be used.Otherwise, any arrangement of characters is fine, except that everycharacter used has to be either a normal ASCII character or part ofthe "extra characters" (already declared) in the ZSCII set.(ii) Defining the "extra characters"Inform normally makes up a block of "extra characters" based on thesource code it reads: if it reads plain ASCII or ISO Latin1 (-C0 or-C1) then the block contains the usual European accents, such ase-acute or i-circumflex, as defined in Z-machine Standard 0.2. (Andif this table is never changed, Inform doesn't then compile thetable at all, as this is the default Z-machine arrangement.)More generally if Inform reads ISO 8859-n (-Cn) then the block is setup to contain all the non-ASCII letter characters in ISO 8859-n.There's room to spare for others to be added, and Zcharacter table + '@{386}';would add Unicode character $0386 (Greek capital Alpha with tonosaccent, as it happens) to the current stock of "extra characters".Alternatively, you can simply give a fresh stock altogether: Zcharacter table '@{9a}' '@{386}' '@^a';would specify a stock of just three, for instance.These directives must be made before the characters in question arefirst used in game text.(iii) Defining terminating charactersIt's also possible to specify which ZSCII character codes are"terminating characters", meaning that they terminate a line of input.Normally, the return key is the only terminating character, but thiscan be added to. For instance, the following directive makesZSCII 132 and 136 terminating: Zcharacter terminating 132 136;The legal values to include are those for the cursor, functionand keypad keys, plus mouse and menu clicks. The special value 255makes all of these characters terminating. 12.4 Sequence points ---------------Inform marks certain positions in the code it compiles as being"sequence points". The idea is that the code can be regarded as asequence of chunks, and the sequence points mark where these chunksbegin. Roughly speaking, each different statement in Inform source codecompiles to a different chunk, so that statements correspond closely tosequence points. Sequence points are marked in assembly trace outputusing the notation "<*>". For instance, the source code [ WorkOutSquares counter; counter = 0; while (counter < 100) { squares-->counter = counter*counter; counter = counter + 1; } ];produces the traced output: 6 +00008 [ WorkOutSquares counter 7 +00009 <*> store counter short_0 8 +0000c .L0 8 +0000c <*> jl counter short_100 to L1 if FALSE 9 +00011 <*> mul counter counter -> sp 9 +00015 storew long_480 counter sp 10 +0001b <*> add counter short_1 -> counter 11 +0001f jump L0 11 +00022 .L1 12 +00022 <*> rtrue The "<*>" in front of an instruction means "the position where thisinstruction begins is a sequence point". We could mark the fivepositions in the original source code as: [ WorkOutSquares counter; <*> counter = 0; <*> while (counter < 100) { <*> squares-->counter = counter*counter; <*> counter = counter + 1; } <*> ];Note that the open and close braces and square brackets don't normallycause sequence points. The exact rule is that every statement,action < > command, assignment or expression in void context is ata sequence point, except as shown in the examples below: for (<*> i=0: <*> i<45: <*> i++) ..."for" loops contain 0 to 3 sequence points, depending on whether there'sany code compiled in the three parts of the specification. For instance for (::) <*> print "Madness!";contains no sequence point corresponding to the "for" specification. <*> objectloop (<*> O ofclass Coin) <*> print (name) O;"objectloop" normally generates two sequence points: at the start,where the variable is initialised, and then where it's tested. However,loops over the contents of particular objects work differently: <*> objectloop (O in Mailbox) <*> print (name) O;(Because the test "O in Mailbox" is not actually being performed atrun-time: instead, O is looping through the tree.) do <*> print counter++, " "; <*> until (counter < 17);Here the sequence point generated by the loop itself is attached tothe "until" clause, not the "do" clause, because that's where thetest is performed."switch", "while" and "if" statements are not exceptions to theusual rule (1 statement = 1 sequence point at the beginning), but itmight be useful to give some examples anyway: <*> switch(counter) { 1: <*> print "One^"; 2, 3: <*> print "Two or three^"; default: <*> print "Neither^"; } <*> if (i == 17) <*> print "i is 17"; else <*> print "i isn't 17"; <*> while (i<100) <*> print i++;The following is true: Except possibly in code explicitly assembled using the "@" notation, at each sequence point the Z-machine stack is empty and no important information is held in the global variables reserved by Inform as "registers": thus, it's safe for a debugger to switch execution from any sequence point in a routine to any other. No two sequence points can be at the same position in either the source code or the compiled code. Every sequence point corresponds to a definite position in the source code (because the veneer, i.e. the code compiled from within Inform itself, contains no sequence points).But the following is _not_ true: Sequence points occur in the same order in the source code as they do in compiled code Every routine contains at least one sequence point (a very few "stub" routines are excluded)Inform uses sequence points only to generate debugging informationfiles, and to annotate assembly tracing output. They do not affect thecode compiled. 12.5 Format of debugging information files -------------------------------------This is a provisional specification of a format which will probablychange slightly in future releases. Support for the old -k option hasbeen re-introduced in Inform 6.12 to assist development of Infix, theprojected source-level debugger for Inform. (See the minor utilityprogram "infact", updated to 6.12 format, which prints out the contentsof a debugging information file in legible form.)A debugging information file begins with a six-byte header: 0,1 the bytes $DE and then $BF (DEBF = "Debugging File") 2,3 a word giving the version number of the format used (currently 0) 4,5 a word giving the current Inform version number, in its traditional decimal form: e.g. 1612 means "6.12"The remainder of the file consists of a sequence of records, terminatedby an end-of-file record. These records may be in _any_ order unlessotherwise noted. Each record begins with an identifying byte, for whichconstants looking like *_DBR are defined in Inform's source code.A "string" is a null-terminated string of ASCII chars.A "word" is a 16-bit unsigned number, high byte first.A "line" is a sequence of four bytes: the first is the file number, the next two are a line number (a word), and the last is a character number within that line. In all three cases -- file numbers, line numbers, character numbers -- counting begins at 1. The line reference 0:0:0 is however used to mean "no such line": for instance, the metaclass "Routine" is defined at line 0:0:0, because it's defined by the compiler, not in any source code. Character positions greater than 255 in any line are recorded simply as 255.An "address" is a 24-bit unsigned number, a sequence of three bytes (high byte, middle byte, low byte). All addresses are counted in bytes (rather than being Z-machine packed addresses). EOF_DBR (byte: 0) End of the debugging file. FILE_DBR (byte: 1) <file number> 1 byte, counting from 1 <include name> string <actual file name> string One of these records always appears before any reference to the source code file in question. CLASS_DBR (byte: 2) <name> string <defn start> line <defn end> line OBJECT_DBR (byte: 3) <number> word <name> string <defn start> line <defn end> line GLOBAL_DBR (byte: 4) <number> byte <name> string ARRAY_DBR (byte: 12) <byte address> word <name> string The byte address is an offset within the "array space" area, which always begins with the 480 bytes storing the values of the global variables. ATTR_DBR (byte: 5) <number> word <name> string PROP_DBR (byte: 6) <number> word <name> string FAKE_ACTION_DBR (byte: 7) <number> word <name> string Note that the numbering of fake actions differs in Grammar Versions 1 and 2. ACTION_DBR (byte: 8) <number> word <name> string HEADER_DBR (byte: 9) <the header> 64 bytes This is provided in order to check that a debugging information file (probably) does match a given story file. ROUTINE_DBR (byte: 11) <routine number> word <defn start> line <PC start> address <name> string then for each local variable: <local name> string terminated by a zero byte. Note that the PC start address is in bytes, relative to the start of the story file's code area. Routines are numbered upward from 0, and in each case the ROUTINE_DBR, LINEREF_DBR and ROUTINE_END_DBR records occur in order. LINEREF_DBR (byte: 10) <routine number> word <number of sequence points> word and then, for each sequence point: <source code position> line <PC offset> word The PC offset for each sequence point is in bytes, from the start of the routine. (Note that the initial byte of the routine, giving the number of local variables for that routine, is at PC offset 0: thus the actual code begins at PC offset 1.) It is possible for a routine to have no sequence points (as in the veneer, or in the case of code reading simply "[; ];"). ROUTINE_END_DBR (byte: 14) <routine number> word <defn end> line <next PC value> address MAP_DBR (byte: 13) A sequence of records consisting of: <name of structure> string <location> address terminated by a zero byte. The current names of structures consist of: "abbreviations table" "header extension" "alphabets table" "Unicode table" "property defaults" "object tree" "common properties" "class numbers" "individual properties" "global variables" "array space" "grammar table" "actions table" "parsing routines" "adjectives table" "dictionary" "code area" "strings area" Other names made be added later, and some of the above won't be present in all files ("Unicode table", for instance). Locations are byte addresses inside the story file.LINEREF_DBR records will probably be compressed in future releases. 12.6 Notes on how to syntax-colour Inform source code ------------------------------------------------"Syntax colouring" is an automatic process which some text editors applyto the text being edited: the characters are displayed just as they are,but with artificial colours added according to what the text editor thinksthey mean. The editor is in the position of someone going through a bookcolouring all the verbs in red and all the nouns in green: it can only doso if it understands how to tell a verb or a noun from other words.Many good text editors have been programmed to syntax colour for languagessuch as C, and a few will allow users to reprogram them to other languages.One such is the popular Acorn RISC OS text editor "Zap", for which theauthor has written an extension mode called "ZapInform". ZapInformcontributes colouring rules for the Inform language and this sectiondocuments its algorithm, which has since been successfully adapted byPaul Gilbert's "PIDE" environment and John Wood's C++ code for Informsyntax styling, both running under Windows 95/NT. (My thanks to Johnfor making two corrections to the previously-published algorithm.)(a) State valuesZapInform associates a 32-bit number called the "state" with everycharacter position.The "state" is as follows. 11 of the upper 16 bits hold flags, therest being unused: 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 comment single-quoted text double-quoted text statement after marker highlight flag highlight all flag colour backtrack after-restart-flag wait-direct (waiting for a directive) dont-know-flagThese flags make up the "outer state" while the lower 16 bits holdsa number pompously called the "inner state": 0 after WS (WS = white space or start of line or comma) 1 after WS then "-" 2 after WS then "-" and ">" [terminal] 3 after WS then "*" [terminal] 0xFF after junk 0x100*N + S after WS then an Inform identifier N+1 characters long itself in state S: 101 w 202 wi 303 wit 404 with 111 h 212 ha 313 has 121 c 222 cl 323 cla 424 clas 525 class same + 0x8000 when complete [terminal] In practice it would be madness to try to actually store the stateof every character position in memory (it would occupy four times asmuch space as the file itself). Instead, ZapInform caches just onestate value, the one most recently calculated, and uses a processcalled "scanning" to determine new states. That is, given that weknow the state at character X and want to know the state at characterY, we can find out by scanning each character between X and Y,altering the state according to each one.It might possible save some time to cache more state values thanthis (say, the state values at the start of every screen-visibleline of text, or some such) but the complexity of doing this doesn'tseem worthwhile on my implementation. Scanning is a quick processbecause the Zap text editor stores the entire file in almost contiguousmemory, easy to run through, and the state value can be kept in asingle CPU register while this is done.(b) Scanning textLet us number the characters in a file 1, 2, 3, ...The state before character 1 is always 0x02000000: that is, innerstate zero and outer state with only the waiting-for-directive flag set.(One can think of this as the state of an imaginary "character 0".)The state at character N+1 is then a function of the state atcharacter N and what character is actually there. Thus, State(0) = 0x02000000and for all N >= 0, State(N+1) = Scanning_function(State(N), Character(N+1))And here is what the scanning function does: 1. Is the comment bit set? Is the character a new-line? If so, clear the comment bit. Stop. 2. Is the double-quote bit set? Is the character a double-quote? If so, clear the double-quote bit. Stop. 3. Is the single-quote bit set? Is the character a single-quote? If so, clear the single-quote bit. Stop. 4. Is the character a single quote? If so, set the single-quote bit and stop. 5. Is the character a double quote? If so, set the double-quote bit and stop. 6. Is the character an exclamation mark? If so, set the comment bit and stop. 7. Is the statement bit set? If so: Is the character "]"? If so: Clear the statement bit. Stop. If the after-restart bit is clear, stop. Run the inner finite state machine. If it results in a keyword terminal (that is, a terminal which has inner state 0x100 or above): Set colour-backtrack (and record the backtrack colour as "function" colour). Clear after-restart. Stop. If not: Is the character "["? If so: Set the statement bit. If the after-marker bit is clear, set after-restart. Stop. Run the inner finite state machine. If it results in a terminal: Is the inner state 2 [after "->"] or 3 [after "*"]? If so: Set after-marker. Set colour-backtrack (and record the backtrack colour as "directive" colour). Zero the inner state. [If not, the terminal must be from a keyword.] Is the inner state 0x404 [after "with"]? If so: Set colour-backtrack (and record the backtrack colour as "directive" colour). Set after-marker. Set highlight. Clear highlight-all. Is the inner state 0x313 ["has"] or 0x525 ["class"]? If so: Set colour-backtrack (and record the backtrack colour as "directive" colour). Set after-marker. Clear highlight. Set highlight-all. If the inner state isn't one of these: [so that recent text has formed some alphanumeric token which might or might not be a reserved word of some kind] If waiting-for-directive is set: Set colour-backtrack (and record the backtrack colour as "directive" colour) Clear waiting-for-directive. If not, but highlight-all is set: Set colour-backtrack (and record the backtrack colour as "property" colour) If not, but highlight is set: Clear highlight. Set colour-backtrack (and record the backtrack colour as "property" colour). Is the character ";"? If so: Set wait-direct. Clear after-marker. Clear after-restart. Clear highlight. Clear highlight-all. Is the character ","? If so: Set after-marker. Set highlight. Stop.The "inner finite state machine" adjusts only the inner state, andalways preserves the outer state. It not only changes an old innerstate to a new inner state, but sometimes returns a "terminal" flagto signal that something interesting has been found. State Condition Go to state Return terminal-flag? 0 if "-" 1 if "*" 3 yes if space, "#", newline 0 if "_" 0x100 if "w" 0x101 if "h" 0x111 if "c" 0x121 other letters 0x100 otherwise 0xFF 1 if ">" 2 yes otherwise 0xFF 2 always 0 3 always 0 0xFF if space, newline 0 otherwise 0xFF all 0x100+ states: if not alphanumeric, add 0x8000 to the state yes then for the following states: 0x101 if "i" 0x202 otherwise 0x200 0x202 if "t" 0x303 otherwise 0x300 0x303 if "h" 0x404 otherwise 0x400 0x111 if "a" 0x212 otherwise 0x200 0x212 if "s" 0x313 otherwise 0x300 0x121 if "l" 0x222 otherwise 0x200 0x222 if "a" 0x323 otherwise 0x300 0x323 if "s" 0x424 otherwise 0x400 0x424 if "s" 0x525 otherwise 0x500 but for all other 0x100+ states: if alphanumeric, add 0x100 to the state 0x8000+ always 0(Note that if your text editor stores tabs as characters in their ownright (usually 0x09) rather than rows of spaces, tab should be includedwith space and newline in the above.)Briefly, the finite state machine can be left running until it returnsa terminal, which means it has found "->", "*" or a completed Informidentifier: and it detects "with", "has" and "class" as special keywordsamongst these identifiers.(c) Initial colouringZapInform colours one line of visible text at a time. For instance, itmight be faced with this: Object -> bottle "~Heinz~ bottle"And it outputs an array of colours for each character position in theline, which the text editor can then use in actually displaying the text.It works out the state before the first character of the line (the "O"),then scans through the line. For each character, it determines theinitial colour as a function of the state at that character: If single-quote or double-quote is set, then quoted text colour. If comment is set, then comment colour. If statement is set: Use code colour unless the character is "[" or "]", in which case use function colour, or is a single or double quote, in which case use quoted text colour. If not: Use foreground colour unless the character is "," or ";" or "*" or ">", in which case use directive colour, or the character is "[" or "]", in which case use function colour, or is a single or double quote, in which case use quoted text colour. However, the scanning algorithm sometimes signals that a block oftext must be "backtracked" through and recoloured. For instance,this happens if the white space after the sequence "c", "l", "a","s" and "s" is detected when in a context where the keyword "class"is legal. The scanning algorithm does this by setting the "colourbacktrack" bit in the outer state. Note that the number of characterswe need to recolour backwards from the current position has beenrecorded in bits 9 to 16 of the inner state (which has been countingup lengths of identifiers), while the scanning algorithm has alsorecorded the colour to be used. For instance, in Object -> bottle "~Heinz~ bottle" ^ ^ ^backtracks of size 6, 2 and 6 are called for at the three markedspaces. Note that a backtrack never crosses a new-line.ZapInform uses the following chart of colours: name default actual colour foreground navy blue quoted text grey comment light green directive black property red function red code navy blue codealpha dark green assembly gold escape character redbut note that at this stage, we've only used the following: function colour [ and ] as function brackets, plus function names comment colour comments directive colour initial directive keywords, plus "*", "->", "with", "has" and "class" when used in a directive context quoted text colour singly- or doubly-quoted text foreground colour code in directives code colour code in statements property colour property, attribute and class names when used within "with", "has" and "class"For instance, Object -> bottle "~Heinz~ bottle"would give us the array DDDDDDDDDDFFFFFFFQQQQQQQQQQQQQQQQ(F being foreground colour; it doesn't really matter what colourvalues the spaces have).(d) Colour refinementThe next operation is "colour refinement", which includes a numberof things.Firstly, any characters with colour Q (quoted-text) which have specialmeanings are given "escape-character colour" instead. This appliesto "~", "^", "\" and "@" followed by (possibly) another "@" and anumber of digits.Next we look for identifiers. An identifier for these purposes includesa number, for it is just a sequence of: "_" or "$" or "#" or "0" to "9" or "a" to "z" or "A" to "Z".The initial colouring of an identifier tells us its context. We'reonly interested in those in foreground colour (these must be usedin the body of a directive) or code colour (used in statements).If an identifier is in code colour, then: If it follows an "@", recolour the "@" and the identifier in assembly-language colour. Otherwise, unless it is one of the following: "box" "break" "child" "children" "continue" "default" "do" "elder" "eldest" "else" "false" "font" "for" "give" "has" "hasnt" "if" "in" "indirect" "inversion" "jump" "metaclass" "move" "new_line" "nothing" "notin" "objectloop" "ofclass" "or" "parent" "print" "print_ret" "provides" "quit" "random" "read" "remove" "restore" "return" "rfalse" "rtrue" "save" "sibling" "spaces" "string" "style" "switch" "to" "true" "until" "while" "younger" "youngest" we recolour the identifier to "codealpha colour".On the other hand, if an identifier is in foreground colour, then wecheck it to see if it's one of the following interesting keywords: "first" "last" "meta" "only" "private" "replace" "reverse" "string" "table"If it is, we recolour it in directive colour.Thus, after colour refinement we arrive at the final colour scheme: function colour [ and ] as function brackets, plus function names comment colour comments quoted text colour singly- or doubly-quoted text directive colour initial directive keywords, plus "*", "->", "with", "has" and "class" when used in a directive context, plus any of the reserved directive keywords listed above property colour property, attribute and class names when used within "with", "has" and "class" foreground colour everything else in directives code colour operators, numerals, brackets and statement keywords such as "if" or "else" occurring inside routines codealpha colour variable and constant names occurring inside routines assembly colour @ plus assembly language opcodes escape char colour special or escape characters in quoted text(e) An exampleConsider the following example stretch of code (which is not meant tobe functional or interesting, just colourful): ! Here's the bottle: Object -> bottle "bottle marked ~DRINK ME~" with name "bottle" "jar" "flask", initial "There is an empty bottle here.", before [; LetGo: ! For dealing with water if (noun in bottle) "You're holding that already (in the bottle)."; ], has container; [ ReadableSpell i j k; if (scope_stage==1) { if (action_to_be==##Examine) rfalse; rtrue; } @set_cursor 1 1; ]; Extend "examine" first * scope=ReadableSpell -> Examine;Here are the initial colourings: ! Here's the bottle: CCCCCCCCCCCCCCCCCCCC Object -> bottle "bottle marked ~DRINK ME~" DDDDDDDDDDFFFFFFFQQQQQQQQQQQQQQQQQQQQQQQQQQ with name "bottle" "jar" "flask", FFDDDDDPPPPPQQQQQQQQFQQQQQFQQQQQQQD initial "There is an empty bottle here.", FFFFFFFPPPPPPPPQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQD before FFFFFFFPPPPPP [; LetGo: ! For dealing with water FFFFFFFfSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSCCCCCCCCCCCCCCCCCCCCCCCC if (noun in bottle) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS "You're holding that already (in the bottle)."; SSSSSSSSSSSSSSSSSQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQS ], SSSSSSSfD has container; FFDDDDDPPPPPPPPPD [ ReadableSpell i j k; fffffffffffffffSSSSSSS if (scope_stage==1) SSSSSSSSSSSSSSSSSSSSS { if (action_to_be==##Examine) rfalse; SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS rtrue; SSSSSSSSSSSS } SSS @set_cursor 1 1; SSSSSSSSSSSSSSSSSS ]; fD Extend "examine" first DDDDDDDQQQQQQQQQFFFFFF * scope=ReadableSpell -> Examine; FFFFFFFFFFFFFFFFDDFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFDDDFFFFFFFD(Here F=foreground, D=directive, f=function, S=code (S for"statement"), C=comment, P=property, Q=quoted text.) And here isthe refinement: ! Here's the bottle: CCCCCCCCCCCCCCCCCCCC Object -> bottle "bottle marked ~DRINK ME~" DDDDDDDDDDFFFFFFFQQQQQQQQQQQQQQQEQQQQQQQQEQ with name "bottle" "jar" "flask", FFDDDDDPPPPPQQQQQQQQFQQQQQFQQQQQQQD initial "There is an empty bottle here.", FFFFFFFPPPPPPPPQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQD before FFFFFFFPPPPPP [; LetGo: ! For dealing with water FFFFFFFfSSIIIIISSSSSSSSSSSSSSSSSSSSSSSCCCCCCCCCCCCCCCCCCCCCCCC if (noun in bottle) SSSSSSSSSSSSSSSSSIIIISSSSIIIIIIS "You're holding that already (in the bottle)."; SSSSSSSSSSSSSSSSSQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQS ], SSSSSSSfD has container; FFDDDDDPPPPPPPPPD [ ReadableSpell i j k; fffffffffffffffSSSSSSS if (scope_stage==1) SSSSSSIIIIIIIIIIISSIS { if (action_to_be==##Examine) rfalse; SSSSSSSSSSIIIIIIIIIIIISSIIIIIIIIISSSSSSSSS rtrue; SSSSSSSSSSSS } SSS @set_cursor 1 1; SSAAAAAAAAAAASISIS ]; fD Extend "examine" first DDDDDDDQQQQQQQQQFDDDDD * scope=ReadableSpell -> Examine; FFFFFFFFFFFFFFFFDDFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFDDDFFFFFFFD(where E = escape characters, A = assembly and I = "codealpha", thatis, identifiers cited in statement code). 13 What little I remember ---------------------- 13.1 Publication history -------------------The language has existed in the following different forms on the Internet: Inform 1 v0.5 April 30th 1993 Inform 2 v0.6 (I can't find any record of this date) Inform 3 v1.0 November 17th 1993 Inform 3a December 7th 1993 Inform 4 January 20th 1994 Inform 5 June 1st 1994 Inform 5.1 June 12th 1994 Inform 5.2 June 19th 1994 Inform 5.3 (released on an "Acorn User" cover disc) Inform 5.4 October 2nd 1994 update later in October 1994 Inform 5.5 v1501 late May/early June (private circulation) v1502 June 30th 1995 Inform 6 v6.01 April 30th 1996: declared a "beta" release v6.02 May 5th 1996 v6.03 May 10th 1996 v6.04 September 12th 1996: declared no longer "beta" v6.05 September 24th 1996 Inform 6.1 v6.10 December 16th 1996 v6.11 January 27th 1997 v6.12 March 26th 1997 v6.13 April 5th 1997 v6.14 September 8th 1997 v6.15 March 22nd 1998 Inform 6.2 v6.20 December 10th 1998: declared a "beta" v6.21 April 30th 1999There have been a few other very slightly modified states, consisting ofbug fixes and source code portability improvements on the above (e.g.Inform 4 went through a fairly long process of tidying-up once it reachedthe porters), and a number of CD ROM releases.This base of users has continuously grown, from size 1 (myself) to what attime of writing is a very large group of at least casual users and a group ofperhaps 50 "serious" ones whom I actually know. Inform's early versions werevery hard to use, as will become clear, and only attracted attention because: (a) it did something undeniably "cool", to aficionados of 1980s adventure games - it revived the Infocom run-time format, and the early Inform manuals were for a while the fullest documents on the Web describing the Infocom format; and (b) since "Curses" had been written in it, the compiler must at least work. (Even though the source code for "Curses" has never been made public, the fact that it was known to exist underwrote the project and made it seem worthwhile to the many porters who invested considerable time and effort on initially complex and poorly written code.)I first posted Inform to the Internet in the spirit of "glasnost" rather thanwith any pretension that it would be a widely used program. (It worked forme, which was all that was then required.) By mid-1994 my ambitions for ithad grown: in looking back, that first year is the interesting one. 13.2 Design history --------------Inform was designed specifically as a language to express programs whichwould run on the Z-machine, a run-time format which already existed. Inturn, the Z-machine was designed specifically as a run-time format foran existing language (ZIL, or Zork Implementation Language, a form of MDL).Inform is therefore the result of a game of Chinese whispers, since I had noidea what ZIL looked like until much later.In fact Inform and ZIL are entirely dissimilar. One reason for this is thatZIL is syntactically a very high-level language, not unlike LISP, and itscompiler (ZILCH) made a substantial translation down to Z-machine level:whereas Inform was designed very close to the Z-machine level. It is oftenremarked that Inform resembles C, and although this is partly because somesyntax conventions were borrowed from C by a designer who felt comfortablewith it, another reason is that both languages were designed for easytranslation to efficient low-level code.Inform began as an assembler called "zass", whose syntax continues to exerta (perhaps malign) influence on the language of today. The zass assemblymnemonics were different from the modern set (this was at a time when nostandardisation of opcode names had taken place), and were copied from theset of names output by Mark Howell's disassembler "txd". (The opcodenames recognised by Inform were tinkered with until Inform 5 when the standardset was agreed.) But the format of routines and labels remains. A typical"zass" routine looked like this: [ Routine v1 v2 ... vn an assembly line .alabel another assembly line ]and semicolons were introduced as line separators when "zass" became Inform1. Individual assembly lines then took the form opcode operand1 ... operandn store;(if they were store operands), with no syntactic indication that the lastoperand was not really an operand but a store destination. This persisteduntil Inform 5, and is still recognised in Inform 6, though the preferredsyntax is now opcode operand1 ... operandn -> store;Right up until Inform 5, similarly, "sp" was a reserved word meaning "theassembly-language stack pointer", preventing its use as a local variablename (for example). Only in Inform 6 has it disappeared from the set ofreserved words (except in the context of assembly lines).One of the most unusual features of Inform is its handling of local variablesto functions, in that (a) functions can be called with any number of arguments; (b) there's no specification by the programmer of any expected number of arguments; (c) local variables and argument-receiving variables are treated as the same; (d) there are at most 15 local variables per routine.All four arise from the fact that this is how Z-machine programs behave:these were the rules for zass, and they're still the rules for Inform 6.It's efficient to identify Inform's idea of local variables with theZ-machine implementation of locals; nevertheless, the decision to do sowas made by default.This illustrates a general point about restrictions in the Inform syntax.The Z-machine is rich, well-equipped with features making it a good targetfor compilation. It implements, let us say, feature F subject to a modestrestriction (for example, it implements local variables subject to a limitof 15 per routine). Since this is good enough for most purposes, andefficient, and easy to compile to, an early release of Inform decided tomake use of the Z-machine's implementation directly; and consequently thedesign of the Inform language was shaped around the restriction neededto make it possible to use F.In other words, if a sparer "RISC" target machine had been chosen, Informwould have been obliged to make a new implementation of feature F, whichwould not have been subject to arbitrary Z-machine restrictions.Perversely, it is therefore Inform's greatest weakness (as well as itsgreatest selling point) that it compiles to the Z-machine. At least, though,there is a not inconsiderable reward from using the Z-machine's featuresso directly: compact and fast code results.In any case, the history of Inform is a continuous process of syntax movingaway from direct expression of the Z-machine and towards a higher-leveldescription of a program.Something that partially frustrated this was that Inform acquired a serioususer base "too soon", before the language design was mature enough: by Inform4, a good many people were using Inform, and this began to make me nervous ofchanging the established syntax. But if a syntax feature is clearly "wrong"(that is, anomalous and inconvenient) then clearly one must bite the bulletand make a change. In retrospect, I feel I have always been too hesitantover this.For example, I realised when working on Inform 5.1 that the notation print "You swallow ", (the) food, " with glee.";to mean "print the name of the object whose number follows, preceding itby a suitable definite article", would be desirable. But I had wantedInform 5 to be a point of stability, not rapid change, and so I shied awayfrom adding this feature (besides, I was worried about complicating theinteraction between the library and the language itself). I did not in factadd the feature until Inform 5.5, but it would have been easier for allconcerned had I done so earlier.The moral to draw would be that any really big changes still needed oughtto be made now. Unfortunately, they are just too big (the syntax forcharacter and dictionary literals, for example, or for array and propertyvalue initialisation) and so I have failed to learn my lesson, after all.Another problem with an evolutionary design is that vestigial features tendto survive, especially if one is trying to serve a user base who may stillbe using old syntaxes. (For instance, using old library files of code.)A problem P, say, is given a crude solution S1 in 1993, a slightly bettersolution S2 in 1994 and a proper solution S3 in 1995. By this time, doesInform still have to recognise the syntaxes applying to S2 and S1? UsuallyI have decided to make it do so; the formal concept of an "obsolete usage"(which would be recognised but cause "please change this" warnings to beissued) was not added to Inform until 5.5.For example, how do we set a property value? In Inform 1, by using a lineof assembly code: put_prop lamp oil_left 15;By Inform 3 this had become painful, and a shift away from assembly languagewas taking place, but not a drastic one. The statement "write" was devisedfor writing a sequence of property values (in this example, just one): write lamp oil_left 15;(Even this existed partly because of a horrid form of "object recycling"present in Autumn 1993 forms of "Curses" and very early drafts of "Jigsaw",causing objects to be heavily over-written: it became necessary to changemany properties in one go. The "write" statement was a great help.)Only in Inform 4 did the C-like form appear: lamp.oil_left = 15;And not until Inform 6 was it possible to apply ++ and -- to a propertyspecified in this way. Likewise, only in Inform 6 was the "write" statementfinally withdrawn from recognition, though it was flagged as obsolete inInform 5.5.Such obsolete syntaxes do not merely increase the "clutteredness" of thelanguage; they also block more desirable syntaxes by causing clashes. Andthe tendency to provide partial solutions to problems has been unfortunatein putting off the point where significant change is needed. I did notrealise in "zass" that a notation was needed to refer to dictionary wordsas operands of opcodes, automatically creating them if need be; then I didrealise this, in Inform 1, but made a directive called "Dictionary" to setconstant symbol names equal to their dictionary addresses; in Inform 3,I found this was cumbersome and instead used the syntax #n$... to indicate"new dictionary word" (it seemed logical since #w$... was being used for"existing dictionary word"); in Inform 5, the syntax '...' was used, andthis was perhaps a mistake, since it clashes slightly with the syntax forspecifying literal characters.If I had realised at Inform 1 that a syntax was needed, this would have beenavoided. As a footnote, #n$... is still present in the language; #w$...and "Dictionary" were finally deleted in Inform 6.A happier point of view on this evolution is that, at every stage, the subsetof Z-machine features which Inform could use expanded. Inform 1 could produceonly version 3 games; Inform 4 was a major breakthrough in making version 5games; and so on. (V5 did not become the default version compiled tountil Inform 5, but this reflects a lack of confidence in the interpretersfor high-version-number Z-machines then available.) And an evolved syntaxis at least adapted to what its users want of it. (i) Inform 1 and 2 --------------Inform 1, then, was the assembler "zass" rewritten and with certain shorthandsadded. For example, var = number operator number;was permitted, though full expressions were not. Most of the "creating"directives were present -- Object, Global, Property and Attribute -- thoughdirectives were not syntactically distinguished from statements, and couldbe mixed freely. Nor were statements distinguished from assembly lines:indeed, most of them were assembly lines.In addition, control structures were provided: if ... else ... while ... ... do ... until ...These took their modern form, except that conditions did not need to bebracketed, and braces were compulsory around all code blocks, even thosewhich contained only one statement. A form of "for" loop was provided,but took the BASIC-like form "for var 1 to 10". (This syntax existedalongside the modern "for" syntax until Inform 6.)Grammar and actions existed in their modern form from the first (exceptfor the <...>, <<...>> notation for causing them). Object definitions tooktheir modern form (except that specifying a parent was compulsory, there wasno "class" segment and routines could not be given as property values).And that was about all. The language appears from this description to bevery primitive, and it was, but it was not so different in appearance from a"real" language (except for the lack of compound expressions) and "Curses"was written in Inform 1 code for about two years before being translatedinto modern syntax.Inform 2 was only a tidying-up release of Inform 1. (ii) Inform 3 and 3a ---------------Inform 3 was a more substantial upgrade, with various worthy features(command line switches; control over header flag bits) added. But thedriving force was that "Curses" was being expanded, and running severely intothe limits of the Z-machine Version 3 architecture: in particular, the limitof 255 objects. "Curses" apparently needed a minimum of about 270: RichardTucker suggested to me that there was no need for the 255 physical Z-machineobjects to correspond directly to the 270 game objects. So the idea of"recycling" was born. Object O might be a carpet in one room and a door inanother: as long as it was impossible for these ever to be simultaneouslypresent. When either room was entered, everything about O was over-writtento make it the appropriate definition.This made it necessary to be able to drastically alter objects. The "write"statement (see above) was introduced to fix properties, a syntax #n$...was introduced to refer to dictionary words (because so many were needed asproperty values to over-write with) and the @.. string escape was addedto make use of the Z-machine's abbreviations table, with the "string"statement added for changing entries in it. (This last made it possible forthe apparently unalterable object short-name to consist of a string reading"print abbreviation 1", so that by changing abbreviation 1 it was possible toeffectively alter short names.) Finally, object declaration syntax wasaltered to make it possible to give the same object more than one symbolname.In retrospect this substantial effort to break the game-object Z-objectcorrespondence was misguided; the proper solution to the problem was to make Inform Version-5 capable (as happened in Inform 4). It ought to beremembered, though, that at this time many people were using Version 3interpreters not capable of running Version 5 games (indeed, the Version 5release of "Curses" was one of the nudges towards a wider awareness ofVersion 5 and the interpreters capable of running it). Inform hung backbecause interpreters were not ready; yet as events turned out, interpretersbecame ready partly because of Inform overtaking them.Inform 3a was a tidying-up release of Inform 3. (iii) Inform 4 --------Inform truly became a high-level language, rather than an assembler withdelusions of grandeur, only with the release of Inform 4.The important new addition was a full expression evaluator, recognisingoperators (and an expanded set of these, including for instance ++ and --),precedence levels, brackets and so on. Conditions were also permitted tobe compound, using the new logical operators && and ||. Properties couldbe referred to using the new "." operator (along with ".&" and ".#").System functions (such as "children" and "random") were added.This was the decisive break of the correspondence between Inform statementsand Z-machine instructions: expressions no longer looked like three-addresscode by any other name, and might compile to arbitrarily long sequences ofinstructions. Opcodes like "random" were no longer accessed directly, butvia system functions.The most annoying syntax point remaining was that braces were stillcompulsory. A typical Inform 3 line might have been: if i == 2 { print "Two"; }This restriction was now lifted. Implementation considerations(specifically, in the source line-reader which fed into the tokeniser) meantthat some indication was needed of where the condition ended, and so bracketswere imposed on conditions. Thus, the corresponding Inform 4 line: if (i == 2) print "Two";Making the parallel with C even closer, new-style "for" loops wereintroduced: for (i=0:i<10:i++) print i;(and the desire to make Inform capable of compiling "for" loops as complex asthose available to C caused the operators ++, -- and , to be added). Moreoriginally, "objectloop" was added.Although it had little impact on the syntax, to the outside world the majordevelopment was that Inform was now capable of compiling Version 5 files(which had no limit on object numbers and could be twice as large as Version3 ones). The main impact of this on the language (which I now regret) wasthat statements like "box" and "style" were added to control newly availabletext features of the Z-machine.Otherwise I took the decision that in principle all Inform programs should becross-compilable to all versions (subject to the limits applying in eachversion), and this remains substantially true. Since certain game operationssuch as saving, loading and reading the keyboard were commanded by verydifferent assembly language in versions 3 and 5, these were abstracted intostatements so that, for instance, what looked like version-3 assemblylanguage to read the keyboard would in fact be compiled to appropriate butdifferent version-5 assembly. (iv) Inform 5 --------The final stage in Inform's arrival as a programming language took place alittle over a year after Inform 1's creation. Only then did I look back inleisure at the language, and only then did I begin to seriously find out aboutthe rival adventure-game languages to see what they were capable of. I wasunimpressed except by TADS (which I continue to have a great respect for):the point was rubbed home when I was adapting the original mainframeAdventure, "Colossal Cave", to Inform for use as an example. Dave Baggett'sTADS translation was constantly at my side, and it became obvious that thecorresponding Inform 4 code was far less legible.In some ways, then, the Inform 5 syntax was worked out by constructing thelanguage so as to make the implementation of "Advent" look neat.Although Inform had, from the first, associated code closely with objects(the concept of before/after routines was part of the original conception),only now was this brought out in syntax by making explicit routine definitionsproperty values. (Previous code had been full of property values like"FireplacePre", which were names of what would now be called before-routines.)The syntax now seems obvious and natural, but at the time it felt like abreakthrough: what had been broken through, I think, was the residual feelingthat Inform was an assembler at heart, whose programs were a list of routinesinterspersed with some overgrown variable definitions.Class definitions (and the inheritance rules) followed in the same, somewhatbewildering, couple of days, and a brace of aesthetic changes followed:dictionary words written 'thus', ##Names for actions (rather than theprevious #a$Name syntax), the use of bare strings as implied "print_ret"statements. Most importantly, the use of <...> and <<...>> to triggeractions, and the implementation of what amounted to a switch statement on thecurrent action within embedded routines.I have come to realise that adventure game programs are unusually dense inselections and comparison against lists of alternative possibilities. Informnow contains many syntaxes to make such comparisons concise: switching on actions in embedded routines; the "switch" statement (added in Inform 5.5) - note that both of these deliberately take a more concise form from, say, C, by not allowing case fall-through and therefore not requiring the incessant use of "break" statements; switch ranges such as "5, 7, 8 to 19: ..."; the "or" alternative operator in conditions, as in "if (x == 1 or 5) ...".And this, I feel, is something for which I should make no apology.Later developments (in 5.5) included the Array directive (previously, it wascustomary to use global variables to point to arrays) and the printingsyntaxes for "print (The) ..." and so forth (see above). An extension wasmade to "Versions 7 and 8" (new hybrids of the Z-machine allowing largergames to be compiled), though this had little impact on the language or thecompiler.The release of Inform 5.5 notwithstanding, the language remained essentiallystable during the two years between Inform 5 and Inform 6. For the firsttime it became possible for people other than myself to seriously use Inform,rather than toy with it. From Inform 5 onwards, the issue has not beenadequacy of the language but adequacy of its documentation, and I think thisis the point where Inform can be counted a "proper" language. 13.3 Implementation history ----------------------Writing a compiler for a large language from scratch requires glacialpatience, but the result is a mountain. The time invested in tedious detailsduring development is repaid ten times over when the compiler comes to beused. Temporary measures are infernal temptations, and must be rejectedfirmly.Inform was not written this way.The compiler has been through three "design iterations": "zass", which wasdiscarded as soon as I thought I had understood how the Z-machine worked,Inform 1 to 5.5, and now Inform 6. This section briefly describes the seconditeration.Inform 1 was essentially a program for translating assembly language linesinto Z-code statements. It made the generous and unjustified assumptionthat its input would always be well-formed except for the odd spellingmistake, and did not check syntax particularly carefully: lexical analysisconsisted of grabbing a line from the source code files (accessing thesefiles a byte at a time, in an unbuffered way) and cutting out white spaceto leave a set of tokens. Syntax analysis consisted of going throughthese tokens until a line seemed to have worked out all right, and thenforgetting the line altogether (never checking if there were any more tokens).This caused an enormous number of little jobs later, testing for errorconditions like "expected a semicolon after ... but found ..." in a quiteunsystematic way.Not only was there no formal apparatus for syntax analysis, but lexicalanalysis went on all over the place, with strings being cut up and comparedagainst the symbols table erratically (and often repeatedly). Tokens werestored as strings (with all the consequent inefficiencies of manipulation).Direct string comparisons were commonplace, making it difficult to pin downwhat the set of reserved words was, and indeed almost impossible to writea precise definition of what programs would pass Inform syntax checking.When statements required translation into code, this translation wouldactually be made into textual assembly language instructions, fed back intoInform via a "preprocessor". (As Inform became more elaborate, thispreprocessing might take three stages: high level code such as "objectloop"constructs being knocked down to equivalent "while" statements and then toassembly language lines.)In order to cope with the problem of forward references to labels, Informmade two passes through its assembly language source, which is fairly standardbehaviour for assemblers (because it is easy to implement and has lowmemory overheads). Yet it became a more and more nightmarish process toensure that high-level constructs would always translate into identical codeon pass 1 and pass 2, and ever more clear that great effort was being madeto do a long and difficult calculation twice, burning the result the firsttime.Given all this, the wonder is that Inform worked at all. In fact it workedrather well, after a couple of years of polishing and optimisation: althoughslow in theory, profiling runs suggested that the process of translating intotextual assembly language (for instance) was not a significant speed loss(since opcodes could be textually recognised very quickly, while the workin parsing operands had been deferred from earlier on, and would have to havebeen done anyway). Inform performed very rapidly in terms of compilationtime per source line when compared to typical commercial compilers (which,to be fair, were compiling more complex languages) and was unusually lowin memory consumption.This last was a serious issue when Inform was released, as many users wererunning it on machines like 0.5M Amigas. Without a grown-up syntax analyser,there was no need to store large parse trees (such as most compilers do forentire routines at a time: the performance of most compilers seriouslydegrades as the length of a single routine increases).From Inform 1 to 5.5, the main algorithms and issues never changed, despitemuch reorganisation, renaming, rewriting and extension. But after two and ahalf years, it had clearly "rusted" badly: it was difficult to maintain orwork on. Bugs were appearing which seemed ridiculous in a program soestablished and widely used: for example, not until 1995 did anybody realisethat division had been accidentally programmed as right, not left associative(thus "12/6/2" evaluated to 4, not 1). An endless mass of ad-hoc rules wasbeing added to cover up the lack of proper lexical analysis. Moreover, themajor new feature -- the linker -- added another massive complication to analready convoluted piece of code. (I have by now designed and painfully gotworking three different versions of the linker, and would not go through allthat again for all the tea in China.)What began as a general tidying-up and bug-fixing exercise sank into thedepressing realisation that a total rewrite was the only way to makesignificant progress. Six months later, the writing of this sentence finallycompletes that task. 13.4 Modification history -------------------- (i) Changes made between v6.01 and v6.02 ------------------------------------Features added:"Declared but not used" and "no such constant" errors now reported on or near line of occurrence, not at end of source.Error message added for local variable being defined twice for the same routineBugs fixed:"Segmentation fault" (i.e., writing memory out of range) fixed to do with symbol flags being set wronglyConstant binary numbers not lexed correctly"for (p = 0::)" misinterpreted due to "::" being recognised as an operatorBackpatch error under RISC OS (and possibly elsewhere) when linker usedGrammar token routines reported as "declared but not used"Grammar token routines not working at run time"ofclass" failing to recognised inherited class membershipInheritance of negated attributes not working"children" function giving the wrong answer (specifically, always giving the object number of the first child); this had the knock-on effect of making the "tree" debugging verb produce incorrect outputSource code cleaning:Header file rearranged to typedef int32 in the right place, and tidied up a little; ICL_Directory defined rather than left to cause an error when Inform is compiled; INFORM_FILE replaced with MAIN_INFORM_FILE in some of the OS definition blocks; SEEK_SET defined rather than left to the vagaries of ANSIType clashes between int and int32 reconciled for: assemble_label_no, routine_starts_line, read_byte_from_memory_block, write_byte_to_memory_block, parse_label, symbol_indexReturn value for hash_code_from_string cast explicitly to int (rather than implicitly from unsigned int)prec_table given type intMany "dead assignments" (redundant settings of variables) removed"=-" replaced by "= -" to prevent pre-ANSI compilers from misunderstanding "x=-1" as "x=x-1"The veneer string for the source of "CA__Pr" has been contracted to make it smaller than 2048 chars long, since Microsoft Visual C/C++ won't compile strings longer than thatsymbs[] explicitly cast to char * in a few points in "linker", "objects" and "symbols"Format string for process IDs corrected from "_proc%08x" to "_proc%08lx"Format string for serial number copying removed, and a use of strcpy put in its place (ii) Changes made between v6.02 and v6.03 ------------------------------------Feature added:The on/off command-line switches can now be prefixed with ~ to turn them off. (This is useful only in ICL, to undo the effect of previous lines.)Bugs fixed:The "my parent is Class" value in modules changed from 0xffff to 0x7fff to make it valid in type int (rather than int32)The "spaces" statement not being parsed correctly (bug in syntax analyser)Arrays declared using Global rather than Array don't work (this is fixed, but also now causes an obsolete usage warning)Fclose applied to null file handle (effectively, the same file closed twice)"for (::p++)" misinterpreted due to "::" being recognised as an operatorSome -> -> object initialisations getting the structure wrongTable of property names wrongly compiled (resulting in garbled run-time error messages)Serious failure of individual property inheritance from a class when both inheritor and class define individual property valuesMessages sent to a property whose value is NULL now do nothing and reply 0 (as if the property value were 0)Action switches not working in properties called via messages (well, not really a bug: but it's better that they should work, as this makes it possible to call "before"/"after" as messages)The "children" inlined routine leaving redundant values on the stack (which resulted in complex expressions containing uses of children() mis-evaluating)Actions in the form <A x> and <A x y> using a call opcode improperly (with possibly regrettable results -- though they worked on my interpreter!)ICL files not working (due to a bug gratuitously introduced in v6.02), and not understanding tab characters as spaces, and generally being poorly implemented (I found print "Err..."; in the code, to my alarm): generally tidied up nowNegative constants not working as switch() case valuesSlightly better reporting of bad statement errors, and slightly better recoverySource code cleaning:Calls to get_next_char() replaced with (*get_next_char)() which is equivalent to a modern ANSI compiler, but inequivalent to a pre-ANSI oneLogically redundant "break" statements added to parse_routine() to prevent either "unreachable code" or "no return from non-void function" warnings being producedparse_given_directive() has its unnecessary parameter removeduchar types used for characters as read in by the lexer before filtering takes place to, e.g., strip off any set top bits (iii) Changes made between v6.03 and v6.04 ------------------------------------Features added:When double-quoted text is continued from one source line to another, and the first ends in ^, then the new-line is not replaced by a space " ". Thus: print "Shall I compare thee to a summer's day?^ Thou art more..."; results in the "T" being printed underneath the "S", not one space to the right.Statements like "The wyvern swallows ", (the) noun, " in three gulps!"; are now correctly understood to be print_ret statements. (Previously this gave errors because the syntax analyser thought it was going to be a list of three switch cases separated by commas, until it hit the semicolon and realised there should be a colon, by which time it was too late to go back. The language definition has never been clear on whether long implied print_rets are legal or not, so I've decided that they are.)The #n$ construct (for single-character dictionary words) can now take non-alphabetic characters: e.g., #n$1 and #n$# refer to dictionary words '1' and '#'.Bugs fixed:The Verb "newverb" = "oldverb"; syntax not working, due to mistake in syntax analyser (or rather, to inadequate testing)If routine R is being Replaced, then its new definition can now appear either before or after its Library definition (as long as the Replace declaration occurs before both definitions). In 6.01 to 6.03, errors would result if the new definition was earlier than the Library one.Constants not being allowed as switch cases when they happen to be the same as internal statement keywords (such as the single letter "a")Memory allocations of 0 bytes now return NULL, which protects Inform from trying to apply "free" to them, which seems to have caused problems on some ports (not on mine, but it was my mistake)Spurious "Second 'Ifnot' in same 'If...'" errors appearing in certain nested 'If...' configurationsA comma in between object headers and bodies is now optional once again (as it used to be in Inform 5), rather than forbidden.Text or commentary appearing after a string folding character \ now causes an error, as it should (the rest of such a line should be empty)."continue" not properly working inside a "for" loop of the "optimised" kind (that is, whose update code consists of just variable++ or variable--)For two different reasons, "or" did not quite work when used in its extended way (i.e. with conditions other than ==, ~= or with many alternatives): apologies, as the code was in a muddle. Better now.Class classname(N) ... was allowing creation of only N-1 new instances, not N.(a) not recognised as a print specification when "a" is also the name of a local variable."youngest" function failing to work.Source code cleaning:Header declarations of the variables "mv_xref" and "no_stubbed_routines" removed (neither one actually exists in the released state of Inform 6)Miscellaneous comments addedAssembly of store opcodes optimised to pull opcodes in a few cases, saving 1 byte of object code each timeHooks for Robert Pelak's MAC_FACE port (traditionally the most strenuous) insertedError messages reworded slightly for better consistency (iv) Changes made between v6.04 and v6.05 ------------------------------------Feature added:Assembly language tracing slightly tidied up (a cosmetic change only).Bugs fixed:When a file with very long strings in (such as one generated automatically by "infoclues") is read, it can corrupt memory, resulting in the malloc heap being damaged.Spurious backpatching errors (actually, all backpatching errors are spurious - backpatching is supposed to work every time) produced when the Z-code area of the Z-machine exceeds 64K. (This seldom happens even in large games, unless quite long print and print_ret strings are quoted.) The error message in question was *** Backpatch symbol out of range *** and, just to recap, this should never appear. Please send me the offending source code if it persists!The only time I've seen this bug is that it once hung the Z-machine while working on printing an inventory of items which needed grouping using "list_together", but it's actually a mistake in the shift-reduce expression grammar: "(...)" (function call) has precedence level 11, whereas "." has level 12, in order to make messages object.message(...) parse correctly (i.e., as (object.message)(...)). This isn't usually done in SR operator grammars because, as I now realise, it results in function(X).property being misinterpreted as function(X.property). I've corrected this by giving (...) the asymmetric precedence level of 11 on the right, but 14 on the left.Printing dictionary contents to a text transcript file not working on the Amiga port, since an internal buffer was overwritten by one byte (5x16+1 = 81, not 80).Spurious "variable declared but not used" warnings appearing for a variable used only as a store destination in assembly language.The "box" statement producing boxes of the wrong width when the text contains the @ escape character (e.g. "@@92@@92@@92" used to be wrongly calculated as 12 characters wide, when it really consists of three backslash characters).The v6.04 optimisation using "pull" (see above) didn't work in v6, v7 or v8 (for two different reasons, but basically because the opcode behaves differently in these as compared with lower versions).Assembly language in v8 recognising the wrong instruction set (same as the previous bug).Version 6 games not properly compiled in some cases (because of a rounding error in calculating packed address offsets: a new section 8.8 has been added to this manual to document how Inform works these out). "I compiled Inform 6.05 under Purify, and am pleased to report that Purify reported exactly 0 memory access errors and 0 bytes of memory leaked." -- Brad Jones (v) Changes made between v6.05 and v6.10 ------------------------------------Features added:The four ICL path variables which define places to look for input of different kinds (source code, include files, ICL files and modules) can now hold lists of alternative locations, separated by a character called FN_ALT which may be OS-dependant (but is normally ','). These alternatives are tried left-to-right until the desired filename is found. It's legal for an alternative to be empty, meaning the current position (e.g. "+include_path=,library,oldlib" gives three possible paths -- the current directory, then the library and oldlib subdirectories). The on-line examples (in the -h1 help information) now include this new feature.File and pathnames in ICL and on the command line can now contain spaces if written inside double-quotes: e.g., "Games Folder/New Games" would be a valid ICL pathname. (Provided for benefit of Macintosh users.)A new error message format, -E2, uses Macintosh Programmer's Workshop style.Output game files in the MPW port are given the type and creator for the MaxZip interpreter. (With thanks to Tom Emerson for supplying details.)The compiler can now generate a new format of grammar table: if used with old code, it will behave just as it used to, but if used with Library 6/3 or later will generate "GV2" (grammar version 2) tables. Users will notice that several restrictions are lifted, most usefully: the number of tokens per grammar line can be up to 30, not up to 6; the total number of prepositions is now unlimited (it used to be limited to about 80); the total number of routines named in grammar tokens is now unlimited (it used to be limited to 32); the total number of actions per game can be up to 4096, not 256. (Chapter 8 above has been rewritten to document GV1 and GV2 in full.) In addition, there are three new grammar features: (a) you can mark an action as "reverse", as in the following example: Verb 'show' 'present' 'display' * creature held -> Show reverse * held 'to' creature -> Show; "reverse" indicating that the two parameters are to be reversed in order before the action is generated. (b) you can give two or more prepositions (only) as alternatives: Verb 'sit' 'lie' * 'on' 'top' 'of' noun -> Enter * 'on'/'in'/'inside' noun -> Enter; the "/" markers indicating alternative prepositions, any one of which is equally good. (c) there is a new grammar token called "topic", which matches any block of text up to the next preposition or the end of the input. For instance, Verb 'read' * noun -> Examine * 'about' topic 'in' noun -> Consult * topic 'in' noun -> Consult; It's used mostly for reading material and subjects of conversation, hence the name "topic". For how to deal with the results of parsing a topic, see the Designer's Manual on "Consult". These three features are only available when using Inform in conjunction with Library 6/3 or later.The run-time veneer's implementation has been rewritten, especially in the areas of message-sending and superclass access. Several benefits: (a) the :: superclass operator can be used with any property (in Inform 6.05 it only worked with individual properties); (b) it makes sense to look up or send to something.Object::cant_go and this refers to the default value of "cant_go". (So we can think of common properties as being exactly the ones inherited from class Object.) (c) printing out a property name, as in print (property) Crown::colour now correctly prints "Crown::colour". (Previously it got stuck on superclass properties.) (d) the limits on numbers of things are made more sensible. These are now as follows: (i) up to 256 classes per game; (ii) up to 128 different individual properties (and up to 63 common properties) can be provided by any single object; (iii) up to 32703 individual property names throughout the game. These limits are now enforced by the compiler, which used to lazily not check them. (e) in games using the Inform library, when the variable debug_flag has bottom bit set (the Inform library does this when the "routines" verb has been typed), all messages are traced to the screen. (Except any messages caused as a result of trying to perform this printing.) This is fuller and more helpful than the old "routines" output. (Several parts of chapter 9 above have been corrected.)The dictionary data structure is entirely redesigned, as a red-black tree rather than a hash table. The effects of this change are localised to "text.c" (i.e., no other source files were modified to achieve it). For small games no difference will be noticed; for large games there will be a slight speed improvement, becoming more noticeable as the game grows. (E.g.: on my machine "Curses" compiles 4% faster.) See Section 8.4 above for details.As part of a general move toward multilingualism, the notation for dictionary words is extended: 'word//letters' means the word 'word' with some flags attached. At present, the only such flag is 'p', meaning "plural". Note that 'word//' is legal and equivalent to 'word'. Thus, 'w//' is another way of making the dictionary word #n$w (which we can't write as 'w' because that's only a single character). (This information is stored in a previously unused flag in #dict_par1: see the bitmap in section 8.5 above.)Dictionary words may now contain accented characters (this is likely to be essential for some Scandinavian languages). It is also now legal to write a quotation mark within quotation marks, in two cases: ''' (meaning: the literal character ') '@'etude' (to put an acute accent on something)To avoid dictionary resolution falling unacceptably when accented chars are used, it's helpful to move commonly occurring ones into the Z-machine alphabets: the new directive "Zcharacter" does this. See section 12.2 above.The dictionary contents are now given in alphabetical order in text transcript files, and the "Trace dictionary;" output is much more descriptive.Tracing output for calls to functions marked with a * is more legible, giving exactly the arguments supplied and naming embedded routines more sensibly (e.g. as "lantern.time_left" rather than "Embedded__43"). The -g switch now traces only the user's own functions, not the library ones, unless -g2 is set.Inform now checks for the error of giving the same property twice in one definition. This is not always as obvious as (say) giving the "description" of an object twice over, because of the existence of aliases. A typical error message would be: line 15: Error: Property given twice in the same declaration, because the names 'initial' and 'when_open' actually refer to the same propertyA warning is produced if the "box" statement is used in a Version 3 game (since it cannot have any effect in such a game).The default memory allocation for arrays has been increased (from 4000 to 10000 bytes). The original was too conservative, and anyway more will be needed for language definition files in library 6/3.Action and attribute names are now written into story files, so that it's possible to print them (for debugging purposes). Full details of how to do this are in section 9.6 above. (Users with library 6/3 may notice that the "actions" debugging verb now names all actions, not just the ones defined by the library.)Inform now allows the three special constants DEBUG, USE_MODULES and MODULE_MODE to be defined more than once without giving an error. Thus, if your code contains "Constant DEBUG;" already, but you compile it with the "-D" option anyway, no error will be produced.When an object isn't given a textual name, rather than calling it "?" Inform now calls it something like "(red_box)", where "red_box" is its internal symbol name. (This makes debugging output easier to follow and wastes very few bytes.) When the object hasn't got an internal name either (if it's defined just as "Object;" for example) it has the textual name "(102)", 102 being its object number.Bugs fixed:"Extend only ..." not always parsed correctly, sometimes giving a spurious syntax error, other times failing to notice "first/last/replace".In -U mode, failure to report as an error a line consisting of two words, the first of which is a constant or variable name defined in a previously linked module. (This arose when the line "Action RevTakeWith" was accidentally written instead of "Fake_action RevTakeWith" -- what happened was that "Action" was recognised as the variable "action", which Inform knew was a symbol exported from a linked module (ParserM); it took this value as a class, wrongly, and made a new object called RevTakeWith of this "class". The fault was in the syntax analyser, which wrongly assumed that it wouldn't necessarily be known whether an exported symbol was a class-name or not, and so allowed all exported symbols to be used, just to be on the safe side.)ICL fatal error messages quoting a garbled filename.Version 3 games (ah, remember them?) not compiling because of the usage of call opcodes not present in the V3 Z-machineThe "Include ">name"" feature not working when the original source file lies at the current working directory of the host OS, rather than in some subdirectory.The linker had a rarely-occurring bug which prevented the tasks scoring system from working properly in a game which linked rather than included the library modules: if an instruction wrote a value to certain variables whose internal numbers were in a certain narrow range, then these variables would not be correctly reconciled with the story file variables at link time. This same bug possibly also accounts for an even rarer occurrence, which I've had no definite report of, in which the less than professional error message "Ooops" is produced. (I've corrected the error message to something more helpful.)Rather seriously (though this error took a long time to be found), if (condition) rfalse; else ... [or the same thing with rtrue;] producing syntax errors. (A mistake in some optimisation code.)The "elder" function wrongly returning 0.Not always reporting errors when global variables were used wrongly as values of object properties or array elements. (Sometimes a backpatch error would be given.) These are now properly parsed in constant context, not quantity context.Spurious warning about code not being reachable at the close brace of a "for" loop, when this is indeed not reachable. (It can still be sensible code, if the idea is always to "continue" or "return" from the body of the loop.)When the error "Error: System function name used as value "child"" is produced, also printing a spurious *** Emit token error *** message.Backpatch errors sometimes produced when compiling a line (in a module only) which contains an "if" condition making heavy use of "or".Code backpatch errors sometimes produced when the condition "ofclass" is compiled.'->' or 'Nearby' mysteriously failing if the parent object referred to was (a) defined in a module and (b) one of the first four objects defined there."Declared but not used" warnings issued for replacements of routines only accessed from within a linked module.Error message if a non-existent ICL variable set, not being printed.Source code cleaning:A port definition called PC_WIN32 has been added to "header.h", courtesy of Kirk Klobe.Two variables made int32 instead of int (thanks to Evan Day for spotting the desirability of this).The veneer routines are segmented so that no individual literal string is as long as 512 characters. (An irritating requirement imposed by the Macintosh port's compiler.)Six miscellaneous explicit casts of (char *) to (uchar *), suggested by Robert Pelak. More systematically, the opcode names in asm.c are all explicitly cast to (uchar *).A slight change to inform.c makes the Mac front end's arrangement (of calling only sub_main(), while main() itself is not compiled) more generally usable by other ports: see section 2.2 (e). (vi) Changes made between v6.10 and v6.11 ------------------------------------Bugs fixed:An important one -- the optimised form of statements like if (variable) rfalse; was being negated, i.e., was being wrongly compiled as if (~~variable) rfalse; (In Library 6/3, this showed up at one stage as odd behaviour when moving around in darkness.)The statement "spaces 0" printing infinitely many spaces instead of doing nothing. (Negative arguments have always done nothing.) It is possible to provoke this behaviour in calls to the list-writer.Bug to do with parsing quoted filenames in ICL.Spurious "declared but not used" warning for global variables used only in a module being linked in.Source code cleaning:Atari ST definition block updated on advice of Charles Briscoe-Smith.Copyright dates nudged on to 1997.Paranoid test added when closing temporary files.Macintosh interface code added on advice of Robert Pelak.Two long printfs subdivided to assist poor compilers.String subdivision in "CA__Pr" in the veneer moved slightly in a way which makes no logical difference, but which seems to fix a mysterious problem observed by Robert Pelak (possibly with his compiler: it has been observed on no other platform).Text transcript file is now opened as a text file, not a binary file, which will hopefully assist some ports but inconvenience nobody. (All the same, it may be worth testing that text transcripts still look sensible on your port.) Code added to the Macintosh port to ensure closure of the file if compilation aborts. (vii) Changes made between v6.11 and v6.12 ------------------------------------Features added:A new switch, -C, and the ability to read source code files in any ISO 8859 standard extension to the ASCII character set. The default is ISO 8859-1, Latin1, the standard character set on most computers in most European countries. This means that many accented characters can simply be typed directly, a particular advantage with exotic ISO ranges (Arabic, Hebrew, Greek, Cyrillic, etc.). If your computer is unable to use any of 8859-1 to -9, the -C0 switch (for plain ASCII only) can be used. Accent markers such as '@:u' (for u-diaeresis) remain valid, just as usual, but a new string escape '@{..hex number..}' allows the specification of any Unicode character. For instance '@{a9}' produces a copyright sign (Unicode values between $0000 and $00ff are equal to ISO Latin1 values), and '@{2657}' produces in principle a White bishop chess symbol. In practice, if you wish to use rare Unicode symbols, you'll need an interpreter obeying the Z-Machine Standard 1.0, and you'll probably have to supply it with an appropriate font as well; also, any characters not normally resident in the Z-machine need to be declared in advance of use with the "Zcharacter table" directive. But if you're only using (e.g.) ordinary German, Spanish or French accents the new facilities will work with any Standard 0.2 interpreter. The "Zcharacter" directive, for configuring the Z-machine to use non-English alphabets, is greatly enhanced. (In the Z-machine, Inform now always generates a header extension table, and sometimes uses a slot within it to point to a Unicode translation table newly specified in Standard 1.0.)The debugging information file has finally been brought up to date with Inform 6, and old code inherited from Inform 5 (the only such code in the whole compiler) has been cleaned out. The main change is in producing source-code-to-PC-value maps, which is made more difficult by the compression of code at the end of each routine. The exact format will probably change: this release is an interim one, to assist development of a source-level debugger.The concept of sequence points has been introduced, likewise.Bugs fixed:Not a bug as such, but the veneer implementation has been changed so that class-objects never have attributes set. In 6.01 to 6.11, a class object like Monster might have the attribute "animate" and would therefore be picked up in objectloops over all animate objects, and so on. This is undesirable, and illogical since after all class objects don't provide any properties, so why have attributes? Anyway, the result means that objectloops of the form objectloop (x has <attribute>) ... will now not range through classes but only genuine game objects.When compound Boolean conditions are used as expressions (for instance, in an assignment like "x = (y>1) || d;") the wrong answers could result if the top operator was ||. (If it was &&, or if the condition had no &&s or ||s in, or if the condition was being used as a test rather than a numerical value, everything was normal.)Fake actions being defined before grammar version is set were causing mysterious problems. This would happen if the Fake_Action directive were used before "Parser" is included. An error message has been added advising that Fake_Action directives should be moved. (Fake actions are now mostly obsolete anyway.)Any attributes aliased together being given the unhelpful name "<unknown attribute>" in output from the "showobj" debugging verb.Backpatch *** errors *** sometimes being produced as a knock-on effect from already-reported linkage errors. (These *** errors *** are now suppressed if linkage has already broken down.)The character codes for << and >> (Continental European quotation marks) were the wrong way around, following a mistake in version 0.2 of the Z-Machine Standards document. They have therefore been swapped over, following the correction made in version 1.0 of that document.Accented characters deep inside dictionary words sometimes getting lost. (Because of an ambiguity in the Standards document where Inform made the wrong choice, but interpreters made the right one.)The -w (suppress warnings) switch doing nothing. (In writing Inform 6 I simply forgot this.)Source code cleaning:A new UNIX64 port, for 64-bit Unix machines, added (courtesy of Magnus Olsson) which should subtract pointers correctly even in the unlikely event that they lie in different 2^32 byte pages. Otherwise identical to UNIX.And a new BEOS port added (courtesy of Michael van Biesbrouck) but which is identical to the Linux one in all but name.Link error message handling, and character error message handling, slightly more automated.The "Compiled with %d errors and %d warnings" message reworded so as not to mention 0 of anything, and to indicate how many warnings were suppressed (by the -q or -w switches).The table of memory settings is now printed more prettily in response to the ICL command $list.A new memory setting, MAX_LABELS, has been added, along with error checking to test for too many label points in any single routine.Various functions to do with character set handling tidied up and moved into the new section "chars.c". (viii) Changes made between v6.12 and v6.13 ------------------------------------Bug fixed:String escapes in the form "You can't go @00 from here." not working (slip of the keyboard when re-writing string escapes in v6.12).Source code cleaning:The type of alphabet[][] has been altered so as to ensure that the contents are not read-only strings: from char *alphabet[3] to char alphabet[3][27]. (ix) Changes made between v6.13 and v6.14 ------------------------------------Bugs fixed:"Parker's disease": In very large games only, backpatch errors being caused because the Z-code area of the Z-machine had overflowed past the size where the compiler could backpatch it (this tended to happen only if large amounts of text were printed by Inform code rather than being property values). The Inform code generator now automatically writes any string of text longer than 32 characters into the static strings area: designers shouldn't notice any difference in behaviour of the "print" or "print_ret" statements. (The new arrangement is fractionally wasteful of bytes -- about 2 to 4 bytes per string in excess of 32 characters -- but makes a big difference to the relative sizes of the code and strings areas. "Advent" compiled the old way was 60.1% code, 17.3% strings; the new way, it comes out 40.6% code, 37.1% strings and is about 1K bigger. Most of the change occurs when compiling library messages; the effects are less dramatic on typical Inform code.)A different cause of backpatch errors is attempting to link Version 5 modules into a Version 8 game. If you're compiling a V8 game and want to link in the library, for instance, you'll need to compile the library modules with "-v8" as well. Inform used not to check this error condition, but now does produce an error message if there's a version mismatch between game and module.The linker also now checks that module and game are using the same Z-machine character set. (If either one has made a Zcharacter directive which differs from the other, then the sets are inconsistent and linkage can't take place. This will only ever happen if you're working in a language other than English, and trying to use the linker, and it can be cured by copying the Zcharacter directives out of the language definition file into a little include file, and including that in the source for your main file as well as the source for any modules of your own -- i.e., any modules other than the library ones.)Finally, the linker checks for the error condition that the module asked to import a variable which turns out not to have been defined in the main game. (This may cause problems in linking with library 6/6, as Inform is now able to detect an error in 6/6's "linklv.h" which it previously wasn't able to detect: delete the line reading "Import global gender_and_number;" from "linklv.h" if so.)"continue" not working (branching to the non-existent label -1, in fact) if used inside a "switch" statement. (It ought to continue the innermost loop containing the "switch" statement, if there is one, or give an error if there isn't.)The two string escapes @@94 (producing a ^ character) and @@126 (producing ~) in fact producing new-line and double-quote.Conditional branches to rtrue or rfalse in assembly-language being reversed in sense.Source code being garbled after any Include statement which ends at a byte position which is a multiple of 4096, minus 3. (My thanks to Andrew Plotkin for both finding and diagnosing this.)"For" loops in the form "for ( : : )" (with white space between the two colons), which occur in a routine after a previous one which happens to have a designer-defined label in a particular position, causing an incorrect if harmless "Label declared but not used" warning.Very short programs testing "x provides <property>", but never reading, writing or otherwise using <property>, causing stack overflows or restarts. Nobody would ever need such a program anyway.Function calls not working, or having garbled arguments, in Version 3 or Version 4 games (the Inform library requires Version 5 or better nowadays, so the ability to compile V3 or V4 files is only needed for interpreter testing).The error message disallowing use of an assembly opcode if the Z-machine version doesn't support it, now quotes the opcode name.Suppressed "declared but not used" warnings for global objects not being counted towards the total number reported as suppressed when Inform prints its final message.Source code cleaning:Three character conversions amended to unsigned character conversions.Backpatch errors, which I devoutly hope not to see again, are now reported as a new category of error: "compiler errors" which produce an apologetic banner, asking the user to report the fault to me.Link errors are reported more prettily (differently, anyway). (x) Changes made between v6.14 and v6.15 ------------------------------------Features added:Parametrised object creation. That is, if you define a class in which objects can be created, and also give it a "create" property like so: Class Monster with ... create [ start_at new_species; move self to start_at; self.species = new_species; ], ... then you can now make creation calls like so: M = Monster.create(Tomb_of_Kings, Ogre); The same applies to "recreate": Monster.recreate(M, Wall_of_Thorns, Hobbit); An attempt to supply more than 3 parameters to either kind of creation will result in a run-time error message.It is now legal to declare a Class with zero create-able objects: Class Moribund(0) ... This is useful (i) to permit a definition like Class Gadgets(NUM_GADGETS) in some library file, where NUM_GADGETS is supplied by the library file's user and might be zero; and/or (ii) to permit "recreate" to be used on objects of the given class, reinitialising them to the values set in the class definition.The ASCII form feed character (12 decimal) is now legal as white space. More precisely, it is in all contexts treated as a line feed.Bugs fixed:Getting property lengths wrong for common property values referred to using the superclass operator :: (a bug which sometimes manifests itself in a crash when a common property routine belonging to a superclass is called).Crashing when halting on a memory error because an extremely long string of text has caused MAX_STATIC_STRINGS to be exceeded. (The compiler now ensures that MAX_STATIC_STRINGS is at least twice the size of MAX_QTEXT_SIZE, which should be more than plenty.)Crashing when printing an extremely long "Expected ... but found ..." error (where the second ... stands for an awfully long string of text).An inability to assemble the optional third operand to the "@set_colour" opcode (which is available to Version 6 games only).Array sizes are now formally checked to lie in the range 1 to 32767 entries, with an error message produced if they don't.Misinterpretation of array definitions with calculated sizes: Array Muggins -> MAX_SERVERS * 50; In Inform 6.14, this would be wrongly parsed as an array with one entry, initialised to the value given: whereas 6.15 correctly parses it as an array with MAX_SERVERS*50 initially zero entries.When constants are calculated with at compile time (as in the previous bug), Inform 6.15 now detects overflows of signed addition, subtraction and multiplication. So if, for instance, MAX_SERVERS is 100000, then the following error is produced: Error: Signed arithmetic on compile-time constants overflowed the range -32768 to +32767: "100000 * 50 = 500000"Two bugs in the veneer (both found and fixed by Chris Hall): (i) a program using ".create()" but not "provides" will go into a recursive loop (this never happens if the Inform library is included); (ii) more importantly, some creations of objects within classes where deletions had previously occurred were resulting in two different created objects sharing the same properties.Improper use of 'or' not always producing a good error message (sometimes producing instead the compiler error "*** emitter overflow ***"). The more explanatory error message Error: 'or' not between values to the right of a condition should now be produced in all such cases.Expressions featuring ) followed immediately by ( sometimes crashing the expression evaluator. For instance: (ChooseRoutine())(1); BigRoom.(BigDoor.dir_to)(); The grammar resolving whether a function call is intended, or only a bracketed expression, has been refined so that, e.g., <SetTo dial (4*the_time)>; is not misinterpreted as containing one argument, the result of the function call "dial(4*the_time)". (If you want this, you must put brackets around it.)Named action constants (like ##Take) not being correctly numbered after the 256th in order of first usage.The "Default" directive for setting constants not properly handling some values which are other than numerical -- e.g., Default Story "Untitled Story"; would not have worked correctly under Inform 6.14.Backpatching "compiler errors" are now suppressed if errors have already been reported (because in some circumstances, error recovery is good enough to allow compilation to continue but not good enough to leave a self-consistent backpatch table in the affected area). Error recovery from missed close-quotation-marks in object definitions has been marginally improved in some cases.Source code cleaning:Use of the Inform 5 statements 'print_paddr', 'print_addr' and 'print_char' (all illegal in Inform 6) now results in an obsolete usage message which prints up the necessary Inform 6 equivalent.The statistics output now includes the story file's serial codes, in the traditional <release number>.<serial code> format.Note that this manual now contains an algorithm for syntax-colouring Inform source code. (xi) Changes made between v6.15 and v6.20 ------------------------------------Features added:Strict (-S) mode added, under which Inform undertakes to compile a story file which cannot crash the Z-machine (unless the user has compiled some assembly-language) but will instead print helpful error messages when illegal operations are attempted. -S mode is wasteful of story file size (it adds perhaps 10 to 15%) and of execution speed (but on most modern computers, Z-machine interpreters run extremely quickly anyway) but is intended to be helpful for debugging. See section 7.8 of this manual for a list of exactly what strict-mode does. In support of strict mode, several new # system constants have been added, but these should not be used by designers and are not public features.Inform now checks that readable-memory usage (i.e. the top of the dictionary and the bottom of the code area) does not exceed $10000, as this is a Z-machine requirement. The "-s" statistics listing has an added line indicating how much of readable memory a game file has needed. (To test this, try compiling a game with "Array gross --> 32767;" in it.)Bugs fixed:"class.copy(a,b)" not working properly when "a" overrides an individual property defined by some class. (Because of a bug not in the code for "copy" but in "objects.c": individual property tables were accumulating redundant extra values, which never showed up except when "copy" was used. Thanks to Erik Hetzner and Gevan for finding and fixing this.)"class.copy(a,b)" also transferring the class memberships of b to a (or at least whenever a and b both belonged to at least one class).When a message is sent to a common property of an object which that object doesn't provide, it wasn't being properly routed to the routine given in the default value of that common property (which we'd normally expect to be 0 or NULL, either way causing nothing to happen). Instead, a call would be made to 0-->0, with unpredictable consequences. (This was originally reported as a bug in "Balances".)The maximum number of entries in a list given as a common property value wasn't being correctly restricted to 32 in two different situations: when directly giving values, where the limit was wrongly set to 64; and when the list grows through inheritance of an additive property, where no limit was checked.The "Switches" directive can't set "-k" or "-r", because it's too late by then to reopen the question of opening debugging/transcript files, but 6.15 didn't know this. Torbj|rn thus contributed the shortest legal source file yet known to crash Inform, at just 11 characters: "Switches r;".The rennab (the un-banner) printed at the end of compilation claiming "(no output)" if there were no errors during the main compilation pass, but there were errors late in story file construction instead.I'm not sure this is a bug as such, but assembly listing (caused by -a, -t or use of "#Trace assembly") no longer covers the veneer. (Because it's a nuisance and clutters up the listing.)Not a bug either. The return opcode automatically generated by "]", where needed (i.e. when this is reachable) is now a sequence point. This was requested by Mark Musante, to assist debuggers. (xii) Changes made between v6.20 and v6.20a -------------------------------------(These are bug fixes made to the early beta-version v6.20.)The "length" calculation for properties (in "properties_segment()" of "objects.c") calculates the length differently for individual and common properties.Some small test programs not properly working because the wrong combinations of veneer routines were being compiled. (I've rewritten the appropriate routines in "veneer.c" which decide which routines need the presence of which other routines.) In particular, ".#" applied to individual properties might not work in a very small file.Array bounds checking not working on arrays of >= 255 entries. (xiii) Changes made between v6.20a and v6.21 -------------------------------------A new "-X" switch provides support for the Infix debugging library file. (Which it does by compiling additional veneer tables of symbol names.)The maximum number of source files read from in compilation has been raised from 64 to 256, at the request of a user. (No, really. She knows who she is.) The filename storage has been reduced in size, which will reduce the physical size of the Inform program by 8K on most systems.The "-g" switch for tracing function calls now has a third setting, "-g3", which traces even veneer calls: the "-g2" setting now traces game and library calls but not veneer calls.New syntax: "Zcharacter terminating" specifies the terminating character table for games in Z-machine versions >= 5. See section 12.3 for details.Bugs fixed:Loops in the form "objectloop (X in Y) ..." causing stack underflow crashes if compiled with -S checking off.-S checking now guards against attempts to use properties of "nothing", for instance rejecting something like "nothing.name = 'nobody';", and error messages concerning nonexistent objects no longer print garbled names for them.Linking sometimes producing backpatching errors if arrays are already created before the link occurs.The -S (strict checking) and -X (Infix) switches are now disabled for files being compiled as modules. (They wouldn't work, and would anyway make the library modules too large.)-S checking no longer tries to compile "pop" opcodes in Versions 5, 6, 8 to clear values off the stack in some cases of array-boundary checking. (These Z-machine versions lack a "pop" opcode: instead, the same effect is inelegantly produced by "@jz sp" with a null branch, taking up three bytes rather than one. C'est la Z.)A potential crash if the limit on the number of adjectives is exceeded in a grammar version 1 game (which probably means a very big one using library 6/2 or earlier) has been removed.The readable-memory ceiling was incorrect for version 6 games.The error message resulting from asking Inform to read an unopenable ICL file is now coherent.Assembly tracing at level 3 (which you don't want to read) now correctly prints the size of a routine after branch optimization.The checksum is now included in the header block attached to a debugging information file. (Previously this slot contained 0000.)A minor bug in the abbreviations optimizer, so obscure that I'm not even sure what its consequence was, has been removed.Source code cleaning:The MAC_68K machine definition has been removed from the header: there are no longer any specific lines of code for this system. (xiv) Acknowledgements ----------------Many thanks to the following informers, who have reported bugs in Inform 6: Torbj|rn Andersson, Toni Arnold, Jose Luis Diaz de Arriba, Michael Baum, Paul E. Bell, Michael van Biesbrouck, Gerald Bostock, Neil Brown, Russ Bryan, Jesse Burneko, Evan Day, Gevan Dutton, Stephen van Egmond, Tom Emerson, Theresa van Ettinger, Greg Falcon, Roy Fellows, Rob Fisher, David Fletcher, C. E. Forman, Richard Gault, Paul Gilbert, David Glasser, Michael Graham, Bjorn Gustavsson, Chris Hall, Kory Heath, Erik Hetzner, Andreas Hoppler, Paul Horth, Sam Hulick, Francis Irving, Brad Jones, Christoffer Karlsson, Niclas Karlsson, John Kean, John Kennedy, Matt Kimball, Kirk Klobe, Michael A. Krehan, Mary Kuhner, Josh Larios, Jrgen Lerch, Harvey Lodder, Jennifer Maher, Bonni Mierzejewska, Paul Mikell, Tim Muddleton, Olav Mueller, Jeff Nassiff, Ross Nicol, Magnus Olsson, Marnie Parker, Robert Pelak, Jason Penney, Aphoristic Petrofsky, Andrew Plotkin, Richard H. Poser, Fredrik Ramsberg, Gareth Rees, Evin Robertson, Matthew Russotto, Gunther Schmidl, Miron Schmidt, Rene Schnoor, Nyuchezuu Shampoo, Lucian P. Smith, Anson Turner, Lorelle VanFossen, David L. Wagner, John Wood, Uncle Bob Newell and all.------------------------------------------------------------------------------