Add extractors README with CDS mermaid diagram

Adds the `extractors/README.md` file as an evolution of, and a
replace ment for, PR #168 from this `advanced-security/codeql-sap-js`
repository.

Updates the CDS tools documentation to reflect progress in the
multi-stage process of rewriting the CDS extractor to be more
maintainable (WIP) and performant (TODO).

Author: data-douser

extractors/README.md

@@ -0,0 +1,75 @@
+# `advanced-security/codeql-sap-js` : `extractors/README.md`
+
+## CodeQL CDS Extractor : Overview
+
+The CodeQL CDS Extractor is a specialized component designed to process and analyze Core Data Services (CDS) files used in SAP Cloud Application Programming (CAP) model applications. This extractor enables CodeQL's static analysis capabilities to detect security vulnerabilities, bugs, and quality issues in CDS files.
+
+Key capabilities of the extractor include:
+- Compiling `.cds` files to an intermediate JSON representation
+- Handling SAP CAP dependencies and managing compiler versions
+- Integrating with the JavaScript extractor for comprehensive analysis
+- Converting CDS code to CodeQL's TRAP format for database inclusion
+- Supporting both Windows and Unix-like environments through platform-specific wrapper scripts
+
+The extractor operates as an extension to the JavaScript extractor, complementing its ability to analyze JavaScript, TypeScript, and JSON files with support for the CDS domain-specific language.
+
+## CodeQL CDS Extractor : Flowchart
+
+The following flowchart shows the flow of execution for the current implementation of the extractor.
+
+```mermaid
+flowchart TD
+    COM["`export _build_cmd=<br>$(pwd)/extractors/
+javascript/tools/
+pre-finalize.sh`"]
+    DCR[codeql database create<br>--command=$_build_cmd<br>--language=javascript<br>--search-path=./extractors/<br>--<br>/path/to/database]
+    DB[(CodeQL Database)]
+    DINIT[codeql database init]
+    CRE[codeql resolve extractor]
+    JSE[[javascript extractor]]
+    DTRAC[codeql database<br>trace-command]
+    SPF[[pre-finalize.sh]]
+    DIDX[codeql database index-files<br> --language=cds<br>--include-extension=.cds]
+    SIF[[index-files.sh]]
+    SIT[[index-files.ts/js]]
+    NPM[[npm install & build]]
+    DETS[[Determine CDS command]]
+    FIND[[Find package.json dirs]]
+    INST[[Install dependencies]]
+    CC[[cds compiler]]
+    CDJ([.cds.json files])
+    JSA[[javascript extractor<br>autobuild script]]
+    TF([CodeQL TRAP files])
+    DBF[codeql database finalize<br> -- /path/to/database]
+
+    COM ==> DCR
+    DCR ==> |run internal CLI<br>plumbing command| DINIT
+    DINIT ----> |--language=javascript| CRE
+    CRE -..-> |/extractor/path/javascript| DINIT
+    DINIT -.initialize database.-> DB
+
+    DINIT ==> |run the<br>javascript extractor| JSE
+    JSE -.-> |extract javascript files:<br>_.html, .js, .json, .ts_| DB
+    JSE ==> |run autobuild within<br>the javascript extractor| DTRAC
+    
+    DTRAC ==> |run the build --command| SPF
+    SPF ==> |run codeql index-files<br>for CDS files| DIDX
+    DIDX ==> |invoke script via<br>--search-path| SIF
+    SIF ==> |runs TypeScript version<br>after npm install| NPM
+    NPM ==> |executes compiled<br>index-files.js| SIT
+    
+    SIT ==> |finds project directories<br>with package.json| FIND
+    FIND ==> |install CDS dependencies<br>in project directories| INST
+    SIT ==> |determines which<br>cds command to use| DETS
+    DETS ==> |processes each CDS file| CC
+    
+    CC ==> |compile .cds files to<br>create .cds.json files| CDJ
+    CDJ -.-> |stored in same location<br>as original .cds files| DB
+    
+    SIT ==> |configures extraction<br>filters for JSON files| JSA
+    JSA ==> |processes .cds.json files<br>via javascript extractor| CDJ
+    
+    CDJ ==> |javascript extractor<br>generates TRAP files| TF
+    TF ==> |imported during<br>database finalization| DBF
+    DBF ==> |finalize database and<br>cleanup temporary files| DB
+```

extractors/cds/tools/autobuild.md

@@ -14,17 +14,25 @@ This document is meant to be a common reference and a project guide while the it
 
 The current extractor for [CDS] is based on `index-files`, which has several limitations and challenges:
 
-1. **Testability**: The current extractor is difficult to test, and especially difficult to troubleshoot when tests fail, because the current implementation lacks unit tests and relieas heavily on integration tests that are performed in a post-commit workflow that runs via GitHub Actions, which makes it more difficult to track errors back to the source of the problem and adds significant delay to the development process.
+1. **Testability**
 
-2. **Performance**: The current extractor is slow and inefficient, especially when dealing with large projects or complex [CDS] files. This is due to the way `index-files` processes files, which can lead to long processing times and increased resource usage. There are several performance improvements that could be made to the extractor, but they are all related to avoid work that we either do not need to do or that has already been done.
+   The current extractor is difficult to test, and especially difficult to troubleshoot when tests fail, because the current implementation lacks unit tests and relieas heavily on integration tests that are performed in a post-commit workflow that runs via GitHub Actions, which makes it more difficult to track errors back to the source of the problem and adds significant delay to the development process.
+
+2. **Performance**
+
+   The current extractor is slow and inefficient, especially when dealing with large projects or complex [CDS] files. This is due to the way `index-files` processes files, which can lead to long processing times and increased resource usage. There are several performance improvements that could be made to the extractor, but they are all related to avoid work that we either do not need to do or that has already been done.
 
     - As one example of a performance problem, using the `index-files` approach means that we are provided with a list of all `.cds` files in the project and are expected to index them all, which makes sense for CodeQL (as we want our database to have a copy of every in-scope source code file) but is horribly inefficient from a [CDS] perspective as the [CDS] format allows for a single file to contain multiple [CDS] definitions. The extractor is expected to be able to handle this by parsing the declarative syntax of the `.cds` file in order to understand which other `.cds` files are to be imported as part of that top-level file, meaning that we are expected to avoid duplicate imports of files that are already (and only) used as library-style imports in top-level (project-level) [CDS] files. This is a non-trivial task, and the current extractor does not even try to parse the contents of the `.cds` files to determine which files are actually used in the project. Instead, it simply imports all `.cds` files that are found in the project, which can lead to duplicate imports and increased processing times.
 
     - Another example of a performance problem is that the current `index-files`-based extractor spends a lot of time installing node dependencies because it runs a `npm install` command in every "CDS project directory" that it finds, which is every directory that contains a `package.json` file and either directly contains a `.cds` file (as a sibling of the `package.json` file) or contains some subdirectory that contains either a `.cds` file or a subdirectory that contains a `.cds` file. This means that the extractor will install these dependencies in a directory that we would rather not make changes in just to be able to use a specific version of `@sap/cds` and/or `@sap/cds-dk` (the dependencies that are needed to run the extractor). This also means that if we have five project that all use the same version of `@sap/cds` and/or `@sap/cds-dk`, we will install that version five separate times in five separate locations, which is both a waste of time and creates a cleanup challenge as the install makes changes to the `package-lock.json` file in each of those five project directories (and also makes changes to the `node_modules` subdirectory of each project directory).
 
-3. **Modularity**: The current extractor is mostly just one giant script, aka [index-files.js](./index-files.js), which is surrounded by a collection of small wrapper scripts (aka [index-files.sh](./index-files.sh) and [index-files.cmd](./index-files.cmd)) that are used to allow the JavaScript code to be run in different environments (i.e. Windows and Unix-like environments). While we cannot really get away from the wrapper scripts. we should refactor the "one giant script" (in a single `index-files.js` file) into a more modular design that allows us to break the extractor into smaller, more manageable pieces.
+3. **Modularity**
+
+   ~~The current extractor is mostly just one giant script, aka [index-files.js](./index-files.js), which is surrounded by a collection of small wrapper scripts (aka [index-files.sh](./index-files.sh) and [index-files.cmd](./index-files.cmd)) that are used to allow the JavaScript code to be run in different environments (i.e. Windows and Unix-like environments). While we cannot really get away from the wrapper scripts. we should refactor the "one giant script" (in a single `index-files.js` file) into a more modular design that allows us to break the extractor into smaller, more manageable pieces.~~
+
+4. **Maintainability**
 
-4. **Maintainability**: The current implementation is lacking in terms of mandating consistent code style and best practices. For example, there are no linting rules applied or any scripts for applying consistent code style. This makes it difficult to maintain the code at a consistent level of quality, where it would be much better to have basic linting applied as a pre-commit task (i.e. to be performed in the developer's IDE). The current implementation also lacks documentation, which makes it difficult for new developers to understand how the extractor works and how to contribute to it.
+   ~~The current implementation is lacking in terms of mandating consistent code style and best practices. For example, there are no linting rules applied or any scripts for applying consistent code style. This makes it difficult to maintain the code at a consistent level of quality, where it would be much better to have basic linting applied as a pre-commit task (i.e. to be performed in the developer's IDE). The current implementation also lacks documentation, which makes it difficult for new developers to understand how the extractor works and how to contribute to it.~~
 
 ## Goals for the Future Extractor (using `autobuild`)
 
@@ -48,9 +56,9 @@ All other goals are secondary to and/or in support of the above two goals.
 
 - The new [autobuild.ts](./autobuild.ts) script will be a kept as minimal as possible, with object-oriented code patterns used to encapsulate the functionality of the extractor in `.ts` files stored in a new `src` directory (project path would be `extractors/cds/tools/src`). This will allow us to break the extractor into smaller, more manageable pieces, and will also make it easier to test and maintain the code over time. The new `src` directory will contain all of the TypeScript code for the extractor, and will be organized into subdirectories based on functionality. For example, we might have a `parsers` subdirectory for parsing code, a `utils` subdirectory for utility functions, and so on. This will allow us to keep the code organized and easy to navigate.
 
-- Use TypeScript as the primary language for the extractor, rather than JavaScript. This will allow us to take advantage of TypeScript's type system and other features that make it easier to write and maintain code. Ultimately, we will still be using JavaScript when running the extractor, but we will use TypeScript to develop the extractor and then compile it to JavaScript for use in the CodeQL extractor. This will allow us to take advantage of TypeScript's type system and other features that make it easier to write, test, and maintain code. This will also allow us to use TypeScript's type system to catch errors at compile time rather than runtime, which will make the extractor more robust and easier to maintain.
+- ~~Use TypeScript as the primary language for the extractor, rather than JavaScript. This will allow us to take advantage of TypeScript's type system and other features that make it easier to write and maintain code. Ultimately, we will still be using JavaScript when running the extractor, but we will use TypeScript to develop the extractor and then compile it to JavaScript for use in the CodeQL extractor. This will allow us to take advantage of TypeScript's type system and other features that make it easier to write, test, and maintain code. This will also allow us to use TypeScript's type system to catch errors at compile time rather than runtime, which will make the extractor more robust and easier to maintain.~~
 
-- Add unit tests for everything that can be unit tested. This will allow us to catch errors early in the development process and make it easier to maintain the code over time. We will use a combination of testing frameworks to test the extractor as part of the pre-commit build process. This will allow us to catch errors early in the development process and make it easier to maintain the code over time. Setting up such unit tests will require modifications to the `package.json` file to include the necessary dependencies and scripts for running the tests. We will also need to set up a testing framework, such as Jest or Mocha, to run the tests and report the results. To support all of this, we will create unit tests under a new `test` directory (project path would be `extractors/cds/tools/test`) that will contain all of the unit tests for the extractor. This will allow us to keep the tests organized and easy to navigate. The test directory will be organized into subdirectories based on functionality and mirroring the structure of the `src` directory. For example, if we add a `src/parsers/cdsParser.ts` file, we will also add a `test/parsers/cdsParser.test.ts` file that contains the unit tests for the `cdsParser.ts` file. This will allow us to keep the tests organized and easy to navigate.
+- Add unit tests for everything that can be unit tested. This will allow us to catch errors early in the development process and make it easier to maintain the code over time. We will use a combination of testing frameworks to test the extractor as part of the pre-commit build process. This will allow us to catch errors early in the development process and make it easier to maintain the code over time. ~~Setting up such unit tests will require modifications to the `package.json` file to include the necessary dependencies and scripts for running the tests. We will also need to set up a testing framework, such as Jest or Mocha, to run the tests and report the results. To support all of this, we will create unit tests under a new `test` directory (project path would be `extractors/cds/tools/test`) that will contain all of the unit tests for the extractor. This will allow us to keep the tests organized and easy to navigate. The test directory will be organized into subdirectories based on functionality and mirroring the structure of the `src` directory. For example, if we add a `src/parsers/cdsParser.ts` file, we will also add a `test/parsers/cdsParser.test.ts` file that contains the unit tests for the `cdsParser.ts` file.~~ This will allow us to keep the tests organized and easy to navigate.
 
 ## Examples of Improved [CDS] Parsing