Add unit tests for CDS extractor TypeScript code

Adds unit tests, and unit testing configs, for CDS extractor
`extractor/cds/tools/src/*.ts` files. Sets up test coverage reporting
as part of an updated `npm run build:all` command for the CDS extractor
project. Updates other project configs to supporte the "Testability"
phase of a multi-stage rewrite process.

Author: data-douser

extractors/cds/tools/.eslintrc.js

@@ -19,9 +19,9 @@ module.exports = {
     es2018: true
   },
   ignorePatterns: [
-    'index-files.js*',
+    'out/**',
     'node_modules',
-    '*.js.map',
+    'coverage',
     '*.d.ts'
   ],
   rules: {
@@ -73,7 +73,9 @@ module.exports = {
   parserOptions: {
     ecmaVersion: 2018,
     sourceType: 'module',
-    project: './tsconfig.json'
+    project: './tsconfig.json',
+    tsconfigRootDir: __dirname,
+    createDefaultProgram: true // This helps with files not directly included in the tsconfig
   },
   settings: {
     'import/resolver': {
@@ -85,5 +87,27 @@ module.exports = {
         'extensions': ['.js', '.jsx', '.ts', '.tsx']
       }
     }
-  }
+  },
+  overrides: [
+    {
+      files: ['*.ts'],
+      parserOptions: {
+        project: './tsconfig.json',
+        tsconfigRootDir: __dirname
+      }
+    },
+    {
+      files: ['*.test.ts', 'test/**/*.ts', 'index-files.ts'],
+      parserOptions: {
+        project: './tsconfig.json',
+        tsconfigRootDir: __dirname
+      },
+      rules: {
+        '@typescript-eslint/explicit-function-return-type': 'off',
+        '@typescript-eslint/no-unsafe-assignment': 'off',
+        '@typescript-eslint/no-unsafe-call': 'off',
+        '@typescript-eslint/no-unsafe-member-access': 'off'
+      }
+    }
+  ]
 }

extractors/cds/tools/.gitignore

@@ -0,0 +1,7 @@
+# Ignore the entire "out" directory as this is for the .js and .js.map files
+# which are generated by the `tsc` build process. In the current project config,
+# we require the platform-specific "index-files" shell/cmd script to run the
+# `npm run build` command that generates the files for the correct platform and
+# local environment.
+out/
+

extractors/cds/tools/autobuild.md

@@ -15,13 +15,15 @@ This document is meant to be a common reference and a project guide while the it
 The current extractor for [CDS] is based on `index-files`, which has several limitations and challenges:
 
 1. **Testability**: The current extractor is difficult to test, and especially difficult to troubleshoot when tests fail, because the current implementation lacks unit tests and relieas heavily on integration tests that are performed in a post-commit workflow that runs via GitHub Actions, which makes it more difficult to track errors back to the source of the problem and adds significant delay to the development process.
+
 2. **Performance**: The current extractor is slow and inefficient, especially when dealing with large projects or complex [CDS] files. This is due to the way `index-files` processes files, which can lead to long processing times and increased resource usage. There are several performance improvements that could be made to the extractor, but they are all related to avoid work that we either do not need to do or that has already been done.
 
     - As one example of a performance problem, using the `index-files` approach means that we are provided with a list of all `.cds` files in the project and are expected to index them all, which makes sense for CodeQL (as we want our database to have a copy of every in-scope source code file) but is horribly inefficient from a [CDS] perspective as the [CDS] format allows for a single file to contain multiple [CDS] definitions. The extractor is expected to be able to handle this by parsing the declarative syntax of the `.cds` file in order to understand which other `.cds` files are to be imported as part of that top-level file, meaning that we are expected to avoid duplicate imports of files that are already (and only) used as library-style imports in top-level (project-level) [CDS] files. This is a non-trivial task, and the current extractor does not even try to parse the contents of the `.cds` files to determine which files are actually used in the project. Instead, it simply imports all `.cds` files that are found in the project, which can lead to duplicate imports and increased processing times.
 
     - Another example of a performance problem is that the current `index-files`-based extractor spends a lot of time installing node dependencies because it runs a `npm install` command in every "CDS project directory" that it finds, which is every directory that contains a `package.json` file and either directly contains a `.cds` file (as a sibling of the `package.json` file) or contains some subdirectory that contains either a `.cds` file or a subdirectory that contains a `.cds` file. This means that the extractor will install these dependencies in a directory that we would rather not make changes in just to be able to use a specific version of `@sap/cds` and/or `@sap/cds-dk` (the dependencies that are needed to run the extractor). This also means that if we have five project that all use the same version of `@sap/cds` and/or `@sap/cds-dk`, we will install that version five separate times in five separate locations, which is both a waste of time and creates a cleanup challenge as the install makes changes to the `package-lock.json` file in each of those five project directories (and also makes changes to the `node_modules` subdirectory of each project directory).
 
 3. **Modularity**: The current extractor is mostly just one giant script, aka [index-files.js](./index-files.js), which is surrounded by a collection of small wrapper scripts (aka [index-files.sh](./index-files.sh) and [index-files.cmd](./index-files.cmd)) that are used to allow the JavaScript code to be run in different environments (i.e. Windows and Unix-like environments). While we cannot really get away from the wrapper scripts. we should refactor the "one giant script" (in a single `index-files.js` file) into a more modular design that allows us to break the extractor into smaller, more manageable pieces.
+
 4. **Maintainability**: The current implementation is lacking in terms of mandating consistent code style and best practices. For example, there are no linting rules applied or any scripts for applying consistent code style. This makes it difficult to maintain the code at a consistent level of quality, where it would be much better to have basic linting applied as a pre-commit task (i.e. to be performed in the developer's IDE). The current implementation also lacks documentation, which makes it difficult for new developers to understand how the extractor works and how to contribute to it.
 
 ## Goals for the Future Extractor (using `autobuild`)

extractors/cds/tools/index-files.cmd

@@ -20,6 +20,8 @@ if %ERRORLEVEL% neq 0 (
 set "_response_file_path=%~1"
 set "_script_dir=%~dp0"
 REM Set _cwd before changing the working directory to the script directory.
+REM We assume this script is called from the source root directory of the
+REM to be scanned project.
 set "_cwd=%CD%"
 
 echo Checking response file for CDS files to index
@@ -48,7 +50,9 @@ REM     the original working (aka the project source root) directory.
 cd /d "%_script_dir%" && ^
 echo Installing node package dependencies && ^
 npm install --quiet --no-audit --no-fund && ^
+echo Building TypeScript code && ^
+npm run build && ^
 echo Running the 'index-files.js' script && ^
-node "%_script_dir%index-files.js" "%_response_file_path%" "%_cwd%"
+node "%_script_dir%out\index-files.js" "%_response_file_path%" "%_cwd%"
 
 exit /b %ERRORLEVEL%

extractors/cds/tools/index-files.js

@@ -20,6 +20,9 @@ then
     exit 3
 fi
 
+# Set the _cwd variable to the present working directory (PWD) as the directory
+# from which this script was called, which we assume is the "sourc rooot" directory
+# of the project that to be scanned / indexed.
 _cwd="$PWD"
 _response_file_path="$1"
 _script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
@@ -49,10 +52,12 @@ fi
 #  3. passing the original working directory as a parameter to the
 #     index-files.js script;
 #  4. expecting the index-files.js script to immediately change back to
-#     the original working (aka the project source root) directory.
+#     original working (aka the project source root) directory.
 
 cd "$_script_dir" && \
 echo "Installing node package dependencies" && \
 npm install --quiet --no-audit --no-fund && \
+echo "Building TypeScript code" && \
+npm run build && \
 echo "Running the 'index-files.js' script" && \
-node "$(dirname "$0")/index-files.js" "$_response_file_path" "${_cwd}"
+node "$(dirname "$0")/out/index-files.js" "$_response_file_path" "${_cwd}"

extractors/cds/tools/index-files.js.map

@@ -0,0 +1,29 @@
+/** @type {import('ts-jest').JestConfigWithTsJest} */
+module.exports = {
+  preset: 'ts-jest',
+  testEnvironment: 'node',
+  testMatch: ['**/test/**/*.test.ts'],
+  roots: ['<rootDir>/src/', '<rootDir>/test/'],
+  collectCoverageFrom: [
+    'src/**/*.ts',
+    '!src/**/*.d.ts',
+    '!**/node_modules/**',
+  ],
+  coverageReporters: ['text', 'lcov', 'clover', 'json'],
+  coverageDirectory: 'coverage',
+  verbose: true,
+  moduleDirectories: ['node_modules', 'src'],
+  moduleFileExtensions: ['ts', 'js', 'json', 'node'],
+  transform: {
+    '^.+\\.ts$': [
+      'ts-jest',
+      {
+        tsconfig: 'tsconfig.json',
+        diagnostics: {
+          warnOnly: true
+        },
+        isolatedModules: true,
+      }
+    ]
+  }
+};