feat apilinks.json generator #153
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
@flakey5 do you need review here? I think I forgot to review this PR! |
As far as the approach yes please |
| .filter(path => path !== undefined && path.endsWith('.js')) | ||
| ); | ||
|
|
||
| const parsedJsFiles = await parseJsSources(sourceFiles); |
There was a problem hiding this comment.
Does this process needs to be blocking? Do we need to parse all these sources beforehand and not on-demand?
There was a problem hiding this comment.
We don't need too, this is just consistent to how to we parse markdown sources. We could definitely just store the paths of the sources and parse them in the generator instead though. It'd definitely lower the amount of memory that's allocated for the lifetime of the program
| @@ -0,0 +1,215 @@ | |||
| // @ts-check | |||
There was a problem hiding this comment.
Temporary comment just enabling type checking in the file
There was a problem hiding this comment.
I wonder how expensive these functions are, as it seems that this can quickly get bloated depending on the size of source files.
BTW, could you use https://unifiedjs.com/explore/package/estree-util-visit/ to walk/visit the nodes? This is what I was referring to, instead of plain forEaches and while loops.
There was a problem hiding this comment.
Also, it might be worth to move "estree" moving logic to a dedicated "estree parser" IDK
There was a problem hiding this comment.
Since we have a parser for the Markdown files, having one just for Estree makes sense I guess.
There was a problem hiding this comment.
How about having a createMarkdownLoader and createJsLoader? Then the same with createMarkdownParser and createJsParser
| return filePaths.map(async filePath => { | ||
| const fileContents = await readFile(filePath, 'utf-8'); | ||
|
|
||
| return new VFile({ path: filePath, value: fileContents }); |
There was a problem hiding this comment.
I am a bit concerned of the memory allocation that this will cause, some of the source files can be humungous. Have you made some comparisons of heap size differences?
|
Also, @flakey5 this https://github.com/syntax-tree/estree-util-visit is what I was referring to use to walk through the Nodes of the source code. Since acorn generates an AST that is compatible with estree. |
Closes #152 Signed-off-by: flakey5 <[email protected]>
Co-authored-by: Claudio W <[email protected]>
Signed-off-by: flakey5 <[email protected]>
Closes #152
Opening this as a draft currently to get feedback on the approach since it's a bit non-trivial relative to the other generators.Theapilinks.jsonfile maps things exported by modules to their source locations on Github.Example:
{ "SomeClass.prototype.var": "github.com/nodejs/node/tree/<hash>/lib/file.js#L100" }This means we need to parse the module's javascript source in addition to its markdown.
What the current approach does is doing:
acornis used for parsing the source files.jsfiles, so to run it you need to do pass innode/lib/*.jsas the inputThis is dependent on the Markdown source parsing since it uses thesource_linkmetadata in the docsast-jsgeneratorapi-linksgenerator is based off of theast-jsresult