@carlosparadis here is my first pass at the issue for part I.

Yes, this is great!

The one thing I would emphasize is:

Understand XPath queries, and create queries for the new functions (file and class level documentation).

The better mindset is to consider:

1. Imagine a table that every row is a file in the project. The first column is the filepath, the second column contain in the cell all of its content as words.
2. If we wrote programs purely in english, then every word would be meaningful. But the words "for", "if", etc, do not carry a lot of meaning on its own. So, in another column, we could consider creating a cell without them (let's skip this for now).
3. Then in a subsequent column we could say, "let's just focus on docstrings since that's actual english". Now you have another column that only contain words coming from docstrings.
4. You could also have another column that in addition to the docstrings, include the variable names, since they carry meaning. Or function names, etc.

Every column is a different "extract" from all the words in each file. The functions you will add, will extract these words from a project files. This would enable the rest of the pipeline to experiment with which set of words is more meaningful.

@carlosparadis I added a notebook for syntax extraction. Could you skim it and see if this is what you are looking for? The section on XPath queries is definitely a WIP, it was more to gather my own understanding. It will be updated as I work on getting the actual queries needed for the new functions. New config file is a placeholder as of now.

@daomcgill did you open a PR for it?

@carlosparadis I had not. I just opened one here.

@carlosparadis I made the changes we discussed. I also added functions for extracting function names (opt. params) and variable names (opt. types). I do need to do a more thorough manual check to make sure the results are accurate, but it appears to be working so far.
Should I continue adding functions--maybe imports or some way of determining inheritance structure (I would need to think about this some more)--or move on and come back once the other pieces start to come together?

@daomcgill the code that @RavenMarQ did already determines dependency types, so no need to go after that. If you already covered variables, functions, classes, is there anything else on the source code you thing carries semantic meaning?

Also, you did this for Java, right? What were the other available languages srcML covered again?

And lastly, did you do the parser functions for these too or just the command to create the XML?

@carlosparadis I do not think operator, specifier or control statements would be particularly meaningful. Do you have any ideas of what might be useful? The only thought I had was imports.

I did this for Java. It looks like srcML is also available for C, C++ and C#.

I am not sure what you mean by parser functions vs command to create the XML. I created functions that query the XML file generated by the preexisting function, and extract the class/file level comments, functions and variables.

Hi Dao,

I honestly can't think of anything else. I would say go ahead and add the imports. We can just stick with Java for now. Python would have been nice if they covered.

Once this issue is done, I suggest we also skip ahead to play with the Python Notebook that process this data on the other repo, since this will give you a better idea how this data is being used, and then we can cycle back to the issue that represent files as commit and comments associated to it.

Have you been using your functions to parse a table of 1 file, or the entire project out? We will want to ideally get a table that contains the filepaths the classes, methods, docstring belongs so it helps with subsetting when we move to the Python script.

I recall in the Text GoF notebook I had to create a filepath and a classpath to be able to connect both together, but maybe the xpath can give you both.

@carlosparadis I will add the imports. It looks like srcML should be adding support for more languages soon, but we do not know when.

Also, I realized what you meant by parser functions. I did create those as well.

So far, I have been using the functions on one specific src folder in the Maven repo. It contains multiple files, but not the entire project. I can work on that. The tables generated show filepath + the element that specific function is querying, e.g. filepath and classes. I could make a function that calls each of the query functions and compiles that extracted data in a single table.