"codeql database create" command cannot recognize the relative path in "import" (Python/Go)

[zhenshiping]

If the project use relative path to import module lilke below:

from ...moduleA import ClassA
import .moduleB
from ..mouduleC import ClassC

The . .. ... will cause WARN when you use codeql database create, here is the log:

[2023-05-05 09:32:15] [build-stdout] [WARN] [1] Failed to find .xxx, no parent package of Python module at xxx.py
[2023-05-05 09:32:15] [build-stdout] [WARN] [2] Failed to find ..xxx, no parent package of Python module at xxxxx.py
[2023-05-05 09:32:15] [build-stdout] [WARN] [2] Failed to find ....xxxx, no parent package of Python module at xxxxxx.py

Python and Go has the same problem, how can I fix this WARN?

[owen-mc]

I am only familiar with CodeQL for Go, not python. I will try to get a colleague who works on python to answer when they are available.

We extract relative imports in Go correctly as long as the build is set up correctly. Can you point me to a repository where this doesn't work? Alternatively, the build logs might help me diagnose what is happening (along with some idea of the layout of the repo).

One situation I am aware of where it doesn't work is when there is a go.mod file which is not in the root of the repository. Say it is in folder a/b/c. In this case the solution is to add a build command which moves into the approriate folder first. The default build command is go build ./... (... has a special wildcard meaning here). You could instead specify the build command (cd a/b/c && go build ./...) and that should work.

[zhenshiping]

Solved this problem on python project , using a absolufy-imports tool to preprocess the repo ,it will cconvert all the relative path to abslute path and database create get no WARN, but this is just a tempory solution, for your conference here: https://stackoverflow.com/questions/27549291/convert-all-relative-imports-to-absolute-automatically-in-python

[EndlessConf]

For GO，does CodeQL only find module dependencies through go mod? I am not using go mod to manage modules, but instead specifying the module path through GOPATH at compile time. In this case, I found that codeql database create command will enter an infinite loop.

[owen-mc]

@EndlessConf In the absence of a go.mod we do the best that we can. Are you running locally or in CI? Can you point me to an example repo where it doesn't work? If it doesn't work then you can provide a build command, which normally solves the problem.

[EndlessConf]

Sorry, this issue came up about a month ago when I was trying to write a Go demo to test CodeQL.
At the time, I used my own encapsulation, and then found that "Codeql database create" did not end for a long time.
Today I tried to reproduce the problem, but failed.
I looked up the previous problem log and it kept looping like this:

Warning: SEMMLE_EXEC and SEMMLE_EXECP not set. Falling back to path lookup on argv[0].
==== Candidate to intercept: git (connonical: git) ====
Will try to resolve git in path /usr/local/sin/:/usr/local/bin:/usr/bin:/usr/sbin:/home/zy/test
......
......
Warning: SEMMLE_EXEC and SEMMLE_EXECP not set. Falling back to path lookup on argv[0].
==== Candidate to intercept: git (connonical: git) ====
Will try to resolve git in path /usr/local/sin/:/usr/local/bin:/usr/bin:/usr/sbin:/home/zy/test
......
......

[RasmusWL]

@y0utiao can you please link to a open-source repo, or create a minimal working example repo that has this problem?

@EndlessConf please open a new issue for asking new questions next time 😊