Open and load --code file and --py-args arguments in C++ #60134

btzy · 2025-01-13T17:35:37Z

Description

I took a stab at #60120 to see if we can design a better alternative to #60121.

Instead of doing string substitution into Python code, we use the proper Python C API functions to assign to sys.argv and execute a Python script from a file. This avoids the issues that arise due to improper escaping of strings (e.g. ' and \). Unix file names in particular may contain any non-null character (including non-printable characters), and trying to escape such characters manually would otherwise be difficult to get right. String substitution is also possibly vulnerable to code injection attacks, as it is possible to carefully craft a string containing code that will be executed.

There was a hack that converts Windows-style directory separators to Unix-style directory separators. This is no longer necessary, and therefore has been removed.

Fixes #60120.

domi4484 · 2025-01-16T16:30:53Z

@btzy when we introduced --py-args we decided on purpose to make the args available for each python execution in QGIS independently from the --code argument.
But I can't remember exactly what's the use case thought, maybe @m-kuhn do you remember more?

@btzy If you need arguments valid only in the context of the --code script I would suggest to add a new parameter --code-args. But then arise the question what to do if they are both set?

m-kuhn · 2025-01-16T20:55:49Z

No strong opinion. What would be the reason to not make it also available for other python code than --code?

btzy · 2025-01-17T12:58:13Z

@domi4484 I misunderstood the help string of --py-args. I'll modify this PR. Likely we'll need another function in QgsPythonUtils to set the sys.argv variable (separate from runFile).

btzy · 2025-01-18T16:03:17Z

I'm not sure how to trigger the automatic Windows build that other PRs have; can someone help me get the GitHub Action to run?

btzy · 2025-01-18T17:56:37Z

Separately, I made #60183 to fix the grammar error in the help, which was what originally misled me to think that --py-args is only used with --code.

github-actions · 2025-01-19T13:40:23Z

🪟 Windows builds

Download Windows builds of this PR for testing.
Debug symbols for this build are available here.
(Built from commit 3e1671a)

🪟 Windows Qt6 builds

Download Windows Qt6 builds of this PR for testing.
(Built from commit 3e1671a)

btzy · 2025-01-23T16:01:59Z

I've tested that the Windows build works:

Passing --code runs the Python code
Passing --code with --py-args make the extra args available in the Python code in sys.argv
Passing --py-args, and then later opening the Python console and running sys.argv shows the arguments

btzy · 2025-01-28T16:11:42Z

@m-kuhn Can I get a review for this?

github-actions · 2025-02-12T02:46:39Z

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

that all unit tests are passing
that all comments by reviewers have been addressed
that there is enough information for reviewers, in particular
- link to any issues which this pull request fixes
- add a description of workflows which this pull request fixes
- add screenshots if applicable
that you have written unit tests where possible
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
If there is no further activity on this pull request, it will be closed in a week.

src/core/qgspythonrunner.h

m-kuhn · 2025-02-12T05:59:15Z

src/python/qgspythonutilsimpl.cpp

+  if ( !file.open( QIODevice::ReadOnly | QIODevice::Text ) )
+  {
+    ret = "Cannot open file";
+    goto error;


Please avoid goto

src/python/qgspythonutilsimpl.cpp

m-kuhn · 2025-02-12T06:02:42Z

src/python/qgspythonutilsimpl.cpp

+  if ( !argsobj )
+  {
+    ret = "Error occurred in PyList_New";
+    goto error;


m-kuhn · 2025-02-12T06:02:57Z

src/python/qgspythonutilsimpl.cpp

+      ret = QString( "SetArgvTraceback" ) + getTraceback();
+    else
+      ret = "Error occurred in PyImport_ImportModule";
+    goto error;


How would you like me to rewrite this code? We need to execute the cleanup code at the end whether or not this function succeeds.

The most C++-ic way to do this is to write an RAII wrapper class around PyGILState_Ensure/PyGILState_Release and PyObject/Py_XDECREF. Or for a more ad-hoc mechanism, I might have used the equivalent of std::experimental::scope_exit] (both Folly and Abseil have this in some form, but QGIS uses neither of them). I would have written code in one of the above two ways if this was for my own project, but I do not see existing code in QGIS that leverages either of them.

If the above solutions are unacceptable, we would have to either invert the guard clauses into nested if-statements, which is generally considered to be bad practice. We are thus left with goto statements, which seem to be an idiom for writing cleanup code in C.

What would be most appropriate for the QGIS codebase? Writing my own scope guard?

Sorry for the slow feedback, time is very limited recently!

I realize that we still do have some goto: in this file, so you are building on precendence in which case I do think this is acceptable.

What I did in the past was introducing a custom deleter for std::unique_ptr

QGIS/src/core/qgssqliteutils.h

Line 67 in 41d5fc2

class CORE_EXPORT sqlite3_statement_unique_ptr : public std::unique_ptr< sqlite3_stmt, QgsSqlite3StatementFinalizer>

QGIS/src/core/qgssqliteutils.cpp

Lines 37 to 40 in 41d5fc2

void QgsSqlite3StatementFinalizer::operator()( sqlite3_stmt *statement ) const

{

sqlite3_finalize( statement );

}

Something along these lines would be great to have (something similar to https://pythonextensionpatterns.readthedocs.io/en/latest/cpp_and_cpython.html ) but is of bigger scope.

So, this is good to merge, but I would love to hear your thoughts about this.

I read through section 20.1 of your linked webpage.

What they call DecRefDtor is exactly what I mean when I said "an RAII wrapper class around ... PyObject/Py_XDECREF". I would perhaps just change the implicit conversion to PyObject* to operator* and operator-> like how std::unique_ptr does it, because having the implicit conversion makes it too easy to perform that conversion by accident (and also, once we start using DecRefDtor everywhere, we'd start to see plain PyObject* as something to be discouraged, and so we shouldn't make it easy to convert to it).

Their BorrowedRef and NewRef are pointless and dangerous though - they're dangerous because you'd be able to cast a BorrowedRef into a DecRefDtor (which people will be tempted to do, to avoid writing one version of a function for BorrowedRef and another version for NewRef) and end up with slicing, and they're pointless because BorrowedRef's functionality can be written as a copy constructor for DecRefDtor, and NewRef's functionality is already in DecRefDtor (it already has a constructor taking a PyObject*). Another way to see it is that you might have multiple of these DecRefDtor instances referencing the same PyObject, but since they're all equivalent, there's no reason at all to encode in the type system that one was constructed by a borrow while the other was constructed fresh.

I think using std::unique_ptr with a custom deleter is nice; I did not think of that, and I think it would be quite appropriate for wrapping a PyObject (unless we want shared ownership - but we don't need that here, and it's quite rare to actually need shared ownership in properly written C++ code). I would write it as an alias, like using UniquePyObject = std::unique_ptr<PyObject, decltype([](PyObject* obj) { Py_XDECREF(obj); })>;, instead of using inheritance, because there's no need to inherit. Whether we would want just a UniquePyObject alias or a full-fledged DecRefDtor (i.e. that is copyable, and hence behaves more like a shared_ptr) is probably a matter of preference - UniquePyObject is simpler to write and reason about, but a full-fledged DecRefDtor abstracts the underlying Python API more fully without any extra overhead (as the Python API already has built-int ref-counting which we can't opt out of).

IMO your use of public inheritance in the definition of sqlite3_statement_unique_ptr is a code smell - you probably don't need or want the user of this class to twiddle with the std::unique_ptr inside it directly, and so I would go with composition instead (and add a get() member function that returns a plain sqlite3_stmt*). Inheritance may be justified to save characters if sqlite3_stmt had many member functions that you would like users to call directly, but it seems to be an opaque struct in a C library, so this isn't the case.

I would be happy to write a version of DecRefDtor if it's something you want for QGIS, but it's a separate change and so should not be combined into this PR.

m-kuhn · 2025-02-12T06:04:55Z

Thanks, using sys.argv does look like the correct approach!

github-actions · 2025-03-08T02:38:20Z

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

that all unit tests are passing
that all comments by reviewers have been addressed
that there is enough information for reviewers, in particular
- link to any issues which this pull request fixes
- add a description of workflows which this pull request fixes
- add screenshots if applicable
that you have written unit tests where possible
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
If there is no further activity on this pull request, it will be closed in a week.

btzy · 2025-03-08T03:46:27Z

This PR is ready for review.

github-actions · 2025-03-25T02:51:51Z

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

that all unit tests are passing
that all comments by reviewers have been addressed
that there is enough information for reviewers, in particular
- link to any issues which this pull request fixes
- add a description of workflows which this pull request fixes
- add screenshots if applicable
that you have written unit tests where possible
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
If there is no further activity on this pull request, it will be closed in a week.

btzy · 2025-03-25T02:54:18Z

Ready for review

github-actions bot added this to the 3.42.0 milestone Jan 13, 2025

agiudiceandrea mentioned this pull request Jan 15, 2025

Add py-args and -- as command line parameters qgis/QGIS-Documentation#9556

Merged

1 task

btzy force-pushed the codearg branch from 0120199 to 45ae0c0 Compare January 18, 2025 15:53

btzy marked this pull request as ready for review January 18, 2025 15:54

btzy force-pushed the codearg branch from 45ae0c0 to 6dd2003 Compare January 18, 2025 17:44

btzy force-pushed the codearg branch 4 times, most recently from ef2a7f6 to 20e2318 Compare January 19, 2025 13:10

btzy force-pushed the codearg branch 6 times, most recently from 64c5dda to 1f44ac4 Compare January 22, 2025 15:34

btzy force-pushed the codearg branch from 1f44ac4 to 4efb330 Compare January 24, 2025 17:43

btzy changed the title ~~Open and load --code file and arguments in C++~~ Open and load --code file and --py-args arguments in C++ Jan 25, 2025

nyalldawson mentioned this pull request Jan 31, 2025

[App] Allow to use single quotes in file name with the --code command line option (Fix #60120) #60121

Closed

github-actions bot added the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Feb 12, 2025

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/core/qgspythonrunner.h Outdated Show resolved Hide resolved

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/python/qgspythonutilsimpl.cpp Outdated Show resolved Hide resolved

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/python/qgspythonutilsimpl.cpp

if ( !argsobj )

{

ret = "Error occurred in PyList_New";

goto error;

Copy link

Member

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

m-kuhn reviewed Feb 12, 2025

View reviewed changes

github-actions bot removed the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Feb 12, 2025

btzy force-pushed the codearg branch from 4efb330 to 31fbb4a Compare February 12, 2025 15:46

Open and load --code file in C++

3e1671a

btzy force-pushed the codearg branch from 31fbb4a to 3e1671a Compare February 12, 2025 16:10

nyalldawson added Frozen Feature freeze - Do not merge! and removed Frozen Feature freeze - Do not merge! labels Feb 20, 2025

github-actions bot added the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Mar 8, 2025

github-actions bot removed the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Mar 8, 2025

github-actions bot added the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Mar 25, 2025

github-actions bot removed the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Mar 25, 2025

lbartoletti requested a review from m-kuhn March 25, 2025 05:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open and load --code file and --py-args arguments in C++ #60134

Open and load --code file and --py-args arguments in C++ #60134

btzy commented Jan 13, 2025 •

edited

Loading

domi4484 commented Jan 16, 2025

m-kuhn commented Jan 16, 2025

btzy commented Jan 17, 2025 •

edited

Loading

btzy commented Jan 18, 2025

btzy commented Jan 18, 2025

github-actions bot commented Jan 19, 2025 •

edited

Loading

btzy commented Jan 23, 2025 •

edited

Loading

btzy commented Jan 28, 2025

github-actions bot commented Feb 12, 2025

m-kuhn Feb 12, 2025

m-kuhn Feb 12, 2025

m-kuhn Feb 12, 2025

btzy Feb 12, 2025

m-kuhn Mar 10, 2025

btzy Mar 10, 2025 •

edited

Loading

m-kuhn commented Feb 12, 2025

github-actions bot commented Mar 8, 2025

btzy commented Mar 8, 2025

github-actions bot commented Mar 25, 2025

btzy commented Mar 25, 2025

	void QgsSqlite3StatementFinalizer::operator()( sqlite3_stmt *statement ) const
	{
	sqlite3_finalize( statement );
	}

Open and load --code file and --py-args arguments in C++ #60134

Are you sure you want to change the base?

Open and load --code file and --py-args arguments in C++ #60134

Conversation

btzy commented Jan 13, 2025 • edited Loading

Description

domi4484 commented Jan 16, 2025

m-kuhn commented Jan 16, 2025

btzy commented Jan 17, 2025 • edited Loading

btzy commented Jan 18, 2025

btzy commented Jan 18, 2025

github-actions bot commented Jan 19, 2025 • edited Loading

🪟 Windows builds

🪟 Windows Qt6 builds

btzy commented Jan 23, 2025 • edited Loading

btzy commented Jan 28, 2025

github-actions bot commented Feb 12, 2025

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

btzy Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn Mar 10, 2025

Choose a reason for hiding this comment

btzy Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

m-kuhn commented Feb 12, 2025

github-actions bot commented Mar 8, 2025

btzy commented Mar 8, 2025

github-actions bot commented Mar 25, 2025

btzy commented Mar 25, 2025

btzy commented Jan 13, 2025 •

edited

Loading

btzy commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 19, 2025 •

edited

Loading

btzy commented Jan 23, 2025 •

edited

Loading

btzy Mar 10, 2025 •

edited

Loading