Decoding and path problems on windows

[gijzelaerr]

When running this script on Windows:

https://github.com/MonetDBSolutions/monetdbe-examples/blob/master/python/taxi.py

we see the error:

ParseException:SQLparser:42000!COPY INTO: filename must have absolute path: "C:ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ\020\016`>g\002"
Traceback (most recent call last):
  File "taxi.py", line 156, in <module>
    loaddb(path)
  File "taxi.py", line 54, in loaddb
    cursor.execute("""
  File "C:\Python38\lib\site-packages\monetdbe\cursor.py", line 152, in execute
    self.result, self.rowcount = self.connection.lowlevel.query(formatted, make_result=True)  # type: ignore
  File "C:\Python38\lib\site-packages\monetdbe\_cffi.py", line 244, in query
    check_error(lib.monetdbe_query(self._connection, query.encode(), p_result, affected_rows))
  File "C:\Python38\lib\site-packages\monetdbe\_cffi.py", line 82, in check_error
    raise exception(msg)
monetdbe.exceptions.OperationalError: COPY INTO: filename must have absolute path: "C:ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ\020\016`>g\002"

The script issues a COPY INTO command, but the path is not properly escaped. When the script is modified with:

path = os.getcwd().replace("\\", "\\\\") + '\\\\..\\\\'

Above error is not properly decoded by monetdbe though:

Traceback (most recent call last):
  File "taxi.py", line 156, in <module>
    loaddb(path)
  File "taxi.py", line 54, in loaddb
    cursor.execute("""
  File "C:\Python38\lib\site-packages\monetdbe\cursor.py", line 152, in execute
    self.result, self.rowcount = self.connection.lowlevel.query(formatted, make_result=True)  # type: ignore
  File "C:\Python38\lib\site-packages\monetdbe\_cffi.py", line 244, in query
    check_error(lib.monetdbe_query(self._connection, query.encode(), p_result, affected_rows))
  File "C:\Python38\lib\site-packages\monetdbe\_cffi.py", line 69, in check_error
    decoded = ffi.string(msg).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 79: invalid continuation byte

Monetdbe-Python assumes the error strings returned by monetdb are UTF-8, but in this case the are somehow latin-1. If we change the decoding of the monetdbe errors to to latin-1 we get the proper error.

[gijzelaerr]

@njnes can you give your opinion about this issue? :)

[njnes]

monetdbe (and server) expect input 2 be utf8, ie how could monetdbe generate latin-1?

[gijzelaerr]

I think the issue was misdiagnosed, and the path is containing illegal characters.

[gijzelaerr]

This is still an issue, If I run the query:

COPY OFFSET 2 INTO yellow_tripdata_2016_01 FROM 'C:\Users\gijs\Code\monetdbe-examples\data\yellow_tripdata_2016-01.csv' delimiters ',','

I get the error:

monetdbe.exceptions.OperationalError: COPY INTO: filename must have absolute path: "C:"

[njnes]

the string passed to the c level should be
COPY OFFSET 2 INTO yellow_tripdata_2016_01 FROM 'C:\Users\gijs\Code\monetdbe-examples\data\yellow_tripdata_2016-01.csv' delimiters ',','

[gijzelaerr]

0.9 has been release, i just didn't manage to test this on Windows yet, so we still have to see if that is resolved.

[gijzelaerr]

This now gives a ParseException:SQLparser:42000!NULL byte in string on all platforms, double escaped or not.

[gijzelaerr]

All these give the same errors:

con = connect()
q1 = "COPY OFFSET 2 INTO t FROM 'C:\\does\\not\\exist.csv' delimiters ',','"
con.execute(q1)
q2 = "COPY OFFSET 2 INTO t FROM 'C:\\\\does\\\\not\\\\exist.csv' delimiters ',','"
con.execute(q1)
q13= r"COPY OFFSET 2 INTO t FROM 'C:\does\not\exist.csv' delimiters ',','"
con.execute(q1)