add Linux support

[jkirk]

Thank you for this wonderful project.

My setup might be unusual but I run Windows + Excel in Virtualbox on a GNU/Linux Debian/stretch machine. My project folder is a shared folder so I can use all the command line stuff in Linux and work with Excel in the Virtualbox.

It would be fantastic if git xltrail was available in Linux too. Currently the setup/installation fails (after cloning) :

% pwd
~proj/misc/git-xltrail
% python3 --version
Python 3.5.3

% virtualenv -p $(which python3) venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/jkirk/projects/misc/git-xltrail/venv/bin/python3
Also creating executable in /home/jkirk/projects/misc/git-xltrail/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

% . venv/bin/activate
% pip install -r requirements.txt

% pip install -r requirements.txt
Collecting colorama==0.3.9 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting pythonnet==2.3.0 (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/89/3b/a22cd45b591d6cf490ee8b24d52b9db1f30b4b478b64a9b231c53474731e/pythonnet-2.3.0.tar.gz
Collecting pyinstaller==3.3.1 (from -r requirements.txt (line 3))
Collecting pefile>=2017.8.1 (from pyinstaller==3.3.1->-r requirements.txt (line 3))
Requirement already satisfied: setuptools in ./venv/lib/python3.5/site-packages (from pyinstaller==3.3.1->-r requirements.txt (line 3)) (41.0.1)
Collecting macholib>=1.8 (from pyinstaller==3.3.1->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Collecting future (from pefile>=2017.8.1->pyinstaller==3.3.1->-r requirements.txt (line 3))
Collecting altgraph>=0.15 (from macholib>=1.8->pyinstaller==3.3.1->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py) ... error
  ERROR: Complete output from command /home/jkirk/projects/misc/git-xltrail/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-jilzgp3v/pythonnet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-vk_d05qc --python-tag cp35:
  ERROR: running bdist_wheel
  running build
  running build_ext
  Could not load file or assembly or one of its dependencies.
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 405, in <module>
      zip_safe=False,
    File "/home/jkirk/projects/misc/git-xltrail/venv/lib/python3.5/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/jkirk/projects/misc/git-xltrail/venv/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 139, in build_extension
      self._install_packages()
    File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 269, in _install_packages
      subprocess.check_call(cmd, shell=use_shell)
    File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command 'mono tools/nuget/nuget.exe update -self' returned non-zero exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: colorama, pythonnet, future, pefile, altgraph, macholib, pyinstaller
  Running setup.py install for pythonnet ... error
    ERROR: Complete output from command /home/jkirk/projects/misc/git-xltrail/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-jilzgp3v/pythonnet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v19pscej/install-record.txt --single-version-externally-managed --compile --install-headers /home/jkirk/projects/misc/git-xltrail/venv/include/site/python3.5/pythonnet:
    ERROR: running install
    running build
    running build_ext
    Could not load file or assembly or one of its dependencies.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 405, in <module>
        zip_safe=False,
      File "/home/jkirk/projects/misc/git-xltrail/venv/lib/python3.5/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/jkirk/projects/misc/git-xltrail/venv/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
        self.run_command('build')
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.5/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/lib/python3.5/distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/usr/lib/python3.5/distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 139, in build_extension
        self._install_packages()
      File "/tmp/pip-install-jilzgp3v/pythonnet/setup.py", line 269, in _install_packages
        subprocess.check_call(cmd, shell=use_shell)
      File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command 'mono tools/nuget/nuget.exe update -self' returned non-zero exit status 1
    ----------------------------------------
ERROR: Command "/home/jkirk/projects/misc/git-xltrail/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-jilzgp3v/pythonnet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v19pscej/install-record.txt --single-version-externally-managed --compile --install-headers /home/jkirk/projects/misc/git-xltrail/venv/include/site/python3.5/pythonnet" failed with error code 1 in /tmp/pip-install-jilzgp3v/pythonnet/

I tracked it down a bit and it seems to be related to pythonnet/pythonnet#729.

Do you have any idea what to do or how I can help to make this happen?

[bstiel]

Good question.

I've actually never tried to run this on Linux but in theory this should work as long as you have mono installed (which in turn supports nuget). But clearly it doesn't work ;-)

I'll dig a bit deeper and report back here, feel free to do the same.

[PavelTurk]

The project seems to be very interesting. The reason why I can't say exactly is that I use Linux. To tell the truth it is a very strange approach to product vision - the product was developed for developers (who else uses git) however doesn't support Linux (main OS among developers). So, please, implement support for it a little bit faster :)

[jkirk]

The pythonnet dependency has been removed in 107fbac so the "pip install" error doesn't hit me anymore.

I started an attempt to create a build script for Linux (Debian/buster) here: https://github.com/jkirk/git-xl/tree/linux-support

And it seems to do something:

% scripts/linux/build                                         
-----------
Version tag: 0.0.0
Build number: 0
Commit hash: dev
Generate file version: 0.0.0.0
-----------
110 INFO: PyInstaller: 3.3.1
110 INFO: Python: 3.7.3
110 INFO: Platform: Linux-5.7.0-0.bpo.2-amd64-x86_64-with-debian-10.6
111 INFO: wrote /home/jkirk/projects/misc/git-xltrail/git-xl-diff-x64.spec
112 INFO: UPX is not available.
113 INFO: Extending PYTHONPATH with paths
['/home/jkirk/projects/misc/git-xltrail/src',
 '/home/jkirk/projects/misc/git-xltrail']
113 INFO: checking Analysis
116 INFO: checking PYZ
117 INFO: checking PKG
117 INFO: Bootloader /home/jkirk/projects/misc/git-xltrail/venv3/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
117 INFO: checking EXE
118 WARNING: ignoring version, manifest and resources, platform not capable
118 WARNING: ignoring icon, platform not capable
102 INFO: PyInstaller: 3.3.1
102 INFO: Python: 3.7.3
103 INFO: Platform: Linux-5.7.0-0.bpo.2-amd64-x86_64-with-debian-10.6
106 INFO: wrote /home/jkirk/projects/misc/git-xltrail/git-xl-x64.spec
111 INFO: UPX is not available.
113 INFO: Extending PYTHONPATH with paths
['/home/jkirk/projects/misc/git-xltrail/src',
 '/home/jkirk/projects/misc/git-xltrail']
113 INFO: checking Analysis
116 INFO: Building because /home/jkirk/projects/misc/git-xltrail/src/cli.py changed
116 INFO: Initializing module dependency graph...
118 INFO: Initializing module graph hooks...
119 INFO: Analyzing base_library.zip ...
1984 INFO: running Analysis out00-Analysis.toc
2008 INFO: Caching module hooks...
2014 INFO: Analyzing /home/jkirk/projects/misc/git-xltrail/src/cli.py
2093 INFO: Loading module hooks...
2094 INFO: Loading module hook "hook-xml.py"...
2277 INFO: Loading module hook "hook-encodings.py"...
2321 INFO: Loading module hook "hook-pydoc.py"...
2331 INFO: Looking for ctypes DLLs
2369 INFO: Analyzing run-time hooks ...
2376 INFO: Looking for dynamic libraries
2598 INFO: Looking for eggs
2598 INFO: Python library not in binary dependencies. Doing additional searching...
2647 INFO: Using Python library /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0
2651 INFO: Warnings written to /home/jkirk/projects/misc/git-xltrail/build/git-xl-x64/warngit-xl-x64.txt
2680 INFO: Graph cross-reference written to /home/jkirk/projects/misc/git-xltrail/build/git-xl-x64/xref-git-xl-x64.html
2686 INFO: checking PYZ
2688 INFO: Building because toc changed
2688 INFO: Building PYZ (ZlibArchive) /home/jkirk/projects/misc/git-xltrail/build/git-xl-x64/out00-PYZ.pyz
2943 INFO: Building PYZ (ZlibArchive) /home/jkirk/projects/misc/git-xltrail/build/git-xl-x64/out00-PYZ.pyz completed successfully.
2945 INFO: checking PKG
2946 INFO: Building because toc changed
2946 INFO: Building PKG (CArchive) out00-PKG.pkg
5044 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
5045 INFO: Bootloader /home/jkirk/projects/misc/git-xltrail/venv3/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
5045 INFO: checking EXE
5045 INFO: Building because toc changed
5045 INFO: Building EXE from out00-EXE.toc
5046 INFO: Appending archive to ELF section in EXE /home/jkirk/projects/misc/git-xltrail/dist/git-xl-x64
5060 INFO: Building EXE from out00-EXE.toc completed successfully.
scripts/linux/build  5.14s user 0.31s system 100% cpu 5.435 total

But the tools git-xl-diff-x64 + git-xl-x64 do not work:

% cd dist
% ./git-xl-diff-x64 
Fatal Python error: initfsencoding: Unable to get the locale encoding
zipimport.ZipImportError: can't find module 'encodings'

Current thread 0x00007f8de665d740 (most recent call first):
[1]    9577 abort      ./git-xl-diff-x64
% ./git-xl-x64 
Fatal Python error: initfsencoding: Unable to get the locale encoding
zipimport.ZipImportError: can't find module 'encodings'

Current thread 0x00007f4a05881740 (most recent call first):
[1]    9597 abort      ./git-xl-x64

How can I proceed from here? I need help. Thank you!

[Navax99]

Fatal Python error: initfsencoding: Unable to get the locale encoding
zipimport.ZipImportError: can't find module 'encodings'

This is an issue with pyinstaller 3.3.1. Upgrading to 4.2 fixes the issue :).
Testing if the package works now...

[Navax99]

Doesn't work. git diff runs ok, it reports a clean diff. No error at all.
I modified the macros using Libreoffice. Maybe is something related

[demo-git-xl]$ git diff dev..master
diff --xl a/demo.xlsx b/demo.xlsx
[demo-git-xl]$ echo $?
0

[Jeroendevr]

Does any of the above commenters have been able to create a working package? Could one of the developers point out where they their opinion is on supporting multiple platforms. Seems interesting to work on it in some spare time.

[fzumstein]

You may want to have a look at the discussion here: #75

[Adrien-LUDWIG]

As @Navax99 reported, even after installing it on Linux (for example using @watercrossing solution described in #75), it does not detect any difference for any file.

The issue seems to come from the underlying library oletools. This line returns an empty list in all my test:

https://github.com/xlwings/git-xl/blob/37ce4d3c04a2885b19ac54a731808dd955d5fdb8/src/diff.py#L12

I was not able to find the precise source. It may be related to this oletools issue.

[Jython1415]

Would it be accurate to say that making git-xl installable via pip (#75) is blocked by the fact that vba_modules = vba_parser.extract_all_macros() if vba_parser.detect_vba_macros() else [] returns an empty list on Linux, but not on Windows? I am interested in seeing this open-source project become available on other platforms.

[fzumstein]

the quoted issue in the previous comment doesn't look like it's a platform-dependent issue, so no, I wouldn't expect this to be a blocker.

[Jython1415]

@fzumstein thank you for the quick response. Would you be open to a new PR to follow-up on the work of #75, but use a pyproject.toml file instead of setup.py?

[fzumstein]

Yes, but I guess what might make even more sense is to provide git-xl as pre-commit hook: https://pre-commit.com/

[Jython1415]

I see, but would this work on a non-windows system (e.g. Linux)?

[fzumstein]

sure, pre-commit isn't platform-dependent

[Jython1415]

Apologies for the confusion. I'm confused as to how to set this up on a non-windows machine. I am trying to follow the instructions on the README. Are the following steps correct?

1. Clone the repo
2. Set up a virtual environment and install requirements with pip install -r requirements.txt
3. git xl install (I think I'm missing something here)

[fzumstein]

Actually, forget pre-commit again, you'll want git diff to work, not git commit.

[Jython1415]

Got it. Yeah I've used pre-commit to enforce linting on my Python packages before but the key feature of this project is the git diff view right? How does one configure git diff to use this without the pre-compiled windows installer?

[PavelTurk]

@fzumstein Could you say if git-xl can be used on Linux now?

[fzumstein]

We currently don't have the resource to look into this, but I don't see any reason why this can't be done nowadays. It's pure python nowadays.

[jkirk]

I just updated my linux-support branch and created a PR #83 to make it a bit more visible.

git-xl compiles under Linux (Debian/bookworm), but as far as I can see git xl install (./git-xl-x64 install) and git-xl-diff-x64 need to be fixed.