New importer

[brandonwillard]

This is an update (rebase to current master) of #1518, along with implemented hyc byte-compilation (copy of py_compile), removed Python 2.x compatibility (Hy no longer supports it), and importer tests converted to the new importlib.

(I couldn't find a good way to contribute changes to @vodik's existing PR; that's why this exists)

Also, a lot of utility code was removed in 5960c90, because it came directly from importlib._bootstrap and/or importlib._bootstrap_external and, since we were already importing from importlib._bootstrap, it seemed just as reasonable to use the necessary functions from their original source; however, these modules are not public-facing. At the very least, we can have everything in working order by simply accounting for discrepancies (e.g. importlib._bootstrap_external appearing after 3.5) across versions.

[Kodiologist]

I don't think we're ready to drop Python 2 support, sorry. The Python folks are still supporting it till 1 Jan 2020.

[Kodiologist]

I should add that even if you want to drop Python 2, it's a big enough change that it should get its own PR.

[brandonwillard]

Agh, yeah; well, I'm in the process of reducing the import machinery to the bare essentials in 3.x, so, once I've figured out exactly what that should look like, I'll re-add 2.7 support.

[brandonwillard]

In the search for minimal integration with Python >= 3.4 importlib, I found this small monkey-patch sufficient for at least the basics (e.g. regular imports of .hy files) and even direct use of standard py_compile:

import importlib

from hy.compiler import hy_compile
from hy.lex import tokenize
from hy.models import HyObject, HyExpression, HySymbol

importlib.machinery.SOURCE_SUFFIXES += ['.hy']

# Keep a copy of the original method
_source_to_code = importlib.machinery.SourceFileLoader.source_to_code

def hy_parse(source):
    source = re.sub(r'\A#!.*', '', source)
    return HyExpression([HySymbol("do")] + tokenize(source + "\n"))

# New, patched method that considers .hy files
def _hy_source_to_code(self, data, path, *, _optimize=-1):
    if os.path.isfile(path) and path.rpartition('.')[-1] == 'hy':
            data = hy_compile(hy_parse(data.decode("utf-8")), self.name)

    return _source_to_code(self, data, path, _optimize=_optimize)

# Patch the standard file loader
importlib.machinery.SourceFileLoader.source_to_code = _hy_source_to_code

If this approach works across the board, we could remove the custom copy of py_compile and almost all existing importlib code in this branch. Patches like this aren't usually ideal, but at least it's patching a method specified in the importlib API/spec without blatantly compromising that spec. As well. the addition to SOURCE_SUFFIXES doesn't seem too out of place, since importlib itself updates it "dynamically".

The current approach is a non-trivial work-around for FileFinder's inflexibilities (see here) that effectively supplants the standard import mechanisms, so it's probably worth seriously considering an extremely terse patch that provides the same functionality and more.

[Kodiologist]

Okay, that sounds like a decent thing to try.

[vodik]

Hey, really happy to see this work continued! Sorry I haven't had the time or energy to keep up my open source work. I'll try and give you a good code review tomorrow.

…

On August 20, 2018 12:59:47 AM EDT, "Brandon T. Willard" ***@***.***> wrote: Closed #1671. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1671 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.