diff --git a/.gitignore b/.gitignore index fb7ad6f..df32d53 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +.venv *.pyc /build/ /dist/ diff --git a/README.md b/README.md index bd3d798..ad7b167 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ [![Build Status](https://travis-ci.org/neurobin/mdx_include.svg?branch=release)](https://travis-ci.org/neurobin/mdx_include) -Include extension for Python Markdown. It lets you include local or remote (downloadable) files into your markdown at arbitrary positions. +Include extension for Python Markdown. It lets you include local or remote (downloadable) files into your markdown at arbitrary positions. This project is motivated by [markdown-include](https://github.com/cmacmackin/markdown-include) and provides the same functionalities with some extras. @@ -14,6 +14,8 @@ Circular inclusion by default raises an exception. You can change this behavior **You should not use markdown-include along with this extension, choose either one, not both.** +--- + # Syntax 1. **Simple:** `{! file_path_or_url !}` @@ -24,13 +26,11 @@ Circular inclusion by default raises an exception. You can change this behavior 6. **Escaped syntax:** You can escape it to get the literal. For example, `\{! file_path_or_url !}` will give you literal `{! file_path_or_url !}` and `\\\{! file_path_or_url !}` will give you `\{! file_path_or_url !}` 7. **File slice:** You can slice a file by line and column number. The syntax is `{! file_path [ln:l.c-l.c,l.c-l.c,...] !}`. No spaces allowed inside file slice syntax `[ln:l.c-l.c,l.c-l.c,]`. See more detals in [File slicing section](#file-slicing). - **General syntax:** `{!recurs_state apply_indent file_path_or_url [ln:slice_syntax] | encoding !}` -> The spaces are not necessary. They are just to make it look nice :) . No spaces allowed between `{!` and recurs_state (`+-`). If apply indentation is specified then it must follow recurse_state immediately or the `{!` if recurse_state is not specified. - +> The spaces are not necessary. They are just to make it look nice :) . No spaces allowed between `{!` and recurs_state (`+-`). If apply indentation is specified then it must follow recurse_state immediately or the `{!` if recurse_state is not specified. -## You can change the syntax!!! +## You can change the syntax If you don't like the syntax you can change it through configuration. @@ -40,6 +40,7 @@ There might be some complications with the syntax `{!file!}`, for example, confl A paragraph {!our syntax!} ``` + would produce: ```html @@ -50,8 +51,9 @@ If you really want to avoid this type of collision, find some character sequence [See the configuration section for details](#configuration) +--- -# Install +# Installation Install from Pypi: @@ -59,34 +61,7 @@ Install from Pypi: pip install mdx_include ``` -# Usage - -```python -text = r""" -some text {! test1.md !} some more text {! test2.md | utf-8 !} - -Escaping will give you the exact literal \{! some_file !} - -If you escape, then the backslash will be removed. - -If you want the backslash too, then provide two more: \\\{! some_file !} -""" -md = markdown.Markdown(extensions=['mdx_include']) -html = md.convert(text) -print(html) -``` - -**Example output:** - -(*when test1.md contains a single line `**This is test1.md**` and test2.md contains `**This is test2.md**`*) - -```html -

some text This is test1.md some more text This is test2.md

-

Escaping will give you the exact literal {! some_file !}

-

If you escape, then the backslash will be removed.

-

If you want the backslash too, then provide two more: \{! some_file !}

-``` - +--- # Configuration @@ -118,7 +93,7 @@ Config param | Default | Details ```python configs = { 'mdx_include': { - 'base_path': 'mdx_include/test/', + 'base_path': 'tests/', 'encoding': 'utf-8', 'allow_local': True, 'allow_remote': True, @@ -148,7 +123,37 @@ html = md.convert(text) print(html) ``` -# File slicing +--- + +# Usage + +```python +text = r""" +some text {! test1.md !} some more text {! test2.md | utf-8 !} + +Escaping will give you the exact literal \{! some_file !} + +If you escape, then the backslash will be removed. + +If you want the backslash too, then provide two more: \\\{! some_file !} +""" +md = markdown.Markdown(extensions=['mdx_include']) +html = md.convert(text) +print(html) +``` + +**Example output:** + +(*when test1.md contains a single line `**This is test1.md**` and test2.md contains `**This is test2.md**`*) + +```html +

some text This is test1.md some more text This is test2.md

+

Escaping will give you the exact literal {! some_file !}

+

If you escape, then the backslash will be removed.

+

If you want the backslash too, then provide two more: \{! some_file !}

+``` + +## File slicing You can include part of the file from certain line/column number to certain line/column number. @@ -172,7 +177,7 @@ Multiple slicing can be done by adding more slice expressions with commas (s`,`) More details on the [rcslice doc](https://github.com/neurobin/rcslice) -# Manual cache control +## Manual Cache Control The configuration gives you enough cache control, but that's not where it ends :). You can do manual cache cleaning instead of letting the extension handle it for itself. First turn the auto cache cleaning off by setting `content_cache_clean_local` and/or `content_cache_clean_remote` to `False` (this is default), then call the cache cleaning function manually on the markdown object whenever you want: @@ -188,7 +193,7 @@ local_cache_dict = md.mdx_include_get_content_cache_local() remote_cache_dict = md.mdx_include_get_content_cache_remote() ``` -# How circular inclusion works +## How Circular Inclusion Works Let's say, there are three files, A, B and C. A includes B, B includes C and C inclues A and we are doing recursive include. @@ -200,17 +205,21 @@ If `allow_circular_inclusion` is set to `True`, then it will work like this: 2. B includes C normally too 3. C includes A which is a circular inclusion (`C>A>B>C>A>B>C...`). Thus A will be included in non-recursive mode as `allow_circular_inclusion` is set to `True` i.e C will include A literally without parsing A anymore. -# An example of including a gist +--- -The following markdown: +# Example +## Including a Gist - Including a gist: - - ```python - {! https://gist.github.com/drgarcia1986/3cce1d134c3c3eeb01bd/raw/73951574d6b62a18b4c342235006ff89d299f879/django_hello.py !} - ``` +The following markdown: + +````text +Including a gist: +```python +{! https://gist.github.com/drgarcia1986/3cce1d134c3c3eeb01bd/raw/73951574d6b62a18b4c342235006ff89d299f879/django_hello.py !} +``` +```` will produce (with fenced code block enabled): @@ -257,3 +266,38 @@ if __name__ == '__main__': ``` + +--- + +# Testing + +mdx_include uses the native Python unittest framework. + +1. Create a virtual environment to isolate the project dependencies + + ```bash + pip3 install venv + python3 -m venv .venv + ``` + +2. Install the project dependencies + + ```bash + python3 setup.py install + ``` + +3. Run the test suite, using one of the three options below + + ```bash + # 1. As a script + python3 tests/test.py + + # 2. Using setup.py + python3 setup.py test + + # 3. With the unittest module + python -m unittest discover tests + ``` + + - The tests will display a variety of error messages but should display `OK` as the last line; this indicates the + tests were successful diff --git a/mdx_include/mdx_include.py b/mdx_include/mdx_include.py index bc807e8..0f43d38 100644 --- a/mdx_include/mdx_include.py +++ b/mdx_include/mdx_include.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -''' +""" Include Extension for Python-Markdown =========================================== @@ -11,7 +11,7 @@ License: [BSD](http://www.opensource.org/licenses/bsd-license.php) -''' +""" from __future__ import absolute_import from __future__ import unicode_literals import markdown @@ -21,6 +21,7 @@ import pkgutil import encodings import logging + try: # python 3 from urllib.parse import urlparse @@ -42,9 +43,10 @@ MARKDOWN_MAJOR = (markdown.__version_info__ if hasattr(markdown, "__version_info__") else markdown.version_info)[0] logging.basicConfig() -LOGGER_NAME = 'mdx_include-' + __version__ +LOGGER_NAME = "mdx_include-" + __version__ log = logging.getLogger(LOGGER_NAME) + def encoding_exists(encoding): """Check if an encoding is available in Python""" false_positives = set(["aliases"]) @@ -53,30 +55,32 @@ def encoding_exists(encoding): if encoding: if encoding in found: return True - elif encoding.replace('-', '_') in found: + elif encoding.replace("-", "_") in found: return True return False -def get_remote_content_list(url, encoding='utf-8'): + +def get_remote_content_list(url, encoding="utf-8"): """Follow redirect and return the content""" try: - log.info("Downloading url: "+ url) - return ''.join([build_opener(HTTPRedirectHandler).open(url).read().decode(encoding), '\n']).splitlines(), True + log.info("Downloading url: " + url) + return "".join([build_opener(HTTPRedirectHandler).open(url).read().decode(encoding), "\n"]).splitlines(), True except Exception as err: # catching all exception, this will effectively return empty string log.exception("E: Failed to download: " + url) return [], False + def get_local_content_list(filename, encoding): """Return the file content with status""" textl = [] stat = False try: - with open(filename, 'r', encoding=encoding) as f: - textl = ''.join([f.read(), '\n']).splitlines() + with open(filename, "r", encoding=encoding) as f: + textl = "".join([f.read(), "\n"]).splitlines() stat = True except Exception as e: - log.exception('E: Could not find file: {}'.format(filename,)) + log.exception("E: Could not find file: {}".format(filename)) return textl, stat @@ -85,33 +89,65 @@ class IncludeExtension(markdown.Extension): def __init__(self, configs={}): self.config = { - 'base_path': [ '.', 'Base path from where relative paths are calculated',], - 'encoding': [ 'utf-8', 'Encoding of the files.', ], - 'allow_local': [ True, 'Allow including local files.', ], - 'allow_remote': [ True, 'Allow including remote files.', ], - 'truncate_on_failure': [True, 'Truncate the include markdown if failed to get the content.'], - 'recurs_local': [True, 'Whether the inclusion is recursive for local files.'], - 'recurs_remote': [False, 'Whether the inclusion is recursive for remote files.'], - 'syntax_left': [r'\{!', 'The left mandatory part of the syntax'], - 'syntax_right': [r'!\}', 'The right mandatory part of the syntax'], - 'syntax_delim': [r'\|', 'Delemiter used to separate path from encoding'], - 'syntax_recurs_on': ['+', 'Character to specify recurs on'], - 'syntax_recurs_off': ['-', 'Character to specify recurs off'], - 'syntax_apply_indent': ['>', 'Character to specify apply indentation'], - 'content_cache_local': [True, 'Whether to cache content for local files'], - 'content_cache_remote': [True, 'Whether to cache content for remote files'], - 'content_cache_clean_local': [False, 'Whether to clean content cache for local files after processing all the includes.'], - 'content_cache_clean_remote': [False, 'Whether to clean content cache for remote files after processing all the includes.'], - 'allow_circular_inclusion': [False, 'Whether to allow circular inclusion.'], - 'line_slice_separator': [['',''], 'A list of lines that will be used to separate parts specified by line slice syntax: 1-2,3-4,5 etc.'], - 'recursive_relative_path': [False, 'Whether include paths inside recursive files should be relative to the parent file path'], - } + "base_path": [".", "Base path from where relative paths are calculated"], + "encoding": ["utf-8", "Encoding of the files."], + "allow_local": [True, "Allow including local files."], + "allow_remote": [True, "Allow including remote files."], + "truncate_on_failure": [True, "Truncate the include markdown if failed to get the content."], + "recurs_local": [True, "Whether the inclusion is recursive for local files."], + "recurs_remote": [False, "Whether the inclusion is recursive for remote files."], + "syntax_left": [r"\{!", "The left mandatory part of the syntax"], + "syntax_right": [r"!\}", "The right mandatory part of the syntax"], + "syntax_delim": [r"\|", "Delemiter used to separate path from encoding"], + "syntax_recurs_on": ["+", "Character to specify recurs on"], + "syntax_recurs_off": ["-", "Character to specify recurs off"], + "syntax_strip_indent": ["<", "Strip indentation common to all included lines"], + "syntax_apply_indent": [">", "Character to specify apply indentation"], + "content_cache_local": [True, "Whether to cache content for local files"], + "content_cache_remote": [True, "Whether to cache content for remote files"], + "content_cache_clean_local": [ + False, + "Whether to clean content cache for local files after processing all the includes.", + ], + "content_cache_clean_remote": [ + False, + "Whether to clean content cache for remote files after processing all the includes.", + ], + "allow_circular_inclusion": [False, "Whether to allow circular inclusion."], + "line_slice_separator": [ + ["", ""], + "A list of lines that will be used to separate parts specified by line slice syntax: 1-2,3-4,5 etc.", + ], + "recursive_relative_path": [ + False, + "Whether include paths inside recursive files should be relative to the parent file path", + ], + } + # ~ super(IncludeExtension, self).__init__(*args, **kwargs) # default setConfig does not preserve None when the default config value is a bool (a bug may be or design decision) for k, v in configs.items(): self.setConfig(k, v) - # self.compiled_re = r'(?P\\)?\{!(?P[+-])?\s*(?P[^]|[]+?)(\s*\[ln:(?P[\d.,-]+)\])?\s*(\|\s*(?P.+?)\s*)?!\}' - self.compiled_re = re.compile( ''.join([r'(?P\\)?', self.config['syntax_left'][0], r'(?P[', self.config['syntax_recurs_on'][0], self.config['syntax_recurs_off'][0], r'])?(?P', self.config['syntax_apply_indent'][0], r'?)?\s*(?P[^]|[]+?)(\s*\[ln:(?P[\d.,-]+)\])?\s*(', self.config['syntax_delim'][0], r'\s*(?P.+?)\s*)?', self.config['syntax_right'][0], ])) + + regex_str = "".join( + [ + r"(?P\\)?", + self.config["syntax_left"][0], + r"(?P[", + self.config["syntax_recurs_on"][0], + self.config["syntax_recurs_off"][0], + r"])?(?P", + self.config["syntax_strip_indent"][0], + r"?)?(?P", + self.config["syntax_apply_indent"][0], + r"?)?\s*(?P[^]|[]+?)(\s*\[ln:(?P[\d.,-]+)\])?\s*(", + self.config["syntax_delim"][0], + r"\s*(?P.+?)\s*)?", + self.config["syntax_right"][0], + ] + ) + log.debug(regex_str) + self.compiled_re = re.compile(regex_str) def setConfig(self, key, value): """Sets the config key value pair preserving None value and validating the value type.""" @@ -129,19 +165,20 @@ def setConfig(self, key, value): def extendMarkdown(self, *args): if MARKDOWN_MAJOR == 2: - args[0].preprocessors.add( 'mdx_include', IncludePreprocessor(args[0], self.config, self.compiled_re),'_begin') + args[0].preprocessors.add("mdx_include", IncludePreprocessor(args[0], self.config, self.compiled_re), "_begin") else: - args[0].preprocessors.register(IncludePreprocessor(args[0], self.config, self.compiled_re), 'mdx_include', 101) + args[0].preprocessors.register(IncludePreprocessor(args[0], self.config, self.compiled_re), "mdx_include", 101) class IncludePreprocessor(markdown.preprocessors.Preprocessor): - ''' + """ This provides an "include" function for Markdown. The syntax is {! file_path | encoding !} or simply {! file_path !} for default encoding from config params. file_path can be a remote URL. This is done prior to any other Markdown processing. All file names are relative to the location from which Markdown is being called. - ''' + """ + def __init__(self, md, config, compiled_regex): md.mdx_include_content_cache_clean_local = self.mdx_include_content_cache_clean_local md.mdx_include_content_cache_clean_remote = self.mdx_include_content_cache_clean_remote @@ -149,51 +186,49 @@ def __init__(self, md, config, compiled_regex): md.mdx_include_get_content_cache_remote = self.mdx_include_get_content_cache_remote super(IncludePreprocessor, self).__init__(md) self.compiled_re = compiled_regex - self.base_path = config['base_path'][0] - self.encoding = config['encoding'][0] - self.allow_local = config['allow_local'][0] - self.allow_remote = config['allow_remote'][0] - self.truncate_on_failure = config['truncate_on_failure'][0] - self.recursive_local = config['recurs_local'][0] - self.recursive_remote = config['recurs_remote'][0] - self.syntax_recurs_on = config['syntax_recurs_on'][0] - self.syntax_recurs_off = config['syntax_recurs_off'][0] - self.syntax_apply_indent = config['syntax_apply_indent'][0] - self.mdx_include_content_cache_local = {} # key = file_path_or_url, value = content - self.mdx_include_content_cache_remote = {} # key = file_path_or_url, value = content - self.content_cache_local = config['content_cache_local'][0] - self.content_cache_remote = config['content_cache_remote'][0] - self.content_cache_clean_local = config['content_cache_clean_local'][0] - self.content_cache_clean_remote = config['content_cache_clean_remote'][0] - self.allow_circular_inclusion = config['allow_circular_inclusion'][0] - self.line_slice_separator = config['line_slice_separator'][0] - self.recursive_relative_path = config['recursive_relative_path'][0] + self.base_path = config["base_path"][0] + self.encoding = config["encoding"][0] + self.allow_local = config["allow_local"][0] + self.allow_remote = config["allow_remote"][0] + self.truncate_on_failure = config["truncate_on_failure"][0] + self.recursive_local = config["recurs_local"][0] + self.recursive_remote = config["recurs_remote"][0] + self.syntax_recurs_on = config["syntax_recurs_on"][0] + self.syntax_recurs_off = config["syntax_recurs_off"][0] + self.syntax_strip_indent = config["syntax_strip_indent"][0] + self.syntax_apply_indent = config["syntax_apply_indent"][0] + self.mdx_include_content_cache_local = {} # key = file_path_or_url, value = content + self.mdx_include_content_cache_remote = {} # key = file_path_or_url, value = content + self.content_cache_local = config["content_cache_local"][0] + self.content_cache_remote = config["content_cache_remote"][0] + self.content_cache_clean_local = config["content_cache_clean_local"][0] + self.content_cache_clean_remote = config["content_cache_clean_remote"][0] + self.allow_circular_inclusion = config["allow_circular_inclusion"][0] + self.line_slice_separator = config["line_slice_separator"][0] + self.recursive_relative_path = config["recursive_relative_path"][0] self.row_slice = RowSlice(self.line_slice_separator) - def mdx_include_content_cache_clean_local(self): - """Clean the cache dict for local files """ + """Clean the cache dict for local files""" self.mdx_include_content_cache_local = {} def mdx_include_content_cache_clean_remote(self): - """Clean the cache dict for remote files """ + """Clean the cache dict for remote files""" self.mdx_include_content_cache_remote = {} def mdx_include_get_content_cache_local(self): - """Get the cache dict for local files """ + """Get the cache dict for local files""" return self.mdx_include_content_cache_local def mdx_include_get_content_cache_remote(self): - """Get the cache dict for remote files """ + """Get the cache dict for remote files""" return self.mdx_include_content_cache_remote - def mdx_include_get_cyclic_safe_processed_line_list(self, textl, filename, parent): """Returns recursive text list if cyclic inclusion not detected, otherwise returns the unmodified text list if cyclic is allowed, otherwise throws exception. - """ if not self.cyclic.is_cyclic(filename): textl = self.mdx_include_get_processed_lines(textl, filename) @@ -204,15 +239,17 @@ def mdx_include_get_cyclic_safe_processed_line_list(self, textl, filename, paren raise RuntimeError("Circular inclusion not allowed; detected in file: " + parent + " when including " + filename + " whose parents are: " + str(self.cyclic.root[filename])) return textl - def get_remote_content_list(self, filename, encoding='utf-8'): + def get_remote_content_list(self, filename, encoding="utf-8"): """Get remote content list from cache or by download""" if self.content_cache_remote and filename in self.mdx_include_content_cache_remote: textl = self.mdx_include_content_cache_remote[filename] stat = True else: textl, stat = get_remote_content_list(filename, encoding) + if stat and self.content_cache_remote: self.mdx_include_content_cache_remote[filename] = textl + return textl, stat def get_local_content_list(self, filename, encoding): @@ -234,28 +271,33 @@ def get_recursive_content_list(self, textl, filename, parent, recursive, recurse # it's in a neutral position, check recursive state if recurse_state == self.syntax_recurs_on: textl = self.mdx_include_get_cyclic_safe_processed_line_list(textl, filename, parent) + return textl def mdx_include_get_processed_lines(self, lines, parent): """Process each line and return the processed lines""" new_lines = [] + for line in lines: resll = [] - c = 0 # current offset + c = 0 # current offset ms = self.compiled_re.finditer(line) + for m in ms: textl = [] stat = True total_match = m.group(0) d = m.groupdict() - escape = d.get('escape') - apply_indent = d.get('apply_indent') + escape = d.get("escape") + apply_indent = d.get("apply_indent") + if not escape: - filename = d.get('path') + filename = d.get("path") filename = os.path.expanduser(filename) - encoding = d.get('encoding') - recurse_state = d.get('recursive') - file_lines = d.get('lines') + encoding = d.get("encoding") + recurse_state = d.get("recursive") + file_lines = d.get("lines") + if not encoding_exists(encoding): if encoding: log.warning("W: Encoding (%s) not recognized . Falling back to: %s" % (encoding, self.encoding,)) @@ -266,12 +308,12 @@ def mdx_include_get_processed_lines(self, lines, parent): if urlo.netloc: # remote url if self.allow_remote: - filename = urlunparse(urlo).rstrip('/') + filename = urlunparse(urlo).rstrip("/") # push the child parent relation self.cyclic.add(filename, parent) - #get the content split in lines handling cache + # get the content split in lines handling cache textl, stat = self.get_remote_content_list(filename, encoding) # if slice sytax is found, slice the content, we must do it before going recursive because we don't @@ -296,11 +338,10 @@ def mdx_include_get_processed_lines(self, lines, parent): else: filename = os.path.normpath(os.path.join(self.base_path, filename)) - - #push the child parent relation + # push the child parent relation self.cyclic.add(filename, parent) - #get the content split in lines handling cache + # get the content split in lines handling cache textl, stat = self.get_local_content_list(filename, encoding) # if slice sytax is found, slice the content, we must do it before going recursive because we don't @@ -324,44 +365,61 @@ def mdx_include_get_processed_lines(self, lines, parent): if not stat and not self.truncate_on_failure: # get content failed and user wants to retain the include markdown textl = [total_match] + s, e = m.span() + if textl: - #textl has at least one element + # textl has at least one element + + if d.get("strip_indent"): + # Find the minimum number of spaces common to every line + num_common_spaces = min(len(item) - len(item.lstrip()) for item in textl if item.strip()) + + # Remove common spaces from each line + textl = [item[num_common_spaces:] for item in textl] + if resll: - resll[-1] = ''.join([resll[-1], line[c:s], textl[0] ]) + resll[-1] = "".join([resll[-1], line[c:s], textl[0]]) resll.extend(textl[1:]) else: - if apply_indent != '': - resll = [''.join([line[c:s], element]) for element in textl] + if apply_indent != "": + resll = ["".join([line[c:s], element]) for element in textl] else: - resll.append(''.join([line[c:s], textl[0]])) + resll.append("".join([line[c:s], textl[0]])) resll.extend(textl[1:]) else: resll.append(line[c:s]) # set the current offset to the end offset of this match c = e + # All replacements are done, copy the rest of the string if resll: - resll[-1] = ''.join([resll[-1], line[c:]]) + resll[-1] = "".join([resll[-1], line[c:]]) else: resll.append(line[c:]) + new_lines.extend(resll) return new_lines - def run(self, lines): """Process the list of lines provided and return a modified list""" self.cyclic = Cyclic() - new_lines = self.mdx_include_get_processed_lines(lines, '') + new_lines = self.mdx_include_get_processed_lines(lines, "") + if self.content_cache_clean_local: self.mdx_include_content_cache_clean_local() + if self.content_cache_clean_remote: self.mdx_include_content_cache_clean_remote() + return new_lines + def makeExtension(*args, **kwargs): # pragma: no cover return IncludeExtension(kwargs) + if __name__ == "__main__": import doctest + doctest.testmod() diff --git a/mdx_include/test/testa.md b/mdx_include/test/testa.md deleted file mode 100644 index acde232..0000000 --- a/mdx_include/test/testa.md +++ /dev/null @@ -1,7 +0,0 @@ --------------- testa.md -------------- - - -This just includes: \{! mdx_include/test/test2.md !} - -{! mdx_include/test/test2.md !} --------------- testa.md -------------- diff --git a/mdx_include/test/testi.md b/mdx_include/test/testi.md deleted file mode 100644 index 888b7f7..0000000 --- a/mdx_include/test/testi.md +++ /dev/null @@ -1,10 +0,0 @@ --------------- testi.md -------------- - -Including test1.md - -{! mdx_include/test/test1.md !} - -Including testm.md - -{! mdx_include/test/testm.md !} testm.md finally includes test2.md after following through several includes. --------------- testi.md -------------- diff --git a/mdx_include/test/testm.md b/mdx_include/test/testm.md deleted file mode 100644 index ef50254..0000000 --- a/mdx_include/test/testm.md +++ /dev/null @@ -1,6 +0,0 @@ --------------- testm.md -------------- - -This just includes: \{! mdx_include/test/testa.md !} - -{! mdx_include/test/testa.md !} --------------- testm.md -------------- diff --git a/mdx_include/test/tmc.html b/mdx_include/test/tmc.html deleted file mode 100644 index 6e9a41b..0000000 --- a/mdx_include/test/tmc.html +++ /dev/null @@ -1,3 +0,0 @@ -{u'mdx_include/test/test1.md': [u'**This is test1.md**']} -{u'mdx_include/test/test1.md': [u'**This is test1.md**'], u'mdx_include/test/test2.md': [u'**This is test2.md**']} -

modified

diff --git a/setup.py b/setup.py index 9303933..20cc149 100644 --- a/setup.py +++ b/setup.py @@ -5,42 +5,46 @@ from codecs import open from setuptools import setup -sys.path[0:0] = ['mdx_include'] +sys.path[0:0] = ["mdx_include"] from version import __version__ + def get_readme(filename): content = "" try: - with open(os.path.join(os.path.dirname(__file__), filename), 'r', encoding='utf-8') as readme: + with open(os.path.join(os.path.dirname(__file__), filename), "r", encoding="utf-8") as readme: content = readme.read() except Exception as e: pass return content -setup(name="mdx_include", - version=__version__, - author="Md. Jahidul Hamid", - author_email="jahidulhamid@yahoo.com", - description="Python Markdown extension to include local or remote files", - license="BSD", - keywords="markdown include local remote file", - url="https://github.com/neurobin/mdx_include", - packages=["mdx_include"], - long_description=get_readme("README.md"), - long_description_content_type="text/markdown", - classifiers=[ + +setup( + name="mdx_include", + version=__version__, + author="Md. Jahidul Hamid", + author_email="jahidulhamid@yahoo.com", + description="Python Markdown extension to include local or remote files", + license="BSD", + keywords="markdown include local remote file", + url="https://github.com/neurobin/mdx_include", + packages=["mdx_include"], + long_description=get_readme("README.md"), + long_description_content_type="text/markdown", + classifiers=[ # See: https://pypi.python.org/pypi?:action=list_classifiers - 'Development Status :: 5 - Production/Stable', - 'Intended Audience :: Developers', - 'Intended Audience :: Information Technology', - 'License :: OSI Approved :: BSD License', - 'Operating System :: OS Independent', - 'Programming Language :: Python', - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 3', - 'Topic :: Text Processing :: Filters', - 'Topic :: Text Processing :: Markup', - ], - install_requires=["Markdown>=2.6", "rcslice>=1.1.0", "cyclic"], -test_suite="mdx_include.test.test") + "Development Status :: 5 - Production/Stable", + "Intended Audience :: Developers", + "Intended Audience :: Information Technology", + "License :: OSI Approved :: BSD License", + "Operating System :: OS Independent", + "Programming Language :: Python", + "Programming Language :: Python :: 2", + "Programming Language :: Python :: 3", + "Topic :: Text Processing :: Filters", + "Topic :: Text Processing :: Markup", + ], + install_requires=["Markdown>=2.6", "rcslice>=1.1.0", "cyclic"], + test_suite="tests.test", +) diff --git a/mdx_include/test/__init__.py b/tests/__init__.py similarity index 100% rename from mdx_include/test/__init__.py rename to tests/__init__.py diff --git a/mdx_include/test/c.md b/tests/c.md similarity index 100% rename from mdx_include/test/c.md rename to tests/c.md diff --git a/mdx_include/test/d.md b/tests/d.md similarity index 100% rename from mdx_include/test/d.md rename to tests/d.md diff --git a/mdx_include/test/md/a.md b/tests/md/a.md similarity index 100% rename from mdx_include/test/md/a.md rename to tests/md/a.md diff --git a/mdx_include/test/md/b.md b/tests/md/b.md similarity index 100% rename from mdx_include/test/md/b.md rename to tests/md/b.md diff --git a/mdx_include/test/t.html b/tests/t.html similarity index 86% rename from mdx_include/test/t.html rename to tests/t.html index 3946d82..5f16636 100644 --- a/mdx_include/test/t.html +++ b/tests/t.html @@ -49,9 +49,9 @@

This is test1.md

Including testm.md

-------------- testm.md --------------

-

This just includes: {! mdx_include/test/testa.md !}

+

This just includes: {! tests/testa.md !}

-------------- testa.md --------------

-

This just includes: {! mdx_include/test/test2.md !}

+

This just includes: {! tests/test2.md !}

This is test2.md -------------- testa.md --------------

-------------- testm.md -------------- @@ -59,7 +59,7 @@ -------------- testi.md --------------

Forcing non-recursive include: -------------- testi.md --------------

Including test1.md

-

{! mdx_include/test/test1.md !}

+

{! tests/test1.md !}

Including testm.md

-

{! mdx_include/test/testm.md !} testm.md finally includes test2.md after following through several includes. +

{! tests/testm.md !} testm.md finally includes test2.md after following through several includes. -------------- testi.md --------------

diff --git a/mdx_include/test/tc.html b/tests/tc.html similarity index 88% rename from mdx_include/test/tc.html rename to tests/tc.html index e51813f..bccff57 100644 --- a/mdx_include/test/tc.html +++ b/tests/tc.html @@ -1,6 +1,6 @@

This is a test with custom configuration

-

Including test1.md This is test1.md where base path is set to mdx_include/test/

-

Including test2.md This is test2.md where base path is set to mdx_include/test/

+

Including test1.md This is test1.md where base path is set to tests/

+

Including test2.md This is test2.md where base path is set to tests/

Including a gist:

# -*- coding: utf-8 -*-
 
@@ -49,8 +49,8 @@
 

Include is here -> {! https://no.no/ !} <- This will produce download failed warning but won't strip off the include markdown because truncate_on_failure is False in the config.

Forcing recursive include when recurs_local is set to None: -------------- testi.md --------------

Including test1.md

-

{! mdx_include/test/test1.md !}

+

{! tests/test1.md !}

Including testm.md

-

{! mdx_include/test/testm.md !} testm.md finally includes test2.md after following through several includes. +

{! tests/testm.md !} testm.md finally includes test2.md after following through several includes. -------------- testi.md --------------

This is test2.md

diff --git a/mdx_include/test/tcache.html b/tests/tcache.html similarity index 100% rename from mdx_include/test/tcache.html rename to tests/tcache.html diff --git a/mdx_include/test/test.py b/tests/test.py similarity index 55% rename from mdx_include/test/test.py rename to tests/test.py index bf0de2c..ac07007 100644 --- a/mdx_include/test/test.py +++ b/tests/test.py @@ -9,25 +9,27 @@ import unittest from mdx_include.mdx_include import IncludeExtension -LOGGER_NAME = 'mdx_include_test' +LOGGER_NAME = "mdx_include_test" log = logging.getLogger(LOGGER_NAME) + def get_file_content(path): - cont = '' + cont = "" try: - with open(path, 'r') as f: - cont = f.read(); + with open(path, "r") as f: + cont = f.read() except Exception as e: log.exception("E: could not read file: " + path) return cont + def assertEqual(self, html, output): if tuple(markdown.__version_info__ if hasattr(markdown, "__version_info__") else markdown.version_info) >= (3, 3): html = html.replace('ass="language-', 'ass="') - html = html.replace('\n\n

', '

') - html = html.replace('\n

', '

') - output = output.replace('\n\n

', '

') - output = output.replace('\n

', '

') + html = html.replace("\n\n

", "

") + html = html.replace("\n

", "

") + output = output.replace("\n\n

", "

") + output = output.replace("\n

", "

") self.assertEqual(html, output) @@ -37,9 +39,9 @@ def test_default(self): text = r""" This is a simple text -Including test1.md {! mdx_include/test/test1.md !} +Including test1.md {! tests/test1.md !} -Including test2.md {! mdx_include/test/test2.md | utf-8 !} +Including test2.md {! tests/test2.md | utf-8 !} Including a gist: @@ -49,15 +51,13 @@ def test_default(self): Writing the syntax literally: \{! file_path !} (you just escape it with a backslash \\\{! file_path !} -> this one will show the backslash before the syntax in HTML) -Recursive include: {! mdx_include/test/testi.md !} +Recursive include: {! tests/testi.md !} -Forcing non-recursive include: {!- mdx_include/test/testi.md !} +Forcing non-recursive include: {!- tests/testi.md !} """.strip() - output = get_file_content('mdx_include/test/t.html') - md = markdown.Markdown(extensions=[IncludeExtension(), - 'markdown.extensions.extra', - ]) + output = get_file_content("tests/t.html") + md = markdown.Markdown(extensions=[IncludeExtension(), "markdown.extensions.extra"]) html = md.convert(text) # print(html) assertEqual(self, html, output.strip()) @@ -73,20 +73,19 @@ def test_non_existent(self): Include was here -> {! https://no.no/ !} <- Non existent URL also strips off the include markdown. """ - output = get_file_content('mdx_include/test/tne.html') - md = markdown.Markdown(extensions=[IncludeExtension(), 'markdown.extensions.extra']) + output = get_file_content("tests/tne.html") + md = markdown.Markdown(extensions=[IncludeExtension(), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) self.assertEqual(html, output.strip()) - def test_config(self): text = r""" This is a test with custom configuration -Including test1.md {! test1.md !} where base path is set to mdx_include/test/ +Including test1.md {! test1.md !} where base path is set to tests/ -Including test2.md {! test2.md | utf-8 !} where base path is set to mdx_include/test/ +Including test2.md {! test2.md | utf-8 !} where base path is set to tests/ Including a gist: @@ -107,64 +106,61 @@ def test_config(self): {! test2.md | Invalid !} """.strip() - output = get_file_content('mdx_include/test/tc.html') + output = get_file_content("tests/tc.html") configs = { - 'mdx_include': { - 'base_path': 'mdx_include/test/', - 'encoding': 'utf-8', - 'allow_local': True, - 'allow_remote': True, - 'truncate_on_failure': False, - 'recurs_local': None, - 'recurs_remote': False, - 'syntax_left': r'\{!', - 'syntax_right': r'!\}', - 'syntax_delim': r'\|', - 'syntax_recurs_on': '+', - 'syntax_recurs_off': '-', - 'content_cache_local': True, - 'content_cache_remote': True, - 'content_cache_clean_local': False, - 'content_cache_clean_remote': False, - - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra']) + "mdx_include": { + "base_path": "tests/", + "encoding": "utf-8", + "allow_local": True, + "allow_remote": True, + "truncate_on_failure": False, + "recurs_local": None, + "recurs_remote": False, + "syntax_left": r"\{!", + "syntax_right": r"!\}", + "syntax_delim": r"\|", + "syntax_recurs_on": "+", + "syntax_recurs_off": "-", + "content_cache_local": True, + "content_cache_remote": True, + "content_cache_clean_local": False, + "content_cache_clean_remote": False, + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) assertEqual(self, html, output.strip()) - def test_recurs(self): text = r""" -Forcing recursive include when recurs_local is set to None: {!+ mdx_include/test/testi.md !} +Forcing recursive include when recurs_local is set to None: {!+ tests/testi.md !} """.strip() - output = get_file_content('mdx_include/test/tr.html') + output = get_file_content("tests/tr.html") configs = { - 'mdx_include': { - 'base_path': '', - 'encoding': 'utf-8', - 'allow_local': True, - 'allow_remote': True, - 'truncate_on_failure': False, - 'recurs_local': None, - 'recurs_remote': False, - 'syntax_left': r'\{!', - 'syntax_right': r'!\}', - 'syntax_delim': r'\|', - 'syntax_recurs_on': '+', - 'syntax_recurs_off': '-', - 'content_cache_local': True, - 'content_cache_remote': True, - 'content_cache_clean_local': False, - 'content_cache_clean_remote': False, - - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra']) + "mdx_include": { + "base_path": "", + "encoding": "utf-8", + "allow_local": True, + "allow_remote": True, + "truncate_on_failure": False, + "recurs_local": None, + "recurs_remote": False, + "syntax_left": r"\{!", + "syntax_right": r"!\}", + "syntax_delim": r"\|", + "syntax_recurs_on": "+", + "syntax_recurs_off": "-", + "content_cache_local": True, + "content_cache_remote": True, + "content_cache_clean_local": False, + "content_cache_clean_remote": False, + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) self.assertEqual(html, output.strip()) @@ -180,19 +176,18 @@ def test_manual_cache(self): """ configs = { - 'mdx_include': { - 'base_path': 'mdx_include/test/', - - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra',]) + "mdx_include": { + "base_path": "tests/", + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) print(md.mdx_include_get_content_cache_local()) prevr = md.mdx_include_get_content_cache_remote() html = md.convert("{!test2.md!}") print(md.mdx_include_get_content_cache_local()) - md.mdx_include_get_content_cache_local()['mdx_include/test/test2.md'] = ['modified'] + md.mdx_include_get_content_cache_local()["tests/test2.md"] = ["modified"] print(md.convert("{!test2.md!}")) self.assertEqual(md.mdx_include_get_content_cache_remote(), prevr) md.mdx_include_content_cache_clean_local() @@ -205,11 +200,11 @@ def test_cache(self): Including the same file should use the content from cache instead of reading them from files every time. -Including test1.md {! mdx_include/test/test1.md !} +Including test1.md {! tests/test1.md !} -Including test1.md {! mdx_include/test/test1.md !} +Including test1.md {! tests/test1.md !} -Including test1.md {! mdx_include/test/test1.md !} +Including test1.md {! tests/test1.md !} Including a gist: @@ -230,15 +225,14 @@ def test_cache(self): ``` """.strip() - output = get_file_content('mdx_include/test/tcache.html') - md = markdown.Markdown(extensions=[IncludeExtension(), 'markdown.extensions.extra']) + output = get_file_content("tests/tcache.html") + md = markdown.Markdown(extensions=[IncludeExtension(), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) assertEqual(self, html, output.strip()) md.mdx_include_content_cache_clean_local() md.mdx_include_content_cache_clean_remote() - def test_cyclic(self): text = r""" This is a test with circular inclusion @@ -246,14 +240,14 @@ def test_cyclic(self): {! testcya.md !} """.strip() - output = get_file_content('mdx_include/test/testcy.html') + output = get_file_content("tests/testcy.html") configs = { - 'mdx_include': { - 'base_path': 'mdx_include/test/', - 'allow_circular_inclusion': True, - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra']) + "mdx_include": { + "base_path": "tests/", + "allow_circular_inclusion": True, + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) self.assertEqual(html, output.strip()) @@ -269,40 +263,63 @@ def test_file_slice(self): {! testfls.md [ln:1.2-2.13,6.4-2.3] !} """.strip() - output = get_file_content('mdx_include/test/tfls.html') + output = get_file_content("tests/tfls.html") configs = { - 'mdx_include': { - 'base_path': 'mdx_include/test/', - 'allow_circular_inclusion': True, - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra']) + "mdx_include": { + "base_path": "tests/", + "allow_circular_inclusion": True, + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) self.assertEqual(html, output.strip()) - def test_relative_include(self): text = r""" This is a test with relative include -{! mdx_include/test/c.md !} +{! tests/c.md !} -{! mdx_include/test/md/b.md !} +{! tests/md/b.md !} """.strip() - output = get_file_content('mdx_include/test/trl.html') + output = get_file_content("tests/trl.html") configs = { - 'mdx_include': { - 'allow_circular_inclusion': True, - 'recursive_relative_path': True, - }, - } - md = markdown.Markdown(extensions=[IncludeExtension(configs['mdx_include']), 'markdown.extensions.extra']) + "mdx_include": { + "allow_circular_inclusion": True, + "recursive_relative_path": True, + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "markdown.extensions.extra"]) html = md.convert(text) # ~ print(html) self.assertEqual(html, output.strip()) + def test_strip_indent(self): + configs = { + "mdx_include": { + "base_path": "tests/", + }, + } + md = markdown.Markdown(extensions=[IncludeExtension(configs["mdx_include"]), "fenced_code"]) + + # Test that indentation is removed from docstring snippet + text = "{!< testindent.py [ln:6-8] !}" + html = md.convert(text) + # ~ print(html) + output = "

Defines an Example object

\n

This docstring should be sliced from this file and the four leading spaces should be stripped from each line

" + self.assertEqual(html, output) + + # Test that indentation is removed from a code snippet, so we can add our own and specify the language + text = "```python\n{!< testindent.py [ln:13-14] !}\n```" + html = md.convert(text) + # ~ print(html) + output = ( + '
def __init__(self, attr1):\n    self.attribute1 = attr1\n
' + ) + self.assertEqual(html, output) + if __name__ == "__main__": unittest.main() diff --git a/mdx_include/test/test1.md b/tests/test1.md similarity index 100% rename from mdx_include/test/test1.md rename to tests/test1.md diff --git a/mdx_include/test/test2.md b/tests/test2.md similarity index 100% rename from mdx_include/test/test2.md rename to tests/test2.md diff --git a/tests/testa.md b/tests/testa.md new file mode 100644 index 0000000..4621702 --- /dev/null +++ b/tests/testa.md @@ -0,0 +1,7 @@ +-------------- testa.md -------------- + + +This just includes: \{! tests/test2.md !} + +{! tests/test2.md !} +-------------- testa.md -------------- diff --git a/mdx_include/test/testcy.html b/tests/testcy.html similarity index 100% rename from mdx_include/test/testcy.html rename to tests/testcy.html diff --git a/mdx_include/test/testcya.md b/tests/testcya.md similarity index 100% rename from mdx_include/test/testcya.md rename to tests/testcya.md diff --git a/mdx_include/test/testcyb.md b/tests/testcyb.md similarity index 100% rename from mdx_include/test/testcyb.md rename to tests/testcyb.md diff --git a/mdx_include/test/testcyc.md b/tests/testcyc.md similarity index 100% rename from mdx_include/test/testcyc.md rename to tests/testcyc.md diff --git a/mdx_include/test/testcyd.md b/tests/testcyd.md similarity index 100% rename from mdx_include/test/testcyd.md rename to tests/testcyd.md diff --git a/mdx_include/test/testfls.md b/tests/testfls.md similarity index 100% rename from mdx_include/test/testfls.md rename to tests/testfls.md diff --git a/tests/testi.md b/tests/testi.md new file mode 100644 index 0000000..8a7aa8e --- /dev/null +++ b/tests/testi.md @@ -0,0 +1,10 @@ +-------------- testi.md -------------- + +Including test1.md + +{! tests/test1.md !} + +Including testm.md + +{! tests/testm.md !} testm.md finally includes test2.md after following through several includes. +-------------- testi.md -------------- diff --git a/tests/testindent.py b/tests/testindent.py new file mode 100644 index 0000000..b0ca00a --- /dev/null +++ b/tests/testindent.py @@ -0,0 +1,14 @@ +"""This file is used to test the strip_indent feature""" + + +class Example: + """ + Defines an Example object + + This docstring should be sliced from this file and the four leading spaces should be stripped from each line + """ + + attribute1 = None + + def __init__(self, attr1): + self.attribute1 = attr1 diff --git a/tests/testm.md b/tests/testm.md new file mode 100644 index 0000000..0dda6b7 --- /dev/null +++ b/tests/testm.md @@ -0,0 +1,6 @@ +-------------- testm.md -------------- + +This just includes: \{! tests/testa.md !} + +{! tests/testa.md !} +-------------- testm.md -------------- diff --git a/mdx_include/test/tfls.html b/tests/tfls.html similarity index 100% rename from mdx_include/test/tfls.html rename to tests/tfls.html diff --git a/tests/tmc.html b/tests/tmc.html new file mode 100644 index 0000000..a27b0cb --- /dev/null +++ b/tests/tmc.html @@ -0,0 +1,3 @@ +{u'tests/test1.md': [u'**This is test1.md**']} +{u'tests/test1.md': [u'**This is test1.md**'], u'tests/test2.md': [u'**This is test2.md**']} +

modified

diff --git a/mdx_include/test/tne.html b/tests/tne.html similarity index 100% rename from mdx_include/test/tne.html rename to tests/tne.html diff --git a/mdx_include/test/tr.html b/tests/tr.html similarity index 81% rename from mdx_include/test/tr.html rename to tests/tr.html index c10017f..7dffbe3 100644 --- a/mdx_include/test/tr.html +++ b/tests/tr.html @@ -3,8 +3,8 @@

This is test1.md

Including testm.md

-------------- testm.md --------------

-

This just includes: {! mdx_include/test/testa.md !}

-

{! mdx_include/test/testa.md !} +

This just includes: {! tests/testa.md !}

+

{! tests/testa.md !} -------------- testm.md -------------- testm.md finally includes test2.md after following through several includes. -------------- testi.md --------------

diff --git a/mdx_include/test/trl.html b/tests/trl.html similarity index 100% rename from mdx_include/test/trl.html rename to tests/trl.html