-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Liam Childs
committed
Sep 14, 2016
1 parent
b3c13de
commit 560e23a
Showing
4 changed files
with
127 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
[distutils] | ||
index-servers=pypi | ||
|
||
[pypi] | ||
repository = https://pypi.python.org/pypi | ||
username = childsish |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
language: python | ||
python: | ||
- "3.5" | ||
install: | ||
- python setup.py install | ||
script: | ||
- python -m unittest discover -s tests | ||
addons: | ||
apt_packages: | ||
- pandoc | ||
before_deploy: | ||
- pandoc --from=markdown --to=rst --output=README.rst README.md | ||
deploy: | ||
provider: pypi | ||
user: childsish | ||
password: | ||
secure: BJg0b1taCp/9Q+W59OYPHMERGSMKCF2i0u9uiHM3Q11pyC7Uh5nFQ0ScWUrQ5qLtg6aAFqoSruD2yxzbYYwdt2hBChlt/B5k5Mx3B/UH0fZiSFLnQf4zaz+1TjKLKZKOph/Erg/92I3NQXJIdg+mgmZ7wfzCWuxyts3kIJHRru+hOoP7bLdyDVV9wArZyuzEuG03p6a11HFgdVdcZZeTGPW3TouR0GtlInUeJpjJhnpTqKT/jkASyWBYnToHRvxiPlmeMZr+/7u50t/VoqUFAU9lyEMzKYhTN3xIyJzv+NxTCLOz+ebfejN1p3HuTGMW9/1eYUgj7kPMLwtdi1c/5ZpvxJH5f0jaXc0e1D+jyBWSqrHywhJDBalWPcUTaV28odtlNywPYxvhi7X+XK9nk7REZlzATGXMw2mBMSr4sCucS2EzAkp4uWQe9FOxP9otYvFmYq89qgjqs8a3ecRGki6NOBH9CAZxb+pijvg5T8JAUloxuoO4pIOyNehFKwrUlSDJZjt6yGNdPkcSdKbeNDyt6XwRIYzrNF/W5uxKVDrWDsLHnTEX68J7f3ddKdVnNRjovILDij04JKHGRco9RgTId23Q+vwYiNmSoZv8w3DJBrWLl2qzI3cPkuwMQokwyNFq24WjEwcKr+0WfPr7uWJB737iMvGXDtMqUV7N+U8= | ||
on: | ||
tags: true | ||
distributions: sdist bdist_wheel | ||
repo: childsish/mimo |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,81 @@ | ||
# mimo | ||
A Python multi-input multi-output streaming library | ||
[![Build Status](https://travis-ci.org/childsish/mimo.svg?branch=master)](https://travis-ci.org/childsish/mimo) | ||
|
||
mimo | ||
==== | ||
|
||
MiMo is a multi-input multi-output Python streaming library. It allows users to define a stream with multiple inputs and multiple outputs and run them completely from beginning to end. Back-pressure has also been implemented to prevent too much memory from being used. | ||
|
||
Usage | ||
----- | ||
|
||
There are two core components in MiMo; the `Stream` and the `Workflow`. Streams to the computational processing but do not handle how the data is passed between streams. Workflows pass the data around streams but do no processing of their own. To create a workflow, the user needs to implement streams and pipe them together. | ||
|
||
### Streams | ||
|
||
Implementing a stream can be done through inheriting a sub-class from the `Stream` class or creating a `Stream`class with a custom function as the `fn` parameter. The following code shows the same implementation of a stream that will produce the numbers from 0 to 99. | ||
|
||
|
||
```python | ||
from mimo import Stream | ||
|
||
|
||
# Method 1 (inheritance) | ||
|
||
class MyStream(Stream): | ||
|
||
IN = [] | ||
OUT = ['entity'] | ||
|
||
def __init__(self): | ||
super().__init__() | ||
self.iterator = None | ||
|
||
def run(self, ins, outs): | ||
if self.iterator is None: | ||
self.iterator = iter(range(100)) | ||
for item in self.iterator: | ||
if not outs.entity.pueh(item): | ||
return True | ||
|
||
|
||
# Method 2 (constructor) | ||
|
||
my_stream = Stream(outs=['entity], fn=my_stream_fn) | ||
|
||
def my_stream_fn(ins, outs, state): | ||
if 'iterator' not in state: | ||
state['iterator'] = iter(range(100)) | ||
for item in state['iterator']: | ||
if not outs.entity.push(item): | ||
return True | ||
``` | ||
|
||
There are a few things to note about the `run` function. | ||
1. It takes two parameters, `ins` and `outs`, that contain the input streams and the output streams. The names of the input and output streams are defined by the `IN` and `OUT` member variables and accessing the input and output streams can be done through the attributes. From the example above, accessing the `entity` output stream can be done with `outs.entity`. | ||
2. Input streams can be popped and peeked. Input streams haven't been used in the above example, but the entities in the stream can be accessed one at a time with the functions `pop` and `peek`. Popping an entity will remove it from the input stream, and peeking will look at the top-most entity without removing it from the stream. | ||
2. Output streams can be pushed. Pushing an entity to an output stream will make it available to any connected downstream streams. The `push` function return a boolean to indicate whether the stream is full or not (`True` if still pushable). A full stream ca still be pushed to, but users can make their custom streams back-pressure aware by testing this value. | ||
3. The return value is a boolean. If a stream did not fully complete it's task (possibly due to back-pressure), then it should return `True` to indicate that it can be run again after downstream streams have completed. Otherwise a `False` (or equivalent like `None`) will prevent further execution of the stream until new input is available. | ||
|
||
### Workflows | ||
|
||
Workflows are created by piping streams together. First a workflow must be instantiated and populated with the desired streams. The steps returned by populating a workflow can then be used to make the connections between the streams using the `pipe` function. The function returns the stream being piped to, so `pipe` calls can be chained. | ||
|
||
```python | ||
from mimo import Stream, Workflow | ||
|
||
|
||
def main(): | ||
workflow = Workflow() | ||
step1 = workflow.add_stream(Stream(outs=['a'])) | ||
step2 = workflow.add_stream(Stream(['b'], ['c'])) | ||
step3 = workflow.add_stream(Stream(['d'])) | ||
|
||
step1.pipe(step2).pipe(step3) | ||
|
||
print(str(workflow)) | ||
workflow.run() | ||
|
||
if __name__ == '__main__': | ||
import sys | ||
sys.exit(main()) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
import os | ||
|
||
from setuptools import setup, find_packages | ||
|
||
long_description = open('README.rst').read() if os.path.exists('README.rst') else\ | ||
open('README.md').read() | ||
|
||
setup( | ||
name='mimo', | ||
version='1.0.0', | ||
author='Liam H. Childs', | ||
author_email='[email protected]', | ||
packages=find_packages(exclude=['test']), | ||
url='https://github.com/childsish/mimo', | ||
license='LICENSE.txt', | ||
description='A Python multi-input, multi-output streaming library', | ||
long_description=long_description, | ||
install_requires=['lhc-python'] | ||
) |