Skip to content

Commit c22c12f

Browse files
avolkov1avolkov1
authored
[RELEASE] Merge pull request #94 from rapidsai/develop
gQuant 0.5 release
2 parents 63616ba + 5d53970 commit c22c12f

48 files changed

Lines changed: 1682 additions & 767 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Changelog
2+
3+
## [0.5](https://github.com/rapidsai/gQuant/tree/0.5) (2020-07-10)
4+
5+
[Full Changelog](https://github.com/rapidsai/gQuant/compare/0.4.1...0.5)
6+
7+
**Implemented enhancements:**
8+
9+
- \[FEA\] csvStockLoader.py and stockNameLoader.py - Use cudf.read\_csv\(\) insteand of pandas.read\_csv\(\) [\#24](https://github.com/rapidsai/gQuant/issues/24)
10+
11+
**Fixed bugs:**
12+
13+
- \[BUG\] Using a UDF via Series.rolling.apply\(\) results in KeyError in numba [\#88](https://github.com/rapidsai/gQuant/issues/88)
14+
- \[BUG\] download\_data.sh seems to do not be in containers anymore [\#66](https://github.com/rapidsai/gQuant/issues/66)
15+
16+
**Closed issues:**
17+
18+
- \[FEA\] Conda resolves too slow in the latest versions of the container [\#67](https://github.com/rapidsai/gQuant/issues/67)
19+
- \[FEA\] Comprehensive refactoring of indicator\_demo.ipynb notebook [\#46](https://github.com/rapidsai/gQuant/issues/46)
20+
- \[FEA\] Rename viz\_graph\(\) to viz\(\), save\_taskgraph\(\) to save\(\) [\#34](https://github.com/rapidsai/gQuant/issues/34)
21+
22+
**Merged pull requests:**
23+
24+
- \[REVIEW\] Fix mortgage e2e example for rapids 0.14. [\#93](https://github.com/rapidsai/gQuant/pull/93) ([avolkov1](https://github.com/avolkov1))
25+
- \[REVIEW\] Update RAPIDS to version 0.14 [\#92](https://github.com/rapidsai/gQuant/pull/92) ([yidong72](https://github.com/yidong72))
26+
- \[REVIEW\]Multiple gpu xgboost - Dask performance fix [\#91](https://github.com/rapidsai/gQuant/pull/91) ([yidong72](https://github.com/yidong72))
27+
- \[REVIEW\]Mutliple GPU xgboost [\#90](https://github.com/rapidsai/gQuant/pull/90) ([yidong72](https://github.com/yidong72))
28+
29+
## [0.4.1](https://github.com/rapidsai/gQuant/tree/0.4.1) (2020-05-26)
30+
31+
[Full Changelog](https://github.com/rapidsai/gQuant/compare/0.4...0.4.1)
32+
33+
**Merged pull requests:**
34+
35+
- \[REVIEW\] hot fix for 0.4 release [\#86](https://github.com/rapidsai/gQuant/pull/86) ([yidong72](https://github.com/yidong72))
36+
- \[REVIEW\] fix the cuIndicator notebook and RSI perf notebook [\#85](https://github.com/rapidsai/gQuant/pull/85) ([yidong72](https://github.com/yidong72))
37+
- Add cuda102 docker support and update version against development branch [\#84](https://github.com/rapidsai/gQuant/pull/84) ([jbaron](https://github.com/jbaron))
38+
39+
## [0.4](https://github.com/rapidsai/gQuant/tree/0.4) (2020-05-19)
40+
41+
[Full Changelog](https://github.com/rapidsai/gQuant/compare/v0.2...0.4)
42+
43+
**Implemented enhancements:**
44+
45+
- \[REVIEW\]Feature adding fractional differencing computation [\#56](https://github.com/rapidsai/gQuant/pull/56) ([yidong72](https://github.com/yidong72))
46+
47+
**Fixed bugs:**
48+
49+
- \[BUG\] Dask computation fails with 0.8 build script [\#28](https://github.com/rapidsai/gQuant/issues/28)
50+
51+
**Closed issues:**
52+
53+
- \[FEA\] Add cuda 10.1.2 support [\#64](https://github.com/rapidsai/gQuant/issues/64)
54+
- \[FEA\] Use RAPIDS 0.9 container in build.sh [\#54](https://github.com/rapidsai/gQuant/issues/54)
55+
- \[FEA\] Rename notebook to notebooks [\#50](https://github.com/rapidsai/gQuant/issues/50)
56+
- \[FEA\] Add Jupyterlab extension to display GPU usage [\#49](https://github.com/rapidsai/gQuant/issues/49)
57+
- \[FEA\] Merge develop branch to master [\#47](https://github.com/rapidsai/gQuant/issues/47)
58+
- \[FEA\] implement the fractional difference operation [\#42](https://github.com/rapidsai/gQuant/issues/42)
59+
60+
**Merged pull requests:**
61+
62+
- \[REVIEW\] merge develop to master and release it as 0.4 [\#82](https://github.com/rapidsai/gQuant/pull/82) ([yidong72](https://github.com/yidong72))
63+
- \[REVIEW\]update to latest version of RAPIDS 0.13 [\#81](https://github.com/rapidsai/gQuant/pull/81) ([yidong72](https://github.com/yidong72))
64+
- fixed the gamma computation error [\#79](https://github.com/rapidsai/gQuant/pull/79) ([doyend](https://github.com/doyend))
65+
- \[REVIEW\]asian barrier option tutorial [\#77](https://github.com/rapidsai/gQuant/pull/77) ([yidong72](https://github.com/yidong72))
66+
- \[REVIEW\] upgrade to RAPIDS 0.11 [\#76](https://github.com/rapidsai/gQuant/pull/76) ([yidong72](https://github.com/yidong72))
67+
- \[skip ci\] Merge CI Scripts [\#75](https://github.com/rapidsai/gQuant/pull/75) ([avolkov1](https://github.com/avolkov1))
68+
- \[REVIEW\] Add CI scripts and conda recipe [\#74](https://github.com/rapidsai/gQuant/pull/74) ([raydouglass](https://github.com/raydouglass))
69+
- \[WIP\] CUQ-36: fix typechecking nodes multi input dataframes [\#68](https://github.com/rapidsai/gQuant/pull/68) ([avolkov1](https://github.com/avolkov1))
70+
- \[REVIEW\] Upgrade to RAPIDS 0.10 [\#63](https://github.com/rapidsai/gQuant/pull/63) ([yidong72](https://github.com/yidong72))
71+
- \[REVIEW\] stable master merge [\#62](https://github.com/rapidsai/gQuant/pull/62) ([yidong72](https://github.com/yidong72))
72+
- \[REVIEW\]upgrade to RAPIDS 0.9, FIX the rebase problem [\#61](https://github.com/rapidsai/gQuant/pull/61) ([yidong72](https://github.com/yidong72))
73+
- Revert "\[REVIEW\]upgrade to RAPIDS 0.9" [\#59](https://github.com/rapidsai/gQuant/pull/59) ([yidong72](https://github.com/yidong72))
74+
- Revert "\[REVIEW\]upgrade to RAPIDS 0.9" [\#58](https://github.com/rapidsai/gQuant/pull/58) ([avolkov1](https://github.com/avolkov1))
75+
- \[REVIEW\]upgrade to RAPIDS 0.9 [\#57](https://github.com/rapidsai/gQuant/pull/57) ([yidong72](https://github.com/yidong72))
76+
- \[REVIEW\] change the text for notebook 05 [\#55](https://github.com/rapidsai/gQuant/pull/55) ([yidong72](https://github.com/yidong72))
77+
- Fix \#50b - Rename notebook folder to notebooks [\#52](https://github.com/rapidsai/gQuant/pull/52) ([miguelusque](https://github.com/miguelusque))
78+
- Fix \#50 - Rename notebook folder to notebooks [\#51](https://github.com/rapidsai/gQuant/pull/51) ([miguelusque](https://github.com/miguelusque))
79+
80+
## [v0.2](https://github.com/rapidsai/gQuant/tree/v0.2) (2019-08-16)
81+
82+
[Full Changelog](https://github.com/rapidsai/gQuant/compare/v0.1...v0.2)
83+
84+
**Implemented enhancements:**
85+
86+
- \[FEA\] Refactor 04\_portfolio\_trade.ipynb notebook [\#39](https://github.com/rapidsai/gQuant/issues/39)
87+
- \[FEA\] Refactor notebook 01\_tutorial.ipynb [\#35](https://github.com/rapidsai/gQuant/issues/35)
88+
- \[FEA\] Add error message \(or warning\) if replace node does not exist [\#32](https://github.com/rapidsai/gQuant/issues/32)
89+
- \[FEA\] Add new issue templates [\#26](https://github.com/rapidsai/gQuant/issues/26)
90+
- \[FEA\] cuIndicator notebook plot widget is too complicated [\#17](https://github.com/rapidsai/gQuant/issues/17)
91+
92+
**Fixed bugs:**
93+
94+
- \[BUG\] Remove debug info from barPlotNode.py and cumReturnNode.py [\#40](https://github.com/rapidsai/gQuant/issues/40)
95+
- \[BUG\] 04\_portfolio\_trade.ipynb - Number of filtered stocks differs from text [\#23](https://github.com/rapidsai/gQuant/issues/23)
96+
97+
**Merged pull requests:**
98+
99+
- Fix \#17 - cuIndicator notebook plot widget is too complicated \(WIP\) [\#45](https://github.com/rapidsai/gQuant/pull/45) ([miguelusque](https://github.com/miguelusque))
100+
- Fix \#39 - Refactor 04\_portfolio\_trade.ipynb notebook [\#44](https://github.com/rapidsai/gQuant/pull/44) ([miguelusque](https://github.com/miguelusque))
101+
- Merge develop to master [\#43](https://github.com/rapidsai/gQuant/pull/43) ([yidong72](https://github.com/yidong72))
102+
- Fix \#40 - Remove debug info [\#41](https://github.com/rapidsai/gQuant/pull/41) ([miguelusque](https://github.com/miguelusque))
103+
- Update mortgage example using TaskGraph API. [\#38](https://github.com/rapidsai/gQuant/pull/38) ([avolkov1](https://github.com/avolkov1))
104+
- fixed the issue 32 [\#37](https://github.com/rapidsai/gQuant/pull/37) ([yidong72](https://github.com/yidong72))
105+
- Fix \#35 - Refactor 01\_tutorial.ipynb notebook [\#36](https://github.com/rapidsai/gQuant/pull/36) ([miguelusque](https://github.com/miguelusque))
106+
- Fix \#26b - Add new issue templates [\#30](https://github.com/rapidsai/gQuant/pull/30) ([miguelusque](https://github.com/miguelusque))
107+
- Revert "fix \#26 - Add new issues template" [\#29](https://github.com/rapidsai/gQuant/pull/29) ([yidong72](https://github.com/yidong72))
108+
- Fix \#26 - Add new issues template [\#27](https://github.com/rapidsai/gQuant/pull/27) ([miguelusque](https://github.com/miguelusque))
109+
- added workflow class [\#22](https://github.com/rapidsai/gQuant/pull/22) ([yidong72](https://github.com/yidong72))
110+
- Fix \#19b - Combine OS/Cuda versions user input [\#21](https://github.com/rapidsai/gQuant/pull/21) ([miguelusque](https://github.com/miguelusque))
111+
- Fix \#19 - build.sh - Move pip dependencies to conda dependencies [\#20](https://github.com/rapidsai/gQuant/pull/20) ([miguelusque](https://github.com/miguelusque))
112+
- Fix \#13, \#14, \#16 in cuIndicator.ipynb notebook [\#18](https://github.com/rapidsai/gQuant/pull/18) ([miguelusque](https://github.com/miguelusque))
113+
- update the build.sh [\#15](https://github.com/rapidsai/gQuant/pull/15) ([yidong72](https://github.com/yidong72))
114+
- Feature xgb notebook [\#11](https://github.com/rapidsai/gQuant/pull/11) ([yidong72](https://github.com/yidong72))
115+
- CUQ-5: Mortgage example using gQuant. [\#10](https://github.com/rapidsai/gQuant/pull/10) ([avolkov1](https://github.com/avolkov1))
116+
- CUQ-5: Mortgage example using gQuant. [\#9](https://github.com/rapidsai/gQuant/pull/9) ([avolkov1](https://github.com/avolkov1))
117+
- Feature indicator node [\#8](https://github.com/rapidsai/gQuant/pull/8) ([yidong72](https://github.com/yidong72))
118+
- Feature mulit assets indicator [\#7](https://github.com/rapidsai/gQuant/pull/7) ([yidong72](https://github.com/yidong72))
119+
- Update build.sh [\#6](https://github.com/rapidsai/gQuant/pull/6) ([phogan-nvidia](https://github.com/phogan-nvidia))
120+
- Feature environment [\#5](https://github.com/rapidsai/gQuant/pull/5) ([yidong72](https://github.com/yidong72))
121+
122+
## [v0.1](https://github.com/rapidsai/gQuant/tree/v0.1) (2019-08-13)
123+
124+
[Full Changelog](https://github.com/rapidsai/gQuant/compare/e4a967fc9e3289fdbfa37e7a7b84887579332b42...v0.1)
125+
126+
**Implemented enhancements:**
127+
128+
- \[FEA\] build.sh - Move pip dependencies to conda dependencies [\#19](https://github.com/rapidsai/gQuant/issues/19)
129+
130+
**Fixed bugs:**
131+
132+
- \[BUG\] Update build.sh to 0.7 until issue \#28 is fixed [\#31](https://github.com/rapidsai/gQuant/issues/31)
133+
- \[BUG\] cuIndicator.ipyng - Wrong series names [\#16](https://github.com/rapidsai/gQuant/issues/16)
134+
- \[BUG\] cuIndicator.ipynb - Runtime error in cell \#3 - Missing file [\#14](https://github.com/rapidsai/gQuant/issues/14)
135+
- \[BUG\] cuIndicator.ipynb - Incorrect path to dataset [\#13](https://github.com/rapidsai/gQuant/issues/13)
136+
137+
**Merged pull requests:**
138+
139+
- Revert "gQuant34 - Update build.sh to make use of RAPIDS v0.8 container" [\#33](https://github.com/rapidsai/gQuant/pull/33) ([yidong72](https://github.com/yidong72))
140+
- gQuant34 - Update build.sh to make use of RAPIDS v0.8 container [\#12](https://github.com/rapidsai/gQuant/pull/12) ([miguelusque](https://github.com/miguelusque))
141+
- Synch master with develop [\#4](https://github.com/rapidsai/gQuant/pull/4) ([avolkov1](https://github.com/avolkov1))
142+
- added unit tests for the cuindicator [\#3](https://github.com/rapidsai/gQuant/pull/3) ([yidong72](https://github.com/yidong72))
143+
- CUQ-21: Improving tutorials for gQuant [\#2](https://github.com/rapidsai/gQuant/pull/2) ([avolkov1](https://github.com/avolkov1))
144+
- Add download script and instructions in the readme [\#1](https://github.com/rapidsai/gQuant/pull/1) ([yidong72](https://github.com/yidong72))
145+
146+
147+
148+
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*

‎docker/build.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ echo -e "\nPlease, select your cuda version:\n" \
3333

3434
read -p "Enter your option and hit return [1]-4: " CUDA_VERSION
3535

36-
RAPIDS_VERSION="0.13"
36+
RAPIDS_VERSION="0.14"
3737

3838
CUDA_VERSION=${CUDA_VERSION:-1}
3939
case $CUDA_VERSION in

‎gquant/dataframe_flow/_node_flow.py‎

Lines changed: 42 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,41 @@ def __translate_column(self, columns):
124124

125125
return output
126126

127+
def __delayed_call_noports(self, inputs):
128+
129+
def get_pout(df_out):
130+
'''Used for delayed unpacking.'''
131+
if isinstance(df_out, cudf.DataFrame):
132+
# Needed for the same reason as __make_copy. To prevent columns
133+
# addition in the input data frames. In python everything is
134+
# by reference value and dataframes are mutable.
135+
# Handle the case when dask_cudf.DataFrames are source frames
136+
# which appear as cudf.DataFrame in a dask-delayed function.
137+
return df_out.copy(deep=False)
138+
139+
return df_out
140+
141+
# handle the dask dataframe automatically
142+
# use the to_delayed interface
143+
# TODO, currently only handles first input is dask_cudf df
144+
i_df = inputs[0]
145+
rest = inputs[1:]
146+
if isinstance(i_df, dask_cudf.DataFrame):
147+
output_df_dly_list = []
148+
for input_dly in i_df.to_delayed():
149+
inputs_ = [input_dly] + rest
150+
output_df_dly = dask.delayed(self.decorate_process())(inputs_)
151+
output_df_dly_per = output_df_dly.persist()
152+
df_out = dask.delayed(get_pout)(output_df_dly_per)
153+
output_df_dly_list.append(df_out.persist())
154+
155+
output_df = dask_cudf.from_delayed(output_df_dly_list)
156+
157+
else:
158+
output_df = self.decorate_process()(inputs)
159+
160+
return output_df
161+
127162
def columns_flow(self):
128163
"""
129164
Flow the graph to determine the input output dataframe column names and
@@ -406,7 +441,7 @@ def __valide(self, node_output, ref_cols):
406441
cudf_types_tuple = (cudf.DataFrame, dask_cudf.DataFrame)
407442

408443
if out_type in cudf_types_tuple:
409-
if len(out_val) == 0 and out_optional:
444+
if len(out_val.columns) == 0 and out_optional:
410445
continue
411446

412447
if out_type in cudf_types_tuple:
@@ -439,6 +474,10 @@ def __input_columns_ready(self):
439474
if iport not in self.input_columns:
440475
return False
441476

477+
if (self._using_ports() and len(self._get_input_ports(
478+
)) != 0 and len(self.inputs) == 0):
479+
return False
480+
442481
return True
443482

444483
def __get_input_df(self):
@@ -673,9 +712,9 @@ def get_pout(out_dict, port):
673712
return output_df
674713

675714
def decorate_process(self):
676-
import time
677715

678716
def timer(*argv):
717+
import time
679718
start = time.time()
680719
result = self.process(*argv)
681720
end = time.time()
@@ -711,18 +750,7 @@ def __call__(self, inputs_data):
711750
else:
712751
output_df = self.decorate_process()(inputs)
713752
else:
714-
# handle the dask dataframe automatically
715-
# use the to_delayed interface
716-
# TODO, currently only handles first input is dask_cudf df
717-
i_df = inputs[0]
718-
rest = inputs[1:]
719-
if isinstance(i_df, dask_cudf.DataFrame):
720-
d_fun = dask.delayed(self.decorate_process())
721-
output_df = dask_cudf.from_delayed([
722-
d_fun([item] + rest)
723-
for item in i_df.to_delayed()])
724-
else:
725-
output_df = self.decorate_process()(inputs)
753+
output_df = self.__delayed_call_noports(inputs)
726754

727755
if self.uid != OUTPUT_ID and output_df is None:
728756
raise Exception("None output")

‎gquant/dataframe_flow/task.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import copy
44
from .taskSpecSchema import TaskSpecSchema
55
from ._node import _Node
6+
from pathlib import Path
67

78

89
__all__ = ['Task']
@@ -62,6 +63,9 @@ def get_node_obj(self, replace=None, profile=False, tgraph_mixin=False):
6263
if modulepath is not None:
6364
spec = importlib.util.spec_from_file_location(node_id,
6465
modulepath)
66+
modulename = Path(modulepath).stem
67+
spec = importlib.util.spec_from_file_location(
68+
modulename, modulepath)
6569
mod = importlib.util.module_from_spec(spec)
6670
spec.loader.exec_module(mod)
6771
NodeClass = getattr(mod, node_type)

‎gquant/plugin_nodes/analysis/barPlotNode.py‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
from gquant.dataframe_flow import Node
22
from bqplot import Axis, LinearScale, DateScale, Figure, OHLC, Bars, Tooltip
3+
import cupy as cp
4+
import cudf
5+
import dask_cudf
36

47

58
class BarPlotNode(Node):
@@ -39,16 +42,33 @@ def process(self, inputs):
3942
ax_y = Axis(label='Price', scale=sc, orientation='vertical',
4043
tick_format='0.0f')
4144
# Construct the marks
42-
ohlc = OHLC(x=stock['datetime'][::stride],
43-
y=stock[['open', 'high', 'low', 'close']]
44-
.as_gpu_matrix()[::stride, :],
45-
marker='candle', scales={'x': dt_scale, 'y': sc},
46-
format='ohlc', stroke='blue',
47-
display_legend=True, labels=[label])
48-
bar = Bars(x=stock['datetime'][::stride],
49-
y=stock['volume'][::stride],
50-
scales={'x': dt_scale, 'y': sc2},
51-
padding=0.2)
45+
if (isinstance(stock,
46+
cudf.DataFrame) or isinstance(stock,
47+
dask_cudf.DataFrame)):
48+
ohlc = OHLC(x=stock['datetime'][::stride].to_array(),
49+
y=cp.asnumpy(stock[['open',
50+
'high',
51+
'low',
52+
'close']].values[::stride, :]),
53+
marker='candle', scales={'x': dt_scale, 'y': sc},
54+
format='ohlc', stroke='blue',
55+
display_legend=True, labels=[label])
56+
bar = Bars(x=stock['datetime'][::stride].to_array(),
57+
y=stock['volume'][::stride].to_array(),
58+
scales={'x': dt_scale, 'y': sc2},
59+
padding=0.2)
60+
else:
61+
ohlc = OHLC(x=stock['datetime'][::stride],
62+
y=stock[['open',
63+
'high',
64+
'low', 'close']].values[::stride, :],
65+
marker='candle', scales={'x': dt_scale, 'y': sc},
66+
format='ohlc', stroke='blue',
67+
display_legend=True, labels=[label])
68+
bar = Bars(x=stock['datetime'][::stride],
69+
y=stock['volume'][::stride],
70+
scales={'x': dt_scale, 'y': sc2},
71+
padding=0.2)
5272
def_tt = Tooltip(fields=['x', 'y'], formats=['%Y-%m-%d', '.2f'])
5373
bar.tooltip = def_tt
5474
bar.interactions = {

0 commit comments

Comments
 (0)