This repository offers a comprehensive range of dataflow examples from basic to advanced levels, designed to familiarize you with the concepts and techniques essential for deploying Stateful Dataflows.
-
Install and Updated Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh rustup update
-
Install Fluvio and SDF
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash fvm install sdf-beta11 -
Run SDF setup to install dependencies
sdf setupYou might be prompted to install wasm dependency. Copy paste the command and run.
-
Start Cluster
fluvio cluster start
Dataflow examples:
- bank-processing
- car-processing
- hackernews-notify
- helsinki-transit
- ny-transit
- http-callout
- mask-user-pii
- openai-callout
- split-sentence
- word-counter
- word-probe
Examples centered around primitive concepts and operations.
- map
- filter
- filter-map
- flat-map
- split (filter)
- split (filter-map)
- merge
- regex
- sql (cli)
- sql (join)
- update-state
- key-value (input)
- key-value (output)
- key-value (chained)
Step-by-step instructions on how to build your own package:
Pre-built packages used in dataflows:
Stateful DataFlow workers continue to run even if you remove a fluvio cluster. If you delete a cluster and start a new one, or if you unintentionally start a dataflow in a cluster and need to clean up the workers, you can use the following command:
ps aux | grep sdf
kill -9 <PID>
rm -rf ~/.sdf/local/In the above example local is the cluster name & profile name. If you have a different cluster name, replace local with your cluster name.