v1.6.0

Major Features 🔥

• Rust Dataplane is the default Dataplane (Golang will be deprecated in next release). To switch to Golang Runtime, set NUMAFLOW_RUNTIME

    - name: vtx-name
      containerTemplate:
        env:
          - name: NUMAFLOW_RUNTIME
            value: "golang"

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex

🚨 Breaking Change 🚨

• If you have a Reduce vertex with persistence, pipeline should be recreated. The WAL format has changed.

What's Changed

• doc: accumulator doc by @vigith in #2609
• feat: enable pod template support for serving deployment by @whynowy in #2620
• feat: exponential backoff retry strategy for sink by @adarsh0728 in #2614
• fix: potential incorrect k8s minor version by @whynowy in #2647
• feat: Kafka source - Rust implementation by @BulkBeing in #2636
• feat: Aligned Window for Reduce Async Data Movement by @yhl25 in #2618
• feat: Rust pipeline metrics by @adarsh0728 in #2666
• fix: conditional forwarding in Async Dataplane by @vigith in #2668
• fix: return 400 if serving ID header contains '.' by @vigith in #2669
• fix: retry attempts calculation and logs in retry strategy by @adarsh0728 in #2653
• feat(doc): update quick start and UI docs by @adarsh0728 in #2622
• feat: Kafka sink implementation in Rust by @BulkBeing in #2672
• feat: extension for http source by @vigith in #2687
• feat: optionally enable Wire compression to optimize network and ISB stability by @vigith in #2709
• feat: add serving http server by @cosmic-chichu in #2633
• feat: http source should honor response status codes sent to clients by @yhl25 in #2732
• feat: insert id into generator payload by @vigith in #2735
• feat: Pulsar Sink implementation by @BulkBeing in #2745
• docs: enhance user-defined sinks docs with message-response contract by @sapkota-aayush in #2747
• test(ui): partially fix UI test by @tmenjo in #2726
• fix(UI): pending and ackPending buffer info on Edges by @yogesh1801 in #2746
• feat: Unaligned Window for Async Data Movement by @yhl25 in #2699
• docs: Add user guide for GPU annotation and resource requests in vertices by @sapkota-aayush in #2753
• fix: use kafka timestamp as event time by @yhl25 in #2781
• fix: jetstream source should use published timestamp by @yhl25 in #2778
• doc: Example PDB and Anti-Affinity config for ISB by @vigith in #2787
• feat: e2e tests with rust runtime by @adarsh0728 in #2693
• fix: kafka writes messages to same partition when setKey is not set by @yhl25 in #2797
• feat: Daemon server to use http based watermark fetcher by @yhl25 in #2798
• fix(UI): health filter & support for mono-vertex health on pipeline listing page by @adarsh0728 in #2761
• doc: User documentation for Jetstream source by @BulkBeing in #2826
• fix: Detect tokio worker threads by @BulkBeing in #2831
• fix: handle stream already existing when making consumer by @saty9 in #2832
• fix: add serving pipeline transformer config by @cosmic-chichu in #2835
• feat: Filter subjects and deliver policy as user options by @BulkBeing in #2837
• feat: enable rust runtime by default by @yhl25 in #2833
• feat: unpause pipelines vertices and mvtx from min replicas by @kohlisid in #2840
• fix: graceful shutdown during reduce udf panic by @yhl25 in #2845
• fix: requeue reconciliation after seeing container restart by @whynowy in #2855

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.5.0...v1.6.0

v1.5.2

What's Changed

• chore: option to disable read ahead in udf (#2851)
• feat: option to unpause pipelines vertices and mvtx from min replicas (#2840)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.2/validating-webhook-install.yaml

Full Changelog: v1.5.1...v1.5.2

v1.5.1

What's Changed

• fix: Jetstream source should use published timestamp – #2777
• fix: Generator source – decode Base64 blob – #2757
• fix: Jetstream source – Unlimited reconnects – #2772
• fix: Use Kafka timestamp as event time – #2781
• fix: Add -- in mvtx name for memory/CPU metrics – #2784
• fix: Conditional exit code rendering – #2789
• chore: Avoid publishing same watermark
• chore: Create new hasher for every hash
• chore: Shuffle based on reduce key
• chore: Revert Golang to 1.23.1 due to compatibility

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.1/validating-webhook-install.yaml

Full Changelog: v1.5.0...v1.5.1

v1.5.0

Major Features 🔥

• Introducing Serving
• Accumulator
• Contextual Debugging

Recommend SDK

• Please use 0.9 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex

🚨 Breaking Change 🚨

• None

Experimental 🚀

• Async data movement in Numaflow backed by Rust

What's Changed

• feat: add sdk infomation metrics by @whynowy in #2208
• feat: use sidecar for ud containers [Breaking k8s < v1.29] by @whynowy in #2230
• feat: Built-in Pulsar source by @BulkBeing in #2237
• feat: Asynchronous Streaming for Source and Sink in Numaflow Core by @yhl25 in #2251
• feat: counter metrics visualizer for pipeline by @adarsh0728 in #2238
• feat: Introducing Tracker to track completeness of the Message by @yhl25 in #2264
• feat: Asynchronous Map Implementation for Pipeline by @yhl25 in #2295
• feat: Added support for kafka OAuth authentication by @SzymonZebrowski in #2257
• feat: contextual flow for metrics by @veds-g in #2345
• feat: rows/page dropbox. Fixes #1764 by @SaniyaKalamkar in #2337
• feat: adaptive lookback for monovertex by @kohlisid in #2373
• feat: add sqs source by @cosmic-chichu in #2355
• feat: Accumulator by @yhl25 in #2475
• feat: validating webhook for MonoVertex by @whynowy in #2497
• feat: surface application errors by @adarsh0728 in #2499
• feat: errors tab for application errors by @veds-g in #2473
• feat: [experimental] start golang vs. rust image based on env (#2537) by @vigith in #2538
• feat: persist error in case of partial/no responses from udf by @adarsh0728 in #2539
• feat: Builtin serve sink by @BulkBeing in #2551
• feat: Support for pending count with Adaptive lookback by @RohanAshar in #2486
• feat: Improves Serving Latency and Throughput by @yhl25 in #2564

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0/validating-webhook-install.yaml

New Contributors

• @SzymonZebrowski made their first contribution in #2257
• @SaniyaKalamkar made their first contribution in #2337
• @jacque1ine made their first contribution in #2360
• @sesame0224 made their first contribution in #2457
• @t-wata made their first contribution in #2466
• @ccpeng made their first contribution in #2504
• @borkox made their first contribution in #2526

Full Changelog: v1.4.6...v1.5.0

v1.5.0-rc5

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc5/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc5/validating-webhook-install.yaml

v1.5.0-rc4

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc4/validating-webhook-install.yaml

v1.5.0-rc3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc3/validating-webhook-install.yaml

v1.4.6

What's Changed

• chore: calculate pending in replica 0 only
  Full Changelog: v1.4.5...v1.4.6

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.6/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.6/validating-webhook-install.yaml

v1.4.5

What's Changed

• fix: broken retry strategy in pipeline sink by @yhl25 in #2625
• feat: expose metrics for pipeline/mvtx desired and current phase by @whynowy in #2563

Full Changelog: v1.4.4...v1.4.5

v1.5.0-rc2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc2/validating-webhook-install.yaml