Skip to content

Commit 8a250a4

Browse files
committed
Add streaming execution examples to DataFrame documentation
1 parent d6e8132 commit 8a250a4

File tree

1 file changed

+24
-0
lines changed

1 file changed

+24
-0
lines changed

docs/source/user-guide/dataframe/index.rst

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,30 @@ To work with the stream directly, use
203203
for batch in stream:
204204
...
205205
206+
Execute as Stream
207+
^^^^^^^^^^^^^^^^^
208+
209+
For finer control over streaming execution, use
210+
:py:meth:`~datafusion.DataFrame.execute_stream` to obtain a
211+
:py:class:`pyarrow.RecordBatchReader`:
212+
213+
.. code-block:: python
214+
215+
reader = df.execute_stream()
216+
for batch in reader:
217+
... # process each batch as it is produced
218+
219+
When partition boundaries are important,
220+
:py:meth:`~datafusion.DataFrame.execute_stream_partitioned`
221+
returns an iterable of :py:class:`pyarrow.RecordBatchReader` objects, one per
222+
partition:
223+
224+
.. code-block:: python
225+
226+
for stream in df.execute_stream_partitioned():
227+
for batch in stream:
228+
... # each stream yields RecordBatches
229+
206230
See :doc:`../io/arrow` for additional details on the Arrow interface.
207231

208232
HTML Rendering

0 commit comments

Comments
 (0)