Add generator-style run_batch function

[xingyaoww]

This PR adds a new generator_style parameter to run_batch that allows yielding results as they become available, while maintaining the performance benefits of batch processing. This is particularly useful when you want to process results as soon as they are ready, for example to save them to disk.

Changes

• Added a new generator_style parameter to run_program_batch in interpreter.py that defaults to False
• Modified the implementation to support both the original behavior and the new generator behavior
• Added the generator_style parameter to the run_batch method in ir.py to expose it to users

Usage Examples

# Original batch mode (default):
states = my_program.run_batch(args)
for state in states:
   save_result(state)

# New generator mode:
for state in my_program.run_batch(args, generator_style=True):
   save_result(state)

# With progress bar:
for state in tqdm.tqdm(my_program.run_batch(args, generator_style=True), total=len(args)):
   save_result(state)

The generator mode yields results as soon as they are available, which is useful for:

• Saving results to disk immediately to avoid memory pressure
• Processing results in real-time while others are still being generated
• Getting a more accurate progress indication

Implementation Details

• The implementation maintains the same efficient batching and threading mechanisms as before
• When generator_style=True, results are yielded through Python generators as they complete
• Progress bar support is maintained in both modes
• The change is fully backward compatible - existing code will continue to work without modification

Fixes #303

---

PR co-authored by OpenHands: https://www.all-hands.dev/share?share_id=b1757eabec18e7204a615b889819333dca4cb4388a7da4fe5b8b074f3595a582

[merrymercy]

Please fix the CI test cases https://github.com/sgl-project/sglang/actions/runs/12506724584/job/34892097249?pr=2513

[merrymercy]

The CI stil fails

[xingyaoww]

@merrymercy Sorry for the delay! Finally got it pass now and can confirm this works locally for me as well

EDIT: ok.. when call next on the generator, it didn't really yield an item until all the jobs are done..

[merrymercy]

Let me know when it is fully ready.

[xingyaoww]

@merrymercy Now i confirm it actually works in my case -- feel free to take a look when you have time

[merrymercy]

You chunking logic changes the behavior of the old code. Please revert it. When you add this functionality, please do not change any existing behavior and only add additional code to support the new functionality. Make sure when generator_style == False, it runs exactly the old code.