--progressive flag works in coordination with --timestamp-delta=0d, so that only logs that are in the past and new since the last submission are submitted. This is helpful if you are trying to minimize difference between event_time and ingestion_time. It requires either bookkeeping for the highwater mark of last submission or an additional flag for ignoring events below some threshold (relative: e.g. 30 mins or absolute and use the last run time as an input for the next.
--progressive flag works in coordination with --timestamp-delta=0d, so that only logs that are in the past and new since the last submission are submitted. This is helpful if you are trying to minimize difference between event_time and ingestion_time. It requires either bookkeeping for the highwater mark of last submission or an additional flag for ignoring events below some threshold (relative: e.g. 30 mins or absolute and use the last run time as an input for the next.