Introduce sampling to StateBackedIterable#33621

Closed

stankiewicz wants to merge 130 commits intoapache:masterfrom

stankiewicz:sample_size_in_iterable

Commits on Jan 16, 2025

Introduce sampling to StateBackedIterable - for cheap coders it will still encode as is, for others it will sample.
stankiewicz
committed
fix test
stankiewicz
committed
Bump golang.org/x/text from 0.20.0 to 0.21.0 in /sdks (#33305)

dependabot[bot]
authored and
stankiewicz
committed
Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#33303)

dependabot[bot]
authored and
stankiewicz
committed
Enable managed service in Python tests (#33190)

ahmedabu98
authored and
stankiewicz
committed
[yaml] various inline provider doc fixes (#33296)

authored and
stankiewicz
committed
Add throttling metrics and retries to vertex embeddings (#33311)

damccorm
authored and
stankiewicz
committed
Fix FlatMapTuple typehint bug (#33307)

hjtran
authored and
stankiewicz
committed
Polish FlatMapTuple and MapTuple docstrings (#33316)

hjtran
authored and
stankiewicz
committed
Reapply "bump hadoop version (#33011)" (#33312)

Abacn
authored and
stankiewicz
committed
Add Dataflow Cost Benchmark framework to Beam Python (#33297)

jrmccluskey
authored and
stankiewicz
committed
[infra] #33309 automatically apply spotless

je-ik
authored and
stankiewicz
committed
Replace None metric values with 0.0 in Dataflow Cost Benchmark (#33336)

jrmccluskey
authored and
stankiewicz
committed
[java] BQ: add missing avro conversions to BQ TableRow (#33221)

RustedBones
authored and
stankiewicz
committed
Remove the use of google-github-actions/auth@v1 (#33338)

damondouglas
authored and
stankiewicz
committed
Add hadoop-auth (#33342)

ahmedabu98
authored and
stankiewicz
committed
Bump golang.org/x/net from 0.31.0 to 0.32.0 in /sdks (#33326)

dependabot[bot]
authored and
stankiewicz
committed
Remove mandatory beam-sdks-io-kafka dependency for dataflow worker jar (#33302)

Abacn
authored and
stankiewicz
committed
Bump github.com/nats-io/nats-server/v2 from 2.10.22 to 2.10.23 in /sdks (#33352)

dependabot[bot]
authored and
stankiewicz
committed
fix Error processing result file: CData section too big found in publish-int-test-result-action #29966 (#33353)
stankiewicz
committed
Reapply "SolaceIO.Read: handle occasional cases when finalizeCheckpoint is not executed (#32962)" (#33259) (#33269)

bzablocki
authored and
stankiewicz
committed
Update confluent version to fix CVE-2024-26308 CVE-2024-25710 (#32674)
stankiewicz
committed
Add bounded Trie metric type.

robertwb
authored and
stankiewicz
committed
Bounded Trie proto representation.

robertwb
authored and
stankiewicz
committed
Bounded Trie translation.

robertwb
authored and
stankiewicz
committed
Update reference.

robertwb
authored and
stankiewicz
committed
Bump cloud.google.com/go/profiler from 0.4.1 to 0.4.2 in /sdks (#33351)

dependabot[bot]
authored and
stankiewicz
committed
Bump cloud.google.com/go/storage from 1.47.0 to 1.48.0 in /sdks (#33327)

dependabot[bot]
authored and
stankiewicz
committed
Fix custom coder not being used in Reshuffle (global window) (#33339)

shunping
authored and
stankiewicz
committed
Bump golang.org/x/crypto from 0.30.0 to 0.31.0 in /sdks (#33366)

dependabot[bot]
authored and
stankiewicz
committed
Update numpy requirement from <2.2.0,>=1.14.3 to >=1.14.3,<2.3.0 in /sdks/python (#33325)

authored and
stankiewicz
committed
Switch to use unshaded hive-exec for io expansion service (#33349)

Abacn
authored and
stankiewicz
committed
Bump cloud.google.com/go/bigquery from 1.64.0 to 1.65.0 in /sdks (#33365)

dependabot[bot]
authored and
stankiewicz
committed
[#32211] Support OnWindowExpiration in Prism. (#33337)

lostluck
authored and
stankiewicz
committed
Bump google.golang.org/api from 0.210.0 to 0.211.0 in /sdks (#33374)

dependabot[bot]
authored and
stankiewicz
committed
Fix Apache snapshot repo usage (#33370)

adoroszlai
authored and
stankiewicz
committed
Batch optimized SparkRunner groupByKey (#33322)

twosom
authored and
stankiewicz
committed
remove flaky tests that terminate before read. (#33371)

lostluck
authored and
stankiewicz
committed
Clean up Java Tests GitHub workflow (#33354)

damondouglas
authored and
stankiewicz
committed
Fix custom coders not being used in Reshuffle (non global window) (#33363)

authored and
stankiewicz
committed
fix BoundedTrieData get_result return type hint

Polber
authored and
stankiewicz
committed
[yaml] Add use cases for Enrichment transform in YAML (#32289)

authored and
stankiewicz
committed
Fix duplicate yml entry causing test result dashboard sync to fail

kennknowles
authored and
stankiewicz
committed
Fix yaml error in Flink Java8 ValidatesRunner

kennknowles
authored and
stankiewicz
committed
[YAML] Make datetime available for jinja templatization.

robertwb
authored and
stankiewicz
committed
[YAML] Document the yaml provider include syntax.

robertwb
authored and
stankiewicz
committed
chore: upgrade google_cloud_bigdataoss_version to 2.2.26 in order to be protobuf 3 and 4 compatible (#33395)

ldetmer
authored and
stankiewicz
committed
Bump google.golang.org/protobuf from 1.35.2 to 1.36.0 in /sdks (#33397)

dependabot[bot]
authored and
stankiewicz
committed
Rag chunking embedding (#33364)

claudevdm
authored and
stankiewicz
committed
Bump google.golang.org/api from 0.211.0 to 0.212.0 in /sdks (#33398)

dependabot[bot]
authored and
stankiewicz
committed
Faster default coder for unknown windows. (#33382)

robertwb
authored and
stankiewicz
committed
Support creating BigLake managed tables (#33125)

ahmedabu98
authored and
stankiewicz
committed
End to end BoundedTrie metrics through worker and runner APIs.

robertwb
authored and
stankiewicz
committed
Add missing pxd definition.

robertwb
authored and
stankiewicz
committed
Skip unsupported runners.

robertwb
authored and
stankiewicz
committed
yapf

robertwb
authored and
stankiewicz
committed
Docstring fix.

robertwb
authored and
stankiewicz
committed
[#32929] Add OrderedListState support to Prism. (#33350)

lostluck
authored and
stankiewicz
committed
Add TF MNIST classification cost benchmark (#33391)

jrmccluskey
authored and
stankiewicz
committed
Do no throw exceptions if commitSync fails in KafkaUnboundedReader. (#33402)

an2x
authored and
stankiewicz
committed
Add missing to_type_hint to WindowedValueCoder (#33403)

shunping
authored and
stankiewicz
committed
Python ExternalTransformProvider improvements (#33359)

ahmedabu98
authored and
stankiewicz
committed
[Managed Iceberg] Support partitioning time types (year, month, day, hour) (#32939)

ahmedabu98
authored and
stankiewicz
committed
Validate circular reference for yaml (#33208)

mravi
authored and
stankiewicz
committed
Added support for SparkRunner streaming stateful processing (#33267)

twosom
authored and
stankiewicz
committed
In contributor docs, don't try to download releases from snapshot repo

raboof
authored and
stankiewicz
committed
[YAML] Better docs for Filter and MapToFields. (#33274)

authored and
stankiewicz
committed
Fix env variable loading in Cost Benchmark workflow (#33404)

jrmccluskey
authored and
stankiewicz
committed
Revert three commits related to supporting custom coder in reshuffle

shunping
authored and
stankiewicz
committed
fix error on start-build-env.sh (#33401)

marblejenka
authored and
stankiewicz
committed
Moving to 2.63.0-SNAPSHOT on master branch.

kennknowles
authored and
stankiewicz
committed
Bump Iceberg to v1.6.1 (#33294)

regadas
authored and
stankiewicz
committed
Add 2.63.0 section to CHANGES.md

kennknowles
authored and
stankiewicz
committed
Improve existing Python multi-lang SchemaTransform examples (#33361)

ahmedabu98
authored and
stankiewicz
committed
Migrate lineage counters to bounded tries.

robertwb
authored and
stankiewicz
committed
Add lineage support for local files.

robertwb
authored and
stankiewicz
committed
Remove file-specific lineage bounding.

robertwb
authored and
stankiewicz
committed
Add missing imports.

robertwb
authored and
stankiewicz
committed
Deserialization fix.

robertwb
authored and
stankiewicz
committed
Update lineage query function.

robertwb
authored and
stankiewicz
committed
Move Dataflow Python Managed tests to prod (#33426)

ahmedabu98
authored and
stankiewicz
committed
Bump github.com/docker/docker in /sdks (#33420)

dependabot[bot]
authored and
stankiewicz
committed
Bump google.golang.org/api from 0.212.0 to 0.214.0 in /sdks (#33429)

dependabot[bot]
authored and
stankiewicz
committed
[yaml] add RunInference support with VertexAI (#33406)

Polber
authored and
stankiewicz
committed
Clean up Python Tests workflow (#33396)

damondouglas
authored and
stankiewicz
committed
[yaml] allow logging bytes in LogForTesting (#33433)

Polber
authored and
stankiewicz
committed
Update .GitHub/workflows README.md (#33341)

authored and
stankiewicz
committed
[Managed BigQuery] use file loads with Avro format for better performance (#33392)

ahmedabu98
authored and
stankiewicz
committed
Support Iceberg partition identity transform (#33332)

regadas
authored and
stankiewicz
committed
python multi-lang with schematransforms guide (#33362)

ahmedabu98
authored and
stankiewicz
committed
BigQueryIO : control StorageWrite parallelism in batch, by reshuffling before write on the number of streams set for BigQueryIO.write() using .withNumStorageWriteApiStreams(numStorageWriteApiStream…

razvanculea
authored and
stankiewicz
committed
Clean up Typescript Tests workflow (#33458)

damondouglas
authored and
stankiewicz
committed
[Managed Iceberg] refactor integration tests; make it easy to add a new catalog to test suite (#33444)

ahmedabu98
authored and
stankiewicz
committed
[yaml] add mysql, oracle, postgres and sql server providers (#33124)

Polber
authored and
stankiewicz
committed
Update Python Dependencies (#33459)

authored and
stankiewicz
committed
Remove aws v1 from infrastructure (#33455)

damccorm
authored and
stankiewicz
committed
Fix typo of output type in the docs. (#33463)

damjad
authored and
stankiewicz
committed
Fix iceberg ITs (#33462)

ahmedabu98
authored and
stankiewicz
committed
Implement "stage_file_with_retry" for robust file staging (#33436)

shunping
authored and
stankiewicz
committed
Update xlang kinesis to v2 (#33416)

damccorm
authored and
stankiewicz
committed
Remove amazon-web-services and kinesis folders + settings.gradle cleanup (#33456)

damccorm
authored and
stankiewicz
committed
Don't block bot on neutral responses (#33465)

damccorm
authored and
stankiewicz
committed
Remove aws v1 dependencies (#33466)

damccorm
authored and
stankiewicz
committed
Update Java expansion service to use distroless (#33464)

damccorm
authored and
stankiewicz
committed
Fix missing worker_id from interceptor (#33453)

damondouglas
authored and
stankiewicz
committed
Make Beam expansion service shadowJar multi-release so it processes multi-release dependencies correctly (#33472)

Abacn
authored and
stankiewicz
committed
Revert "Update Java expansion service to use distroless (#33464)" (#33476)

damccorm
authored and
stankiewicz
committed
Bump github.com/aws/aws-sdk-go-v2/config from 1.28.6 to 1.28.7 in /sdks (#33440)

dependabot[bot]
authored and
stankiewicz
committed
fix error on local-env-setup.sh (#33461)

marblejenka
authored and
stankiewicz
committed
Stage beam_Publish_Python_SDK_Distroless_Snapshots.yml (#33470)

damondouglas
authored and
stankiewicz
committed
Stage beam_Publish_Java_SDK_Distroless_Snapshots.yml (#33469)

damondouglas
authored and
stankiewicz
committed
Update BEAM_DEV_SDK_CONTAINER_TAG for deps update. (#33480)

authored and
stankiewicz
committed
Add descriptive tags to republished containers (#33477)

damccorm
authored and
stankiewicz
committed
Fix bounded trie merge when merging on empty trie

rohitsinha54
authored and
stankiewicz
committed
Update gcp bom to 26.50.0 (#33192)

damccorm
authored and
stankiewicz
committed
[#32167][Prism] Use the worker-id gRPC metadata (#33438)

damondouglas
authored and
stankiewicz
committed
Better error message on incorrect package listing.

robertwb
authored and
stankiewicz
committed
[Distroless] Publish Java SDK distroless variants (#33485)

damondouglas
authored and
stankiewicz
committed
Use other fallback coders for protobuf message base class (#33432)

shunping
authored and
stankiewicz
committed
Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#33500)

dependabot[bot]
authored and
stankiewicz
committed
[java] Fix avro bytes conversion for BQ storage (#33412)

RustedBones
authored and
stankiewicz
committed
Fix republish docker branch (#33501)

damccorm
authored and
stankiewicz
committed
Adds references to KeyCommitTooLargeException docs whever we throw the exception and also mentions --experiments=throw_exceptions_on_large_output as an option. (#33380)

acrites
authored and
stankiewicz
committed
[#31912] Fix bug in finalization for Prism & fix the Java bundle finalizing tests. (#33493)

lostluck
authored and
stankiewicz
committed
Build wheels on gha runners instead of local runners (#33505)

damccorm
authored and
stankiewicz
committed
Add support for Iceberg table identifiers with special characters (#33293)

regadas
authored and
stankiewicz
committed
Update 2024.md (#33502)

liferoad
authored and
stankiewicz
committed
Adds an SdkHarnessOption that controls whether logging is redirected through the FnApi (#33418)

t2h6
authored and
stankiewicz
committed
[feat 32473] Added soft deadline logic to Spanner Change Stream IO connector. (#32474)

dedocibula
authored and
stankiewicz
committed
apply scaling factor
stankiewicz
committed