Skip to content

Documentation fixes for scio parquet testing harness #5675

@johnwalk-spotify

Description

@johnwalk-spotify

in the testing section for scio.io.Parquet There are several points where the examples appear out of date or unclear.

  1. predicate filtering: the function withPredicate appears to have been renamed to withFilter

  2. the sequencing of predicate vs projection is inverted - in the first example (avro projection) we have

records withProjection projection withPredicate predicate

however this fails to compile due to apparently interpreting withFilter as a scala collection method, leading to a type mismatch expecting a method returning a Boolean rather than a parquet FilterPredicate.

Fiipping the order to

records withFilter predicate withProjection projection

should resolve this (tested with the case-class projection approach, which requires other changes - see below)

  1. postfix operators do not work for the case-class projection method - in the second example we have
records withProjection[MyProjection]

but this fails with type application is not allowed for postfix operators. Using a normal method call records.withProjection[MyProjection] resolves this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions