Upgrade to AWS SDK V3

[pfrank13]

Steps to reproduce

1. Using https://www.npmjs.com/package/@aws-sdk/client-s3, pass that AWS Client into https://github.com/LibertyDSNP/parquetjs/blob/main/lib/reader.js#L115
2. Notice it errors out client.getObject() is undefined

Expected behaviour

1. Ideally would work with the newer version of the AWS V3 SDK

Actual behaviour

Seems it used AWS V2 SDK

Any other comments?

Not so much a bug per se but rather a request to update this library to support AWS SDK V3

This is how we solved it in our application

// This is a hack file to support things that @dsnp/parquetjs doesn't support quite yet
import { GetObjectCommand, HeadObjectCommand, S3Client, GetObjectCommandInput } from "@aws-sdk/client-s3";
import { Readable } from "stream";
import { Blob } from "buffer";
const parquet = require("@dsnp/parquetjs");
const { ParquetReader, ParquetEnvelopeReader } = parquet;

export const openS3Reader = async (
  client: S3Client,
  params: GetObjectCommandInput,
  options?: any
): Promise<typeof ParquetReader> => {
  const fileStat = async () => {
    const headObjectResult = await client.send(new HeadObjectCommand(params));
    return headObjectResult.ContentLength;
  };

  const readFn = async (offset: number, length: number, file: string): Promise<Buffer> => {
    if (file) {
      return Promise.reject("external references are not supported");
    }
    const Range = `bytes=${offset}-${offset + length - 1}`;
    const response = await client.send(new GetObjectCommand({ ...{ Range }, ...params }));

    const body = response.Body;
    if (body) {
      return streamToBuffer(body);
    }
    return Buffer.of();
  };

  const closeFn = () => ({});

  const envelopeReader = new ParquetEnvelopeReader(readFn, closeFn, fileStat, options);

  return ParquetReader.openEnvelopeReader(envelopeReader, options);
};

async function streamToBuffer(body: any): Promise<Buffer> {
  const blob = body as Blob;
  if (blob.arrayBuffer !== undefined) {
    const arrayBuffer = await blob.arrayBuffer();
    const uint8Array: Uint8Array = new Uint8Array(arrayBuffer);
    return new Buffer(uint8Array);
  }

  //Assumed to be a Readable like object
  const readable = body as Readable;
  return await new Promise((resolve, reject) => {
    const chunks: Uint8Array[] = [];
    readable.on("data", (chunk) => chunks.push(chunk));
    readable.on("error", reject);
    readable.on("end", () => resolve(Buffer.concat(chunks)));
  });
}

Notes

• It is OK to lose support for AWS V2 when doing this update

Points: 3

[wilwade]

Might be worth looking at ZJONSSON#64