Skip to content

Commit d94a4af

Browse files
rchowellrchowell
committed
Adds DatumWriter for most types
1 parent 5446de5 commit d94a4af

File tree

11 files changed

+1118
-64
lines changed

11 files changed

+1118
-64
lines changed

‎docs/wiki/v1/streams.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# PartiQL Data Streams
2+
3+
*This document defines the PSink and PSource in relation to Datum and Java primitives*
4+
5+
* * *
6+
7+
### Background
8+
9+
We have defined
10+
an [encoding of PartiQL values using the Ion data format](https://quip-amazon.com/5Su8AQhKG5xA/PartiQL-Values-in-Ion),
11+
but how does this fit in? Let’s look at two questions.
12+
13+
1. How does PartiQL represent values in memory?
14+
2. How does PartiQL read values from a stream into memory?
15+
16+
An in-memory PartiQL value has a layer of indirection between the Java primitive and its view to the rest of the
17+
program. This is called the “Datum” and is a fat interface which allows the partiql-lang-kotlin engine to not worry
18+
about a value’s Java type, and instead switch on an int tag (ptype) to then pull out a value. Effectively the fat
19+
interface removes checking classes and casting with tag checking then accessing without a cast. It’s effectively a
20+
unifying interface over the old values, so how does the variant fit in?
21+
22+
A variant is an implementation of a Datum whose value is opaque to the rest of the system. When the system checks the
23+
tag, it simply gets back “variant<T>” where T might tell us a set of capabilities (or traits) this type system / value
24+
has. This value is not lowered into a PartiQL value such as an INT or ARRAY, but is kept in its “container” or “box”.
25+
Think of the variant types of other engines or jsonb of PostgreSQL.
26+
27+
So how does PartiQL read values from a stream into Datums, and how does it handle variants? It depends because an
28+
encoding may include a data type or it may not. Also, the reader itself may expect a type (or not). Consider that a
29+
PartiQL value carries a type with it along with the value itself.
30+
31+
## Writing Data
32+
33+
### PSink
34+
35+
The PSink interface is used to write PartiQL data. It has APIs just like the IonWriter, and similarly, it has different
36+
implementations for the actual encoding like how Ion has both a text and a binary encoding. A PSink is used without any
37+
assumptions about the actual encoding.
38+
39+
### DatumWriter
40+
41+
The DatumWriter is a class which facilitates writing datums via a PSink implementation; it is handles materializing a
42+
datum and calling the appropriate sink methods.
43+
44+
**Example**
45+
46+
```kotlin
47+
val writer = DatumWriter.standard(sink)
48+
writer.write(datum1)
49+
writer.write(datum2)
50+
writer.write(datum3)
51+
writer.close()
52+
```
53+
54+
### IonSink
55+
56+
This example shows how to encode a datum as Ion; type decorations are omitted where possible.
57+
58+
```kotlin
59+
val sink = IonSink(System.out) // printing
60+
val writer = DatumWriter(sink)
61+
62+
// bool
63+
writer.write(Datum.bool(true)) // >> true
64+
65+
// ints
66+
writer.write(Datum.smallint(1)) // >> smallint::1
67+
writer.write(Datum.int(2)) // >> int::2
68+
writer.write(Datum.bigint(3)) // >> 3
69+
70+
// exact and approx numeric
71+
writer.write(Datum.decimal(BigDecimal("3.14"), 3, 2)) // >> ((decimal 3 2) 3.14)
72+
writer.write(Datum.real(3.14f)) // >> real::3.14e0
73+
writer.write(Datum.doublePrecision(3.14)) // >> 3.14e0
74+
75+
// char strings
76+
writer.write(Datum.char("abc", 3)) // >> ((char 3) "abc")
77+
writer.write(Datum.varchar("abc", 3)) // >> ((varchar 3) "abc")
78+
writer.write(Datum.string("abc")) // >> "abc"
79+
80+
// lobs
81+
writer.write(Datum.clob("hello".toByteArray()), 5) // >> {{ "hello" }}
82+
writer.write(Datum.blob("hello".toByteArray()), 5) // >> {{ aGVsbG8= }}
83+
84+
// datetime
85+
// TODO blocked on https://github.com/partiql/partiql-lang-kotlin/pull/1656
86+
87+
// ion
88+
89+
```
90+
91+
## Reading Data
92+
93+
### DatumReader
94+
95+
### PSource
96+
97+
PLACEHOLDER

‎partiql-spi/api/partiql-spi.api‎

Lines changed: 129 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -464,8 +464,67 @@ public final class org/partiql/spi/function/Routine$DefaultImpls {
464464
public static fun getParameters (Lorg/partiql/spi/function/Routine;)[Lorg/partiql/spi/function/Parameter;
465465
}
466466

467+
public abstract interface class org/partiql/spi/stream/PSink {
468+
public fun close ()V
469+
public fun finish ()V
470+
public fun flush ()V
471+
public abstract fun setType (Lorg/partiql/types/PType;)V
472+
public abstract fun stepIn (I)V
473+
public abstract fun stepOut ()V
474+
public abstract fun writeBigint (J)V
475+
public abstract fun writeBlob ([B)V
476+
public abstract fun writeBool (Z)V
477+
public abstract fun writeChar (Ljava/lang/String;)V
478+
public abstract fun writeClob ([B)V
479+
public abstract fun writeDate (Lorg/partiql/value/datetime/Date;)V
480+
public abstract fun writeDecimal (Ljava/math/BigDecimal;)V
481+
public abstract fun writeDouble (D)V
482+
public abstract fun writeField (Ljava/lang/String;)V
483+
public abstract fun writeInt (I)V
484+
public abstract fun writeMissing ()V
485+
public abstract fun writeNull ()V
486+
public abstract fun writeNumeric (Ljava/math/BigDecimal;)V
487+
public abstract fun writeReal (F)V
488+
public abstract fun writeSmallint (S)V
489+
public abstract fun writeString (Ljava/lang/String;)V
490+
public abstract fun writeTime (Lorg/partiql/value/datetime/Time;)V
491+
public abstract fun writeTimestamp (Lorg/partiql/value/datetime/Timestamp;)V
492+
public abstract fun writeTimestampz (Lorg/partiql/value/datetime/Timestamp;)V
493+
public abstract fun writeTimez (Lorg/partiql/value/datetime/Time;)V
494+
public abstract fun writeTinyint (B)V
495+
public abstract fun writeVarchar (Ljava/lang/String;)V
496+
public abstract fun writeVariant (Ljava/lang/Object;)V
497+
}
498+
499+
public abstract interface class org/partiql/spi/stream/PSource {
500+
public fun close ()V
501+
public abstract fun next ()Lorg/partiql/types/PType;
502+
public abstract fun readBigint ()J
503+
public abstract fun readBlob ()[B
504+
public abstract fun readBool ()Z
505+
public abstract fun readChar ()Ljava/lang/String;
506+
public abstract fun readClob ()[B
507+
public abstract fun readDate ()Lorg/partiql/value/datetime/Date;
508+
public abstract fun readDecimal ()Ljava/math/BigDecimal;
509+
public abstract fun readDouble ()D
510+
public abstract fun readField (Ljava/lang/String;)Ljava/lang/String;
511+
public abstract fun readInt ()I
512+
public abstract fun readReal ()F
513+
public abstract fun readSmallint ()S
514+
public abstract fun readString ()Ljava/lang/String;
515+
public abstract fun readTime ()Lorg/partiql/value/datetime/Time;
516+
public abstract fun readTimestamp ()Lorg/partiql/value/datetime/Timestamp;
517+
public abstract fun readTimestampz ()Lorg/partiql/value/datetime/Timestamp;
518+
public abstract fun readTimez ()Lorg/partiql/value/datetime/Time;
519+
public abstract fun readTinyint ()B
520+
public abstract fun readVarchar ()Ljava/lang/String;
521+
public abstract fun stepIn ()V
522+
public abstract fun stepOut ()V
523+
}
524+
467525
public abstract interface class org/partiql/spi/value/Datum : java/lang/Iterable {
468526
public static fun array (Ljava/lang/Iterable;)Lorg/partiql/spi/value/Datum;
527+
public static fun array (Ljava/lang/Iterable;Lorg/partiql/types/PType;)Lorg/partiql/spi/value/Datum;
469528
public static fun bag (Ljava/lang/Iterable;)Lorg/partiql/spi/value/Datum;
470529
public static fun bigint (J)Lorg/partiql/spi/value/Datum;
471530
public static fun blob ([B)Lorg/partiql/spi/value/Datum;
@@ -538,8 +597,10 @@ public class org/partiql/spi/value/DatumReader$Builder {
538597
public fun register (Lorg/partiql/spi/value/Encoding;Lorg/partiql/spi/value/DatumReader;)Lorg/partiql/spi/value/DatumReader$Builder;
539598
}
540599

541-
public abstract interface class org/partiql/spi/value/DatumWriter : java/lang/AutoCloseable {
542-
public abstract fun write (Lorg/partiql/spi/value/Datum;)Lorg/partiql/spi/value/DatumWriter;
600+
public final class org/partiql/spi/value/DatumWriter : java/lang/AutoCloseable {
601+
public fun <init> (Lorg/partiql/spi/stream/PSink;)V
602+
public fun close ()V
603+
public final fun write (Lorg/partiql/spi/value/Datum;)V
543604
}
544605

545606
public class org/partiql/spi/value/Encoding : org/partiql/spi/Enum {
@@ -555,6 +616,72 @@ public abstract interface class org/partiql/spi/value/Field {
555616
public static fun of (Ljava/lang/String;Lorg/partiql/spi/value/Datum;)Lorg/partiql/spi/value/Field;
556617
}
557618

619+
public final class org/partiql/spi/value/ion/IonSink : org/partiql/spi/stream/PSink {
620+
public static final field Companion Lorg/partiql/spi/value/ion/IonSink$Companion;
621+
public fun <init> (Lcom/amazon/ion/IonWriter;Ljava/util/BitSet;)V
622+
public static final fun binary (Ljava/io/OutputStream;)Lorg/partiql/spi/value/ion/IonSink;
623+
public static final fun binary (Ljava/io/OutputStream;[I)Lorg/partiql/spi/value/ion/IonSink;
624+
public fun close ()V
625+
public static final fun decorated ()Lorg/partiql/spi/value/ion/IonSink$Builder;
626+
public static final fun elided ()Lorg/partiql/spi/value/ion/IonSink$Builder;
627+
public fun finish ()V
628+
public fun flush ()V
629+
public static final fun pretty (Ljava/lang/Appendable;)Lorg/partiql/spi/value/ion/IonSink;
630+
public static final fun pretty (Ljava/lang/Appendable;[I)Lorg/partiql/spi/value/ion/IonSink;
631+
public fun setType (Lorg/partiql/types/PType;)V
632+
public static final fun standard ()Lorg/partiql/spi/value/ion/IonSink$Builder;
633+
public static final fun standard (Lcom/amazon/ion/IonWriter;)Lorg/partiql/spi/value/ion/IonSink;
634+
public fun stepIn (I)V
635+
public fun stepOut ()V
636+
public static final fun text (Ljava/lang/Appendable;)Lorg/partiql/spi/value/ion/IonSink;
637+
public static final fun text (Ljava/lang/Appendable;[I)Lorg/partiql/spi/value/ion/IonSink;
638+
public fun writeBigint (J)V
639+
public fun writeBlob ([B)V
640+
public fun writeBool (Z)V
641+
public fun writeChar (Ljava/lang/String;)V
642+
public fun writeClob ([B)V
643+
public fun writeDate (Lorg/partiql/value/datetime/Date;)V
644+
public fun writeDecimal (Ljava/math/BigDecimal;)V
645+
public fun writeDouble (D)V
646+
public fun writeField (Ljava/lang/String;)V
647+
public fun writeInt (I)V
648+
public fun writeMissing ()V
649+
public fun writeNull ()V
650+
public fun writeNumeric (Ljava/math/BigDecimal;)V
651+
public fun writeReal (F)V
652+
public fun writeSmallint (S)V
653+
public fun writeString (Ljava/lang/String;)V
654+
public fun writeTime (Lorg/partiql/value/datetime/Time;)V
655+
public fun writeTimestamp (Lorg/partiql/value/datetime/Timestamp;)V
656+
public fun writeTimestampz (Lorg/partiql/value/datetime/Timestamp;)V
657+
public fun writeTimez (Lorg/partiql/value/datetime/Time;)V
658+
public fun writeTinyint (B)V
659+
public fun writeVarchar (Ljava/lang/String;)V
660+
public fun writeVariant (Ljava/lang/Object;)V
661+
}
662+
663+
public final class org/partiql/spi/value/ion/IonSink$Builder {
664+
public final fun build (Lcom/amazon/ion/IonWriter;)Lorg/partiql/spi/value/ion/IonSink;
665+
public final fun decorate (I)Lorg/partiql/spi/value/ion/IonSink$Builder;
666+
public final fun elide (I)Lorg/partiql/spi/value/ion/IonSink$Builder;
667+
}
668+
669+
public final class org/partiql/spi/value/ion/IonSink$Companion {
670+
public final fun binary (Ljava/io/OutputStream;)Lorg/partiql/spi/value/ion/IonSink;
671+
public final fun binary (Ljava/io/OutputStream;[I)Lorg/partiql/spi/value/ion/IonSink;
672+
public static synthetic fun binary$default (Lorg/partiql/spi/value/ion/IonSink$Companion;Ljava/io/OutputStream;[IILjava/lang/Object;)Lorg/partiql/spi/value/ion/IonSink;
673+
public final fun decorated ()Lorg/partiql/spi/value/ion/IonSink$Builder;
674+
public final fun elided ()Lorg/partiql/spi/value/ion/IonSink$Builder;
675+
public final fun pretty (Ljava/lang/Appendable;)Lorg/partiql/spi/value/ion/IonSink;
676+
public final fun pretty (Ljava/lang/Appendable;[I)Lorg/partiql/spi/value/ion/IonSink;
677+
public static synthetic fun pretty$default (Lorg/partiql/spi/value/ion/IonSink$Companion;Ljava/lang/Appendable;[IILjava/lang/Object;)Lorg/partiql/spi/value/ion/IonSink;
678+
public final fun standard ()Lorg/partiql/spi/value/ion/IonSink$Builder;
679+
public final fun standard (Lcom/amazon/ion/IonWriter;)Lorg/partiql/spi/value/ion/IonSink;
680+
public final fun text (Ljava/lang/Appendable;)Lorg/partiql/spi/value/ion/IonSink;
681+
public final fun text (Ljava/lang/Appendable;[I)Lorg/partiql/spi/value/ion/IonSink;
682+
public static synthetic fun text$default (Lorg/partiql/spi/value/ion/IonSink$Companion;Ljava/lang/Appendable;[IILjava/lang/Object;)Lorg/partiql/spi/value/ion/IonSink;
683+
}
684+
558685
public abstract class org/partiql/value/BagValue : org/partiql/value/CollectionValue {
559686
public fun <init> ()V
560687
public abstract fun copy (Ljava/util/List;)Lorg/partiql/value/BagValue;

0 commit comments

Comments
 (0)