Clarification on fields and partition key usage

[findchris]

Hi there, and thanks for koupler - it is just what we were looking for.

A question on usage:

    -delimiter <arg>          delimiter between fields (default: ',')
    -partitionKeyField <arg>  zero-based index of field containing partition key (default: 0)

Might be just terminology confusion on my part, but the docs say Kinesis message data is a just a blob of data (which, in my case, will be a JSON-encoded string). What is the fields concept here?

Additionally, if I wanted to use asdf123 as my partition key, and a JSON blob as my data, would the call (over HTTP) look like:

 curl -d "asdf123,{\"key\": \"value\"}" http://192.168.99.123:4321/event

I appreciate the clarification here.

[findchris]

Get a chance to check this out @boneill42 ?

[boneill42]

sorry for the delay @findchris.

Yes, that is exactly how you could do it. The code to identify the partition key is here:
https://github.com/monetate/koupler/blob/master/src/main/java/com/monetate/koupler/KinesisEventProducer.java#L78
(really dumb -- just doing a split on the delimiter)

However -- note that the entirety of the data is sent over as the message. You can see that code here:
https://github.com/monetate/koupler/blob/master/src/main/java/com/monetate/koupler/KinesisEventProducer.java#L113-L115

It would be great if koupler could natively handle JSON and use a JSONPath to identify the partition key. Alternatively, it would be great if koupler could use the first field as the partition key and send the remainder of the input as the message.

Would love a PR? ;)

If not, you can work around it, by parsing off the partition key on the receiving end.

[findchris]

Thanks for the response.

I might be able to send over a PR. How do you envision the interface for native JSON handling? Something like:

{
   partitionKey: 'abc123',
   data: { foo: 'bar', ... }
}

As for the alternative of removing the partition key from the data, that would undoubtedly break existing implementations, right?

[boneill42]

Yes, removing the key would break existing. I was thinking you could pass it through, and remove the key on the consumer side. But, we can probably abstract away how the key and data are retrieved on a per format basis....

Something like this:
https://github.com/monetate/koupler/pull/29/files

I didn't test it yet, just a straw man. Let me know what you think.

[findchris]

Looks promising.

I made one specific comment; other than than, the only thing that came to mind is wrapping the concept of a partitionKey/data combination into a message object, as in:

class Message {
  public String partitionKey;
  public String data;
}

The Format/Parser could then pass these Messages around after a single parse.

[boneill42]

Ah, great idea. I like it.

[findchris]

If this were merged, I'd just need to create a new Format that omits the partitionKey from the Message's data then. Sound right?

[boneill42]

exactly... and could support JSON directly if you want.

[findchris]

Very cool. So, 👍 here :)

[boneill42]

merged: would love to see a PR with first-class JSON support. =)

[findchris]

Thanks!
I should be able to look into that this week.