Standardize Mongo document timestamps for all collections

[jimsiff]

Currently the Dex BG data in Mongo has a date string format of "dateString": "06/28/2014 09:58:52 AM", using the PWD's local TZ. The new devicestatus and treatments collections used by the dev branch have a date field format of "$date": "2014-09-07T20:25:07.796Z" and the time appears to be UTC.

In order to maintain data consistency, it might be a good idea to choose one format for all collections. It would make it easier to correlate data when troubleshooting problems with Mongo uploads.

I'm currently using an uploader built from WIP/devicestatus.

[scottleibrand]

When DIYPS writes to the treatment collection, I have found it useful to include a date field (Unix timestamp) as well.

Scott

On Sep 10, 2014, at 1:00 AM, Jim Sifferle [email protected] wrote:

Currently the Dex BG data in Mongo has a date string format of "dateString": "06/28/2014 09:58:52 AM", using the PWD's local TZ. The new devicestatus and treatments collections used by the dev branch have a date field format of "$date": "2014-09-07T20:25:07.796Z" and the time appears to be UTC.

In order to maintain data consistency, it might be a good idea to choose one format for all collections. It would make it easier to correlate data when troubleshooting problems with Mongo uploads.

—
Reply to this email directly or view it on GitHub.

[jasoncalabrese]

Do we want to store date as Date objects of ISO8601 strings? I'm thinking real Dates would be best since they are easier to query. We should also store the source timezone.

[jimsiff]

Would it also be possible to store a 'time received/uploaded' time stamp similar to how syslog can insert a timestamp on received log messages? It would provide a metric that could help detect time drift from Dexcom. Maybe that could be used to reset Android system time? It would reduce time drift, but can introduce another problem if Dex time/date is off by days or years as is sometimes the case.

[jimsiff]

Real dates based on the user's TZ would be nice for troubleshooting assuming it doesn't change when the phone jumps to a tower in an adjacent TZ. Keeping log timestamps sequential is nice.

[scottleibrand]

I don't care about the fancy-formatted date strings: I just want to be able
to easily sort based on Unix timestamp.

On Wed, Sep 10, 2014 at 10:47 AM, Jim Sifferle [email protected]
wrote:

Real dates based on the user's TZ would be nice for troubleshooting
assuming it doesn't change when the phone jumps to a tower in an adjacent
TZ. Keeping log timestamps sequential is nice.

—
Reply to this email directly or view it on GitHub
#31 (comment)
.

[jasoncalabrese]

Unix timestamps are hard to debug, and having a duplicate dateString is a hack. Mongo has good support for Dates sorting and lots more. If we're using the data outside of mongo converting to a timestamp is easy. With 8601, the timestamp is included over the wire, so there's no guessing.

@bewest has a gist with some proposals related to this

[ktind]

Revisiting this issue. It seems that the CRM stores the date as an ISO8601 string for device status while the uploader stores it as a Date object when using mongo.

Was anything decided? This should be an easy fix - we just need to pick one and go with it 😃

[jasoncalabrese]

For the devicestatus api I copied what was in place for settings, for mongo the date was passed. Didn't realize the format was different till later. I want to change the date format everywhere.