Recording raw AIS

[hughfe]

An AIS partner recently wanted details of all movements of a few of their vessels for the last year. We don't record any data, we clean it and pass it on to major aggregators, who collate it and filter it before sharing. This led me to consider the technicalities of recording raw AIS data, in case someone wanted it later. This is unlikely to be commercially attractive, but it might be worthwhile for a port operator.
So, are there any standard formats for recording AIS data? Obviously it needs a time stamp added and needs to be in a format that can be replayed somehow. Initially I have experimented with this format: https://opencpn-manuals.github.io/main/vdr/log_format.html although the documentation doesn't quite match what is created and read by OpenCPN. Once the format is corrected it can be replayed using OpenCPN.
Most other authorities (including AIS-catcher) refer to using a json format. Is there a json format for recording AIS data with timestamps? And if so, how can it be replayed? You mention a binary format for transmitting NMEA data elsewhere.

[jvde-github]

I guess that whatever openCPN does tends to become the de facto standard. Wrt AIS-catcher, you can write the stream to a file in different formats:

-f output.txt MSGFORMAT json_nmea  (bit verbose)
-f output.txt MSGFORMAT binary_nmea. (most compact, but not human readable)
-f output.txt MSGFORMAT json_full (decomposes the messages, but this can be done a posteriori, not a good format for backup)
-f output.txt MSGFORMAT nmea  (misses timestamp)

Only the first three contain timestamp and all of them can be read back into the program with -r txt output.txt. The alternative is that you use the Postgres option in the program (but this is one way) and I would not go that route.

Not sure if there ever will be a standard as it depends very much on how you want to access the data again. I have not seen much on this. For a generic backup, '-f' and some file rotation with zipping might be a decent option. For advanced historical analysis I guess you need a proper geospatial-temporal database.

[hughfe]

I'm not entirely clear about the use-case for reading files into AIS-catcher. I can see how a piped input could be useful, but AIS-catcher seems to be designed around real-time operation. A file could be read, processed and passed on but it seems a bit over engineered for just transferring a replay file like that. I might be missing something but I can't see how to visualise a replay using AIS-catcher. Horses for courses - not a feature request, I think I'll carry on with the assumption that openCPN is the best tool for visualising a replay.

[jvde-github]

The function to read from file is added to read from devices like hardware receivers that can provide the output as a file in Linux. The second use case is for research and testing for which I use it quite often.

You are right and you cannot visualize a replay. The program is not intended as a replacement for openCPN but as a AIS receiver and I think I prefer to keep it that way to keep it clean and comprehensible.

[hughfe]

Totally agree with you.

[hughfe]

Thank for the information. I'll look at those formats, and the option to playback through AIS-catcher. My thinking is to do the minimum amount of processing when recording the data, because 1) probably no one will look at it anyway and 2) without knowing the playback method there is not much point.
All of the PPU & ECDIS software can replay their own recordings, but either the formats are different or they add extra sentences with timestamps.

openCPN might be satisfactory as the de facto standard, and the format is pretty straightforward.
Update - the VDR plugin is recording localtime, but marking it as 'Z', which might make interpreting playback tricky.

[Dirk--]

NMEA 0183 v4.0 format with TAG block seems to be standard, see:

• https://servicedocs-sm.kpler.com/tcp-stream-v2/the-nmea-message-encoding-format/
• https://github.com/M0r13n/pyais/blob/master/README.md#tag-block
• https://rietman.wordpress.com/2016/09/17/nemastudio-now-supports-the-nmea-0183-tag-block/

[jvde-github]

Interesting, added that. Note that to retain the timestamp at the input you need to add -go stamp off. And to provide output in this format use -o 7 for screen or -u xxxx xx msgformat nmea_tag on

[hughfe]

Well spotted. If the NMEA standards were put in the public domain they wouldn't have to spend so much money on lawyers.