Update SMSmissingROWIDs.py

[MetadataForensics]

Updated for both new artifact format and query changes. Additional information is available here: https://github.com/MetadataForensics/RowIDetective

[JamesHabben]

Great information on your repo page, and thanks for referencing the LEAPPs too!

A couple notes:

Timestamp adjustment

1. Would you consider adding a raw timestamp output column for any existing date columns to this query? We are trying to get artifacts to use our api functions for timestamp conversions for consistency in timestamp handling and output. I recognize that it is much easier to maintain these queries if you can copy/paste in all the different places it is being used, so I don't want to ask that you make changes to the query just for iLEAPP. I thought an additional raw column for any dates could be easy enough to maintain, and it would even provide a reference point for examiners to validate the data by doing the conversion calculation manually.
2. We are also working toward being able to process and show sub-second resolution on timestamps, and I note that you have a condition in the query to detect that condition and simplify. If you add a raw timestamp output column and use our conversion api functions, we can take that in and provide the sub-second data immediately when the core code is ready for it.
   see: lines 60-64 as a reference.

Test Data

Do you know of any publicly available valid test data that we can use in our test library to allow for continuous testing?

[SQLMcGee]

I submitted this pull request, this is just my other account.

Glad the repo is helpful, this has been a fun little project.

I don't think those changes will be a problem at all, I started into it but caught up with some other things. It's interesting the changes and improvements, it's been a minute since I submitted anything here - hence, all the other changes to this artifact as well.

I don't know a test dataset offhand that includes this newest change.. basically this covers the most recent sent/received iMessages being removed making a difference between the max ROWID and the sqlite_sequence number for the message table. You can recreate this scenario by adjusting the seq so it's higher and zipping the appropriate folder structure. It's a quick test in a bind.

But, again, I'll keep looking at the timestamps.

Thanks for all you do!

[MetadataForensics]

I reviewed timestamp conversions under iLEAPP/scripts/ilapfuncs.py - is there currently any other handling of timestamp values that may be 9 digit (seconds) or 18 digit (nanoseconds)? I may have missed something that you have defined. Having not seen anything in ilapfuncs.py that accounts for the possible timestamp difference and going away from the case statement I originally had the def fix_ts(val) worked here, for just this artifact. I've been working this query for some time now and Apple has adjusted timestamp recordings within the sms.db over the years so handling both possible timestamps is more universal for datasets.
Also, added in columns of just the raw message.date timestamp values should the users want to validate the data without going to the database itself.

[JamesHabben]

@Johann-PLW has done the work on those timestamp conversions and would know better

[MetadataForensics]

@JamesHabben @Johann-PLW Good afternoon and Happy December! Have you had any time to revisit this and potential approval of this Pull Request?

[JamesHabben]

after looking into this, i realize i wasnt quite clear on what i was asking for above. i was hoping to make this and future edits easier for you by not having to have 2 different versions of the query you share in the other repo. my thought was to leave the existing 'Beginning Timestamp' case block exactly as is. add a new column of 'Beginning Timestamp Raw' beside it. you can add it to your publicly shared query over in your repo and use it here in iLEAPP so it is the same query. the added benefit is examiners using your query in a sqlite tool would have the string converted / formatted date column plus the timestamp raw value column for their review. in iLEAPP, we would simply ignore the string date column when returning the data set for processing. again, the goal being a single version of the query to make it easier for you to maintain.

something like this: (cobled copy / paste, probably not working as is)

SELECT * FROM (
  SELECT * FROM (
    SELECT 
    CASE 
      WHEN length(DATE) = 18 
      THEN LAG(DATETIME(DATE/1000000000 + 978307200, 'UNIXEPOCH'),1) OVER (ORDER BY ROWID) 
      WHEN length(DATE) = 9
      THEN LAG(DATETIME(DATE + 978307200, 'UNIXEPOCH'),1) OVER (ORDER BY ROWID)
      END AS "Beginning Timestamp",
    LAG(message.date,1) OVER (ORDER BY ROWID) AS "Beginning Timestamp (RAW)",   -- added raw value
    CASE 
      WHEN length(DATE) = 18 
      THEN DATETIME(DATE/1000000000 + 978307200, 'UNIXEPOCH') 
      WHEN length(DATE) = 9
      THEN DATETIME(DATE + 978307200, 'UNIXEPOCH')	
      END  AS "Ending Timestamp",
    message.date AS "Ending Timestamp (RAW)",   -- added raw value

that said, you have done the work to convert this module over already, so it is up to you on how you would like to proceed with it. we have some thoughts on getting sub-second timestamp precision into the whole pipeline, and i recently found that we are already supporting it on the iLEAPP portion. a timestamp like this can be returned with a decimal for sub-second values, and that will get stored into the sqlite tables. next step is to get outputs adjusted to accommodate for displaying the stored data.

thoughts?

[JamesHabben]

the future output is LAVA which should be getting a general release soon. a few little things to tidy up and it will be ready. by providing the timestamp value as a raw number, we then designate it for type rendering in LAVA. it gets a little (i) icon beside it, and a pop up displays various other information about that timestamp. image below in case you havent seen it.

we have a discord server where some other module devs have joined. some of the LAVA stuff has been shared there. we have also invited a few developers to try out an alpha version of LAVA. if either of those are of interest to you.
https://discord.gg/AKamYcEQ

[MetadataForensics]

I think I fully understand now, thank you for your reply and suggestions. Text back and forth can get confusing but I believe I'm on the same page as you are now. Submitted is my desired artifact for iLEAPP, this query version will differ from the version in my repo - while I fully recognize the importance of data validation the ROWIDs are provided so examiners/investigators seeking to validate the timestamps could use the ROWIDs to review the database and obtain the raw timestamps. Sub-second timestamps will prove interesting - depending on the timestamp stored in the database (seconds vs nanoseconds) would impact whether you really got the sub-second or just .000 I suppose.