Ability to download Modality agnostic files without raw data

[andreifoldes]

Hello,

I don't know how common this is, but in my current usage of ezBIDS I upload my data in batches and this has the unintended consequence of having to manually stich together the "modality agnostic files" (dataset_description.json, finalized.json, participants.json, participants.tsv).

I think it would be nifty is users could optionally download only these files without "having" to download the whole BIDS folder, similar to the possibility to download the "subject mapping" json at the "finalize" stage.

[dlevitas]

Hi @andreifoldes

Thanks for bringing this up, I would say you're not alone in having this issue. A good example of this problem is when users upload additional data that they acquire during their data collection. In this scenario, users have to update the participants.tsv files with the new subject(s)' information when they download their BIDS-compliant data. When data is pushed to brainlife and/or OpenNeuro, this problem is actually addressed, because the agnostic files are updated based on the newly uploaded data. So a solution would be to store an ongoing dataset as a project on brainlife. If however you'd prefer to download the data back to your computer/server, we'd have to look into some kind of merge function to update the agnostic files accordingly.

[andreifoldes]

Thanks I will try out brainlife for this purpose!
For cases where users insist on downloading directly back: while being able to separately download the agnostic files doesn't help with the merging issue it would decrease the load on your servers that results from downloading the whole dataset :)

[andreifoldes]

When data is pushed to brainlife and/or OpenNeuro, this problem is actually addressed, because the agnostic files are updated based on the newly uploaded data.

When I tried this "Send to brainlife > Import into existing project" I notice that the importing overwrite the previous files.. am I missing something?

[dlevitas]

@soichih do you know why this might be?

[soichih]

@andreifoldes Can you explain what you mean by "importing overwrite the previous files". On brainlife, all objects are immutable, i.e, you can not update the object once it's created - apart from its metadata.

[andreifoldes]

right right. I was referring to what @dlevitas said

When data is pushed to brainlife and/or OpenNeuro, this problem is actually addressed, because the agnostic files are updated based on the newly uploaded data. So a solution would be to store an ongoing dataset as a project on brainlife.

I was attempting this: creating a brainlife project uploading some data and then later uploading more additional data in hopes of procuring updated model agnostic files, but using the "Send to brainlife" feature on EZBids just resulted in me overwriting my previously stored data with the contents of the later batch.

[soichih]

@dlevitas reminded me that the information that's overwritten is the participants info. That is correct, the participants info is re-uploaded every time ezBIDS sends the data to brainlife.

So what we need to do is to update the brainlife's participants upload API so that it will only upsert information for subjects that are listed in the newly uploaded participants info. I will open a new ticket for this issue under brainlife/brainlife.