Moving text from HTML's web storage into the Storage Standard

[annevk]

As some of the advice seems a bit dated, I'm opening this issue to track and discuss what kind of changes we might be making. The relevant PRs are whatwg/html#5560 and #93.

[annevk]

User agents should not expire data from a browsing context's session storage areas, but may do so when the user requests that such data be deleted, or when the UA detects that it has limited storage space, or for security reasons. User agents should always avoid deleting data while a script that could access that data is running. When a top-level browsing context is destroyed (and therefore permanently inaccessible to the user) the data stored in its session storage areas can be discarded with it, as the API described in this specification provides no way for that data to ever be subsequently retrieved.

I think this mostly follows from the requirements we already have around UI, storage pressure, and tying storage to specific objects, except for the script running thingie.

User agents should expire data from the local storage areas only for security reasons or when requested to do so by the user. User agents should always avoid deleting data while a script that could access that data is running.

This doesn't list sufficient reasons, e.g., storage pressure, but fortunately the Storage Standard covers that already. Script running is important though.

[annevk]

Disk space

User agents should limit the total amount of space allowed for storage areas, because hostile authors could otherwise use this feature to exhaust the user's available disk space.

User agents should guard against sites storing data under their origin's other affiliated sites, e.g., storing up to the limit in a1.example.com, a2.example.com, a3.example.com, etc, circumventing the main example.com storage limit.

User agents may prompt the user when quotas are reached, allowing the user to grant a site more space. This enables sites to store many user-created documents on the user's computer, for instance.

User agents should allow users to see how much space each domain is using.

A mostly arbitrary limit of five megabytes per origin is suggested. Implementation feedback is welcome and will be used to update this suggestion in the future.

For predictability, quotas should be based on the uncompressed size of data stored.

https://www.w3.org/Bugs/Public/show_bug.cgi?id=21319#c3 is linked in the source for the last paragraph, which makes me question how much buy-in it has.

As part of Storage Standard discussions we have discussed origin vs site as well and my recollection is that in general we don't really want to put sites on a pedal and instead encourage mitigations that also work against a bad actor that has 10k to buy some registrable domains (or uses github.io or some such).

I think the other requirements are already captured by the existing text. The one exception is the 5 MiB limit. We might well want to keep that for localStorage/sessionStorage, but the current infrastructure doesn't cleanly allow for it. #69 might help with this I suppose.

[inexorabletash]

Refactoring things out of the HTML monolith? 🎉

Script running is important though.

Yep... but those two sentences are somewhat in conflict, so probably deserve more nuance. If the user clears data, it should happen even if script is running. To do so properly may require stopping script from running, so it's not wrong, just complicated.

Re: Compression - yeah, I think that ship has sailed. Chrome reports actual quota usage regardless of compression or overhead (e.g. write-ahead logs etc prior to compaction)

Re: origin vs. site - there's also the challenges of preventing cross-origin information leakage which we probably need to document in Storage at some point.

Re: 5 MiB limit - FWIW, I think Chrome is at 10MiB (@mkruisselbrink ?). Agreed that we should keep the limit for localStorage/sessionStorage. I think it would be fine for any storage endpoint to impose its own bottle size limit in addition to a bucket limit.

[domenic]

In general I find the text in HTML much clearer and more detailed than what is in Storage today. So let me try to raise some specific concerns in this issue, before we proceed with merging whatwg/html#5560.

User agents should not expire data from a browsing context's session storage areas, but may do so when the user requests that such data be deleted, or when the UA detects that it has limited storage space, or for security reasons. User agents should always avoid deleting data while a script that could access that data is running. When a top-level browsing context is destroyed (and therefore permanently inaccessible to the user) the data stored in its session storage areas can be discarded with it, as the API described in this specification provides no way for that data to ever be subsequently retrieved.

I think this mostly follows from the requirements we already have around UI, storage pressure, and tying storage to specific objects, except for the script running thingie.

I have bolded a sentence which I cannot find a counterpart in Storage (after #93).

User agents should limit the total amount of space allowed for storage areas, because hostile authors could otherwise use this feature to exhaust the user's available disk space.

User agents should guard against sites storing data under their origin's other affiliated sites, e.g., storing up to the limit in a1.example.com, a2.example.com, a3.example.com, etc, circumventing the main example.com storage limit.

User agents may prompt the user when quotas are reached, allowing the user to grant a site more space. This enables sites to store many user-created documents on the user's computer, for instance.

User agents should allow users to see how much space each domain is using.

A mostly arbitrary limit of five megabytes per origin is suggested. Implementation feedback is welcome and will be used to update this suggestion in the future.

For predictability, quotas should be based on the uncompressed size of data stored.

I think the other requirements are already captured by the existing text.

I have bolded sentences for which I cannot find counterparts in Storage (after #93). It sounds like maybe you disagree with sentence 2, and I believe Chrome does too (our most-starred bug is about guarding against this "attack", and I think we've decided to not address it). So that one perhaps should be dropped. But the others seem valuable.

[annevk]

1. This is covered by https://whatpr.org/storage/93.html#management and being much more explicit about when data is created. The same rules as elsewhere apply. Unless anything suggests it's allowed, it's not.
2. What it calls storage areas are storage bottles and they are still restricted as per Add the ability for bottles to have a quota #97. Making that advice more general would warrant more discussion. In general I like the balance between https://whatpr.org/storage/93.html#usage-and-quota and https://whatpr.org/storage/93.html#storage-pressure giving user agents more ability to let applications get hold of more storage if they need it (as well as removing things if they don't).
3. I covered this upthread.
4. I don't think we should be prompting the user and this model isn't really workable for a site that wants to store a lot of data. It gets an exception, then the user gets prompted, but how does the site know?
5. Allowing users to clear sites is covered by https://whatpr.org/storage/93.html#ui-guidelines and I think that's good enough. I'm not convinced we should make the UI requirements that specific.
6. Discussed above.

[domenic]

1. This is covered by https://whatpr.org/storage/93.html#management and being much more explicit about when data is created. The same rules as elsewhere apply. Unless anything suggests it's allowed, it's not.

I can't find anything in Storage that says user agents are allowed to clear session storage upon user request or for security reasons. I can only find storage pressure.

2. What it calls storage areas are storage bottles and they are still restricted as per Add the ability for bottles to have a quota #97. Making that advice more general would warrant more discussion. In general I like the balance between https://whatpr.org/storage/93.html#usage-and-quota and https://whatpr.org/storage/93.html#storage-pressure giving user agents more ability to let applications get hold of more storage if they need it (as well as removing things if they don't).

The sentence (2) talks about a limit on the total size consumed by all bottles. I cannot find a counterpart in Storage.

3. I covered this upthread.

Are you referring to

As part of Storage Standard discussions we have discussed origin vs site as well and my recollection is that in general we don't really want to put sites on a pedal and instead encourage mitigations that also work against a bad actor that has 10k to buy some registrable domains (or uses github.io or some such).

If so, I'd like to get general confirmation that we want to remove this restriction. As I said above, it seems like Chrome does not intend to implement the restriction, and I guess maybe you're saying Firefox doesn't either?

Additionally, if we're really moving to a world where "Unless anything suggests it's allowed, it's not", then it'd be good to ensure that nobody is doing this today, since such a mitigation would become disallowed.

4. I don't think we should be prompting the user and this model isn't really workable for a site that wants to store a lot of data. It gets an exception, then the user gets prompted, but how does the site know?

This seems reasonable, but I'd like us to note it as a normative change, since we're disallowing user agents from doing something. We should also ensure that no user agents are currently doing it.

5. Allowing users to clear sites is covered by https://whatpr.org/storage/93.html#ui-guidelines and I think that's good enough. I'm not convinced we should make the UI requirements that specific.

I'm hesitant to remove user-friendly normative text from the specification.

6. Discussed above.

Should we add the opposite statement? This seems important for interoperability... otherwise the 5 MiB quota is pretty meaningless.

[annevk]

1:

Instead user agents should offer users the ability to clear all storage for each group of schemelessly same site origins.

2:

The storage quota of a storage shelf is a conservative estimate of the total amount of bytes it can hold. This amount should be less than the total available storage space on the device to give users some wiggle room.

I guess I could add implementation-defined to that: #108.

3: Firefox currently has this restriction, but we are thinking of removing it. As quotas are implementation-defined as per 2, I don't think we would be really disallowing anything. We're just not putting specific solutions in the specification that might not be ideal.

4: Does it matter if implementers agree it's a bad model? Exhaustively testing this across all APIs feels like a waste of time if we all want to move past it.

5: How is requiring particular UI user-friendly? I'm not sure we even know what the best UI would be here in our future partitioned world.

6: Unless we define the cost of each operation exactly we're not going to get interoperability, so I'd rather have it be somewhat vague as I'm not sure we want to commit on how to serialize code units and such.