Support Content-Encodings in http.request

[daurnimator]

Especially notable is gzip, as Transfer-Encoding is not supported in HTTP2.

Need to be mindful of interaction with things like Content-Length and ETag.

[daurnimator]

The simplest way I can think to implement this is to override the get_next_chunk function on the stream returned by http.request:go(). This option doesn't seem clean.

An alternative would be a 'filtering' framework; chunks read and written from/to a stream pass through a series of user defined filters.

Is this in scope for the lua-http project?

Hrm....

[daurnimator]

From https://tools.ietf.org/html/rfc7231#section-5.3.4

A request without an Accept-Encoding header field implies that the user agent has no preferences regarding content-codings. Although this allows the server to use any content-coding in a response, it does not imply that the user agent will be able to correctly process all encodings.

I want to support reading/writing from files in the content-encoded form.
I'm currently thinking about a stream field .decode_content_encoding or similar that turns on decoding of content-encoding. Would want a similar field for encoding/writing.

Should support at least the codings mentioned https://tools.ietf.org/html/rfc7231#section-3.1.2.1

• compress
• x-compress
• deflate
• gzip
• x-gzip

See also https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding

[Mehgugs]

bump on this?

Having a look at the code, reading content encodings inflatable by zlib.inflate could use the same codepath that transfer encoding takes. Perhaps using self.body_read_inflate as a more general function that can handle the other kinds is workable?

The issue with this is that partial or non standard support seems to be quite common, and how does the user know that the library handled the compression perhaps a stream:decode_body_as_* interface?

[daurnimator]

The issue with this is that partial or non standard support seems to be quite common

Indeed. Content-Encoding is a place where the standards (and designers) completely disagree from the implementation(s). With a confounding factor that subsequent standards (like Etags) are written to the standards, rather than reality.

perhaps a stream:decode_body_as_* interface?

Users would still probably want "read up to newline" or "read N bytes" functions. I'd prefer that these didn't proliferate too much, instead there should be some flag on a stream about whether to decode or not I think? This leads to the question of where to retain decoder state, and if the content_decode flag was turned on/off, what happens with decoder state?