Proposal: add brotli encoding by dustdfg · Pull Request #13196 · TeamNewPipe/NewPipe

dustdfg · 2026-02-05T11:24:20Z

What is it?

Feature (user facing) ⚠️ Your PR must target the refactor branch
Meta improvement to the project (dev facing)

Description of the changes in your PR

⚠️ DISCLAIMER: ⚠️ This is a "proposal" PR to show off the idea and ask for help with further testing and evaluating. It is not like it is draft because it is rather finished prototype. I am unable to evaluate RAM, CPU and BATTERY usage in my setup

Bluntly adds Accept-Encoding: brotli gzip support to every http request for every service (so maybe it needs special handling to add it only for youtube)
Gzip support is by default added by OkHttp so it is currently used by newpipe
If brotli is somehow used for thumbs, it is probably very bad because uses cpu and doesn't reduce bandwidth consumption. Though gzip also uses cpu. But it seems like thumbs and other images doesn't use brotli. How to check? ¯\_(ツ)_/¯
PeerTube and media.ccc.de don't support even gzip
All services ignore zstd so there is no sense to add it to newpipe
Brotli is supported only for youtube
If we are going to add it (or some ephemeral implementaion which adds brotli only to youtube) we probably need to update extractor mocks too
Descriptions are small and they are already compressed by gzip. They don't profit from brotli at all

	0.28.2	brotli_test	difference	difference %
APK size¹	12494057 bytes	12573083 bytes	+ 79026 bytes	+ 0.63%
Feed update 53 channels	4.1 MB	3.7 MB	- 0.4 MB	- 9.75%
Feed update 133 channels	9.3 MB	8.3 MB	- 1.0 MB	- 10.75%
	-	-	-	-
Feed load thumbs only 1	8.4 MB	8.3 MB	- 0.1 MB	- 1.19% ²
Feed load thumbs only 2	9.6 MB	9.6 MB	0 MB	- 0% ²
	-	-	-	-
Opening video	106.72 KB	94.7 KB	12.02 KB	- 11.26%
Opening video + description	106.96 KB	95.0 KB	11.96	- 11.18%
Opening video + comments	227.42 KB	194.56 KB	32.86 KB	- 14.44%
Opening chnannel	65.48 KB	60.44 KB	5.04 KB	- 7.69%
	-	-	-	-
RAM usage	How to measure?	How to measure?	How to measure?	How to measure?
CPU usage	How to measure?	How to measure?	How to measure?	How to measure?
Battery usage	How to measure?	How to measure?	How to measure?	How to measure?
Feed update speed³	How to measure?	How to measure?	How to measure?	How to measure?

Fixes the following issue(s)

Inspired by Feed update compression #13181 (not sure if it is correct to say fixes)

APK testing

The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration on this wiki page.

Due diligence

I read the contribution guidelines.
The proposed changes follow the AI policy.
I tested the changes using an emulator or a physical device.

Showed numbers for clean 0.28.2 and 0.28.2 + brotli_test commit, dev branch will probably give other results especially taking into account recent messing with resource optimizations ↩
Probably it is just a measurement error and there is no difference ↩ ↩²
because of artificial limit we have to not be rate limited they can be the same ↩

TobiGr · 2026-02-06T10:27:40Z

just out of iterest: could you also test how much data is saved when opening a video detail page and when loading comments? Thank you.

dustdfg · 2026-02-06T10:39:20Z

just out of iterest: could you also test how much data is saved when opening a video detail page and when loading comments? Thank you.

Yeah good idea. I concentrated too much on profiling/debugging that forgot I can test other things 😅

I will test it soon. Moreover it finally clicked that I can easily remove any image data noise by using Image quality -> dont load images in tests. I think I will also test playlist loading, channel loading and search

dustdfg · 2026-02-07T17:30:28Z

@TobiGr the table is updated! Descriptions are small and they are already compressed by gzip. They don't profit from brotli at all

TobiGr · 2026-02-08T14:28:04Z

I'd be in favor of adding brotli. I would not expect any improvements for non text-based content(images, audio, video, ...).

dustdfg · 2026-02-08T15:53:15Z

I'd be in favor of adding brotli

So there comes mainly two questions:

Should I start working on implementation which adds brotli to youtube only? (It probably will be messy because I already somewhat tried). I am not sure but other services can use it as fingerprinting if we tell the brotli is available?
I we should create some new setting (yeah it is bad but as I know brotli consumes more resources which I can't check because can't test physical devices now...)

I would not expect any improvements for non text-based content(images, audio, video, ...).

I agree.

jpds · 2026-02-08T16:32:17Z

PeerTube and media.ccc.de don't support even gzip

Where do you see this? Both of these return content-encoding: gzip in Firefox's inspect window for me.

I also do not expect a well-configured webserver to ever compress non-text content, this is what Caddy does out of the box: https://caddyserver.com/docs/caddyfile/directives/encode

Stypox · 2026-02-08T17:54:14Z

Thanks for your research! Given your findings, I'd also be ok with merging this PR as is. If problems come up, it will be very easy to revert.

Should I start working on implementation which adds brotli to youtube only?

Since all browsers since quite some time already support brotli, and they include 'br' in 'Accept-Encoding', we become less fingerprintable if we also support brotli. So I think this is not needed.

I we should create some new setting

No. If it actually is a problem on some phones, I wouldn't add brotli at all, rather than add a new setting that noone will ever find out about.

Bluntly adds Acept-Encoding: brotli gzip support to every http request for every service

I guess you meant 'br' right? https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Encoding

dustdfg · 2026-02-08T22:19:31Z

Thanks for your research! Given your findings, I'd also be ok with merging this PR as is. If problems come up, it will be very easy to revert.

Wait a bit, I finally got an opportunity to test/profile on real device. I hope api 26 is enough because google isn't certain by itself if I need api 26 or 29.

Should I start working on implementation which adds brotli to youtube only?

Since all browsers since quite some time already support brotli, and they include 'br' in 'Accept-Encoding', we become less fingerprintable if we also support brotli. So I think this is not needed.

I mean that gzip is common br zstd gzip and zstd b gzip too. zstd gzip is also "widely" supported. But br gzip is the less common from all possible...

I we should create some new setting

No. If it actually is a problem on some phones, I wouldn't add brotli at all, rather than add a new setting that noone will ever find out about.

We support old devices (android 5.0) too. Just imagine how additional 20MB of ram would look for 3-4GB device with android 10 vs 20MB of ram for device with android 5.0 and only 1GB of ram. And you know newer androids have better ART compilation strategies which improves performance (if it uses native .so library it isn't very applicable but I didn't check) so android 5.0 devices can suffer while all new feel normal. But I hope I will be able to profile it with device I got access to and give some results

I guess you meant 'br' right? https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Encoding

Yeah

dustdfg · 2026-02-08T22:27:08Z

Where do you see this? Both of these return content-encoding: gzip in Firefox's inspect window for me.

By doing something like

OkHttpClient client = new OkHttpClient.Builder()
    .addNetworkInterceptor(new SomeDefaultNetworkInterceptor(SHOW_HEADERS))
    .build();

And then looking in the logcat output. It shows what our app actually sends and receives not what is shown in browser. I also just tried to look on it via firefox.

Peertube really compresses with gzip but only site part not api part peertube.instance/api/*. We parse api part. In general it probably should be regulated by instance settings
media.ccc.de doesn't compress at all, firefox showed no compression ¯\_(ツ)_/¯

dustdfg · 2026-02-09T10:12:31Z

Thanks for your research! Given your findings, I'd also be ok with merging this PR as is. If problems come up, it will be very easy to revert.
Wait a bit, I finally got an opportunity to test/profile on real device. I hope api 26 is enough because google isn't certain by itself if I need api 26 or 29.

I was wrong. The device I got access to can't be used for profiling so everything returns to start position. I can't really do any further testing so it can be merged if you want

Stypox · 2026-02-09T13:39:38Z

I mean that gzip is common br zstd gzip and zstd b gzip too. zstd gzip is also "widely" supported. But br gzip is the less common from all possible...

I don't understand. Firefox desktop, Chromium desktop, Firefox mobile all send Accept-Encoding: gzip, deflate, br, zstd for me. So not sending br is actually more fingerprintable.

We support old devices (android 5.0) too

Yeah, that's why I'm saying, if it turns out to be too slow on those devices, let's not add brotli at all. Adding an option for a technical detail nobody knows about would not make sense.

I was wrong. The device I got access to can't be used for profiling so everything returns to start position. I can't really do any further testing so it can be merged if you want

I have a Fairphone 3 which is a pretty slow device. If you share some instructions I can test myself.

dustdfg · 2026-02-09T16:28:25Z

I don't understand. Firefox desktop, Chromium desktop, Firefox mobile all send Accept-Encoding: gzip, deflate, br, zstd for me. So not sending br is actually more fingerprintable.

You look on it from perspective of separate independent bool toggles? I look on them from perspective of whole coupled string as identifier because even order matters here

Yeah, that's why I'm saying, if it turns out to be too slow on those devices, let's not add brotli at all. Adding an option for a technical detail nobody knows about would not make sense.

There is plan B. Refactor branch which now targets minimum api 23 (android 6). Not big difference but who is going to use refactor on android 6? Taking into account it was unavailable for downloads for a very long time

I have a Fairphone 3 which is a pretty slow device. If you share some instructions I can test myself.

I've never did by myself except castrated experience with waydroid so expect my instruction to be a dead end

Requirements:

You probably need google play. I am not quite sure probably it will work without... They say that you need api 26 and then say you need api 29 and then again 26. And I could run very castrated and not functional version of profiling with waydroid which doesn't have google play so there is no other way then check ¯\_(ツ)_/¯
Some hardwork for your phone. You need to export backup with lots of subscription 50+ ideally. But the more the best
Enthusiasm

Setup:

https://developer.android.com/studio/profile
https://developer.android.com/studio/profile/inspect-app-live
It should work right away with just adb but maybe you will need your shamanic drum

Steps:

Stop other apps as much as possible to reduce noise
Run dev build on your device.
You probably need only this https://developer.android.com/studio/profile/inspect-app-live or first option from the menu (but on your place I would play with other too)
Import the dataset with subscriptions
Go to the tab of subscriptions feed update
Wait till app gives you something like flatline on cpu/ram usage (not sure if it is possible but I hope you understood what I mean). If you don't disable in settings image loading, I would scroll the feed to preload all the images (after import they aren't in cache)
Run subscription update
See values on the chart, save values (I don't know how, I was going to just use pen and max/min/average values 🤪)
Repeat all above for brotli version
Compare...

Offtopic: my xiaomi redmi 8 (which seems to belong to the same performance category) was really slow but after installing lineage os 23 (android 16) it is quite fast. Though when I firstly installed lineage 20/21 it was a downgrade in at least half of use cases but now it is like new.

dustdfg · 2026-02-17T15:08:45Z

Samsung Galaxy A02S

OS: vanilla android 12
RAM: 3GB
CPU: Qualcomm Snapdragon 450
- Cores: Cortex A53 x8
- Cache:
  - L1d: 32KB x8
  - L1i: 32KB x8
  - L2: 512KB + 1MB

Benchmark action: update of 167 subscriptions

Screenshots:

Don't ask me which one is brotli. I don't remeber... They both show peak usage as 155MB. Both have roughly 60% (of the app) peak CPU usage

RAM decryption color keys:

Why I said roughly about CPU usage? There is no exact numbers. Thanks android studio. And that's what I got for profiling: android studio which lags and can't even show exact numbers. And even shows visual lags like on first screenshot in ram section (grey color) :/ Android settings tell that peak usage of the ram of the app is 780MB :/

About the gaps. They happened in both runs at the same numbers of already updated channels. But the PR about shuffling subscriptions is already merged so it is not specific channels. Is it throttling? Maybe but it wouldn't be so synchronous? Idk doesn't look like artificial channel limit update we have ¯\_(ツ)_/¯

Oh and one more thing. For these benches I was for some reason forced for tests increase minSdk from 21 to 29 which could probably allow for some better compile time optimization ¯\_(ツ)_/¯

My conclusion: These benches are crap. Maybe they even show that brotli version doesn't have difference from normal but I don't believe these benches at all and android studio too so you can think these benches don't exist

dustdfg · 2026-02-18T22:52:27Z

Huawei MediaPad AGS2

OS: vanilla android 8
RAM: 4GB
CPU: HiSiliconKirin 650
- Cores:
  - Cortex A53 2Gz x4
  - Cortex A53 1.7Gz x4
- Cache:
  - L1d: 32KB x8
  - L1i: 32KB x8
  - L2: 512KB

Benchmark action: update of 119 subscriptions

Screenshots:

Dev:

Brotli:

Peak RAM usage is roughly 256MB on both. CPU peaks roughly 50-60% on both without much difference between them

dustdfg · 2026-02-18T23:05:01Z

Conclusion:

RAM is generally not affected at all
CPU: well it is hard to say especially because it is in percentage values which would be much higher on low end devices with arm32 and two cores.

But do we need to be so stupid to lock (all) users from this feature? Especially if benches are correct and impact on modern devices is negligible? We can just add check if api >= 26 (or whatever) and enable it for those who certainly won't be impacted?

For us it doesn't really matter amount of cores available for us. We need only physical cores count as we use it for filtering weak devices. _SC_NPROCESSORS_CONF - amount of physical cores (android devices doesn't have extrnal cpu cores hotplug). **Used** _SC_NPROCESSORS_ONLN - amount of cores available for OS right now. Often smaller than _SC_NPROCESSORS_CONF because OS can deactivate cores for powersave Runtime.getRuntime().availableProcessors() - a core count available to jvm. Has the same limitations as _SC_NPROCESSORS_ONLN but also can be decreased because some cores are allocated to other more priority processes (?)

dustdfg · 2026-02-26T09:05:26Z

I added check based on how many cores processor have (>2) to filter out very weak devices (I am not sure if they are really gonna struggle with brotli but just want to play safely)

(Only) Checking if arch is arm32 is wrong solution because "Samsung Galaxy A02S" (on which I did one of the benchmarks) is arm32 and it didn't struggle

github-actions bot added the size/small PRs with less than 50 changed lines label Feb 5, 2026

TobiGr added the performance Related to performance and optimisation; affects the speed or memory efficiency of the application label Feb 8, 2026

TobiGr approved these changes Feb 18, 2026

View reviewed changes

dustdfg added 2 commits February 26, 2026 10:40

Add brotli support to downloader

985b302

dustdfg force-pushed the brotli_test branch from a267c03 to facf576 Compare February 26, 2026 09:03

Uh oh!

Conversation

dustdfg commented Feb 5, 2026 • edited by absurdlylongusername Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is it?

Description of the changes in your PR

Fixes the following issue(s)

APK testing

Due diligence

Footnotes

Uh oh!

TobiGr commented Feb 6, 2026

Uh oh!

dustdfg commented Feb 6, 2026

Uh oh!

dustdfg commented Feb 7, 2026

Uh oh!

TobiGr commented Feb 8, 2026

Uh oh!

dustdfg commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpds commented Feb 8, 2026

Uh oh!

Stypox commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustdfg commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustdfg commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustdfg commented Feb 9, 2026

Uh oh!

Stypox commented Feb 9, 2026

Uh oh!

dustdfg commented Feb 9, 2026

Uh oh!

dustdfg commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustdfg commented Feb 18, 2026

Uh oh!

dustdfg commented Feb 18, 2026

Uh oh!

dustdfg commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dustdfg commented Feb 5, 2026 •

edited by absurdlylongusername

Loading

dustdfg commented Feb 8, 2026 •

edited

Loading

Stypox commented Feb 8, 2026 •

edited

Loading

dustdfg commented Feb 8, 2026 •

edited

Loading

dustdfg commented Feb 8, 2026 •

edited

Loading

dustdfg commented Feb 17, 2026 •

edited

Loading

dustdfg commented Feb 26, 2026 •

edited

Loading