Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTA download and decompress on the fly #43

Merged

Conversation

andreagilardoni
Copy link
Contributor

@andreagilardoni andreagilardoni commented Dec 5, 2023

This PR aims to provide a easy to use utility class LZSSDecoder that can be used to perform an OTA update on the fly.
Since in the mbed core we provide apis for http download using callbacks it was required to adapt the code for the decode function.

Some examples for the usage of LZSSDecoder were added and they made it possible to measure the performance improvements obtained from a sequential download and decompress to a download and decompress on the fly:

Starting download to QSPI ...
819600 bytes stored.

download elapsed 31.07s speed: 25.76KBps
decompress elapsed 82.13s size 1048576

Starting download & decompress on the fly
downloaded 819600 bytes 1048576 bytes stored.
download elapsed 39.27s speed: 20.38KBps

This way the OTA download and decompress operations now take 35% of the time.

This PR requires arduino/ArduinoCore-mbed#785

@CLAassistant
Copy link

CLAassistant commented Dec 5, 2023

CLA assistant check
All committers have signed the CLA.

@per1234 per1234 added type: enhancement Proposed improvement topic: code Related to content of the project itself labels Dec 5, 2023
@andreagilardoni andreagilardoni force-pushed the lzss-streaming branch 5 times, most recently from cdfa12a to 085768d Compare January 15, 2024 16:28
@pennam pennam self-requested a review January 16, 2024 08:24
Copy link
Contributor

@pennam pennam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreagilardoni i've tested the code and it works. Just a couple of minor things that i think can be improved:

  • commit history can be improved a little bit more. All subsequentials changes to the LZSSDecoder can be squashed in a single commit.
  • Alway put a space after if for ... to be consistent to the code style

} ota_progress;

int bytes = socket->download(url, is_https, [&decoder, &ota_header, &ota_progress](const char* buffer, uint32_t size) {
for(char* cursor=(char*)buffer; cursor<buffer+size; ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can directly declare cursor as uint8_t* and also cast buffer to uint8_t* this should avoid cast in line 177

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The correct approach will be to define buffer as char* const, but this requires lots of changes in the core code. Then cast cursor to uint8_t* in decompress call

examples/LZSS/LZSS.ino Outdated Show resolved Hide resolved
src/decompress/utility.cpp Outdated Show resolved Hide resolved
@andreagilardoni andreagilardoni force-pushed the lzss-streaming branch 2 times, most recently from 34304da to 394c825 Compare January 16, 2024 12:12
Comment on lines +85 to +91
enum FSM_STATES: uint8_t {
FSM_0 = 0,
FSM_1 = 1,
FSM_2 = 2,
FSM_3 = 3,
FSM_EOF
} state;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreagilardoni i would give a more meaningful name to this FSM states if it possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no meaningful names for those states

@pennam
Copy link
Contributor

pennam commented Feb 5, 2024

@andreagilardoni CI is failing because is using an old deprecated mbed core for portenta. See

@andreagilardoni andreagilardoni force-pushed the lzss-streaming branch 2 times, most recently from eb58243 to 6038ad6 Compare February 5, 2024 16:00
@arduino-libraries arduino-libraries deleted a comment from github-actions bot Feb 5, 2024
@arduino-libraries arduino-libraries deleted a comment from github-actions bot Feb 5, 2024
@andreagilardoni andreagilardoni force-pushed the lzss-streaming branch 3 times, most recently from e7a8d74 to 9c13139 Compare February 13, 2024 09:20
Copy link

Memory usage change @ f3db4bc

Board flash % RAM for global variables %
arduino:mbed_giga:giga 🔺 0 - +584 0.0 - +0.03 💚 -4224 - 0 -0.81 - 0.0
arduino:mbed_nicla:nicla_vision 🔺 0 - +520 0.0 - +0.03 💚 -4264 - 0 -0.81 - 0.0
arduino:mbed_opta:opta 🔺 0 - +528 0.0 - +0.03 💚 -4256 - 0 -0.81 - 0.0
arduino:mbed_portenta:envie_m7 N/A N/A N/A N/A
Click for full report table
Board examples/OTA_Qspi_Flash
flash
% examples/OTA_Qspi_Flash
RAM for global variables
% examples/OTA_Usage_Portenta
flash
% examples/OTA_Usage_Portenta
RAM for global variables
% examples/LZSS
flash
% examples/LZSS
RAM for global variables
% examples/OTA_Qspi_Flash_download_onthefly
flash
% examples/OTA_Qspi_Flash_download_onthefly
RAM for global variables
% examples/OTA_Qspi_Flash_Ethernet
flash
% examples/OTA_Qspi_Flash_Ethernet
RAM for global variables
% examples/OTA_SD_Portenta
flash
% examples/OTA_SD_Portenta
RAM for global variables
%
arduino:mbed_giga:giga 584 0.03 -4224 -0.81 0 0.0 0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A
arduino:mbed_nicla:nicla_vision 520 0.03 -4264 -0.81 0 0.0 0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A
arduino:mbed_opta:opta 520 0.03 -4256 -0.81 0 0.0 0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A 528 0.03 -4256 -0.81
arduino:mbed_portenta:envie_m7 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
Click for full report CSV
Board,examples/OTA_Qspi_Flash<br>flash,%,examples/OTA_Qspi_Flash<br>RAM for global variables,%,examples/OTA_Usage_Portenta<br>flash,%,examples/OTA_Usage_Portenta<br>RAM for global variables,%,examples/LZSS<br>flash,%,examples/LZSS<br>RAM for global variables,%,examples/OTA_Qspi_Flash_download_onthefly<br>flash,%,examples/OTA_Qspi_Flash_download_onthefly<br>RAM for global variables,%,examples/OTA_Qspi_Flash_Ethernet<br>flash,%,examples/OTA_Qspi_Flash_Ethernet<br>RAM for global variables,%,examples/OTA_SD_Portenta<br>flash,%,examples/OTA_SD_Portenta<br>RAM for global variables,%
arduino:mbed_giga:giga,584,0.03,-4224,-0.81,0,0.0,0,0.0,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
arduino:mbed_nicla:nicla_vision,520,0.03,-4264,-0.81,0,0.0,0,0.0,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A
arduino:mbed_opta:opta,520,0.03,-4256,-0.81,0,0.0,0,0.0,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,528,0.03,-4256,-0.81
arduino:mbed_portenta:envie_m7,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A

Copy link
Contributor

@pennam pennam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💪 Thanks @andreagilardoni

@arduino-libraries arduino-libraries deleted a comment from github-actions bot Feb 13, 2024
@arduino-libraries arduino-libraries deleted a comment from github-actions bot Feb 13, 2024
@arduino-libraries arduino-libraries deleted a comment from github-actions bot Feb 13, 2024
@andreagilardoni andreagilardoni merged commit 619d5a9 into arduino-libraries:main Feb 13, 2024
7 checks passed
@andreagilardoni andreagilardoni deleted the lzss-streaming branch February 13, 2024 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: code Related to content of the project itself type: enhancement Proposed improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants