File create fails on flash encrypted partition

[gw8484]

I am testing LittleFS 1.16.0 with Arduino 2.0.14/ESP-IDF 4.4.6 on a ESP32-C3 flash encrypted partition. It seems to work fine for file and directory read operations. However simple file create operation:

File dstFile = LittleFS.open("/copy.txt", FILE_WRITE);

always fails with error result like:

[E][vfs_api.cpp:332] VFSFileImpl(): fopen(/ffs/copy.txt) failed

Is this expected behavior? I have not being able to any concrete documentation on the topic. Is there a way to make file create and write operations work on encrypted partition? Any help would be appreciated.

[BrianPugh]

Can you post a full log? And maybe some code? I haven't tested the flash-encryption settings in a while, but esp_littlefs shouldn't care whether or not the partition is encrypted, as thats abstracted by higher layers in the esp-idf stack.

EDIT: more info: esp_littlefs uses esp_partition_read and esp_partition_write, both of which automatically decrypt/encrypt data on partitions marked as encrypted.

[gw8484]

Thanks for confirming that it should work. Is there any example code of writing file to encrypted partition?

Unfortunately, there is no other message besides the fopen() error. The code is really simple:

char fileData[8192];

extern const char *ESP_LITTLEFS_TAG;

void setup() {

  Serial.begin(115200);


  LittleFS.begin(false, "/ffs", 10U, "ffs1");

  File srcFile = LittleFS.open("/anthem.txt");

  fileData[0] = '\0';
  srcFile.readBytes(fileData, srcFile.size());
  srcFile.close();

  Serial.print("anthem.txt:\n");
  Serial.print(fileData);
  Serial.print("\n");

  File dstFile = LittleFS.open("/copy.txt", FILE_WRITE);
  dstFile.print(fileData);
  dstFile.close();
}

Is there a way to confirm the fopen() call is even reaching the esp_littlefs code? And more detailed error info?

Thanks

[BrianPugh]

are you sure the file exists? How are you getting the file on to the device? If you are flashing a pre-made partition, are you encrypting-on-flash?

[gw8484]

I am sure the source file exists. Plain file system image is uploaded before encryption. I was to find a way to get more messages.

Messges showing no problem reading the source file:

[20146.16h.58m.35s.793] I (1850) esp_littlefs: Opening /file.txt
[20146.16h.58m.35s.801] I (1850) esp_littlefs: Done opening /file.txt
[20146.16h.58m.35s.821] file.txt(232):
[20146.16h.58m.35s.826] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
[20146.16h.58m.35s.828] 

And messages showing the open for write errors:

[20146.16h.58m.35s.828] 
                        [20146.16h.58m.35s.846] I (1852) esp_littlefs: Opening /copy.txt
[20146.16h.58m.36s.092] W (1858) esp_littlefs: components/esp_littlefs/src/littlefs/lfs.c:2108:warn: Superblock 0x0 has become unwritable
[20146.16h.58m.36s.093] 
[20146.16h.58m.36s.095] E (1858) esp_littlefs: Failed to open file /copy.txt. Error  (-28)
[20146.16h.58m.36s.097] [  1433][E][vfs_api.cpp:332] VFSFileImpl(): fopen(/ffs/copy.txt) failed
[20146.16h.58m.36s.100] File open for write failed: No space left on device (errno=28)

There is plenty of space on the file system (>90% free). So, it seems the likely root issue is "Superblock 0x0 has become unwritable". Any idea how encryption could cause this?

Thanks

[BrianPugh]

Are you trying to encrypt the data partition, or just your firmware?

Possible things:

• Is your partition marked as encrypted in your partition table?
• Are you using encrypted-flash?
• If you get your partition metadata, is the esp_partition_t.encrypted flag true?

Basically, the Superblock 0x0 has become unwritable warning basically means that its reading garbage data from the partition, which could easily come from:

• reading encrypted data on a partition not flagged as encrypted.
• reading plaintext data on a partition flagged as encrypted.

[gw8484]

Thanks for the info. I think I've found the likely issue in esp_littlefs but before explaining I would like to address your questions first.

Encryption is enabled for firmware and file system partitions and they were encrypted successfully (raw flash content no longer show plaintext data after encryption and successfully decrypted in app code). Regarding your questions:

Is your partition marked as encrypted in your partition table?

Yes.

Are you using encrypted-flash?

No, not in this test. However, I have separately tested reflashing with pre-encrypted data and it works.

If you get your partition metadata, is the esp_partition_t.encrypted flag true?

I just checked and yes the flag is set to true.

Now back to the likely issue, based on much testing and a quick review of the source code, I can see the esp_littlefs partition write (littlefs_esp_part_write) only calls esp_partition_write(). This call alone is not enough for successful write to encrypted flash on esp32 in general. In general, the following are required to ensure successful esp_partition_write() to encrypted flash:

1. esp_partition_erase_range() must be called before write to the target range. On esp32c3, the erase offset and size must be multiples of SPI_FLASH_SEC_SIZE=4096. This is required for encrypted and unencrypted flash.
2. The esp_partition_write offset and size must be multiples of 16. This is required for encrypted flash.

Looking at the littlefs code I don't see the above requirements being checked and met? If the requirements are being met then please point out where in the source code. If the requirements are not being met then I would appreciate help on enhancements (I see lots use of caches in many places. So not sure where to best make the necessary enhancements without causing other problems).

Thanks

[BrianPugh]

I think you're making all excellent points, and now I'm questioning the functionality. I'll need to go and refresh myself on espressif's encryption docs.

If the requirements are not being met then I would appreciate help on enhancements

Yup! I'll try and respond again when i have a better understanding of what actions we can take.

[smjothen]

@gw8484 This component only provides the interface for reading/writing/erasing partitions. It is up to the lower LittleFS layer to know that it has to erase before writing, and as far as I can see that is the case. I have not had any problems running with encryption. I have only tested on an ESP32, however, so I'm not sure if there are any differences with the ESP32-C3.

[BrianPugh]

Naively speaking, littlefs works by erasing blocks, and then gradually writing to the bits of the erased block (imagine a bitwise tally in some cases). My worry is that this doesn't actually work properly with encryption, since encryption takes those tallys and requires the whole 16-bytes to be erased/rewritten to something totally different.

Additionally, according to the esp-idf docs:

If writing to an encrypted partition, all write offsets and lengths must be multiples of 16 bytes.
Nowhere in esp_littlefs is this enforce. I'm worried that the current "working" esp_littlefs with encryption is just on the surface, and with heavy use may not actually work as expected.

I suppose my point is, please tread carefully with the encryption! And please report back if things are/are-not working as expected.

[gw8484]

Yes, the erase issue is not that LittleFS is not erasing before writing per se but it's not specifically checking for erasing with 4k-byte alignment blocks on esp32. Having said that, the current erase operation appears to work. The more serious issue is the lack of 16-byte alignment in write operations required for encrypted partitions. I observed at least 8-byte sized writes in the superblock update code (as part of the file creation process) so there is definitely violation of the 16-byte alignment requirement and likely the cause of corruption.

[smjothen]

@gw8484 Then it sounds like you don't have CONFIG_SPI_FLASH_ENABLE_ENCRYPTED_READ_WRITE set in ESP-IDF, because that uses esp_flash_write_encrypted which checks that both the write is 16 byte aligned and the size is a multiple of 16.

[smjothen]

Naively speaking, littlefs works by erasing blocks, and then gradually writing to the bits of the erased block (imagine a bitwise tally in some cases). My worry is that this doesn't actually work properly with encryption, since encryption takes those tallys and requires the whole 16-bytes to be erased/rewritten to something totally different.

Additionally, according to the esp-idf docs:

If writing to an encrypted partition, all write offsets and lengths must be multiples of 16 bytes.
Nowhere in esp_littlefs is this enforce. I'm worried that the current "working" esp_littlefs with encryption is just on the surface, and with heavy use may not actually work as expected.

I suppose my point is, please tread carefully with the encryption! And please report back if things are/are-not working as expected.

I don't think this is correct, because as I mention above there are checks that the writes happen in 16 byte chunks and at 16 byte offsets... Check esp_flash_write_encrypted.

[gw8484]

@gw8484 Then it sounds like you don't have CONFIG_SPI_FLASH_ENABLE_ENCRYPTED_READ_WRITE set in ESP-IDF, because that uses esp_flash_write_encrypted which checks that both the write is 16 byte aligned and the size is a multiple of 16.

CONFIG_SPI_FLASH_ENABLE_ENCRYPTED_READ_WRITE is enabled but I don't know if LittleFS is directly or indirectly using esp_flash_write_encrypted. In any case, since LittleFS is writing non-16-byte aligned data the function would at best fail. So, I don't see how using the function alone would make file creation on encrypted partition work. LittleFS still needs to be changed to ensure the function is called with 16-byte aligned data.

[smjothen]

@gw8484 From what I can see in lfs.c all of the calls to littlefs_esp_part_write are aligned up to the next multiple of CONFIG_LITTLEFS_WRITE_SIZE, which is by default 128 bytes, so should not be possible to write 8 bytes at a time. Where did you observe the 8 byte write?