Power Failure Protection on FDS (Flash Data Storage)

Flash Data Storage moduled has been introduced since SDK 11.0 or later. Comparing to the Pstorge module, it brings some benefits.

• Using the FILE ID instead of the flash address
• Reduce the number of flash erase time (by marking the erase page first and then do the garbage collection if need).

In this blog, I would like to describe how to improve the stability of the FDS module particular in the power failure such as power reset during the garbage collection handling. This improvement would be applied on the SDK 15.3 or before. Because the official improvement would add on the SDK 16.0.

In the Nordic SDK 15.3 documentation for FDS module,

The Flash Data Storage (FDS) module is a minimalistic file system for the flash storage on the chip that minimizes the risk of data corruption and simplifies your interaction with the persistent storage. It does this by organizing data in files, which consist of one or more records. The records contain the actual data and can be written, deleted, updated, or retrieved.

The concept of treating data as files provides a high level of abstraction. You can use the FDS module without detailed knowledge of the actual data format that is used internally. Instead, you can just work with files and records and use the module as a black box.

The module has been designed to provide the following benefits:

• Minimizing the risk of accessing corrupted data by constant validation: In case of power loss, data might be written incompletely. Validation ensures that FDS recognizes invalid data and never returns corrupted data to the user.
• Offering (optional) CRC verification when opening a record to ensure that data has not changed since it was written.
• Minimizing flash operations (updating and deleting): Instead of deleting full pages, FDS stores copies of new data and invalidates outdated data by a single word write.
• Basic wear leveling: Sequential writes and garbage collection provide for an even level of flash usage.
• Making it easy to access data without copying it, which makes the impact of accessing data independent of the size of the data.
• Minimizing memory usage by allowing for flexible data size.
• Impose no restrictions on the content of the data (which means that it can contain special characters).

FDS uses Experimental: Flash Storage as backend to write to flash. Flash Storage, in turn, relies on the SoftDevice to execute the write. Flash Data Storage supports synchronous read operations and asynchronous writes.

See Functionality for an explanation of the functionality that FDS provides. Storage format shows how records are stored in flash. Usage presents code examples and Writable NDEF Message Example shows how to use FDS in an application.

This module is still in the experimental stage.

You can find all the details about the FDS module inside the documentation.

Storage format

Flash Data Storage stores data as records, which are grouped into files. In most use cases, you do not need to understand in detail how FDS stores the data in flash. The following information gives some insight about the data format that FDS uses, but if you are not interested in the details, you can skip this part.

Record layout

Records consist of a header (the record metadata) and the actual content. They are stored contiguously in flash in the order they are written.

Layout of a record

Record header

The record header consists of three words (12 bytes), which are used in the following way:

Field	Size	Description
Record key	16 bits	Key that can be used to find the record. The value FDS_RECORD_KEY_DIRTY (0x0000) is reserved by the system to flag records that have been invalidated. See Restrictions on keys and IDs for further restrictions.
Data length	16 bits	Length of the data that is stored in the record (in 4-byte words).
File ID	16 bits	ID of the file that the record is associated with. The value FDS_FILE_ID_INVALID (0xFFFF) is used by the system to identify records that have not been written correctly. See Restrictions on keys and IDs for further restrictions.
CRC value	16 bits	CRC value of the whole record (checks can be enabled by setting the FDS_CRC_ENABLED compile flag, see Configuration).
Record ID	32 bits	Unique identifier of the record.

When writing the record header to flash, FDS writes the record key and data length first, followed by the record ID. The file ID and the CRC value are written last and finalize a successful write operation. When scanning through records, the FDS module ignores all records where the second word of the header has not been written, because a missing key indicates that the record was not stored completely.

Maximum length

The maximum length of a record depends on the size of a virtual flash page (defined in in fds_config.h, see FDS_VIRTUAL_PAGE_SIZE), the size of the page tag (2 words), and the size of the record header (3 words). By default, the virtual page size is set to the physical page size (1024 words), which results in a maximum data length of 1019 words.

To store larger data, increase the virtual page size or use the Experimental: Flash Storage module instead of FDS.

Page tag

Each virtual page that is used by FDS is marked with a page tag that is used by the system to store information about the page. The 2-word page tag contains information about what the page is used for (data storage or garbage collection) and what version of the file system is installed on the page.

The following page tags are used:

Word 0	Word 1	Description
0xDEADC0DE	0xF11E01FF	Page used for swap during garbage collection.
0xDEADC0DE	0xF11E01FE	Page used to store data.

The page tags are written when FDS is first initialized and updated only during garbage collection.

Improvement on the FDS module during the power failure

There are few discussion on the devzone posts to check with the SDK 14.2 / SDK 15.2.

• https://devzone.nordicsemi.com/f/nordic-q-a/44723/fds—data-corruption-during-an-interrupted-swap
• https://devzone.nordicsemi.com/f/nordic-q-a/43758/fds—data-corruption

For example,

Scenario:

• Given the device started GC(Garbage Collection) procedure
• When the device will reset during the in the certain moment of page swap procedure.
• Then assigning new record id will overlap existing ones.

When just first two words of a record are copied to the swap page, the header of this record will pass header_check function.

During the page_scan procedure, this corrupted header will be used to update m_latest_rec_id. It will result in assigning new record ids from 0.

Workaround

Here are the workaround on the fds.c.

diff --git a/components/libraries/fds/fds.c b/components/libraries/fds/fds.c
index 120ef0c..08c6767 100644
--- a/components/libraries/fds/fds.c
+++ b/components/libraries/fds/fds.c
@@ -185,8 +185,14 @@ static fds_header_status_t header_check(fds_header_t const * p_hdr, uint32_t con
         return FDS_HEADER_CORRUPT;
     }
 
+    // It is important to also check for the record ID to be non-erased.
+    // It might happen that during GC, when records are copied in one operation,
+    // the device powers off after writing the first two words of the header.
+    // In that case the record would be considered valid, but its ID would
+    // corrupt the file system.
     if (   (p_hdr->file_id    == FDS_FILE_ID_INVALID)
-        || (p_hdr->record_key == FDS_RECORD_KEY_DIRTY))
+        || (p_hdr->record_key == FDS_RECORD_KEY_DIRTY)
+        || (p_hdr->record_id  == FDS_ERASED_WORD))
     {
         return FDS_HEADER_DIRTY;
     }
@@ -226,10 +232,26 @@ static fds_page_type_t page_identify(uint32_t const * const p_page_addr)
     }
 }
 
-
-static bool page_is_erased(uint32_t const * const p_page_addr)
+// A page can be tagged if it is entirely erased, or
+// of the first word is fds magic word and the rest of it is erased.
+static bool page_can_tag(uint32_t const * const p_page_addr)
 {
-    for (uint32_t i = 0; i < FDS_PAGE_SIZE; i++)
+    // This function should consider pages that have only the first half
+    // of the fds page tag written as erased (taggable).
+    // That is because the tag is two words long, and if the device
+    // has rebooted after writing the first word, that would cause
+    // the page to be unusable (since undefined and not fully erased).
+    // By considering the first word as erased if it contains fds page tag,
+    // the page can be re-tagged as necessary.
+
+    if ((p_page_addr[FDS_PAGE_TAG_WORD_0] != FDS_ERASED_WORD) &&
+        (p_page_addr[FDS_PAGE_TAG_WORD_0] != FDS_PAGE_TAG_MAGIC))
+    {
+        return false;
+    }
+
+    // Ignore the first word of the tag, we already checked that it is either erased or fds's.
+    for (uint32_t i = FDS_PAGE_TAG_WORD_1; i < FDS_PAGE_SIZE; i++)
     {
         if (*(p_page_addr + i) != FDS_ERASED_WORD)
         {
@@ -654,7 +676,7 @@ static fds_init_opts_t pages_init(void)
         {
             case FDS_PAGE_UNDEFINED:
             {
-                if (page_is_erased(p_page_addr))
+                if (page_can_tag(p_page_addr))
                 {
                     if (m_swap_page.p_addr != NULL)
                     {