A Day with F2FS @ The Pool

Embedded systems are all around us and most people never know about their presence. Most people never give any consideration to the little devices which make modern life possible.

Data caching at The Edge isn't always from Fastter or Cuntter or Cloudstter or whichever CDN happens to be serving your IG, Safeway, NFL, Reddit, Anthropology, or 338% of the internet's concurrent traffic being pointless traversals like jQuery minified & brotli-fied and sent back and forth endlessly through Internet Exchanges.

Edge caching is not limited to Anthropic or OAI or whoever else's Load-Balancer API-side "prompt cache (not yet! after LB!)" packets which "those people" are constantly complaining about, nor the hyperscalers who whitebox big CDN networks under their own headline IP.. those are not what I'm talking about when I talk about Edge Caches.

These little embedded devices which collect climate metrics, air quality data, assembly robotics, municipal awareness and telemetry data, and everything else from a very long list... they need read-heavy churn-heavy persistent storage on a filesystem optimized for tiny sized industrial NAND.

More examples for the ones of all sizes making modern life possible: that ATM you went to, the parking lot video surveillance systems, traffic light coordinator nodes, telco edge ONT routers and OOBs with SDNs, and a thousand other examples abound that are functionally ignored.

Also, robots. And factory hardware. And factories making robots that make tiny factories? Hopefully so.

Embeds Reduce CAPEX & OPEX Efficiently

So, those embedded devices usually are designed for efficiency, while not too often designed to be hammering on NAND for write-heavy storage workloads. This has caveats: devices explicitly designed for write heavy I/O on purpose, and therefore not what I'm referring to, and cases where the provisioning and/or configuration stages of fleet deployment are improperly aligned to hardware realities.

Why? Low wattage, moderate caches, and a desire to not burn out NAND when the device is located in a difficult-to-get-to location for service techs working on hardware replacement.

Most of all - we want efficient storage for embedded devices so that their operationally collected data can be stored in memory with a ECC write-back cache with "operationally consistent" garbage collection and non-invasive wear-leveling, all while the transactions are paged/queued to F2FS before async'ing upstream to a centralized data collector for aggregation.

Edge devices have their own little ecosystem separate from the manner in which datacenter storage network-enabled hardware operates. All by design, and always room for improvement.

Many people consider this boring tech, and would rather move fast and break things — your time is limited, people are sick of perfectly workable systems breaking all the fuckin time and they're tired of being A/B tested on in production — doing those things makes your company's testing policy look like an asshole. Everyone, stop being assholes. Thanks!

These Things - Why Should We Care?

You like your cell phone? How about GPS? SAT-NAV? Skylink? Drones operating up above withour your awareness? How about sector antennas for the police and emergency services? Embedded systems are streaming, caching, processing, and relaying data all over the globe and up into LEO constellations.

You don't have to care, and you shouldn't have to care about FSFS or drones unless you work with that side of tech, or its adjacent use cases - like living in the modern world. Everyone should care because knowledge is fundamentally important, if for no other reason at all. Do you know how the light-switch on the wall works? Same deal. Stop making excuses for saying you don't know or don't care or you will be left behind. No one is saving us but ourselves, and we must have knowledge to do so.

Recent Adventures with Standards-Definitions

I love automation. In part this is because I prefer efficiency over repetitive manual bullshit. Efficiently handling requirements means more time to do what I love:

• Hanging out by the pool with a tablet or waterproof e-ink reader.
  Way back in 2018 my e-book tally was 62 novels over the calendar year. That number has dropped off recently for trivial reasons, but the point remains that I love to read, and I love to lay out by the pool.
• What I do not love:
  Wasting time when I could be at the pool. ♥‿♥

• What I'll never not love:
  The poem on my torso, dedicated to the manner in which my departed twin lived & loved.

Standards for NAND? Standards for SD-NAND!

To the point, here's a script that had to be blasted out a while back, while imaging so many too many industrial-spec SD/TF-cards for a fleet of distibuted data collectors. These fun little RK3588 based systems have a NPU and only an eMMC drive for the OS.

While the eMMC in question has decent specs, if it fails then it's sadly more expensive to de-solder and re-solder and re-image the OS on eMMC, and so on these systems we use the SD NAND for the following data:

• Ephemeral scratch space and logs (eg: /var/log, /var/tmp, /tmp - nope, these go to tmpfs)
• Cache directories which are aptly named .cache or .Cache סּ_סּ
• Files which must be staged locally but have no business being on the OS mount points.
• Several other use cases that I don't feel like describing right now; so here's the discussion for those who love to know more more more.

### F2FS - Embedded Filesystem Performance Options

- Ref: https://docs.kernel.org/filesystems/f2fs.html

#### Option Flags and Reasoning

##### For High-Churn I/O
This is less ideal, but sometimes a `tmpfs` overlay isn't an option, so it's better than burning out the eMMC. A good industrial SD card has ECC, wear-leveling, checksum validations, write-amplification controls, etc. Don't care for those features? You may one day...

Example Writes: data persistent log-sync, package builds on-device, container writable layers, time-series compressible metrics, telemetry buffers, or small-file caches, opt for the non-default `-o 10` flag. 

| Option                      | Rationale              |
| --------------------------- | -----------------------|
| `-l DATA`                   | Stable label for `/etc/fstab`; avoids relying on device enumeration order. UUID is an alternative, but requires programmatic parsing of `blkid` output, or manually specifying a UUID to pass to the FS creation process.. or you can simpy use a GPT label. |
| `-o 5`                      | Reserves 5% of the F2FS volume for the F2FS cleaner. `mkfs.f2fs` documents this as hidden overprovision space used by the cleaner. |
| `-t 1`                      | Performs discard/TRIM at format time; defaults discard to enabled, but some card readers/cards mishandle it; use `-t 0` if format stalls or logs errors |
| `extra_attr`                | Required by several modern F2FS features. |
| `inode_checksum`            | Adds inode metadata checksums; useful for removable flash and unattended systems. |
| `sb_checksum`               | Adds superblock checksum support. |
| `flexible_inline_xattr`     | Modern xattr layout; useful with `extra_attr` and future feature expansion. |
| `inode_crtime`              | Adds inode creation-time support; low cost, useful for forensic/debug visibility. |
| `lost_found`                | Creates lost+found support for recovery workflows. |
| `compression`               | Good for compressible write-heavy data; poor fit for already-compressed media, model files, archives, database pages, and random-write workloads. |
| `compress_algorithm=zstd:3` | Validate performance of zstd compression options before using for production purposes. |
| `compress_log_size=%u`      | Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB. |
| `checkpoint_merge`          | Self-describing (or check the refs)  |
| `nocheckpoint_merge`        | Disable checkpoint merge feature. |

- `checkpoint=%s[:%u[%]]`: Set to "disable" to turn off checkpointing. Set to "enable" to re-enable checkpointing. Is enabled by default. 

- `checkpoint_merge`
When checkpoint is enabled, this can be used to create a kernel daemon and make it to merge concurrent checkpoint requests as much as possible to eliminate redundant checkpoint issues. Plus, we can eliminate the sluggish issue caused by slow checkpoint operation when the checkpoint is done in a process context in a cgroup having low i/o budget and cpu shares.

- `compress_algorithm=%s`
- `compress_algorithm=lzo-rle`

Control compress algorithm, currently f2fs supports "lzo", "lz4", "zstd" and "lzo-rle" algorithm.

- `compress_algorithm=%s:%d`

Control compress algorithm and its compress level.. only "lz4" and "zstd" support compress level config and those algos may be in your kernel config and may not.. but you'll find out quickly if you don't already know the answer.

```
=========      ===========
algorithm      level range
=========      ===========
lz4            3 - 16
zstd           1 - 22
=========      ===========
```

- `compress_log_size=%u`
Support configuring compress cluster size. The size will be `4KB * (1 << %u)`. The default and minimum sizes are 16KB.

- `compress_extension=%s`

Support adding specified extension, so that f2fs can enable compression on those corresponding files.
If all files with `.ext` has high compression rate, we can set the `.ext` on compression extension list and enable compression on these file by default rather than to enable it via ioctl.

For other files, we can still enable compression via ioctl. Note that, there is one reserved special extension `*`, it can be set to enable compression for all files.

- `nocompress_extension=%s`

Support adding specified extension, so that f2fs can disable compression on those corresponding files, just contrary to compression extension.
If you know exactly which files cannot be compressed, you can use this.

> The same extension name can’t appear in both compress and nocompress extension at the same time.

If the compress extension specifies all files, the types specified by the nocompress extension will be treated as special cases and will not be compressed.
After add nocompress_extension, the priority should be: `dir_flag < comp_extention,nocompress_extension < comp_file_flag,no_comp_file_flag`.

- `compress_chksum`
Support verifying chksum of raw data in compressed cluster.

- `compress_mode=%s`
Control file compression mode: "fs" and "user". In "fs" mode (default), f2fs does automatic compression on the compression enabled files. In "user" mode, f2fs disables the automaic compression and gives the user discretion of choosing the target file and the timing.

The user can do manual compression/decompression on the compression enabled files using ioctls.

- `compress_cache`
Support to use address space of a filesystem managed inode to cache compressed block, in order to improve cache hit ratio of random read.

- `inlinecrypt`
When possible, encrypt/decrypt the contents of encrypted files using the blk-crypto framework rather than filesystem-layer encryption.
This allows the use of inline encryption hardware. The on-disk format is unaffected.

- `atgc`
Enable age-threshold garbage collection, it provides high effectiveness and efficiency on background GC. No one likes garbage.

- `discard_unit=%s`
Control discard unit, the argument can be "block", "segment" and "section", issued discard command’s offset/size will be aligned to the unit, by default, "`discard_unit=block`" is set, so that small discard functionality is enabled.

For blkzoned device, "`discard_unit=section`" will be set by default, it is helpful for large sized (fucccck off with SMR, less fucks to ZNS) devices to reduce memory cost by getting rid of fs metadata supports small discard (ouch, ever get burned by SMR? yeah... just don't bother).

- `age_extent_cache`
Enable an age extent cache based on rb-tree. It records data block update frequency of the extent per inode, in order to provide better temperature hints for data block allocation.

- `errors=%s`
Specify f2fs behavior on critical errors. This supports modes: "`panic`", "`continue`" and "`remount-ro`", respectively, trigger panic immediately, continue without doing anything, and remount the partition in read-only mode.

By default it uses "`continue`" mode, not my choice.

```
====================== ===============  
mode                   continue        remount-ro      panic
====================== ===============  
access ops             normal          normal          N/A
syscall errors         -EIO            -EROFS          N/A
mount option           rw              ro              N/A
pending dir write      keep            keep            N/A
pending non-dir write  drop            keep            N/A
pending node write     drop            keep            N/A
pending meta write     keep            keep            N/A
====================== ===============  
```

- `nat_bits`
Enable nat_bits feature to enhance full/empty nat blocks access, by default it’s disabled.

- `lookup_mode=%s`
Control the directory lookup behavior for case-folded directories. This option has no effect on directories that do not have the casefold feature enabled.

```
========================================
Value              Description
========================================
perf               (Default) Enforces a hash-only lookup.
                     The linear search fallback is always
                     disabled, ignoring the on-disk flag.
compat         Enables the linear search fallback for
                compatibility with directory entries
                created by older kernel that used a
                different case-folding algorithm.
                This mode ignores the on-disk flag.
auto               F2FS determines the mode based on the
                on-disk `SB_ENC_NO_COMPAT_FALLBACK_FL`
                flag.
========================================
```

----

#### Benchmarking F2FS Compression

Always validate the compression algo used before using those options, as performance of one algo vs anther can vary widely across different CPU generation and architectures, as well as for hosts with algo/hash compression offloading engines. We can cover benchmarking these in another script.

> lzo, lz4, zstd, lzo-rle algorithms

##### Create Partitions

```
DEV=/dev/sdX
sgdisk --zap-all "$DEV"
sgdisk -a 2048 -n 1:2048:-16G -t 1:8300 -c 1:DATA "$DEV"
partprobe "$DEV"
```

##### Create FS

```
mkfs.f2fs -f \
-l DATA \
-o 10 \
-t 1 \
-O extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found,compression,compress_algorithm=zstd:3
"${DEV}1"
```

#### FS Tab Entry

##### Conservative defaults
```
LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard 0 0
```

##### Compression-capable variant

```
LABEL=DATA /srv/data f2fs rw,noatime,nodiratime,lazytime,nofail,background_gc=on,gc_merge,checkpoint_merge,flush_merge,mode=adaptive,fsync_mode=posix,nodiscard,compress_algorithm=lz4,compress_chksum,compress_mode=fs 0 0
```

#### Selectively Mark Compressible Directories
Do not enable compression broadly for LLM model files, videos, archives, SQLite databases, or opaque binary blobs unless benchmarks show a benefit.

```
mkdir -p /srv/data/_logs /srv/data/_caches.compressible
chattr +c /srv/data/_logs /srv/data/_cache.compressible
```

So, Maybe This is Scriptable?

It sure is, or by Ansible playbook, or Salt sync, perhaps even a CFengine golden image.

That's a lot of blah blah blah. How about that script?

Efficiency, indeed. Here ya go. The benchmarking script is for another day, or you can check out this workload primer about the filesystem from USENIX (w/ link to slide deck), and a benchmarking script which has been customized for F2FS.

#!/usr/bin/env bash
# -*- mode: sh; sh-basic-offset: 4; sh-shell: bash; coding: utf-8; indent-tabs-mode: nil; -*-
# ----------------------------------------------------------------------- #
set -euo pipefail
# ----------------------------------------------------------------------- #
# Purpose: POSIX-syntax wrapper to partition and format a non-boot SD/microSD card as F2FS with optional compression.
# NAND Profile: embedded I/O, kernel 6.1+, ECC industrial spec controller.
# WARNING: this destroys data on the selected block device, so be aware.
# ----------------------------------------------------------------------- #

usage() {
    cat <<'USAGE'
Usage:
  format-f2fs-sd-data.sh -d /dev/DEVICE [-l LABEL] [-r RESERVE_GIB] [-o F2FS_OP_PERCENT] [-c]

Examples:
# ----------------------------------------------------------------------- #
  ./format-f2fs-sd-data.sh -d /dev/sdX -l DATA -r 16 -o 5
  ./format-f2fs-sd-data.sh -d /dev/mmcblk0 -l DATA -r 24 -o 8 -c

Options:
# ----------------------------------------------------------------------- #
  -d  Whole block device to erase and partition: /dev/sdX -OR- /dev/mmcblk0
  -l  F2FS label. Default: DATA
  -r  GiB left unpartitioned at end for card FTL spare area. Default: 16
  -o  F2FS hidden overprovision percentage for the F2FS cleaner. Default: 5
  -c  Enable F2FS compression feature at format time. Mount options and chattr are still required.

Dependencies:
# ----------------------------------------------------------------------- #
  sgdisk, partprobe, mkfs.f2fs, lsblk
# ----------------------------------------------------------------------- #
USAGE
}

# Basic Variables
# ----------------------------------------------------------------------- #
LABEL=DATA
RESERVE_GIB=16
OP_PERCENT=5
COMPRESS=0
DEVICE=

# Define ARGV Mappings
# ----------------------------------------------------------------------- #
while getopts 'd:l:r:o:ch' opt; do
    case "$opt" in
        d) DEVICE=$OPTARG ;;
        l) LABEL=$OPTARG ;;
        r) RESERVE_GIB=$OPTARG ;;
        o) OP_PERCENT=$OPTARG ;;
        c) COMPRESS=1 ;;
        h) usage; exit 0 ;;
        *) usage; exit 2 ;;
    esac
done

# Sanity Checks
# ----------------------------------------------------------------------- #
if [ -z "$DEVICE" ]; then
    usage >&2
    exit 2
fi

case "$DEVICE" in
    /dev/sd?|/dev/sd??|/dev/mmcblk[0-9]|/dev/nvme[0-9]n[0-9]) ;;
    *)
        echo "Refusing suspicious device path: $DEVICE" >&2
        exit 2
        ;;
esac

# Validate Deps & Check Limits
# ----------------------------------------------------------------------- #
for cmd in sgdisk partprobe mkfs.f2fs lsblk; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        echo "Missing required command: $cmd" >&2
        exit 1
    fi
done

if [ ! -b "$DEVICE" ]; then
    echo "Not a block device: $DEVICE" >&2
    exit 1
fi

case "$RESERVE_GIB" in
    ''|*[!0-9]*) echo "Reserve GiB must be an integer" >&2; exit 2 ;;
esac
case "$OP_PERCENT" in
    ''|*[!0-9]*) echo "F2FS OP percentage must be an integer" >&2; exit 2 ;;
esac

case "$DEVICE" in
    /dev/mmcblk*|/dev/nvme*) PART=${DEVICE}p1 ;;
    *) PART=${DEVICE}1 ;;
esac

# Define Default F2FS Options
# ----------------------------------------------------------------------- #
FEATURES='extra_attr,inode_checksum,sb_checksum,flexible_inline_xattr,inode_crtime,lost_found'
if [ "$COMPRESS" -eq 1 ]; then
    FEATURES=${FEATURES},compression
fi

# Make Sure...
# ----------------------------------------------------------------------- #
echo "About to DESTROY all data on: $DEVICE"
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,MOUNTPOINTS "$DEVICE" || true
echo
echo "New partition: $PART"
echo "Label: $LABEL"
echo "Unpartitioned reserve at end: ${RESERVE_GIB} GiB"
echo "F2FS OP: ${OP_PERCENT}%"
echo "F2FS features: $FEATURES"
echo
printf 'Type YES to continue: '
read ans
if [ "$ans" != YES ]; then
    echo "Aborted."
    exit 1
fi

# Best Effort Unmount for Existing Partitions (please don't need this)
# ----------------------------------------------------------------------- #
if command -v findmnt >/dev/null 2>&1; then
    findmnt -rn -S "$DEVICE" -o TARGET 2>/dev/null | while IFS= read -r mp; do
        [ -n "$mp" ] && umount "$mp" || true
    done
fi

# And then...
# ----------------------------------------------------------------------- #
sgdisk --zap-all "$DEVICE"
sgdisk -a 2048 -n "1:2048:-${RESERVE_GIB}G" -t 1:8300 -c "1:${LABEL}" "$DEVICE"
partprobe "$DEVICE" || true
sleep 2

if [ ! -b "$PART" ]; then
    echo "Partition node did not appear: $PART" >&2
    lsblk "$DEVICE" >&2 || true
    exit 1
fi

# OhhhShitHandlers
# ----------------------------------------------------------------------- #
# NOTE: -t 1 performs initial discard when supported; if it's hanging here, rerun manually with -t 0
mkfs.f2fs -f -l "$LABEL" -o "$OP_PERCENT" -t 1 -O "$FEATURES" "$PART" && \
sync

# Display Block Devs
# ----------------------------------------------------------------------- #
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,RM,RO,FSTYPE,LABEL,UUID,PARTUUID,MOUNTPOINTS "$DEVICE"

echo "[COMPLETE]"
# ----------------------------------------------------------------------- #
# EOF #
# ----------------------------------------------------------------------- #