About FrogFS

FrogFS (Fast Read-Only General-purpose File System) is a read-only filesystem designed for embedded use. It can be easily used with a CMake project — including ESP-IDF. It has built-in filters to save space. Files in frogfs-clockwise-demo are reduced by about half using the default filters. Here are links to the example repositories:

• frogfs-standalone-demo
• frogfs-clockwise-demo

Transform filters include:

• babel-convert
• babel-minify
• deflate
• gzip
• brotli
• html-minifier
• rename
• terminate
• uglify-js
• uglifycss

Compression filters include:

• none
• gzip
• heatshrink
• zlib

Transform filters are intended to be compile-time operations that do not incur a run-time cost while compression filters are expected to incur a run-time decompression cost.

This means for an HTTP server, zlib (deflate), brotli (br) or gzip compressed files can be passed through untouched! This saves processing time and bandwidth. Beware however that some browsers don't accept brotli content other HTTP (only HTTPS).

Getting started with ESP-IDF

To use this component with ESP-IDF, within your projects directory run

idf.py add-dependency jkent/frogfs

Embedding a FrogFS image

Embed FrogFS within your project binary with the folowing CMake function:

target_add_frogfs(<target> [CONFIG yaml] [NAME name])

If CONFIG is not specified, frogfs.yaml will be used. If NAME is not specified, it will default to frogfs.

As an example for ESP-IDF, in your project's toplevel CMakeLists.txt:

cmake_minimum_required(VERSION 3.16)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(my_project)

target_add_frogfs(${PROJECT_NAME}.elf)

In C, this results in these two global symbols being available to your application:

extern const uint8_t frogfs_bin[];
extern const size_t frogfs_bin_len;

Making a FrogFS binary and flashing it

You have the option of creating a binary without linking it with your application. A CMake function is provided to output a binary with target generate_${name}.

declare_frogfs_bin(path [CONFIG yaml] [NAME name])

If CONFIG is not specified, frogfs.yaml is used. If NAME is not specifed, frogfs is used.

Here's an example of what you can add to your toplevel CMakeLists.txt:

set(FROGFS_NAME frogfs)
declare_frogfs_bin(NAME ${FROGFS_NAME})

idf_component_get_property(main_args esptool_py FLASH_ARGS)
idf_component_get_property(sub_args esptool_py FLASH_SUB_ARGS)
esptool_py_flash_target(${FROGFS_NAME}-flash "${main_args}" "${sub_args}" ALWAYS_PLAINTEXT)

esptool_py_flash_to_partition(${FROGFS_NAME}-flash storage ${BUILD_DIR}/CMakeFiles/${FROGFS_NAME}.bin)
add_dependencies(${FROGFS_NAME}-flash generate_${FROGFS_NAME}_bin)

You can invoke the flash process by running idf.py frogfs-flash.

Configuration

FrogFS expects a yaml configuration file. There are 3 different sections: define, collect and filter. All but collect is optional.

Define is a list or dict of variable definitions. There are 2 predefined variables: $cwd and $frogfs. You can also reference environment variables with the ${ENV:varname} syntax.

Collect 'gathers' up files and directories and places them in the frogfs root. Glob patterns are allowed in the 'basename' component of the path. There are 3 ways to specify sources; they cn be a string, list, or dictionary. If it's a string, the path(s) become the root directory. If a list, the paths are merged in order and become the root directory. If a dict is used, the paths are merged into the destination of choice; empty string being the root directory. Variables are expanded for both source and destination.

Filter allows you to do post-processing on the files before they are integrated. Filter is a list or dict of dicts; with a glob pattern to a list of verbs. Varibales are expanded and all patterns are evaluated for each file or directory, top down. Transforms are applied first, then an optional final compression before caching the file.

Verbs are applied in descending order. You can prefix a transforms or the compress verb with no to disable it. There are a couple of special verbs: discard which prevents inclusion and cache (default), which caches the file in the build cache. See frogfs_example.yaml for example usage.

Usage

Two interfaces are available: the bare API or when using IDF there is the VFS interface which builds on top of the bare API. You should use the VFS interface in IDF projects, as it uses the portable and familiar posix and stdio C functions with it. There is nothing preventing you from mix and matching both at the same time, however.

Shared initialization

Configuration requries defining a frogfs_config_t structure and passing it to frogfs_init. Two different ways to specify the filesystem:

1. a memory address using the addr variable:

frogfs_config_t frogfs_config = {
    .addr = frogfs_bin,
};

2. a partition name using the part_label string:

frogfs_config_t frogfs_config = {
    .part_label = "storage",
};

Then it is just a matter of passing the frogfs_config to frogfs_init function and checking its return variable:

frogfs_fs_t *fs = frogfs_init(&frogfs_config);
assert(fs != NULL);

When done, and all file handles are closed, you can call frogfs_deinit:

frogfs_deinit(fs);

VFS interface

The VFS interface has a similar method of initialization; you define a frogfs_vfs_conf_t structure:

• base_path - path to mount FrogFS
• fs - a frogfs_fs_t instance
• max_files - max number of files that can be open at a time

frogfs_vfs_conf_t frogfs_vfs_conf = {
    .base_path = "/frogfs",
    .fs = fs,
    .max_files = 5,
};
frogfs_vfs_register(&frogfs_vfs_conf);

Bare API

Filesystem functions:

• frogfs_fs_t *frogfs_init(const frogfs_config_t *conf)
• void frogfs_deinit(frogfs_fs_t *fs)

Object functions:

• const frogfs_entry_t *frogfs_get_entry(const frogfs_fs_t *fs, const char *path)
• char *frogfs_get_name(const frogfs_entry_t *entry)
• char *frogfs_get_path(const frogfs_fs_t *fs, const frogfs_entry_t *entry)
• int frogfs_is_dir(const frogfs_entry_t *entry)
• int frogfs_is_file(const frogfs_entry_t *entry)
• void frogfs_stat(const frogfs_fs_t *fs, const frogfs_entry_t *entry, frogfs_stat_t *st)
• frogfs_fh_t *frogfs_open(const frogfs_fs_t *fs, const frogfs_entry_t *entry, unsigned int flags)
• void frogfs_close(frogfs_fh_t *fh)
• int frogfs_is_raw(frogfs_fh_t *fh)
• size_t frogfs_read(frogfs_fh_t *fh, void *buf, size_t len)
• ssize_t frogfs_seek(frogfs_fh_t *fh, long offset, int mode)
• size_t frogfs_tell(frogfs_fh_t *fh)
• size_t frogfs_access(frogfs_fh_t *fh, void **buf)

Directory Functions:

• frogfs_dh_t *frogfs_opendir(frogfs_fs_t *fs, const frogfs_entry_t *entry)
• void frogfs_closedir(frogfs_dh_t *dh)
• const frogfs_entry_t *frogfs_readdir(frogfs_dh_t *dh)
• void frogfs_seekdir(frogfs_dh_t *dh, long loc)
• long frogfs_telldir(frogfs_dh_t *dh)

How it works

Under the hood there is a hash table consisting of djb2 path hashes to entry offsets, which allow for fast lookups using a binary search algorithm. All entries except the root entry have a parent locator offset. Directory entries have a sorted list of offsets to child entries.

FrogFS binaries can be either embedded in your application, or accessed using memory mapped I/O. It is not possible (at this time) to use FrogFS without the file system binary existing in data address space.

Creation of a FrogFS filesystem is handled by a single tool, tools/mkfrogfs.py. It uses transforms in the tools directory, or you can add your own transforms by creating a tools directory in your projects root directory, with a filename starting with transform- and ending with .js or .py. Transform tools take data on stdin and produce output on stdout.

Both transform and compresors can accept arguments. See frogfs_example.yaml for an example.

History and Acknowledgements

FrogFS was split off of Chris Morgan (chmorgan)'s libesphttpd project (MPL 2.0), which is a fork of Jeroen Domburg (Sprite_tm)'s libesphttpd (BEER-WARE). This project would never have existed without them.

Thank you to all the contributors to this project!