runtime error when no allocator is configured

[orangecms]

I experimented a bit with the zlib-rs crate because I am searching for a decent (de)compression library for use in firmware, specifically https://github.com/oreboot/oreboot.

In the end, I ran into

zlib-rs/zlib-rs/src/inflate.rs

Lines 2151 to 2153 in aee1503

	if stream.zalloc.is_none() || stream.zfree.is_none() {
	return ReturnCode::StreamError;
	}

IIUC - i.e., a runtime error.

This could be checked at compile time instead, I think; is the current way it happens intentional?

Should we add something to the README to tell that an allocator is needed (I'd file a PR)?

Is there a possible intention to offer integration without heap allocation at some point?

[folkertdev]

This error is expected, but also it is already possible to provide a custom allocator. Here is some old code that runs with no_std on embedded hardware: https://github.com/folkertdev/no-std-zlib-rs. zlib-rs has changed so it may not be fully usable with the latest version, but it should be quite close.

this section

https://github.com/folkertdev/no-std-zlib-rs/blob/05c97c32a0639765bc6f1b3bdafdb45c6ec2ae62/src/main.rs#L79-L81

sets the allocator.

Hopefully that gets you unstuck?

We do check on CI that we can compile with no_std, but otherwise we've not been considering embedded use cases that much, because zlib-rs is quite large in terms of code size. If this works out for you tough, we'd be happy to add some additional documentation on use in no_std contexts.

[orangecms]

Oh thanks for the quick reply! For more details, I just did some more research and left my notes in oreboot/oreboot#772.

My goal is to avoid an allocator if possible, and we only need decompression on the target machine, so I would expect code size not to become too much of an issue; I just did a check and my code grows from 34K to 87K.
That is absolutely fine, because our use case is application processors / SoCs where we would run Linux, and we need to (de)compress the kernel to fit in a SPI flash, so a few dozen kilobytes of code vs megabytes of kernel size are a good tradeoff.

Anyway, my question is really whether it is a design decision to only support environments that have an allocator, which is all fine; I would just suggest to error at build time then, because I found the runtime error rather confusing, and it could be avoided, I think.

And I suggest documenting that an allocator is needed. For comparison, miniz_oxide explicitly mentions it in their README: https://github.com/Frommi/miniz_oxide/blob/633e59fd7efa3fe73be7a712503a9b5ede7ef2c1/README.md?plain=1#L6

[folkertdev]

we're using different definitions of allocator here.

So: zlib-rs does not need a rust global allocator. It does not need the alloc crate. It totally should work for your use case, with the right configuration.

What zlib does need is some amount of working memory. Rather than assuming a global allocator, the zlib interface makes it possible to pass in two functions:

pub type in_func = unsafe extern "C" fn(*mut c_void, *mut *const c_uchar) -> c_uint;
pub type out_func = unsafe extern "C" fn(*mut c_void, *mut c_uchar, c_uint) -> c_int;

struct z_stream { 
    // ...
    pub zalloc: Option<alloc_func>,
    pub zfree: Option<free_func>,
    // ..
}

These functions can do anything. We provide default implementations that defer to the rust global allocator, or to malloc and free, but they could just as well be backed by an array. In fact, that is what the example does:

https://github.com/folkertdev/no-std-zlib-rs/blob/05c97c32a0639765bc6f1b3bdafdb45c6ec2ae62/src/main.rs#L17

static USED: AtomicUsize = AtomicUsize::new(0);

struct Buffer<const N: usize>(UnsafeCell<[u8; N]>);
static BUFFER: Buffer<{ 32 * 1024 }> = Buffer::new();

impl<const N: usize> Buffer<N> {
    const fn new() -> Self {
        Self(UnsafeCell::new([0; N]))
    }
}

unsafe impl<const N: usize> Sync for Buffer<N> {}

pub unsafe extern "C" fn zalloc_c(opaque: *mut c_void, items: c_uint, size: c_uint) -> *mut c_void {
    use core::sync::atomic::Ordering;

    let _ = opaque;

    let start = USED.fetch_add((items * size) as usize, Ordering::Relaxed);

    let ptr = unsafe { BUFFER.0.get().cast::<u8>().add(start) as *mut c_void };

    ptr
}

/// # Safety
///
/// The `ptr` must be allocated with the allocator that is used internally by `zcfree`
pub unsafe extern "C" fn zfree_c(opaque: *mut c_void, ptr: *mut c_void) {
    let _ = opaque;

    // no-op
    let _ = ptr;
}

This "allocator" is just an array in static memory, the zalloc_c function updates some bookkeeping and returns a pointer to somewhere in that array.

The library promises that all calls to zalloc happen during initialization, so once that succeeds, you can be sure that actually running (de)compression does not need more memory. There are also some configuration options that reduce the total amount of memory that is needed, which you can find in the example, e.g. the 10 here:

let err = unsafe {
    inflateInit2_(
        &mut stream,
        10,
        zlibVersion(),
        core::mem::size_of::<z_stream>() as _,
    )
};

[orangecms]

Oh interesting, I hadn't seen that - will take a look, thank you!

Is there still something we could add to the README so the next person to try this like me would have a chance to know? =) Or where is it mentioned?

[folkertdev]

yeah, the problem with zlib is that it assumes you already know how it works. We're trying to fix that, but it's a lot.

Technically we do have some documentation on the allocators here, but it is not very discoverable.

[orangecms]

Alright, thank you a lot! 🧡

I'll keep this issue open for tracking, might be useful to others. :)

We may meet at RustWeek, at least I will be attending. Cheers! 🦀