Guidance for cache configuration

[ephemeral-beep]

I want to use a large local cache (8TB) with a filesystem on top of zerofs.
zerofs should trickle-sync to S3 in the background.
The zerofs documentation has an example for "high performance" cache use,
but I would like some visibility into how much cache is in use and dirty, etc.
Should I use something like openCAS or bcache as a layer between zerofs and ZFS?
If so, should I configure the zerofs cache to be small? Is there any guidance I can look to when configuring this?
Thanks!

[Barre]

ZeroFS doesn't yet have writeback caching; the cache is read-only, so you won't be able to achieve "trickle sync" that way.

If you use external cache facilities, I'd still keep a small few-gigabyte cache so that the hottest data is fetched locally.

Should I use something like openCAS or bcache as a layer between zerofs and ZFS?

I think that would be unnecessary, especially if you use ZFS's L2ARC.

[ephemeral-beep]

I think L2ARC is just a read cache, where I need a read/write cache.
I will try with openCAS, and leave a small few-gigabyte cache for zerofs, as you say.
Thanks!

[ephemeral-beep]

This is working well, I've created 64 nbd devices from /mnt/zerofs/.nbd/
and created 64 empty zfs pools on top of the nbd devices (with openCAS in the middle)
however my network is much more busy that I would expect just for having 64 empty filesystems.
It's constantly reading and writing to S3. I see messages like this from zerofs :

2026-02-22T21:52:25.027784Z INFO slatedb::garbage_collector::compacted_gc: deleting SST [id=SsTableId::Compacted(01KJ3MDKX402EA22BY1CHWFJTR)]
2026-02-22T21:52:25.053020Z INFO slatedb::garbage_collector::compacted_gc: deleting SST [id=SsTableId::Compacted(01KJ3MEH7N94RTXG5XW7B1ZM4X)]
2026-02-22T21:52:25.080077Z INFO slatedb::garbage_collector::compacted_gc: deleting SST [id=SsTableId::Compacted(01KJ3MFEK3YRFK8P14P3PK8WGH)]
2026-02-22T21:52:25.113323Z INFO slatedb::garbage_collector::compacted_gc: deleting SST [id=SsTableId::Compacted(01KJ3MFHTZMNCD4NKDXG4PJT3X)]
2026-02-22T21:52:25.131579Z INFO slatedb::garbage_collector::compacted_gc: deleting SST [id=SsTableId::Compacted(01KJ3MFT5D8XKR54Y4QE77S3Q8)]
2026-02-22T21:52:34.804898Z INFO slatedb::compactor: in-flight compaction [compaction=["5242", "5157", "4907", "0"] -> SR(0) (929.49 MB processed)]
2026-02-22T21:52:34.804918Z INFO slatedb::compactor: in-flight compaction [compaction=["5359", "5349", "5330", "5304"] -> SR(5304) (7.45 GB processed)]
2026-02-22T21:52:34.804921Z INFO slatedb::compactor: in-flight compaction [compaction=["5396", "5395", "5394", "5393"] -> SR(5393) (205.14 MB processed)]
2026-02-22T21:52:35.940492Z INFO slatedb::compactor_state: accepted submitted compaction [compaction=["l0", "l0", "l0", "l0", "l0", "l0", "l0"] -> SR(5397)]
2026-02-22T21:52:36.004174Z INFO slatedb::compactor: in-flight compaction [compaction=["5242", "5157", "4907", "0"] -> SR(0) (929.49 MB processed)]
2026-02-22T21:52:36.004194Z INFO slatedb::compactor: in-flight compaction [compaction=["5359", "5349", "5330", "5304"] -> SR(5304) (7.47 GB processed)]
2026-02-22T21:52:36.004198Z INFO slatedb::compactor: in-flight compaction [compaction=["5396", "5395", "5394", "5393"] -> SR(5393) (224.11 MB processed)]
2026-02-22T21:52:36.004201Z INFO slatedb::compactor: in-flight compaction [compaction=["l0", "l0", "l0", "l0", "l0", "l0", "l0"] -> SR(5397)]
2026-02-22T21:52:44.805977Z INFO slatedb::compactor: in-flight compaction [compaction=["5242", "5157", "4907", "0"] -> SR(0) (929.49 MB processed)]
2026-02-22T21:52:44.806004Z INFO slatedb::compactor: in-flight compaction [compaction=["5359", "5349", "5330", "5304"] -> SR(5304) (7.63 GB processed)]
2026-02-22T21:52:44.806009Z INFO slatedb::compactor: in-flight compaction [compaction=["5396", "5395", "5394", "5393"] -> SR(5393) (392.65 MB processed)]
2026-02-22T21:52:44.806012Z INFO slatedb::compactor: in-flight compaction [compaction=["l0", "l0", "l0", "l0", "l0", "l0", "l0"] -> SR(5397) (149.77 MB processed)]
2026-02-22T21:52:50.006455Z INFO slatedb::compactor_state: finished compaction [spec=["l0", "l0", "l0", "l0", "l0", "l0", "l0"] -> SR(5397)]
2026-02-22T21:52:50.006492Z INFO slatedb::compactor: in-flight compaction [compaction=["5242", "5157", "4907", "0"] -> SR(0) (929.49 MB processed)]
2026-02-22T21:52:50.006499Z INFO slatedb::compactor: in-flight compaction [compaction=["5359", "5349", "5330", "5304"] -> SR(5304) (7.72 GB processed)]
2026-02-22T21:52:50.006502Z INFO slatedb::compactor: in-flight compaction [compaction=["5396", "5395", "5394", "5393"] -> SR(5393) (479.29 MB processed)]

Is there a way to increase verbosity or get stats about what zerofs is doing?
Or maybe a way to query to find out which nbd devices are being synced to s3?
When I delete a nbd device (via rm -f /mnt/zerofs/.nbd/deviceX) will zerofs know to stop saving it to s3?

Thanks!

[Barre]

Devices are not synced to S3; devices basically “live” on S3.

You can find out what sort of activity is happening by running zerofs fatrace -c config.toml.

Empty devices by themselves wouldn’t be a source of traffic.

Compaction is a normal process that takes place to optimize the way data is stored. It is expected to have to read large chunks of data and subsequently rewrite a bunch of it.

[ephemeral-beep]

Devices are not synced to S3; devices basically “live” on S3.

Right, dirty data from the nbd device is synced to S3.

You can find out what sort of activity is happening by running zerofs fatrace -c config.toml.

This is perfect for understanding the activity. Thanks!

Empty devices by themselves wouldn’t be a source of traffic.

zfs does a bunch of lazy initialization before settling down.
Next, I'll experiment with 9p directly.

Thanks!