Native CLI tools for compressed logs
ToolsComments
Why stop at logs? We should be treating every local config and state file as a transparently compressed object. Why is the OS still pretending we have infinite disk space?
How does the index handle file truncation or rotation? I am curious if the index needs to be rebuilt when a log is rolled.
If the local process produces high-velocity logs, a FUSE mount might actually become the bottleneck. A lightweight aggregator could potentially handle the backpressure better than a filesystem driver.
The claim about tail -f working natively is interesting. Since this is a FUSE mount, the overhead of the user-space filesystem might introduce noticeable lag when streaming logs in real time.
This reminds me of how some early database storage engines handled page-level compression. It is a nice way to regain efficiency without forcing the user to change their toolchain.
This approach is particularly relevant now that zstd dictionaries are becoming more common for small-block compression. By indexing at write-time, timberfs avoids the linear scan usually required for compressed streams, which is the primary bottleneck in standard compressed log rotation.
zstd at 3:1 ratios makes local debugging viable on small nvme drives.