H5Lens: HDF5 Pickles and Policy Workbench

H5Lens is a machine-readable description of HDF5 on-disk metadata using GNU poke pickles. It is both a format exploration kit and the home of h5policy, an independent metadata preflight oracle for hostile or untrusted HDF5 files. It also includes the early h5patch repair planner for proposing and applying conservative metadata repairs.

What's Here

• pickles/ contains the reusable HDF5 format definitions loaded by GNU poke.
• h5policy/ contains the policy oracle, focused validators, security profiles, regression corpus, differential harness, and fuzzing tools.
• h5patch/ contains the experimental metadata repair planner, JSON patch-plan format, apply workflow, and repair tests.
• h5explain/ contains the interactive GNU poke explorer for byte-level HDF5 metadata navigation.
• src/ contains the marker scanner implementation.
• tools/ collects top-level command symlinks and repository helper scripts such as the documentation generator.
• docs/ contains YAML format notes and generated Markdown reference pages.
• emacs/ contains an Emacs front end for inspecting HDF5 files through GNU poke.
• examples/ contains poke scripts for generating and inspecting HDF5 structures.
• MARKERS.md, TOOLS.md, and TUTORIAL.md explain the format markers, helper tools, and a hands-on exploration path.

Quick Start

Run h5policy against an HDF5 file:

./tools/h5policy --profile untrusted-strict --json file.h5

Try the regression suite:

cd h5policy/tests
./run.sh

Create a what-if metadata repair plan:

./tools/h5patch plan damaged.h5 -o repair.plan.json
./tools/h5patch explain repair.plan.json

Build the marker scanner and generated format docs:

cmake -S . -B build
cmake --build build
cmake --build build --target docs

Explore the sample file interactively:

./tools/h5explain file.h5

See h5explain/README.md for interactive navigation commands and TUTORIAL.md for a guided GNU poke walkthrough.

h5policy In Under A Minute

h5policy maps HDF5 metadata with GNU poke, validates the metadata it can reach, applies a selected security profile, and emits a JSON decision with stable exit codes. It is intentionally metadata-only: it does not call libhdf5, load plugins, decompress data, open external files, repair inputs, write files, or deserialize application payloads.

Use h5policy/README.md for profile behavior, exit codes, coverage, checksum notes, and CLI examples. Use h5policy/tests/README.md for the corpus, differential harness, and fuzzing workflow.

h5patch In Under A Minute

h5patch plans byte-level repairs for damaged HDF5 metadata, applies only an approved JSON plan, writes an audit log, and verifies the result with h5policy. Planning is a what-if operation: it does not modify the input file.

The first repair catalog is intentionally narrow and high-confidence: HDF5 signature restoration, stale v2/v3 superblock flags, v2/v3 superblock checksums, and reachable v2 object-header checksums. See h5patch/README.md for the plan format and workflow.

Acknowledgments

This material is based upon work supported by the U.S. National Science Foundation under Federal Award No. 2534078. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.