Add the hash datatype by u9g · Pull Request #177 · ProtoDef-io/node-protodef

u9g · 2026-09-06T14:32:25Z

Implements the hash datatype proposed in PrismarineJS/prismarine-item#184 (PrismarineJS/prismarine-item#184 (comment)):

["hash", { "alg": "crc32c", "type": "i32", "body": "ItemComponent" }]

Write serializes the value as body, hashes the bytes and writes the digest as type; read yields the digest. crc32 and crc32c are implemented in src/hash.js (the crc package has no CRC32C, which is what Minecraft uses); any other alg goes through node's crypto and produces a Buffer. A CRC written into a signed type such as i32 is written in two's complement, so it reads back as Java would produce it.

Compiler

A compiled writer had no way to size a nested value, so it couldn't serialize the body into a scratch buffer (#169 is about the same gap). This adds WriteCompiler.callTypeSize(value, type) and SizeOfCompiler.callTypeWrite(value, type, offsetExpr), which generate code with the sibling compiler in the current scope (so compareTo references resolve to the same variables) and run it against that compiler's context, exposed as ctx.sizeOfCtx / ctx.writeCtx like #169 does. Documented in doc/compiler.md. SizeOfCompiler also remembers fixed-size natives, so a hash into i32/lu32 is sized without hashing.

Tests

test/misc.js, interpreted and compiled: the CRC check values, signed output, a varint digest, sha256, a body that switches on a field of the enclosing container, and a hash over a list of hashes.

Schema and docs are in the ProtoDef submodule: ProtoDef-io/ProtoDef#65 (the submodule pointer here targets that branch).

Consumer: the HashOps encoding in PrismarineJS/prismarine-item#184 is built on this type.

["hash", { alg, type, body }] writes the value serialized as `body`, hashed with `alg`, as `type`; reading yields the digest. crc32 and crc32c are built in, anything else goes through node's crypto and is a Buffer. A CRC written into a signed type takes its two's complement. The compiled writer needs the size of the body before it can serialize it, which no writer could get at until now: WriteCompiler.callTypeSize and SizeOfCompiler.callTypeWrite generate code with the sibling compiler in the current scope and run it against that compiler's context. SizeOfCompiler now remembers fixed-size natives so hashes into a fixed-width type are sized without hashing.

extremeheat · 2026-09-06T17:47:07Z

+      code += `;((buffer) => ${compiler.callTypeWrite('value', body, '0')})(bodyBuffer)\n`
+      code += `const hash = hashDigest(${JSON.stringify(alg)}, bodyBuffer)\n`
+      code += 'return ' + size
+      return compiler.wrapCode(code)


Hashes are pretty much always fixed in size.

May be nicer to have ProtoDef spec support an explicit list of hashes (SHA1, SHA256, etc) rather than relying on whatever Node.js standard lib exposes.

That will allow hard codeing the hash byte length into a map without having to fake hash first and allows other non-JS ProtoDef implementations to support an explicit list to be spec complaint

Done, the only supported hashes are crc32c for now until a different one is requested.

extremeheat · 2026-09-06T17:50:31Z

@@ -0,0 +1,39 @@
+const crypto = require('crypto')


I don't think this belongs in the lib, we should either import lib (like crc) or at least move the hashing code to src/datatypes/hash.js which would contain the interpreter code plus the hashing code

The latter + importing the crc lib could be best ; you already have to import crypto as we don't do inline SHA hashing or anything like that

src/hash.js is gone, the code lives in src/datatypes/hash.js next to readHash/writeHash/sizeOfHash, and crypto is no longer imported anywhere.

I didn't import crc because it has no CRC32C, it only supports CRC-32.

extremeheat · 2026-09-06T17:52:44Z

+  /**
+   * Generates code with another compiler inside this compiler's scope, so that
+   * field references resolve to the same variables, and binds it to that
+   * compiler's context. Natives are reachable through the context as well.
+   */
+  callTypeIn (other, ctxName, generate) {
+    if (!other) throw new Error(`${ctxName} is only available when compiling with ProtoDefCompiler`)
+    const scopeStack = other.scopeStack
+    other.scopeStack = this.scopeStack
+    try {
+      const code = generate(other)
+      if (!isNaN(code)) return code
+      return `((ctx, native) => ${code})(ctx.${ctxName}, ctx.${ctxName})`
+    } finally {
+      other.scopeStack = scopeStack
+    }
+  }


Should not be this complicated, no need to generate anything at call time

Compile order should be fixed to something like sizeOf=>write=>read so write can always call sizeOf

Agreed, and callTypeIn is deleted. compileProtoDefSync already generated sizeOf → write → read. The only trade-off I took here, is that body must be a named type, which the spec now requires.

extremeheat · 2026-09-06T18:00:36Z

    this.readCtx = readCtx
+    // Code from callTypeSize / callTypeWrite runs against the other context
+    writeCtx.sizeOfCtx = sizeOfCtx
+    sizeOfCtx.writeCtx = writeCtx


SizeOfCompiler shouldn't need to write to figure out the size. That creates a potential cyclic dependency loop.

But it is useful for the WriteCompiler to know size of type such as for writing length prefixes for strings/array/buffer, hash digest, etc. Only reason looks like we didn't have this already is you can size a string/buffer in JS stdlib instead of needing ProtoDef (Buffer.byteLength vs needing to call our own sizeOf functions)

Right, and that direction is gone entirely, SizeOfCompiler.callTypeWrite and the sizeOfCtx.writeCtx back-reference are both removed.

extremeheat · 2026-09-06T18:24:37Z

    // Local variable to provide some context to eval()
    const native = this.native // eslint-disable-line
    const { PartialReadError } = require('./utils') // eslint-disable-line
+    const hashDigest = require('./hash').digest // eslint-disable-line


This is another codesmell, specific data types should not require injecting stuff like this into the pre compile step.

parameterizable types do create duplication but all the types have the same issue, so we shouldn't inject just for this

So that other JS code should be directly copied into the codegen step

Or figure out way to use native/context type, but that would require looking at making them parameterizable

or split the code between a 'parameterizable' type with a parameterizable part that calls some native/context function

Agreed, that line is gone, and nothing is injected into compile()'s scope any more. I went with the context-type option you listed: _crc32c: ['context', crc32c] in Write, which copies the function's source into the generated output, so the compiled code is self-contained and generated code calls ctx._crc32c(bodyBuffer).

With the spec defining an explicit list of algorithms, a digest's width is known before hashing: the compiled sizer is a constant and type has to be of constant size, checked when compiling. That was the only caller of SizeOfCompiler.callTypeWrite, so it and the sizeOf -> write back-reference are gone; a sizer never writes. The hashing code moves to src/datatypes/hash.js beside the interpreter functions, leaving no general-purpose hashing module in the lib, and no crypto import. The digest function is copied into the compiled context as ctx._crc32c rather than injected into compile()'s eval scope, so generated code is self-contained and an unknown algorithm is caught when compiling. A named body type is sized with a direct ctx.sizeOfCtx.<type> call; only an anonymous one, which has no function to call, still has its sizer generated in place.

No schema change, so nothing here reads differently.

…degen callTypeSize now emits a call to the sizer the sizeOf context already holds, ctx.sizeOfCtx.<type>(value), and throws when the body is not a named type. That leaves nothing generating code with another compiler, so callTypeIn and its scope-stack swap are gone: what the write compiler borrows from the sizeOf compiler is one function call, resolved by name. The capability this gives up, a body whose type is selected by a field of the enclosing container, was reachable only from an inline type. A named type cannot reference a parent field at all -- getField throws while the type is being generated -- so hash was alone in being able to do it, and the scope sharing was the whole reason it could. The interpreter rejects an inline body too, rather than accepting schemas the compiler refuses.

u9g mentioned this pull request Sep 6, 2026

Add Item.toHashedNotch for 1.21.5+ window_click slot claims PrismarineJS/prismarine-item#184

Open

extremeheat reviewed Sep 6, 2026

View reviewed changes

u9g mentioned this pull request Sep 12, 2026

My 78 open PRs, grouped PrismarineJS/mineflayer#4119

Open

u9g added 3 commits September 12, 2026 14:11

Point the submodule at the trimmed hash doc

b86c30b

No schema change, so nothing here reads differently.

Navigation Menu

Uh oh!

Add the hash datatype - #177

Add the hash datatype#177
u9g wants to merge 4 commits into
masterfrom
hash-type

u9g commented Sep 6, 2026

Uh oh!

extremeheat Sep 6, 2026 •

edited

Loading

Uh oh!

u9g Sep 12, 2026

Uh oh!

extremeheat Sep 6, 2026

Uh oh!

u9g Sep 12, 2026 •

edited

Loading

Uh oh!

extremeheat Sep 6, 2026

Uh oh!

u9g Sep 12, 2026 •

edited

Loading

Uh oh!

extremeheat Sep 6, 2026

Uh oh!

u9g Sep 12, 2026

Uh oh!

extremeheat Sep 6, 2026 •

edited

Loading

Uh oh!

u9g Sep 12, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

u9g commented Sep 6, 2026

Compiler

Tests

Uh oh!

extremeheat Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

u9g Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

extremeheat Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

u9g Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

extremeheat Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

u9g Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

extremeheat Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

u9g Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

extremeheat Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

u9g Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

extremeheat Sep 6, 2026 •

edited

Loading

u9g Sep 12, 2026 •

edited

Loading

u9g Sep 12, 2026 •

edited

Loading

extremeheat Sep 6, 2026 •

edited

Loading

u9g Sep 12, 2026 •

edited

Loading