chore: upgrade @huggingface/transformers to v4

[PatrickSys]

Why this needs to happen

The current dependency is @huggingface/transformers@^3.8.1, which pins onnxruntime-node@1.21.0. That version has a well-documented C++ mutex destruction bug on macOS (tracked in microsoft/onnxruntime#24579) that causes a hard crash during the embedding phase:

libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

This is the root cause of #68. The fix landed in onnxruntime-node@1.24.x.

Current state of v4

@huggingface/transformers v4 is already in pre-release on npm under the next tag:

Version	onnxruntime-node
3.8.1 (current stable)	1.21.0 ❌
4.0.0-next.3	1.24.1 ✅
4.0.0-next.7 (latest next)	1.24.3 ✅

So the fix is already there — we just need to track when v4 hits stable and do the upgrade.

What to watch out for

v4 is a major version bump so there will likely be breaking changes in the API. Before upgrading, we'll need to:

• Review the v4 migration guide / changelog for anything that affects TransformersEmbeddingProvider
• Check whether the pipeline / model loading API changed
• Run the embedding tests against a v4 install to catch any breakage early

Interim workaround

Until v4 is stable, users hitting #68 can use EMBEDDING_PROVIDER=openai with a local Ollama instance via OPENAI_BASE_URL (see #70) to bypass Transformers.js entirely.

Related

• Bug Report: Mutex Lock Failed During Indexing on Intel Mac #68 — the macOS crash this upgrade would permanently fix
• feat: implement native Ollama embedding provider #71 — native Ollama provider, an alternative path for local embeddings that doesn't depend on Transformers.js at all