PDF metadata with a NUL byte (Canon/Adobe-PSL producer) fails the pgvector insert: UntranslatableCharacter

[nangelovv]

Summary

Embedding a PDF whose metadata contains a NUL byte (\u0000) fails the
pgvector insert, and the file is silently never stored / never searchable.

Symptom

LibreChat logs File embedding failed. The rag_api side logs:

psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
DETAIL:  \u0000 cannot be converted to text.
CONTEXT:  JSON data, line 1: ..."producer": "Adobe PSL 1.3e for Canon\u0000...
[SQL: INSERT INTO langchain_pg_embedding (collection_id, embedding, document, cmetadata, custom_id, uuid) VALUES ...]
ERROR - Failed to store data in vector DB | File ID: ... | Error: ...

The PDF was produced by a Canon device / Adobe PSL PostScript driver that pads
the producer metadata string with a trailing NUL byte. Real-world sources
include Canon/Adobe-PSL scanners and some Google-Docs PDF exports.

Root cause

cmetadata is a JSONB column and PostgreSQL cannot store \u0000 in
text/JSONB, so the entire multi-row batch INSERT into langchain_pg_embedding
is rejected and rolled back. Nothing for that file is stored.

rag_api already strips NUL from document text (clean_text /
remove_null in app/utils/document_loader.py — the fix from LibreChat
Discussion #2243 for Google-Docs PDFs with NUL in the text). But in
_prepare_documents_sync (app/routes/document_routes.py) the raw PDF
metadata is merged into the document unsanitized:

metadata={
    "file_id": file_id,
    "user_id": user_id,
    "digest": generate_digest(doc.page_content),
    **(doc.metadata or {}),   # producer="...Canon\u0000" passes through
},

So a NUL in any metadata field (producer, creator, title, ...) flows
straight into the JSONB column and fails the insert.

Note: the embeddings are generated by the provider before the failing
insert, so the provider cost is incurred and then discarded.

Reproduce

1. Take any PDF and set a NUL in its producer metadata, e.g.
   exiftool -Producer=$'Canon\x00' file.pdf.
2. Embed it via POST /embed.
3. The response is {"status": false, "message": "...UntranslatableCharacter ... \u0000 ..."}
   and nothing is stored.

Proposed fix

Add a recursive clean_metadata() helper next to clean_text that strips NUL
from all string values (including nested dict/list), reusing remove_null, and
apply it where metadata is merged in _prepare_documents_sync. Apply it
unconditionally (not gated behind the PDF-only clean_content flag),
because the JSONB NUL constraint holds for every loader and file type.

PR follows.