Replace libxmljs2 with libxml2-wasm

[arobson-ods]

Overview

This PR address issue #574 - Upgrade version of libxmljs2 used to bring in line with Dashboard validation

What Changed?

• Library Migration: Replaced libxmljs2 0.32.0 with libxml2-wasm 0.7.1
• Primary Fix: Resolved the primary bug in issues #1002 and #845
• Bonus Fix: Resolving this issue inherently fixes an unlogged, hidden bug where the previous library incorrectly rejected a 30-digit decimal.
• Housekeeping: Fixed several typos and broken URLs in the example .env file that were previously preventing the application from booting up locally. Added a pre-commit hook and CI check to prevent inadvertent push of unsecured functions to remote. Applied linting fixes.

Architectural & Scope Decisions

• Out of Scope: The existing limitation regarding "Line numbers for Schema Errors in activities longer than 65535 lines" documented in the README persists in this service. Decided this would best be tracked on a separate ticket if required.
• API Behaviour Impact: The external API contract remains unchanged. However the details object is smaller. This affects only requests that pass ?details=true. details is a documented query parameter — postman/schemas/index.yaml:93 publishes it to developer.iatistandard.org as "Show all context information for advanced use and debugging". Details are additional to the error message e.g.

"message": "Element 'iati-organisations', attribute 'generated-datetime': '2018-11-07' is not a valid value of the atomic type 'xs:dateTime'.\n",

Under libxmljs2, the innermost object was the raw libxml2 xmlError struct, whatever
happened to be populated on it. Measured on iati-org-schema-errors.xml:

{ "domain": 17, "code": 1824, "level": 2, "column": 0, "line": 2, "str1": "2018-11-07" }

It is now exactly:

{ "line": 2, "column": 0 }

Field by field

field	status	reasoning
line	kept	unchanged, still offset-corrected per activity
column	kept, renamed	libxml2-wasm calls it col; remapped so the reported name stays column
str1	dropped	the offending value, e.g. "2018-11-07". Not lost as information — it appears inside message — but no longer separately machine-readable
level	dropped	always 2 (error). Never used: 0.3.1 is hardcoded severity: 'critical'
domain	dropped	always 17 (SCHEMASV) for schema errors
str2, str3, int1	dropped	were never populated for these errors anyway. The old code destructured all eight names unconditionally, so they already serialised as undefined — today's shape is variable per error type
code	dropped — the one real loss	1824 = SCHEMAV_CVC_DATATYPE_VALID_1_2_1, the only machine-readable, version-stable identifier for the kind of violation.

Considered the benefits of migration outweighed this one (minor?) loss.

Verification & Testing

• The pre library upgrade test: commit includes failing unit and integration tests. These turn green after the upgrade.
• Previously schema validation tests only ran in the integration tests (and consequently code base was mergeable to develop even if schema validation was broken). To best target the bug I introduced schema unit tests (at least as a temporary measure) in this PR. This introduces a naming problem as npm run rules:test now runs a few schema tests as well as the ruleset tests.
• The failing integration test also trips a rule set warning (1.18.8) - "The reporting organisation id must start with an approved agency code". This introduces a little noise to the signal but is consistent with the existing practice of using a AA-AAA-123456789 as org identifier in the existing fixtures.

Notes for Reviewer

• We don't make any commitments in the API documentation about the shape of the details information. However we are changing API behaviour. We need to assess how to approach loss of the code param - a) nothing b) document c) patch libxml2-wasm to recover this info.
• Would appreciate guidance on what to do with the new schema and utils unit tests. Useful to me when doing this work but don't fit the current practice.
• Suggest creating a new issue to flag replacing the try/finally{ xmlDoc.dispose() } construct with using when bump Node version.
• See new 'XML Library' section in the README. From v0.7.0 onwards libxml-wasm is built from a submodule that points at the maintainer's own fork of libxml2 containing two (Windows related) commits. Don't know if we want to do that or go back to v0.6.0.
• Once I'd done this I realised I probably should have done docs: , ci:, and style: commits in a separate PR prior to migration.

[simon-20]

This is excellent @arobson-ods - very thorough, and greatly improves the state of the app/repo.

I suggest two changes, but they are extremely minor so I'm marking as approved.

With respect to the decisions:

• I think the handling of ?details=true is correct. I got an AI bot to scan the logs for this application, and IATI Publisher accounts for 93% of calls to the endpoint with details=true, and is the only consumer continuously accessing it, and Publisher doesn't read anything except details[i].error.line so this PR doesn't affect Publisher. The other traffic was all ad hoc (couple of requests on a single day), so it nothing to worry about. So the approach you've taken is good.
• On the version of libxml-wasm used: I think we stick with the version that uses that fork for now, but revisit the next time it is updated.
• Let's change the name of the npm run rules:test command to make it slightly more general so that it captures the types of tests it now runs.
• Could we add a CHANGELOG.md to this repo - that would be a great place to include a couple of bullet points describing the changes.