Skip to content

Implement NetEase Integration Phase C pinned data supply chain - #24

Merged
jacklilyhello merged 3 commits into
mainfrom
feature/netease-pinned-data-supply-chain
Jun 23, 2026
Merged

jacklilyhello merged 3 commits into
mainfrom
feature/netease-pinned-data-supply-chain

Conversation

@jacklilyhello

Copy link
Copy Markdown
Owner

Implementation summary

Implements NetEase Integration Phase C from the merged Phase B state by adding pinned G79/X19 upstream snapshots, deterministic generated OpenAudit Packs/reports, GPL/MIT license-boundary notices, SOURCE.json provenance, an offline verifier/regenerator, a maintainer-only synchronization command, Makefile targets, and committed-pack runtime integration tests.

Modified files

  • cmd/sync-netease-rules/main.go
  • Makefile
  • docs/bundled-rules-phase-c-netease.md
  • internal/engine/bundled_runtime_test.go
  • THIRD_PARTY_NOTICES.md
  • third_party/netease-sensitive-words/LICENSE
  • third_party/netease-sensitive-words/NOTICE.md
  • third_party/netease-sensitive-words/SOURCE.json
  • third_party/netease-sensitive-words/upstream/G79SensitiveWords.json
  • third_party/netease-sensitive-words/upstream/X19SensitiveWords.json
  • data/bundled/NETEASE-NOTICE.md
  • data/bundled/netease-g79.json.gz
  • data/bundled/netease-g79.report.json
  • data/bundled/netease-x19.json.gz
  • data/bundled/netease-x19.report.json

Pinned upstream

Source SHA-256 values

  • SensitiveWords/G79SensitiveWords.json: f398f68c5af1b2b92e01b8dc895ddecef86b72cf2783c798566bab604b5aa07d
  • SensitiveWords/X19SensitiveWords.json: 5f8f7e4d6b2436a4b47e5d8c5fa2e936d06b6c18e45689ebf8aa712097baeba4
  • LICENSE: 3972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986

Generated artifact SHA-256 values

  • data/bundled/netease-g79.json.gz: 0f8d487d940685753896f20d6473482d1de5743e2392e2b7acfb4e36fe4ef2d7
  • data/bundled/netease-g79.report.json: 4925fc6ec4dea0adffdac09294587ef9b3c7899089a0008cbd0bda8b9430e975
  • data/bundled/netease-x19.json.gz: be30a7d16847b25446d630b01164088a203b320b6fccfe364ae9dbbafb66a3e5
  • data/bundled/netease-x19.report.json: 45c7f1fc578ef4c2263ed60e8bfb0afd96647293b40a42702daddabfbe8d87e2

File locations

  • Source snapshots: third_party/netease-sensitive-words/upstream/G79SensitiveWords.json, third_party/netease-sensitive-words/upstream/X19SensitiveWords.json
  • Upstream license: third_party/netease-sensitive-words/LICENSE
  • Provenance manifest: third_party/netease-sensitive-words/SOURCE.json
  • Generated Packs/reports: data/bundled/netease-g79.json.gz, data/bundled/netease-g79.report.json, data/bundled/netease-x19.json.gz, data/bundled/netease-x19.report.json

Actual counts and compatibility

G79:

  • Total/imported/parsed: 1,679 / 1,679 / 1,679
  • Groups: shield 609, intercept 167, replace 1, nickname 899, remind 3
  • Empty/malformed/duplicate IDs/duplicate regex content: 0 / 0 / 0 / 358
  • RE2-compatible/incompatible: 519 / 1,160
  • Incompatible reason breakdown: lookbehind 833, unsupported_escape 152, invalid_syntax 171, lookahead 3, backreference 1

X19:

  • Total/imported/parsed: 1,618 / 1,618 / 1,618
  • Groups: shield 597, intercept 123, replace 1, nickname 894, remind 3
  • Empty/malformed/duplicate IDs/duplicate regex content: 0 / 0 / 0 / 380
  • RE2-compatible/incompatible: 520 / 1,098
  • Incompatible reason breakdown: lookbehind 816, unsupported_escape 121, invalid_syntax 155, lookahead 4, backreference 2

Synchronization and rollback design

go run ./cmd/sync-netease-rules supports offline verification, deterministic regeneration, and maintainer-only download mode. Download mode rejects floating refs, only accepts the reviewed 40-character pinned commit, downloads only the three allowlisted HTTPS upstream paths, uses finite timeouts, enforces response-size and status checks, restricts redirects, avoids credential forwarding, validates JSON/LICENSE content before replacement, stages through temporary files, and preserves prior snapshots on failure with rollback-oriented replacement.

Offline reproducibility design

make verify-bundled-netease performs no network access and should not modify tracked files. It reads committed source snapshots, regenerates G79/X19 Packs and reports in memory using the pinned commit timestamp, validates SOURCE.json with typed JSON decoding and hash/path/provenance checks, and compares regenerated bytes and SHA-256 values to committed artifacts. make regenerate-bundled-netease rewrites deterministic artifacts from committed snapshots; git diff --exit-code passes after regeneration.

GPL-3.0 handling and MIT boundary

The full upstream GPL-3.0 license is preserved verbatim. Notices state that the upstream JSON and generated Packs/reports derived from it are GPL-3.0-covered material, while OpenAudit's root MIT license applies only to independently authored OpenAudit code and does not relicense third-party data. The notices also avoid claiming that repository layout alone is a definitive legal conclusion.

Runtime defaults

Phase B defaults remain conservative: bundled_rules.enabled: false, bundled_rules.netease.enabled: false, both datasets disabled by default, Shield/Intercept group defaults inert until global/provider/dataset enablement is explicit, and Replace/Nickname/Remind remain disabled by default.

Tests and exact results

  • gofmt -w cmd/sync-netease-rules/main.go internal/engine/bundled_runtime_test.go: passed
  • go test ./internal/bundled/... ./internal/engine/... ./internal/rules/... ./internal/config/...: passed
  • go test ./cmd/sync-netease-rules: passed ([no test files])
  • make verify-bundled-netease: passed
  • git diff --check: passed
  • go test ./...: passed
  • go test -race ./internal/bundled/... ./internal/engine/...: passed
  • go vet ./...: passed
  • go build ./...: passed
  • CGO_ENABLED=0 go build ./...: passed
  • make fmt-check: passed
  • make smoke: passed
  • make e2e: passed
  • make gosec: passed with 0 issues
  • make govulncheck: failed because govulncheck could not fetch the vulnerability DB: Get "https://vuln.go.dev/index/modules.json.gz": Forbidden
  • make regenerate-bundled-netease && git diff --exit-code: passed after commit

Known limitations / deferred work

PCRE2 support, Docker/release packaging, release archives, automatic updates, and runtime downloads remain deferred. The runtime remains RE2-only and skips incompatible regexes while reporting them.

@jacklilyhello
jacklilyhello merged commit bff1f6e into main Jun 23, 2026
6 checks passed
@jacklilyhello
jacklilyhello deleted the feature/netease-pinned-data-supply-chain branch June 23, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants