diff --git a/docs/OPTIMIZATION-SUMMARY.md b/docs/OPTIMIZATION-SUMMARY.md deleted file mode 100644 index 7c705fde..00000000 --- a/docs/OPTIMIZATION-SUMMARY.md +++ /dev/null @@ -1,293 +0,0 @@ -# Rules 仓库优化与修复说明 - -本文档总结了对 KuGouGo/Rules 仓库的优化和修复。 - ---- - -## 📋 已实施的优化与修复 - -### 1. 二进制编译并行化(性能优化 - P1) - -**问题**:逐文件串行编译 sing-box/mihomo 二进制规则,无法利用多核 CPU。 - -**修复**: -- 分离 JSON 生成与二进制编译 -- 使用 `xargs -P $BUILD_JOBS` 批量并行编译 -- 新增环境变量 `RULES_BUILD_JOBS`(默认 `nproc`) - -**影响**: -- 4 核 CPU:预期加速 3-4x -- 8 核 CPU:预期加速 6-8x -- 100 个规则集:100s → 15-25s - -**使用方法**: -```bash -# 默认并行(自动检测 CPU 核心数) -make build-custom - -# 指定并行度 -RULES_BUILD_JOBS=8 make build-custom - -# 仅文本产物(不受影响) -make build-custom-text -``` - ---- - -### 2. 规则统计输出(可观测性 - P2) - -**问题**:构建时无法直观看到每个规则集的内容分布。 - -**修复**:增加统计输出函数,打印每个规则集的规则类型分布。 - -**输出示例**: -``` -Building domain rules for emby: 15 rules (DOMAIN=5, SUFFIX=8, KEYWORD=2, REGEX=0) -Building IP rules for apple-apns: 12 CIDRs (IPv4=9, IPv6=3) - -Compiling domain binaries with 8 parallel jobs... - ✓ sing-box: compiled 15 rule-sets - ✓ mihomo: compiled 13 rule-sets - -Compiling IP binaries with 8 parallel jobs... - ✓ sing-box: compiled 5 rule-sets - ✓ mihomo: compiled 5 rule-sets -``` - -**价值**: -- 快速识别规则集规模 -- 发现异常(如规则数突然大幅变化) -- 验证编译产物数量 - ---- - -### 3. DOMAIN-REGEX 备选规则(兼容性 - P2) - -**问题**: -- Surge/QuanX/mihomo 不支持 DOMAIN-REGEX -- `fakeip-filter.list` 中 12 条 REGEX 规则会被跳过 -- 影响 NTP/Xbox Live/STUN 等关键服务 - -**修复**:为关键 REGEX 规则增加 SUFFIX 备选规则,兼容不支持平台。 - -**新增备选规则**: -``` -# NTP 时间同步 -DOMAIN-SUFFIX,time.windows.com # Windows 时间服务 -DOMAIN-SUFFIX,time.nist.gov # NIST 时间服务 -DOMAIN-SUFFIX,ntp.aliyun.com # 阿里云 NTP -DOMAIN-SUFFIX,ntp.tencent.com # 腾讯 NTP - -# Xbox Live -DOMAIN-SUFFIX,xboxlive.com -DOMAIN-SUFFIX,xboxservices.com -DOMAIN-SUFFIX,xboxab.com -DOMAIN-SUFFIX,xbox.com - -# STUN 穿透 -DOMAIN-SUFFIX,stunprotocol.org -DOMAIN-SUFFIX,stun.l.google.com -DOMAIN-SUFFIX,stun.syncthing.net -``` - -**影响**: -- **修复前**:Surge/QuanX/mihomo 仅 86 条可用规则(72% 覆盖) -- **修复后**:98 条可用规则(89% 覆盖) -- **提升**:+14% 覆盖率 - -**权衡**: -- REGEX 规则:精确匹配(如 `^time\.[^.]+\.com$`) -- SUFFIX 备选:范围更广(如 `time.windows.com`) -- 结果:部分域名可能被 Fake-IP 错误分类,但关键服务已覆盖 - ---- - -### 4. 规则类型支持文档(文档 - P1) - -**问题**:用户不清楚各平台支持哪些规则类型,以及不支持时的影响。 - -**修复**:创建 `docs/RULE-TYPE-SUPPORT.md`,详细说明: -- 5 个平台的规则类型支持矩阵 -- DOMAIN-REGEX 不支持的影响与缓解方案 -- 规则顺序与优先级最佳实践 -- 构建输出说明 - -**文档位置**:[docs/RULE-TYPE-SUPPORT.md](docs/RULE-TYPE-SUPPORT.md) - ---- - -### 5. 单标签 TLD 保留(规则完整性 - P0) - -**问题**:China List 中 `cn`/`top`/`wang` 等真实 TLD 被误删。 - -**修复**:增加 13 个单标签 TLD 白名单,防止误删。 - -**影响**:解析从 111444 升至 111450 条(+6 条关键域名)。 - ---- - -### 6. 上游 JSON 严格解析(数据完整性 - P0) - -**问题**:AWS/CloudFront/Fastly/RIPE Stat 源格式变更时可能静默丢失前缀。 - -**修复**:所有 JSON 解析器强制校验 `isinstance(data, dict/list)` 和字段类型。 - -**影响**:防止上游异常时静默失败,构建会报错而非生成错误产物。 - ---- - -### 7. 二进制空文件防御(编译完整性 - P0) - -**问题**:sing-box/mihomo 编译器 OOM 时返回 0 但不产出文件。 - -**修复**:`verify_singbox`/`verify_mihomo` 增加 `st_size > 0` 断言。 - -**影响**:防止空文件被误判成功,CI 会明确失败。 - ---- - -### 8. overlap audit 测试兼容(测试稳定性 - P0) - -**问题**:事务测试只构造部分目录时 overlap audit 失败。 - -**修复**:调用前增加目录存在性检查。 - -**影响**:`test-artifact-transaction-atomic.sh` 通过。 - ---- - -### 9. 平台转换损失统计(可观测性 - P2) - -**问题**:无法直观看到哪些规则因平台不支持被跳过。 - -**修复**:增加 `platform_loss_audit`,汇总每平台跳过规则。 - -**输出示例**: -```json -{ - "platform_conversion_loss": { - "surge": {"skipped_rules": {"DOMAIN-REGEX": 12}}, - "quanx": {"skipped_rules": {"DOMAIN-REGEX": 12}}, - "mihomo": {"skipped_rules": {"DOMAIN-KEYWORD": 3, "DOMAIN-REGEX": 12}} - } -} -``` - -**影响**:写入 `overlap-report.json`,便于分析平台差异。 - ---- - -### 10. sing-box 域名转换防御检查(代码健壮性 - P1) - -**问题**:直接使用 `SINGBOX_KIND_MAP[kind]` 无防御。 - -**修复**:增加 `unsupported_kinds` 检查 + `.get()` + ValueError。 - -**影响**:与 mihomo/Egern 防御机制对齐,避免运行时崩溃。 - ---- - -### 11. Surge/QuanX 跳过规则统计(可观测性 - P2) - -**问题**:列表推导式静默过滤 DOMAIN-REGEX,无输出。 - -**修复**:调用 `print_platform_skip_summary`。 - -**输出示例**: -``` -domain summary: fakeip-filter skips unsupported rules for surge: DOMAIN-REGEX=12 -``` - -**影响**:与 mihomo 对齐,提升可观测性。 - ---- - -## 📊 优化效果总结 - -### 性能提升 -| 指标 | 优化前 | 优化后 | 提升 | -|------|--------|--------|------| -| 构建时间(4核) | 100s | 25-30s | **70-75%** | -| 构建时间(8核) | 100s | 15-20s | **80-85%** | -| 并行度 | 1 | 自动(nproc) | 动态扩展 | - -### 规则覆盖率(fakeip-filter) -| 平台 | 优化前 | 优化后 | 提升 | -|------|--------|--------|------| -| Egern/sing-box | 100% (110条) | 100% (110条) | - | -| Surge/QuanX/mihomo | 72% (86条) | 89% (98条) | **+14%** | - -### 代码质量 -| 维度 | 状态 | -|------|------| -| 子进程调用防御 | ✅ 100% `check=True` | -| 文件编码声明 | ✅ 100% `encoding="utf-8"` | -| 边界条件检查 | ✅ 100% 覆盖 | -| 平台转换防御 | ✅ 5/5 平台完整 | -| 测试覆盖 | ✅ 20/27 通过(7个因环境限制) | - ---- - -## 🎯 使用建议 - -### 推荐配置 - -**构建性能优化**: -```bash -# 默认自动检测(推荐) -make build-custom - -# 手工指定并行度(高性能 CI) -RULES_BUILD_JOBS=16 make build-custom -``` - -**客户端选择**: -- **完整支持**:Egern / sing-box(推荐,支持 DOMAIN-REGEX) -- **兼容支持**:Surge / QuanX / mihomo(已有备选规则,89% 覆盖) - -**规则编写最佳实践**: -1. 优先使用 `DOMAIN` 和 `DOMAIN-SUFFIX`(100% 平台兼容) -2. 避免依赖规则源文件顺序(sing-box/mihomo 不保证) -3. 通过规则精确度控制优先级(DOMAIN > SUFFIX > KEYWORD > REGEX) -4. 关键服务同时提供 REGEX + SUFFIX 备选(兼容性) - ---- - -## 📁 相关文档 - -- [规则类型支持说明](docs/RULE-TYPE-SUPPORT.md) -- [完整 Review 报告](docs/COMPLETE-REVIEW-FINAL.md)(如有) -- [性能优化报告](docs/PERFORMANCE-OPTIMIZATION.md)(如有) - ---- - -## 🔄 后续改进计划 - -### P2:中优先级(1-2 月) -- [ ] 域名后缀合并缓存(Trie 持久化) -- [ ] 上游增量下载(ETag / Last-Modified 缓存) -- [ ] 增加更多常见 NTP/STUN 服务备选规则 - -### P3:低优先级(3+ 月) -- [ ] 平台间一致性交叉验证 -- [ ] 规则重复检测(跨规则集) -- [ ] 自动化性能基准测试 - ---- - -## ✅ 验证清单 - -- [x] Shell 语法检查通过 -- [x] Python 语法检查通过 -- [x] 配置文件 JSON 格式正确 -- [x] 核心测试全部通过 -- [x] 备选规则已生效 -- [x] 统计输出正确 -- [x] 文档完整 - -**当前状态**:✅ **生产就绪,可立即提交** - ---- - -**更新时间**:2026-07-28 -**维护者**:KuGouGo diff --git a/docs/RULE-TYPE-SUPPORT.md b/docs/RULE-TYPE-SUPPORT.md index e8282c87..a05839e4 100644 --- a/docs/RULE-TYPE-SUPPORT.md +++ b/docs/RULE-TYPE-SUPPORT.md @@ -1,162 +1,43 @@ -# 规则类型平台支持说明 +# 规则类型与平台转换 -## 域名规则类型支持矩阵 +仓库以规范化的 classical 规则作为中间表示,再按客户端能力生成文本或二进制产物。 -| 规则类型 | Surge | QuanX | Egern | sing-box | mihomo | -|---------|-------|-------|-------|----------|--------| -| DOMAIN | ✅ | ✅ | ✅ | ✅ | ✅ | -| DOMAIN-SUFFIX | ✅ | ✅ | ✅ | ✅ | ✅ | -| DOMAIN-KEYWORD | ✅ | ✅ | ✅ | ✅ | ❌ | -| DOMAIN-REGEX | ❌ | ❌ | ✅ | ✅ | ❌ | +## 域名规则 ---- +| 规范类型 | Surge | Quantumult X | Egern | sing-box | mihomo MRS | +| --- | --- | --- | --- | --- | --- | +| `DOMAIN` | `DOMAIN` | `HOST` | `domain_set` | `domain` | plain domain | +| `DOMAIN-SUFFIX` | `DOMAIN-SUFFIX` | `HOST-SUFFIX` | `domain_suffix_set` | `domain_suffix` | leading-dot domain | +| `DOMAIN-KEYWORD` | `DOMAIN-KEYWORD` | `HOST-KEYWORD` | `domain_keyword_set` | `domain_keyword` | unsupported | +| `DOMAIN-REGEX` | unsupported | unsupported | `domain_regex_set` | `domain_regex` | unsupported | -## DOMAIN-REGEX 不支持的影响 +不支持的规则不会被近似转换,因为用更宽泛的 `DOMAIN-SUFFIX` 或 `DOMAIN-KEYWORD` 替代正则会改变匹配集合。构建日志和 `overlap-report.json` 会记录平台转换损失。 -### 受影响平台 -- **Surge**:不支持 DOMAIN-REGEX,规则会被跳过 -- **QuanX**:不支持 DOMAIN-REGEX,规则会被跳过 -- **mihomo**:不支持 DOMAIN-REGEX,规则会被跳过 +如果一个规则集必须在所有平台保持同等覆盖,应在源规则中维护语义明确的兼容条目,而不是由转换器自动猜测降级规则。 -### 当前使用情况 -本仓库中 DOMAIN-REGEX 规则主要用于 `fakeip-filter.list`,用于排除 Fake-IP 以避免连接失败。 +## IP 规则 -**关键场景**: -- **NTP 时间同步**:`^time\.[^.]+\.com$` 匹配 `time.windows.com`、`time.apple.com` 等 -- **Xbox Live**:`^xbox\.[^.]+\.microsoft\.com$` 匹配 Xbox 服务发现域名 -- **STUN 穿透**:`^(?:.+\.)?stun(?:\.[^.]+){2,5}$` 匹配各类 STUN 服务器 -- **微信本地调试**:`^localhost\.[^.]+\.weixin\.qq\.com$` +| 规范类型 | Surge | Quantumult X | Egern | sing-box | mihomo MRS | +| --- | --- | --- | --- | --- | --- | +| `IP-CIDR` | `IP-CIDR` | `IP-CIDR` | `ip_cidr_set` | `ip_cidr` | plain CIDR | +| `IP-CIDR6` | `IP-CIDR6` | `IP6-CIDR` | `ip_cidr6_set` | `ip_cidr` | plain CIDR | -### 缓解方案 +IP 规则在进入平台渲染前统一规范化、精确去重并折叠包含或相邻网段。sing-box 的 `ip_cidr` 字段允许 IPv4 和 IPv6 混合。 -#### 方案一:使用支持 REGEX 的客户端(推荐) -- **Egern**(iOS):完整支持 DOMAIN-REGEX -- **sing-box**(全平台):完整支持 DOMAIN-REGEX +## 完整性原则 -#### 方案二:使用 SUFFIX 备选规则(已实施) -`fakeip-filter.list` 已为关键服务增加 SUFFIX 备选规则: +1. 源格式解析失败时终止构建,不静默忽略有效行中的错误。 +2. 只做可证明语义等价的规范化,例如域名大小写统一、CIDR 主机位归一化、CIDR 并集折叠。 +3. 每个规则集可独立订阅,因此跨规则集重叠只审计,不自动删除。 +4. 平台不支持的类型只记录损失,不自动扩大或缩小匹配范围。 +5. sing-box 和 mihomo 二进制产物必须存在、非空,并通过反编译往返验证。 -``` -# REGEX(精确) -DOMAIN-REGEX,^xbox\.[^.]+\.microsoft\.com$ - -# SUFFIX 备选(兼容 Surge/QuanX/mihomo,稍宽松) -DOMAIN-SUFFIX,xboxlive.com -DOMAIN-SUFFIX,xboxservices.com -``` +## 并行编译 -**优点**:兼容所有平台 -**缺点**:SUFFIX 匹配范围更广,可能包含不需要排除的子域名 +二进制编译使用仓库统一的 `RULES_COMPILE_JOBS` 配置。未设置时自动检测 CPU,但最多使用 4 个并发任务,避免在 CI 和低内存设备上产生过高峰值内存。 -#### 方案三:手工为不支持平台添加白名单 -在客户端配置中手工添加常见服务域名到 DNS 直连白名单: - -**Surge 示例**: -``` -[Rule] -DOMAIN-SUFFIX,time.windows.com,DIRECT -DOMAIN-SUFFIX,xboxlive.com,DIRECT -DOMAIN-SUFFIX,stunprotocol.org,DIRECT +```bash +RULES_COMPILE_JOBS=2 make build-custom ``` -**mihomo (Clash) 示例**: -```yaml -rules: - - DOMAIN-SUFFIX,time.windows.com,DIRECT - - DOMAIN-SUFFIX,xboxlive.com,DIRECT - - DOMAIN-SUFFIX,stunprotocol.org,DIRECT -``` - ---- - -## DOMAIN-KEYWORD 不支持的影响 - -### 受影响平台 -- **mihomo**:不支持 DOMAIN-KEYWORD,规则会被跳过 - -### 缓解方案 -DOMAIN-KEYWORD 通常可以拆分为多条 DOMAIN-SUFFIX 规则: - -**转换前**: -``` -DOMAIN-KEYWORD,google -``` - -**转换后**: -``` -DOMAIN-SUFFIX,google.com -DOMAIN-SUFFIX,google.co.jp -DOMAIN-SUFFIX,google.com.hk -DOMAIN-SUFFIX,googleapis.com -DOMAIN-SUFFIX,googleusercontent.com -``` - ---- - -## IP 规则类型支持矩阵 - -| 规则类型 | Surge | QuanX | Egern | sing-box | mihomo | -|---------|-------|-------|-------|----------|--------| -| IP-CIDR | ✅ | ✅ | ✅ | ✅ | ✅ | -| IP-CIDR6 | ✅ | ✅ | ✅ | ✅ | ✅ | - -**注**:QuanX 使用 `IP6-CIDR` 代替 `IP-CIDR6`,本仓库自动转换。 - ---- - -## 规则顺序与优先级 - -### 不保证顺序的平台 -- **sing-box**:rule-set 内部按类型分组,类型间顺序不保证 -- **mihomo**:二进制格式,顺序由编译器决定 - -### 最佳实践 -❌ **不应依赖规则源文件顺序** -✅ **应通过规则精确度控制匹配优先级** - -规则精确度(从高到低): -1. `DOMAIN`:完全匹配(最精确) -2. `DOMAIN-SUFFIX`:后缀匹配 -3. `DOMAIN-KEYWORD`:子串匹配 -4. `DOMAIN-REGEX`:正则匹配(取决于模式复杂度) - -**示例**: -``` -# 好的实践(通过精确度控制) -DOMAIN,exact.example.com # 优先级最高 -DOMAIN-SUFFIX,example.com # 次优先 -DOMAIN-KEYWORD,example # 最低 - -# 不推荐(依赖顺序) -DOMAIN-KEYWORD,example # 可能误匹配 -DOMAIN,exact.example.com # 顺序不保证 -``` - ---- - -## 构建输出说明 - -构建时会输出每个规则集的统计信息: - -``` -Building domain rules for emby: 15 rules (DOMAIN=5, SUFFIX=8, KEYWORD=2, REGEX=0) -domain summary: emby skips unsupported rules for surge: DOMAIN-REGEX=0 -domain summary: emby skips unsupported rules for quanx: DOMAIN-REGEX=0 -domain summary: emby skips unsupported rules for mihomo: DOMAIN-KEYWORD=2, DOMAIN-REGEX=0 -``` - -**mihomo MRS 警告**: -当跳过规则超过 30% 时会打印警告: -``` -mihomo mrs warning: fakeip-filter skips 50% unsupported rules (threshold 30%) -``` - ---- - -## 相关资源 - -- [Surge 规则语法](https://manual.nssurge.com/rule/) -- [Quantumult X 规则语法](https://github.com/crossutility/Quantumult-X) -- [Egern 规则语法](https://book.egernapp.com/) -- [sing-box rule-set 规范](https://sing-box.sagernet.org/configuration/rule-set/) -- [Clash Premium 规则提供器](https://dreamacro.github.io/clash/premium/rule-providers.html) +该变量必须是正整数。文本转换、规则语义和产物事务不受并行度影响。 diff --git a/scripts/commands/build-custom.sh b/scripts/commands/build-custom.sh index d0cd2fd3..7614e801 100755 --- a/scripts/commands/build-custom.sh +++ b/scripts/commands/build-custom.sh @@ -8,7 +8,6 @@ cd "$ROOT" source "$ROOT/scripts/commands/check-runtime.sh" TEXT_ONLY_MODE="${RULES_BUILD_CUSTOM_TEXT_ONLY:-0}" -BUILD_JOBS="${RULES_BUILD_JOBS:-$(nproc 2>/dev/null || echo 4)}" # Reject malformed or globally conflicting custom sources before creating build # directories, inspecting staged artifacts, or downloading build tools. @@ -175,27 +174,33 @@ GENERATED_CUSTOM_ARTIFACTS="$(collect_generated_custom_artifacts)" print_domain_rule_stats() { local plain_list="$1" local base="$2" - local total domain suffix keyword regex - - total=$(grep -c "^DOMAIN" "$plain_list" 2>/dev/null || echo "0") - domain=$(grep -c "^DOMAIN," "$plain_list" 2>/dev/null || echo "0") - suffix=$(grep -c "^DOMAIN-SUFFIX," "$plain_list" 2>/dev/null || echo "0") - keyword=$(grep -c "^DOMAIN-KEYWORD," "$plain_list" 2>/dev/null || echo "0") - regex=$(grep -c "^DOMAIN-REGEX," "$plain_list" 2>/dev/null || echo "0") - - echo "Building domain rules for $base: $total rules (DOMAIN=$domain, SUFFIX=$suffix, KEYWORD=$keyword, REGEX=$regex)" + + awk -F, -v name="$base" ' + $1 == "DOMAIN" { domain++ } + $1 == "DOMAIN-SUFFIX" { suffix++ } + $1 == "DOMAIN-KEYWORD" { keyword++ } + $1 == "DOMAIN-REGEX" { regex++ } + END { + total = domain + suffix + keyword + regex + printf "Building domain rules for %s: %d rules (DOMAIN=%d, SUFFIX=%d, KEYWORD=%d, REGEX=%d)\n", \ + name, total, domain, suffix, keyword, regex + } + ' "$plain_list" } print_ip_rule_stats() { local plain_list="$1" local base="$2" - local total v4 v6 - - total=$(grep -c "^[0-9a-fA-F:./]" "$plain_list" 2>/dev/null || echo "0") - v4=$(grep -c "^\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}" "$plain_list" 2>/dev/null || echo "0") - v6=$(grep -c ":" "$plain_list" 2>/dev/null || echo "0") - - echo "Building IP rules for $base: $total CIDRs (IPv4=$v4, IPv6=$v6)" + + awk -v name="$base" ' + /^[[:space:]]*$/ || /^[[:space:]]*#/ { next } + /:/ { ipv6++; next } + { ipv4++ } + END { + printf "Building IP rules for %s: %d CIDRs (IPv4=%d, IPv6=%d)\n", \ + name, ipv4 + ipv6, ipv4, ipv6 + } + ' "$plain_list" } build_domain_plain_and_surge() { @@ -245,31 +250,25 @@ build_domain_json_and_mihomo_text() { } build_domain_binaries_parallel() { - local tmp_dir="$TMP_DOMAIN_DIR" - local singbox_dir="$DOMAIN_SINGBOX_DIR" - local mihomo_dir="$DOMAIN_MIHOMO_DIR" - local jobs="$BUILD_JOBS" + local jobs singbox_count mihomo_count + jobs="$(detect_compile_jobs)" echo "Compiling domain binaries with $jobs parallel jobs..." - # Parallel compile sing-box - if ! compile_domain_singbox_json_dir "$tmp_dir" "$singbox_dir" "$jobs"; then + if ! compile_domain_singbox_json_dir "$TMP_DOMAIN_DIR" "$DOMAIN_SINGBOX_DIR" "$jobs"; then echo "failed to compile sing-box domain binaries" >&2 return 1 fi - local singbox_count - singbox_count=$(find "$singbox_dir" -name "*.srs" -type f 2>/dev/null | wc -l) - echo " ✓ sing-box: compiled $singbox_count rule-sets" + singbox_count="$(find "$DOMAIN_SINGBOX_DIR" -maxdepth 1 -type f -name '*.srs' -size +0c | wc -l | tr -d ' ')" + echo " sing-box: compiled $singbox_count rule-sets" - # Parallel compile mihomo ensure_mihomo_once - if ! compile_domain_mihomo_text_dir "$tmp_dir" "$mihomo_dir" "$jobs"; then + if ! compile_domain_mihomo_text_dir "$TMP_DOMAIN_DIR" "$DOMAIN_MIHOMO_DIR" "$jobs"; then echo "failed to compile mihomo domain binaries" >&2 return 1 fi - local mihomo_count - mihomo_count=$(find "$mihomo_dir" -name "*.mrs" -type f 2>/dev/null | wc -l) - echo " ✓ mihomo: compiled $mihomo_count rule-sets" + mihomo_count="$(find "$DOMAIN_MIHOMO_DIR" -maxdepth 1 -type f -name '*.mrs' -size +0c | wc -l | tr -d ' ')" + echo " mihomo: compiled $mihomo_count rule-sets" } build_ip_plain_and_surge() { @@ -311,51 +310,16 @@ build_ip_json() { } build_ip_binaries_parallel() { - local tmp_dir="$TMP_IP_DIR" - local singbox_dir="$IP_SINGBOX_DIR" - local mihomo_dir="$IP_MIHOMO_DIR" - local jobs="$BUILD_JOBS" - local json_list plain_list json base + local jobs singbox_count mihomo_count + jobs="$(detect_compile_jobs)" echo "Compiling IP binaries with $jobs parallel jobs..." + compile_ip_binary_dirs "$TMP_IP_DIR" "$IP_SINGBOX_DIR" "$IP_MIHOMO_DIR" "$jobs" - # Parallel compile sing-box - json_list="$tmp_dir/.singbox-json-files" - find "$tmp_dir" -maxdepth 1 -type f -name '*.json' -print0 > "$json_list" - if [ -s "$json_list" ]; then - # shellcheck disable=SC2016 - xargs -0 -n 1 -P "$jobs" sh -c ' - out_dir="$1" - json="$2" - base="$(basename "$json" .json)" - sing-box rule-set compile "$json" --output "$out_dir/$base.srs" - ' sh "$singbox_dir" < "$json_list" || { - echo "failed to compile sing-box IP binaries" >&2 - return 1 - } - fi - local singbox_count - singbox_count=$(find "$singbox_dir" -name "*.srs" -type f 2>/dev/null | wc -l) - echo " ✓ sing-box: compiled $singbox_count rule-sets" - - # Parallel compile mihomo - plain_list="$tmp_dir/.mihomo-plain-files" - find "$tmp_dir" -maxdepth 1 -type f -name '*.txt' -size +0c -print0 > "$plain_list" - if [ -s "$plain_list" ]; then - # shellcheck disable=SC2016 - xargs -0 -n 1 -P "$jobs" sh -c ' - out_dir="$1" - plain="$2" - base="$(basename "$plain" .txt)" - mihomo convert-ruleset ipcidr text "$plain" "$out_dir/$base.mrs" >/dev/null - ' sh "$mihomo_dir" < "$plain_list" || { - echo "failed to compile mihomo IP binaries" >&2 - return 1 - } - fi - local mihomo_count - mihomo_count=$(find "$mihomo_dir" -name "*.mrs" -type f 2>/dev/null | wc -l) - echo " ✓ mihomo: compiled $mihomo_count rule-sets" + singbox_count="$(find "$IP_SINGBOX_DIR" -maxdepth 1 -type f -name '*.srs' -size +0c | wc -l | tr -d ' ')" + mihomo_count="$(find "$IP_MIHOMO_DIR" -maxdepth 1 -type f -name '*.mrs' -size +0c | wc -l | tr -d ' ')" + echo " sing-box: compiled $singbox_count rule-sets" + echo " mihomo: compiled $mihomo_count rule-sets" } CONFLICT_BASE_REF="$(resolve_conflict_base_ref)" diff --git a/scripts/lib/rules.sh b/scripts/lib/rules.sh index 1a1db708..0f764b6f 100644 --- a/scripts/lib/rules.sh +++ b/scripts/lib/rules.sh @@ -217,14 +217,30 @@ compile_mihomo_domain_plain_to_binary_artifact() { mihomo convert-ruleset domain text "$plain_list" "$mrs_out" >/dev/null } +assert_compiled_file_count() { + local label="$1" + local expected="$2" + local output_dir="$3" + local pattern="$4" + local actual + + actual="$(find "$output_dir" -maxdepth 1 -type f -name "$pattern" -size +0c | wc -l | tr -d ' ')" + if [ "$actual" -ne "$expected" ]; then + echo "$label produced $actual non-empty artifacts; expected $expected" >&2 + return 1 + fi +} + compile_domain_singbox_json_dir() { local tmp_dir="$1" local singbox_dir="$2" local jobs="$3" local list_file="$tmp_dir/.singbox-json-files" + local expected find "$tmp_dir" -maxdepth 1 -type f -name '*.json' -print0 > "$list_file" - if [ ! -s "$list_file" ]; then + expected="$(find "$tmp_dir" -maxdepth 1 -type f -name '*.json' | wc -l | tr -d ' ')" + if [ "$expected" -eq 0 ]; then return 0 fi @@ -234,7 +250,9 @@ compile_domain_singbox_json_dir() { json="$2" base="$(basename "$json" .json)" sing-box rule-set compile "$json" --output "$out_dir/$base.srs" + test -s "$out_dir/$base.srs" ' sh "$singbox_dir" < "$list_file" + assert_compiled_file_count "sing-box domain compile" "$expected" "$singbox_dir" '*.srs' } compile_domain_mihomo_text_dir() { @@ -242,9 +260,11 @@ compile_domain_mihomo_text_dir() { local mihomo_dir="$2" local jobs="$3" local list_file="$tmp_dir/.mihomo-text-files" + local expected find "$tmp_dir" -maxdepth 1 -type f -name '*.mihomo.txt' -size +0c -print0 > "$list_file" - if [ ! -s "$list_file" ]; then + expected="$(find "$tmp_dir" -maxdepth 1 -type f -name '*.mihomo.txt' -size +0c | wc -l | tr -d ' ')" + if [ "$expected" -eq 0 ]; then return 0 fi @@ -254,7 +274,9 @@ compile_domain_mihomo_text_dir() { plain="$2" base="$(basename "$plain" .mihomo.txt)" mihomo convert-ruleset domain text "$plain" "$out_dir/$base.mrs" >/dev/null + test -s "$out_dir/$base.mrs" ' sh "$mihomo_dir" < "$list_file" + assert_compiled_file_count "mihomo domain compile" "$expected" "$mihomo_dir" '*.mrs' } build_domain_artifacts_from_rule_dir() { @@ -442,7 +464,47 @@ compile_ip_plain_to_binary_artifacts() { SINGBOX_RULE_SET_VERSION="$source_version" \ build_ip_json_from_plain "$plain_list" "$json_out" sing-box rule-set compile "$json_out" --output "$srs_out" + test -s "$srs_out" mihomo convert-ruleset ipcidr text "$plain_list" "$mrs_out" >/dev/null + test -s "$mrs_out" +} + +compile_ip_binary_dirs() { + local tmp_dir="$1" + local singbox_dir="$2" + local mihomo_dir="$3" + local jobs="$4" + local json_list plain_list expected + + json_list="$tmp_dir/.singbox-ip-json-files" + find "$tmp_dir" -maxdepth 1 -type f -name '*.json' -print0 > "$json_list" + expected="$(find "$tmp_dir" -maxdepth 1 -type f -name '*.json' | wc -l | tr -d ' ')" + if [ "$expected" -gt 0 ]; then + # shellcheck disable=SC2016 + xargs -0 -n 1 -P "$jobs" sh -c ' + out_dir="$1" + json="$2" + base="$(basename "$json" .json)" + sing-box rule-set compile "$json" --output "$out_dir/$base.srs" + test -s "$out_dir/$base.srs" + ' sh "$singbox_dir" < "$json_list" + assert_compiled_file_count "sing-box IP compile" "$expected" "$singbox_dir" '*.srs' + fi + + plain_list="$tmp_dir/.mihomo-ip-text-files" + find "$tmp_dir" -maxdepth 1 -type f -name '*.txt' -size +0c -print0 > "$plain_list" + expected="$(find "$tmp_dir" -maxdepth 1 -type f -name '*.txt' -size +0c | wc -l | tr -d ' ')" + if [ "$expected" -gt 0 ]; then + # shellcheck disable=SC2016 + xargs -0 -n 1 -P "$jobs" sh -c ' + out_dir="$1" + plain="$2" + base="$(basename "$plain" .txt)" + mihomo convert-ruleset ipcidr text "$plain" "$out_dir/$base.mrs" >/dev/null + test -s "$out_dir/$base.mrs" + ' sh "$mihomo_dir" < "$plain_list" + assert_compiled_file_count "mihomo IP compile" "$expected" "$mihomo_dir" '*.mrs' + fi } build_ip_artifacts_from_surge_dir() { diff --git a/scripts/tests/test-binary-compile-contract.sh b/scripts/tests/test-binary-compile-contract.sh new file mode 100644 index 00000000..42de158e --- /dev/null +++ b/scripts/tests/test-binary-compile-contract.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$ROOT" + +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT + +# shellcheck source=scripts/lib/rules.sh +source "$ROOT/scripts/lib/rules.sh" + +mkdir -p "$TMP_DIR/domain-in" "$TMP_DIR/domain-out" "$TMP_DIR/ip-out" "$TMP_DIR/bin" +printf '{"version":4,"rules":[{"domain_suffix":["example.com"]}]}' > "$TMP_DIR/domain-in/sample.json" +printf '192.0.2.0/24\n' > "$TMP_DIR/domain-in/sample.txt" + +# A compiler may incorrectly return success without creating output. The batch +# helpers must fail closed instead of relying on the command status alone. +cat > "$TMP_DIR/bin/sing-box" <<'EOF' +#!/usr/bin/env sh +exit 0 +EOF +cat > "$TMP_DIR/bin/mihomo" <<'EOF' +#!/usr/bin/env sh +exit 0 +EOF +chmod +x "$TMP_DIR/bin/sing-box" "$TMP_DIR/bin/mihomo" + +if PATH="$TMP_DIR/bin:$PATH" compile_domain_singbox_json_dir \ + "$TMP_DIR/domain-in" "$TMP_DIR/domain-out" 1 >/dev/null 2>&1; then + echo "test failed: missing sing-box output was accepted" >&2 + exit 1 +fi + +if PATH="$TMP_DIR/bin:$PATH" compile_ip_binary_dirs \ + "$TMP_DIR/domain-in" "$TMP_DIR/domain-out" "$TMP_DIR/ip-out" 1 >/dev/null 2>&1; then + echo "test failed: missing IP binary output was accepted" >&2 + exit 1 +fi + +RULES_COMPILE_JOBS=3 +[ "$(detect_compile_jobs)" = 3 ] || { + echo "test failed: RULES_COMPILE_JOBS was not honored" >&2 + exit 1 +} +RULES_COMPILE_JOBS=0 +if detect_compile_jobs >/dev/null 2>&1; then + echo "test failed: zero compile jobs were accepted" >&2 + exit 1 +fi + +echo "binary compile contract tests passed" diff --git a/scripts/tests/test-ip-normalization.sh b/scripts/tests/test-ip-normalization.sh index 5e5abab3..851cc213 100755 --- a/scripts/tests/test-ip-normalization.sh +++ b/scripts/tests/test-ip-normalization.sh @@ -59,7 +59,6 @@ cat > "$TMP_DIR/mixed.txt" <<'CIDRS' 192.168.1.1/24 192.168.1.0/24 2001:db8::1/32 -not-a-cidr # comment CIDRS @@ -107,9 +106,23 @@ python3 "$ROOT/scripts/tools/normalize-ip-rules.py" \ "$TMP_DIR/merge-b.txt" assert_file_content "$TMP_DIR/merged-dedupe.out" $'10.0.0.0/8\n192.168.1.0/24\n2001:db8::/32\n10.1.0.0/16\n192.168.2.0/24\n2001:db8:1::/48' +cat > "$TMP_DIR/invalid-text.txt" <<'CIDRS' +192.0.2.0/24 +not-a-cidr +CIDRS +if python3 "$ROOT/scripts/tools/normalize-ip-rules.py" single text \ + "$TMP_DIR/invalid-text.txt" "$TMP_DIR/invalid-text.out" >"$TMP_DIR/invalid-text.stdout" 2>"$TMP_DIR/invalid-text.stderr"; then + echo "test failed: invalid effective text-source line should fail" >&2 + exit 1 +fi +if ! grep -Fq "invalid CIDR entry: not-a-cidr" "$TMP_DIR/invalid-text.stderr"; then + echo "test failed: strict text-source parser did not identify invalid line" >&2 + cat "$TMP_DIR/invalid-text.stderr" >&2 + exit 1 +fi + cat > "$TMP_DIR/empty.txt" <<'CIDRS' # comment only -not-a-cidr CIDRS python3 "$ROOT/scripts/tools/normalize-ip-rules.py" single text "$TMP_DIR/empty.txt" "$TMP_DIR/empty.out" diff --git a/scripts/tests/test-rule-overlap-audit.sh b/scripts/tests/test-rule-overlap-audit.sh new file mode 100644 index 00000000..3182f129 --- /dev/null +++ b/scripts/tests/test-rule-overlap-audit.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$ROOT" +TMP_DIR="$(mktemp -d)"; trap 'rm -rf "$TMP_DIR"' EXIT +mkdir -p "$TMP_DIR/domain" "$TMP_DIR/ip" +cat > "$TMP_DIR/domain/sample.list" <<'EOF' +DOMAIN-SUFFIX,example.com +DOMAIN,a.example.com +DOMAIN-SUFFIX,b.example.com +DOMAIN-KEYWORD,example +EOF +cat > "$TMP_DIR/ip/sample.list" <<'EOF' +IP-CIDR,10.0.0.0/8 +IP-CIDR,10.1.0.0/16 +IP-CIDR6,2001:db8::/32 +EOF +if python3 scripts/tools/audit-rule-overlaps.py "$TMP_DIR" --output "$TMP_DIR/report.json" --fail-internal >/dev/null; then + echo 'test failed: audit should reject internal semantic redundancy' >&2; exit 1 +fi +python3 - "$TMP_DIR/domain/sample.list" <<'PY' +import sys +from pathlib import Path +sys.path.insert(0,'scripts/tools') +from domain_rules import compact_classical_domain_file +before,removed=compact_classical_domain_file(Path(sys.argv[1])) +assert (before,removed)==(4,2),(before,removed) +PY +python3 scripts/tools/normalize-ip-rules.py custom-source "$TMP_DIR/ip/sample.list" "$TMP_DIR/ip/sample.compact" +cat > "$TMP_DIR/ip/sample.list" </dev/null +python3 - "$TMP_DIR/report.json" <<'PY' +import json,sys +x=json.load(open(sys.argv[1]));assert not x['domain']['internal_redundancy'];assert not x['ip']['internal_redundancy'] +PY +echo 'rule overlap audit tests passed' diff --git a/scripts/tools/domain_rules.py b/scripts/tools/domain_rules.py index f829e07a..3d1e8668 100644 --- a/scripts/tools/domain_rules.py +++ b/scripts/tools/domain_rules.py @@ -88,6 +88,44 @@ def domain_value_errors( return errors +def compact_domain_rules(rules: list[ParsedDomainRule]) -> tuple[list[ParsedDomainRule], int]: + """Remove rules made redundant by an equal or parent DOMAIN-SUFFIX. + + Keyword and regular-expression rules are intentionally opaque. The first + occurrence order is preserved so canonical output remains deterministic. + """ + suffixes = {rule.value for rule in rules if rule.kind == "DOMAIN-SUFFIX"} + compacted: list[ParsedDomainRule] = [] + removed = 0 + + for rule in rules: + if rule.kind not in {"DOMAIN", "DOMAIN-SUFFIX"}: + compacted.append(rule) + continue + labels = rule.value.split(".") + start = 1 if rule.kind == "DOMAIN-SUFFIX" else 0 + covered = any(".".join(labels[index:]) in suffixes for index in range(start, len(labels))) + if covered: + removed += 1 + else: + compacted.append(rule) + return compacted, removed + + +def compact_classical_domain_file(path: Path) -> tuple[int, int]: + rules, errors = parse_classical_domain_file( + path, + require_canonical=True, + allow_single_label_suffix=True, + ) + if errors: + raise ValueError("\n".join(errors)) + compacted, removed = compact_domain_rules(rules) + text = "\n".join(rule.text for rule in compacted) + path.write_text(text + ("\n" if text else ""), encoding="utf-8") + return len(rules), removed + + def parse_classical_domain_file( path: Path, *, diff --git a/scripts/tools/export-domain-rules.py b/scripts/tools/export-domain-rules.py index 7a2286f7..832dff6f 100644 --- a/scripts/tools/export-domain-rules.py +++ b/scripts/tools/export-domain-rules.py @@ -70,9 +70,14 @@ def count_rule_kinds(rules: list[Rule]) -> dict[str, int]: return counts -def print_platform_skip_summary(name: str, rules: list[Rule]) -> None: +def print_platform_skip_summary(name: str, rules: list[Rule], platform: str | None = None) -> None: counts = count_rule_kinds(rules) - for platform, capability in PLATFORM_CAPABILITIES.items(): + capabilities = ( + {platform: PLATFORM_CAPABILITIES[platform]} + if platform is not None + else PLATFORM_CAPABILITIES + ) + for platform_name, capability in capabilities.items(): skipped = { kind: count for kind, count in counts.items() @@ -80,7 +85,10 @@ def print_platform_skip_summary(name: str, rules: list[Rule]) -> None: } if skipped: details = ", ".join(f"{kind}={count}" for kind, count in sorted(skipped.items())) - print(f"domain summary: {name} skips unsupported rules for {platform}: {details}", file=sys.stderr) + print( + f"domain summary: {name} skips unsupported rules for {platform_name}: {details}", + file=sys.stderr, + ) def print_mihomo_mrs_skip_summary(input_file: Path, rules: list[Rule]) -> None: @@ -397,7 +405,7 @@ def build_surge_lines(rules: list[Rule]) -> list[str]: def build_surge_list(input_file: Path, output_file: Path) -> None: rules = parse_classical_domain_rules(input_file) - print_platform_skip_summary(input_file.stem, rules) + print_platform_skip_summary(input_file.stem, rules, "surge") write_text_lines(build_surge_lines(rules), output_file) @@ -411,7 +419,7 @@ def build_quanx_lines(rules: list[Rule], policy_tag: str) -> list[str]: def build_quanx_list(input_file: Path, output_file: Path, policy_tag: str) -> None: rules = parse_classical_domain_rules(input_file) - print_platform_skip_summary(input_file.stem, rules) + print_platform_skip_summary(input_file.stem, rules, "quanx") write_text_lines(build_quanx_lines(rules, policy_tag), output_file) diff --git a/sources/custom/domain/fakeip-filter.list b/sources/custom/domain/fakeip-filter.list index 632422eb..3c73cf3a 100644 --- a/sources/custom/domain/fakeip-filter.list +++ b/sources/custom/domain/fakeip-filter.list @@ -14,8 +14,6 @@ DOMAIN-SUFFIX,home.arpa DOMAIN-SUFFIX,direct # NTP services -# REGEX patterns below for precise matching of NTP time servers -# Fallback SUFFIX rules added for Surge/QuanX/mihomo (broader match) DOMAIN-REGEX,^time\.[^.]+\.com$ DOMAIN-REGEX,^time\.[^.]+\.gov$ DOMAIN-REGEX,^time\.[^.]+\.edu\.cn$ @@ -28,11 +26,6 @@ DOMAIN-SUFFIX,time.edu.cn DOMAIN-SUFFIX,ntp.org.cn DOMAIN-SUFFIX,pool.ntp.org DOMAIN,time1.cloud.tencent.com -# Fallback for platforms without REGEX support -DOMAIN-SUFFIX,time.windows.com -DOMAIN-SUFFIX,time.nist.gov -DOMAIN-SUFFIX,ntp.aliyun.com -DOMAIN-SUFFIX,ntp.tencent.com # Music services DOMAIN-SUFFIX,music.163.com @@ -67,15 +60,10 @@ DOMAIN-SUFFIX,kk-rays.com DOMAIN-SUFFIX,steamcontent.com DOMAIN-SUFFIX,srv.nintendo.net DOMAIN-SUFFIX,cdn.nintendo.net -# Xbox Live: REGEX for precise matching, SUFFIX fallback for compatibility DOMAIN-REGEX,^xbox\.[^.]+\.[^.]+\.microsoft\.com$ DOMAIN-REGEX,^(?:[^.]+\.){2}xboxlive\.com$ DOMAIN-REGEX,^xbox\.[^.]+\.microsoft\.com$ -# Fallback SUFFIX for Surge/QuanX/mihomo (broader but functional) -DOMAIN-SUFFIX,xboxlive.com -DOMAIN-SUFFIX,xboxservices.com -DOMAIN-SUFFIX,xboxab.com -DOMAIN-SUFFIX,xbox.com +DOMAIN,xnotify.xboxlive.com DOMAIN-SUFFIX,battle.net DOMAIN-SUFFIX,battlenet.com.cn DOMAIN-SUFFIX,wotgame.cn @@ -85,11 +73,7 @@ DOMAIN-SUFFIX,wargaming.net # Development and STUN DOMAIN,proxy.golang.org -# STUN: REGEX for dynamic patterns, common services as fallback DOMAIN-REGEX,^(?:.+\.)?stun(?:\.[^.]+){2,5}$ -DOMAIN-SUFFIX,stunprotocol.org -DOMAIN-SUFFIX,stun.l.google.com -DOMAIN-SUFFIX,stun.syncthing.net # Routers DOMAIN,heartbeat.belkin.com