Apply _ prefix to internal columns in RTX-KG2 merged graph

[eKathleenCarter]

Prefix all non-Biolink internal fields in the RTX-KG2 merged node and edge tables with _.

• Audit all RTX-KG2 columns
• Update field names with _ prefix if not biolink-compliant field

[IamMikeHelsel]

Design Proposal: Column Normalization for _ Prefix

Summary

Add a normalization step to the validator that automatically prefixes non-Biolink internal fields with _. The validator will handle this transparently, accepting both old (unprefixed) and new (_-prefixed) column names for backwards compatibility.

Problem

RTX-KG2 merged node and edge tables mix Biolink-standard fields with internal/implementation-specific fields. There is no clear convention to distinguish them. Additionally, some TSV headers include KGX type annotations (e.g., all_names:string[]) that are not currently stripped by the validator.

Proposed Changes

1. New module: src/matrix_validator/normalize.py

A normalize_columns(df, file_type) function that:

1. Strips type annotations (:string[], :int, etc.) from column names
2. Classifies each column as Biolink-standard or internal
3. Prefixes internal columns with _
4. Logs all transformations
5. Returns the modified DataFrame and a changelog

Biolink-standard columns (left unchanged):

Nodes	Edges
id	subject
name	object
category	predicate
description	primary_knowledge_source
iri	aggregator_knowledge_source
publications	knowledge_level
equivalent_identifiers	agent_type
	qualified_predicate
	qualified_object_aspect
	qualified_object_direction
	domain_range_exclusion
	publications
	id
	original_subject
	original_object
	subject_aspect_qualifier
	subject_direction_qualifier
	object_aspect_qualifier
	object_direction_qualifier
	upstream_data_source

Internal columns that get _ prefixed:

Original (nodes)	Normalized
all_names / all_names:string[]	_all_names
all_categories / all_categories:string[]	_all_categories
equivalent_curies / equivalent_curies:string[]	_equivalent_curies
:LABEL	_:LABEL

Original (edges)	Normalized
publications_info	_publications_info
kg2_ids / kg2_ids:string[]	_kg2_ids
:TYPE	_:TYPE
:START_ID	_:START_ID
:END_ID	_:END_ID

2. Validator integration

• Call normalize_columns() immediately after reading TSV data, before any schema or data validation
• Update validator_polars.py schemas (create_nodes_schema, create_edges_schema) to expect _-prefixed internal columns
• Update validator_purepython.py column sets (VALID_KGX_NODE_COLUMNS, VALID_KGX_EDGE_COLUMNS) to use _-prefixed names
• Update check_for_superfluous_columns() — _-prefixed columns are recognized as internal, not flagged as superfluous

3. Configuration updates

• Update config_rtxkg2.toml column references in attribute checks to use _-prefixed names where applicable

4. Backwards compatibility

The normalization function handles both old and new formats transparently:

• kg2_ids:string[] → strip annotation → kg2_ids → not Biolink → _kg2_ids
• _kg2_ids → already prefixed → leave as-is
• publications:string[] → strip annotation → publications → is Biolink → leave as-is
• publications → is Biolink → leave as-is

Data flow

TSV file (raw column names)
  → normalize_columns():
      1. strip type annotations (:string[], etc.)
      2. classify column as Biolink or internal
      3. prefix internal columns with _
      4. log changes
  → schema validation (updated schemas)
  → data validation (Biolink columns only)
  → output report

Open questions

1. Should the normalization step also be available as a standalone CLI command (e.g., matrix-validator normalize) for users who want to pre-process their data?
2. Are there additional Biolink-standard columns that should be added to the allowlist?
3. Should the changelog/log of renamed columns be included in the validation output?