Skip to content

Calibration enhancements - Group weighting#29

Merged
baogorek merged 5 commits intomainfrom
calibration-enhancements
Sep 5, 2025
Merged

Calibration enhancements - Group weighting#29
baogorek merged 5 commits intomainfrom
calibration-enhancements

Conversation

@baogorek
Copy link
Copy Markdown
Collaborator

@baogorek baogorek commented Sep 4, 2025

Closes #30

Summary

This PR enhances the calibration module with group-wise loss averaging and improved training output for better interpretability.

Key Changes

  1. Group-wise Loss Averaging: Added support for balancing loss contributions from targets with different cardinalities. This is particularly useful when calibrating with mixed targets like:

    • National-level targets (singletons with large values)
    • State-level age distributions (groups with many bins)

    The new target_groups parameter ensures each group contributes equally to the loss, preventing high-cardinality groups from dominating the optimization.

  2. Improved Training Output: Enhanced verbose output to show:

    • Relative error percentages instead of raw loss values
    • Sparsity statistics (active weights and percentage)
    • Group-averaged errors when using group-wise averaging
    • More interpretable progress monitoring
  3. Simplified Active Weight Detection: Removed the threshold parameter from get_active_weights() - now simply checks for weights > 0.

Example Usage

# Group targets for balanced optimization
target_groups = np.array([
    0, 1, 2,      # 3 national targets (singletons)
    *[3]*18,      # 18 age bins for state 1
    *[4]*18       # 18 age bins for state 2
])

model.fit(
    M, y,
    target_groups=target_groups,  # Balance group contributions
    verbose=True
)

Test Coverage

  • Added comprehensive tests for group-wise averaging
  • Tested edge cases (single group, all singletons, unbalanced groups)
  • Verified that grouping improves balance between different target types

Output Example

Before:

Epoch  100: loss=0.1234, data_loss=0.1000, mse=0.0500, l0=25.00, active=2500, mean_weight=1.234

After:

Epoch  100: mean_group_loss=2.5%, max_error=8.3%, total_loss=0.123, active=2500/10000 (75.0% sparse)

The new output format is more intuitive and directly shows the calibration quality as percentages.

@baogorek baogorek merged commit 5b74d3a into main Sep 5, 2025
4 checks passed
@baogorek baogorek deleted the calibration-enhancements branch January 21, 2026 03:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add group loss balancing for calibration targets

1 participant