Skip to content

CategoricalImputer fails when fill_value is already a category #1113

Open

Description

@AryanRG

CategoricalImputer raises when fill_value is already present in a pandas categorical dtype, instead of filling the missing values.

import pandas as pd
from feature_engine.imputation import CategoricalImputer

X = pd.DataFrame({
    "region": pd.Categorical(
        ["east", None, "west"],
        categories=["unknown", "west", "east"],
    )
})
CategoricalImputer(fill_value="unknown", variables=["region"]).fit_transform(X)

This raises ValueError: new categories must not include old categories: {'unknown'}. I expected ['east', 'unknown', 'west'], with the categorical dtype preserved. It also fails with an ordered categorical; the equivalent object column works.

Reproduced on current main (d866312), Python 3.12, pandas 2.3.3 and scikit-learn 1.7.2. This is related to #1095, but #1096 changes RareLabelEncoder, not this imputer's separate add_categories call.

I'm happy to contribute the fix and regression tests, and I'm planning to work on this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions