Simplify `Set` mapping and filtering

[Baptouuuu]

The current implementation is a bit all over the place due to previous constraints.

This issue should attempt to simplify the implementation by reducing where filtering and mapping is done.