[Feature Request]: Pipeline-level field type defaults, so type and length don't have to be re-entered in every transform

[sfkeller]

What would you like to happen?

Description

As a pipeline developer, I want to define the data type, length and precision of a field once per pipeline, so I can avoid re-entering the same definition in every transform that creates or redefines that field.

Current situation

Fields that only pass through a transform keep their metadata. But every transform that creates or redefines a field (Calculator, Formula, JavaScript, Add constants, Concat fields, Value mapper, Regex evaluation, and others) needs its own type, length and precision in its dialog.

In a pipeline with around 20 transforms, a field such as CUSTOMER_NAME that should be String(80) has to be configured in almost every one of them. For each transform I must open the dialog, go to the field list and enter the type again. This is repetitive and error-prone. Inconsistent lengths can silently lead to truncation or mismatches further downstream.

Proposal

Add an optional pipeline-level field definition (a "field schema" or "default field types") that maps a field name to a type, length and precision. When a transform creates or redefines a field and no explicit type is set, it would take the definition from this mapping. Possible refinements:

• Pre-fill the type, length and precision in the transform dialog from the mapping, so the user can still see and override them.
• Offer a "learn from first occurrence" option, so the first explicit definition of a field becomes the default for later transforms in the same pipeline (this is how e.g. FME (Safe Inc.) handles attributes).
• Keep explicit settings in a transform as the highest priority, so existing pipelines behave exactly as before.

Goal

Define a field's type once and reuse it consistently across the pipeline, with fewer clicks, less duplicated configuration and fewer type mismatches.

Current workarounds (and their limits)

• A Select values transform with a Meta-data tab right after the source. This only helps for pass-through fields, not for fields created later.
• Overwriting existing fields instead of creating new ones. This is not always possible.
• Variables such as ${NAME_LEN} in the length column. This still requires touching every transform once.
• Metadata injection. This is heavy for a simple pipeline.

Additional information

• Hop version: 2.18.1 (latest release at time of writing)
• Java version: Java 17 (the minimum Hop requires)
• OS: Windows 11 Enterprise
• Happy to discuss the design in the dev chat or on the mailing list first, and to test a prototype.

Issue Priority

Priority: 3

Issue Component

Component: Hop Gui

[hansva]

Sorry, I don't understand the issue here. You only have to define the output type for new or changed fields, never for all existing fields.
This makes sense, as you manipulated the field and have to make sure it complies with the new rules. And most of those transforms have a "get fields" button or it's just part of configuring the calculation (one time thing)

[sfkeller]

Thanks for the quick check. I'll compile specific transforms.

[mattcasters]

Hang on @hansva. I implemented exactly this functionality in my ProjectDataHopper/hopper-edw repo.
The fanciness comes into play when you're dealing with data sources that don't carry length/precision for example. Iceberg, JSON, XML, CSV, ...

Have a look if this is what you had in mind @sfkeller : https://github.com/ProjectDataHopper/hopper-edw/blob/main/docs/data-type-mappings.adoc

This is obviously specific for the Hop Source Models (.hsm files). Perhaps we can't immediately apply it at the pipeline or project level but I'm thinking it might be possible to wire it in at the transform level for these input transforms that suffer the most from the data type fuzziness.

[sfkeller]

Have a look if this is what you had in mind @sfkeller : https://github.com/ProjectDataHopper/hopper-edw/blob/main/docs/data-type-mappings.adoc

That's close to what I understood you there at first glance.

Let me try to rephrase my main issue from the perspective of someone who uses FME (Safe Software) and is now migrating to Hop by creating a Hop pipeline. FME does not require a type when a transformer creates an attribute. Types are provided by the reader or, optionally, by the AttributeManager or AttributeCreator (similar to the "Select Values" and/or "Formula" Hop Transformers). These types then propagate downstream in FME and are picked up automatically by the writer.

[hansva]

Well, I guess in some transforms we could try and guess the type the fields will have but it will be transform specific improvements. I just don't like blanket tickets where someone says, can a developer go over the 300 transforms we have and do X. For example in the calculator we could say, if you are adding 2 integer values the output field will likely be an integer. It would save 1 click.

But it will have to be specific feature requests for specific transforms.

[mattcasters]

Of-course I agree with that @hansva. Even in the example above it's applied in specific places of the source modeler, and with the proper UI elements to show how the types are derived. Otherwise, things get unpredictable and chaotic pretty quickly.

[sfkeller]

Dear Hans, Thank you again for your triage work. I’m not as familiar with Hop as I am with other open-source software projects.
While I won't be able to test all 300 transforms, I'd be happy to identify a few key ones that could be improved using the method described here. In one use case involving a reasonably complex pipeline that I’ve heard about, this feature would save several dozen clicks in Hop.

[mattcasters]

Please keep in mind @sfkeller , that very few people work with GIS/Geospatial tools like FME so we don't have a reference point. Perhaps it's easier if you gave a concrete example of a transform where you saw the need.

[hansva]

And I don't expect that work from you either, let's focus on the quick wins and iterate!