Add the Derived Property Element specification - #23
Open
jwrosewell wants to merge 9 commits into
Open
Conversation
Describes DerivedPropertyElement as a flow element, its components, the three script sources, validation, the pipeline check for source property availability, absent and invalid source properties, per request processing, logging and exposed metadata, configuration and a dotnet example. Drawn from the DerivedProperties design note, sections 1, 4, 6 and 7, and links to the format 1 reference in the derived-properties repository as the normative description of the script format. Produced with AI assistance on 1 September 2026 and needs human review.
The new page was not reachable from pipeline-specification/README.md, where every other element specification is listed. Produced with AI assistance and needs human review.
James Rosewell instructed on 1 September 2026 that the derived property engine must not be over engineered, because it is for simple derived properties whilst complex ones are written as ordinary flow elements in code the way that already works today. Required and optional properties therefore do not matter, as a property is either present or it is not. The page now says that where every source property a script names is available the checks and rules run and a value is chosen, and where any one of them is absent the element writes a value with no value whose message names every property that could not be read and what its source element said about each. The message shape is included with the count reading "1 source property was not available" and "<n> source properties were not available", which is the one wording change from the previous draft. Removed from the page are the required and optional split with the Optional key, the Present operator, the counts of evaluated checks, the unknown third state and the Three-valued term, the pipeline check bullet and the build log line that only optional properties needed, and the optional-property branch of the absent property behaviour. Added are a statement that the element is for properties which follow simply from properties already in the flow data whilst anything more involved is written as a flow element in code, a note that evaluation is two-valued, and a note that every script ends in an Else rule so a script always chooses a value once its source properties have been read, which leaves DefaultValue as metadata that nothing reads while a request is processed. The example now names the four source properties the HumanConfidence script reads today and the three values it can return, because naming a property in a script makes the property necessary and the three properties proposed by Pearl 1484, 1486 and 1488 are not in the data yet. Follows the fourth revision of the normative design at 51D-Task-Management/DerivedProperties/DESIGN.md sections 1, 2, 3, 4 and 7. Written with AI assistance and it needs human review.
…ntation Run book 03 has no definition of the example Node, Java, Python and PHP each owe until this file names one, so the four points the .NET example already shows are written down as the requirement. Checked against Examples/DerivedProperty in pipeline-dotnet before being written, so every point named here has code behind it. The order is left to each language because the .NET example runs configuration first and code second. Produced with AI assistance and needs human review.
The page described the Element as a self-hosted Element only, and said nothing about how the 51Degrees cloud serves a derived Property, which is the part run book 04 depends on. The response shapes here were taken from a test rather than written from memory. JsonBuilderIntegrationTests in pipeline-dotnet pins both the value case and the no-value case, so the two JSON examples have code behind them. Produced with AI assistance and needs human review.
A pipeline reports the elements it holds flattened, with the members of a group that runs in parallel placed after every element at the top level, so a position in that list does not say what ran before what. An implementation that judged position failed pipelines that were correct, including the 51Degrees cloud pipeline, which runs the device detection and IP intelligence engines in one such group and adds this element after it. The failure told the reader to move the element after elements it was already after. The build still fails where no element in the pipeline supplies a source property at all, which does not depend on order. Where an element runs is left to the one rule of the format, being that a source property that cannot be read leaves the derived property with no value and a message naming what was missing. James Rosewell decided this on 2 September 2026.
jwrosewell
added a commit
to 51Degrees/pipeline-dotnet
that referenced
this pull request
Sep 2, 2026
The build check worked out "earlier in the pipeline" from the element list a pipeline reports, which flattens a group that runs in parallel by putting its members after every element at the top level. A position in that list therefore does not say what ran before what. This is not a corner case. The 51Degrees cloud Main pipeline runs CloudDeviceDetectionHashEngine and CloudIpiOnPremiseEngine in one such group, and the derived property element belongs after that group, so the check read the engines as coming after the element and failed the build with advice nobody could follow, being to move the element after elements it was already after. The cloud integration could not have been built with the check as it was, even with this pull request approved. PipelineCheck_SuppliersInAParallelGroupBuildAndRun is that shape, being two sources in a parallel group and the element added after it. Put the old check back and it fails with the real message: The script 'Strict' needs the property 'device.IsVisible', which is supplied by StubSourceElement, placed after the derived property element rather than before it. Move the derived property element after StubSourceElement. That was run rather than reasoned about. What the build still refuses ---------------------------- A source property no element in the pipeline supplies at all, which does not depend on order and is the failure the design asked for. Two elements writing the same derived property. Two elements replacing the same property of another element. An override whose target property no element produces, or produces as another type. What is left to the request --------------------------- An element ordered wrongly, or sharing a parallel group with a source it reads, now produces no value rather than failing the build, and the message on the first request names the property that was missing. That is the one rule of the format doing the work it already does for a property excluded from a resource key or one that JavaScript has not populated yet, which is what James Rosewell decided on 2 September 2026. The two tests that pinned the old behaviour now assert the new one, and they assert the outcome rather than the rule, being that the pipeline builds and the value is absent with the property named, and that an override whose target element runs later leaves that element's own value standing. DESIGN.md section 4.3 and the specification page were changed first, as the play book requires. 51Degrees/specifications#23 150 tests pass, up from 149 with the parallel pipeline test added here.
…rty-element # Conflicts: # pipeline-specification/README.md
Automation51D
pushed a commit
to 51Degrees/pipeline-dotnet
that referenced
this pull request
Sep 3, 2026
…T: Add DerivedPropertyElement, a shared script driven derived property engine'
* Add FiftyOne.Pipeline.DerivedProperty, the derived property element
Computes new properties from properties other elements have already put
into the flow data, following a script written in YAML or JSON. The
element holds no data file, makes no request and uses no resource key,
so it extends FlowElementBase in the shape of the Translation element.
- The shared script format lives in the derived-properties repository,
added here as a submodule at
FiftyOne.Pipeline.Elements/FiftyOne.Pipeline.DerivedProperty/Scripts.
Every script in it is embedded in the package and an MSBuild target
generates the BuiltInScript enumeration from whatever is there, so a
new script needs no change to this project and an absent submodule
still builds.
- Scripts reach the builder three ways, being a built in name, a file
path with wildcards, or a string from code, and the three may be mixed.
Configuration binds Scripts and ScriptFiles.
- Validation collects every fault and raises one exception listing them
all, each with the script, the source, a path such as
Rules[3].When.All[1], a line and a plain message.
- AddPipeline checks that the rest of the pipeline supplies every
required source property, naming an element placed too late where
there is one, and refusing two elements that write the same property.
- The element writes one value per script into the derived element data,
or one value with no value carrying a message naming every absent
required property.
The conformance runner reads the case files from the submodule and runs
them, which is how this implementation is proved to agree with the
JavaScript reference. It passes with a notice when the folder is empty,
so a change in the public repository can never break this build.
Produced with AI assistance and needs human review.
* Move the scripts submodule to the settled format, and tidy the tests
The submodule now points at the derived-properties commit that fixes int
at a signed 32 bit whole number and refuses two keys differing only in
case, both of which this implementation already did and the JavaScript
reference did not. The conformance runner now reads 22 value cases and
25 rejection cases and passes all of them.
Also takes the MSTest 4 analyzer suggestions in the element tests, which
give better failure messages without weakening any check.
Produced with AI assistance and needs human review.
* REFACT: Simplify the derived property element to the two valued model
James Rosewell instructed that the engine must not be over engineered,
because it is for simple derived properties whilst complex ones are
written as ordinary flow elements in code the way that already works
today. His words were that required and optional properties do not
matter, since a property is either present or it is not, and that where
one is not present the derived property has no value and the message
says what was missing and which source it came from. Where every named
property is present, the rules are evaluated and a value is chosen.
That removes the Optional top level key and the whole required and
optional split, the Present operator and the presence condition, the
Evaluated aggregate, the unknown third state everywhere, and aggregate
objects used as a Then or an Else value. DerivedState is gone and a
condition now answers with a plain bool, All is true when every member
is true, Any when at least one is, and Not inverts. DerivedRuleValue is
gone too, because a Then and an Else are literals and nothing else.
Two removals follow from the same instruction rather than being asked
for directly. Every script must now end in an Else, which is a new
validation fault at the path Rules, and that deletes the runtime path
for no rule having matched together with its message. Output.DefaultValue
stays as metadata carried through to DerivedPropertyMetaData, still
validated against ValueType and Values, and nothing reads it while a
request is being processed.
The summary sentence of the no value message drops the word required, so
it now reads "1 source property was not available" and "N source
properties were not available". Everything else in the message is
unchanged, including the usual causes sentence and the per property
reason shapes, because that wording is the contract between the five
language implementations.
The pipeline check of the design section 4.3 now has one outcome rather
than two. Every property a script names is needed, so a source property
with no supplier fails the pipeline build naming the property and any
element placed later that would supply it, and the information level log
line for an optional property with no supplier is deleted.
The conformance runner now builds a source element for every element data
key the script names rather than only the keys a case names, so that a
case leaving a property out exercises the value being absent on the
request rather than failing the pipeline build. A property with no
supplier anywhere is a rule about assembling a pipeline rather than about
evaluating a script, and DerivedPropertyElementTests covers it.
HumanConfidence.canonical.json was regenerated by running the JavaScript
reference at tools/canonical.mjs of the derived-properties repository
over the rewritten HumanConfidence script.
The example was rewritten because it used the Optional list and returned
a count of evaluated checks. It now shows a request answered normally and
a request that has no value because one source property was missing,
which is the behaviour a reader needs to see.
* Stop a count being compared with another count, and move the submodule
Two changes that bring the .NET element level with the JavaScript
reference after the format was cut back to the simple model.
The reference stopped accepting a nested aggregate operand such as
{ Failed: Checks, Gt: { Passed: Checks } } after this element had already
been written, so the two disagreed on a script that one accepted and the
other refused. .NET now refuses it too, with the same message, which is
"an aggregate is compared with a whole number, found ...". Comparing one
count against another is a feature with no user, and the format is not a
programming language. DerivedAggregateComparison loses its Right
property, the writer loses the branch that printed the nested form, and
the validator loses the branch that read it.
The Scripts submodule moves from 39d6418 to 673361e, which is the
reference repository after the same simplification. That commit deletes
HumanConfidenceEvaluated, rewrites HumanConfidence to name only the four
source properties that are in the data today, and takes the rejection
cases from 25 to 29. Six conformance tests were failing only because the
submodule still held the old scripts, and they pass on the new commit.
FiftyOne.Pipeline.Elements.sln passes with 273 JsonBuilder, 41
Translation, 134 DerivedProperty and 71 JavaScriptBuilder tests, and the
examples project passes its 5.
* Move the scripts submodule to the settled HumanConfidence
The submodule moves from 673361e to 058fc8c, which carries the three
decisions James Rosewell took on 1 September 2026. BrowserReleaseAge is
in months so Lt: 2 means two months, the data keeps supplying -1 for an
unrated IP address so the script absorbs it through an Any rather than
counting a failed check against the request, and the Unknown value is
gone along with the DefaultValue that named it and the IsMandatory flag
that a property with no value cannot honestly carry.
HumanConfidence.canonical.json is regenerated from that script by the
JavaScript reference, which is what HumanConfidenceMatchesTheJavaScript
Reference compares against, so the two implementations are still proved
to agree on the shipped script rather than assumed to.
No library or test code changed. The conformance runner reads 14 value
cases and 29 rejection cases from the submodule and passes them all.
FiftyOne.Pipeline.Elements.sln passes with 273 JsonBuilder, 41
Translation, 134 DerivedProperty and 71 JavaScriptBuilder tests.
* Pin the scripts submodule to the pushed derived-properties tip
51Degrees/derived-properties now exists as a private repository and main
is pushed, so the relative submodule URL ../derived-properties resolves
against the pipeline-dotnet remote for the first time. The submodule
checkout moves off the local working path and onto GitHub, and the pin
moves to c52ce1f, which is the tip of main.
c52ce1f changes documents only, so no script and no case moved and the
tests are unaffected. The pin is at the tip rather than at the last
commit that touched a script, so that anyone cloning gets the documents
that describe the scripts being embedded.
FiftyOne.Pipeline.Elements.sln passes with 273 JsonBuilder, 41
Translation, 134 DerivedProperty and 71 JavaScriptBuilder tests.
* Move the scripts submodule onto the rebuilt derived-properties history
The derived-properties history was rebuilt to remove a paragraph that
named an unreleased feature, because the repository is intended to go
public and the paragraph would have gone with it. Every commit id changed
as a result, so the pin this repository held at c52ce1f no longer exists.
The pin moves to a6658db, which is the tip of the rebuilt main. The
content is unchanged, and the tree at the tip is byte for byte what it
was, so no script, case or document moved. Only the history differs.
a6658db also records that Ge: 8 on the HumanProbability scale is
confirmed, which was the last unsettled threshold in HumanConfidence.
FiftyOne.Pipeline.Elements.sln passes with 273 JsonBuilder, 41
Translation, 134 DerivedProperty and 71 JavaScriptBuilder tests, and the
conformance runner reads 14 value cases and 29 rejection cases from the
submodule.
* Address the verification review of the derived property element
Four changes from a review of the .NET work before the cloud and further
language work starts. Nothing here changes what the element computes.
Pin the JSON builder contract. JsonBuilderIntegrationTests builds a
pipeline of a source element, this element and the JSON builder, and
checks both that a value reaches derived.confidence and that a missing
source property reaches derived.confidencenullreason. The cloud serves a
derived property by putting the JSON builder after this element, so that
JSON is the contract between this element and every cloud customer, and
nothing held it. The JSON builder reaches this element only because it
walks every element data rather than only the aspect engines, which is a
detail of the JSON builder that could have changed without anyone
thinking about this element.
Writing that test found that the JSON builder throws where no sequence
number was set, so SequenceElement has to be earlier in the pipeline. The
cloud already has it first, so nothing needs doing there, but it is worth
knowing before building a cut down pipeline.
Add Examples.DerivedProperty to FiftyOne.Pipeline.sln. Every other
example project is listed there. This one built only through the project
reference from FiftyOne.Pipeline.Examples.Tests, so CI passed while the
project was invisible to anyone opening the solution.
Name the element and the example in README.md, in the elements list and
in the examples table, where both were missing.
Drop the unused position parameter from CheckForCollisions in
DerivedPropertyElement. Two elements writing the same derived property
collide wherever they sit, because both write the one element data held
under the derived key, so the parameter claimed that ordering mattered
when the body never read it.
Verified with dotnet test on FiftyOne.Pipeline.Elements.sln, 41
Translation, 136 DerivedProperty, 273 JsonBuilder and 71
JavaScriptBuilder tests, 0 failed, and a dotnet build of
FiftyOne.Pipeline.sln with 0 errors to prove the solution file edit.
Produced with AI assistance and needs human review.
* Move the scripts submodule to the cleaned repository
Two changes in derived-properties, taken together here.
The repository was audited before being made public. The Checks workflow
had never once passed, because it pinned Node 20 and quoted the test
glob, and node only expands a test glob itself from version 22.
tools/check-metadata.mjs held two raw NUL bytes so git treated it as
binary. Twelve links to a private repository named three unreleased
detection properties.
Then James Rosewell settled that a script's Output block is the one
definition of its property, that the repository must not mention any
repository that is not public, and that FiftyOne.MetaData will read the
scripts rather than a second copy being kept. The check that compared the
two, its tool and its token are gone.
Only comments changed in scripts/HumanConfidence.yaml, so the canonical
JSON is unchanged and the .NET conformance and writer tests still pass.
Verified with dotnet test on the derived property test project, 136
passed and 0 failed, against this pin.
Produced with AI assistance and needs human review.
* Point the scripts submodule at the recreated repository
51Degrees/derived-properties was deleted and recreated with a single
initial commit, so that its history carries none of the back and forth
from building it and none of the objects the earlier force push left
reachable. Every commit id therefore changed, and the pin moves to the
tip of the new history.
The repository is now public, which is what the submodule needed. Cloning
this repository with --recurse-submodules works for anyone, and the
scripts embedded in the package are public by intention rather than by
accident.
* Let a script replace the value of a property that already exists
A script whose Output.Name carries an element data key prefix, such as
device.IsCrawler, now replaces that property's value rather than creating
one of its own. A bare name still creates derived.<Name> as before, so
the choice is written in the script and does not depend on what the
pipeline happens to hold.
Where any source property the script names is missing, an override writes
nothing at all and the value the owning element produced survives
untouched, with no message. That is the whole reason the mode exists. A
script has proof where the owning element has inference, and where the
proof is absent the inference stands. A verified agent signature proves
an AI is present, whilst Crawler.IsArtificialIntelligence is, by its own
description, a judgement from what a crawler controller published.
The build fails rather than the request wherever it can. The named
element must be earlier in the pipeline and must publish the property,
and the values the rules choose must satisfy the property's declared
type, unwrapping IAspectPropertyValue<T> before comparing. That last
check matters because IsCrawler is a bool whilst
IsArtificialIntelligence is a string.
An override defines nothing, so it is not advertised in Properties. The
definition belongs to the element that owns the property, which keeps one
definition per property.
Two defects in the half of this written earlier are fixed here.
DerivedScriptValidator.WithDependencies rebuilt the output metadata after
collecting the source properties and dropped the element data key, so
device.IsCrawler reached the element as a plain derived.IsCrawler and no
override could ever have worked. Everything downstream of it was correct
and inert. A test now pins the prefix reaching the model.
The duplicate name check compares qualified names, which is right, but it
had started printing the qualified name in every message, so a create
script reported derived.Same where the existing test expected Same. The
comparison is unchanged and the message now prints the qualified name
only for an override, matching how the canonical writer prints each.
Verified with dotnet test on FiftyOne.Pipeline.Elements.sln: 41
Translation, 147 DerivedProperty, 273 JsonBuilder and 71
JavaScriptBuilder, 0 failed.
* FIX: Four things an independent review found
A review by an agent given no context on this work, and told to prove
rather than suspect, found ten things. These are the four worth changing
now. The rest are recorded on the pull request.
A lazily loaded source engine lost the derived property its value
------------------------------------------------------------------
CompiledScript read each source property with IElementData.TryGet, whose
own comment in AspectDataBase says it deliberately reads the dictionary
as it stands so that it does not wait for lazy loading. An engine
configured with SetLazyLoading has its aspect data added to the flow data
immediately and filled on another thread, so the element read an empty
element data and every request lost its value. It is a race, so on a fast
engine it would have worked sometimes, which is worse than failing
outright, and the pipeline build check could not catch it because the
engine does advertise the property.
It now reads through the indexer, which waits, and which is what the
translation engine already does with its own source values. That also
gives a real reason for a property the engine does not have, being what
the missing property service says, rather than the fixed sentence.
Lazy loading that times out or is cancelled is reported as a property
that could not be read rather than thrown out of an element that is only
reading, because that is a state the format already has.
Element_WaitsForALazilyLoadedSourceProperty covers it, with a source
engine that takes 300 milliseconds. Put the old TryGet back and the test
fails with the no value message, which was checked rather than assumed.
Two elements could replace the same property with nothing said
--------------------------------------------------------------
CheckForCollisions skipped every override, on the reasoning that an
override writes nothing under the derived key and so cannot collide.
That is true of the derived key and false of the property being
replaced. Two elements each replacing device.IsCrawler both wrote the
device element data and whichever ran last silently won.
An override is not advertised in an element's properties, so the
collision is invisible to the check that catches two elements creating
the same derived property. The new check asks the other derived property
elements what they replace, which is why it is a separate method.
PipelineCheck_TwoElementsReplacingOnePropertyFails covers it.
The submodule commit is now actually logged
-------------------------------------------
BuiltInScripts.SubmoduleCommit said in its own documentation that "the
element writes the value to the build log so that anyone holding the log
can find the exact scripts that ran". Nothing read it anywhere in the
repository. It is now logged once per element that holds a built in
script, which makes the documentation true rather than deleting a
sentence that describes something worth having.
The example ends with somewhere to go
-------------------------------------
Examples/DerivedProperty printed no links at all, so somebody watching
the demo had nowhere to follow it up. It now ends with a Find out more
section in the shape the agent signature example uses, naming the shared
scripts repository, the published script tester, this repository and the
two engines that supply the properties a script reads. The UTM lint from
common-ci passes over the whole repository with -Campaign pipeline-dotnet.
Verified
--------
149 derived property tests pass, up from 147 with the two added here, and
the derived property example test passes.
* Ship HumanConfidence 0.4.0 rather than 0.2.0
The scripts submodule was pinned 16 commits behind the shared
repository, so this package shipped a version of the script that
predates the decisions James Rosewell has since made about it. 0.2.0
required a human probability of 8 where the published description puts
the line for high confidence at 7, treated a browser as current for 2
months rather than 24, and read four source properties. 0.4.0 reads
eight, adding the three that say whether the browser is automated or its
window was in view, and the connection type, and it records why an
unknown connection is Medium rather than High.
HumanConfidence.canonical.json is regenerated from the JavaScript
reference by the command in the doc comment of the test that reads it,
so the two languages are compared on the script that now ships rather
than on the one that used to.
The 149 derived property tests pass, including the cross language
comparison and the 14 conformance cases the submodule carries, and the
derived property example test passes.
* FIX: Do not judge where an element sits in the pipeline
The build check worked out "earlier in the pipeline" from the element
list a pipeline reports, which flattens a group that runs in parallel by
putting its members after every element at the top level. A position in
that list therefore does not say what ran before what.
This is not a corner case. The 51Degrees cloud Main pipeline runs
CloudDeviceDetectionHashEngine and CloudIpiOnPremiseEngine in one such
group, and the derived property element belongs after that group, so the
check read the engines as coming after the element and failed the build
with advice nobody could follow, being to move the element after
elements it was already after. The cloud integration could not have been
built with the check as it was, even with this pull request approved.
PipelineCheck_SuppliersInAParallelGroupBuildAndRun is that shape, being
two sources in a parallel group and the element added after it. Put the
old check back and it fails with the real message:
The script 'Strict' needs the property 'device.IsVisible', which is
supplied by StubSourceElement, placed after the derived property
element rather than before it. Move the derived property element after
StubSourceElement.
That was run rather than reasoned about.
What the build still refuses
----------------------------
A source property no element in the pipeline supplies at all, which does
not depend on order and is the failure the design asked for. Two
elements writing the same derived property. Two elements replacing the
same property of another element. An override whose target property no
element produces, or produces as another type.
What is left to the request
---------------------------
An element ordered wrongly, or sharing a parallel group with a source it
reads, now produces no value rather than failing the build, and the
message on the first request names the property that was missing. That
is the one rule of the format doing the work it already does for a
property excluded from a resource key or one that JavaScript has not
populated yet, which is what James Rosewell decided on 2 September 2026.
The two tests that pinned the old behaviour now assert the new one, and
they assert the outcome rather than the rule, being that the pipeline
builds and the value is absent with the property named, and that an
override whose target element runs later leaves that element's own value
standing.
DESIGN.md section 4.3 and the specification page were changed first, as
the play book requires.
51Degrees/specifications#23
150 tests pass, up from 149 with the parallel pipeline test added here.
* FEAT: Say once, in the log, when an element is ordered wrongly
There is no accounting for a pipeline built in the wrong order, and the
build cannot catch it, so the next best thing is to say so clearly the
first time it happens and then be quiet about it.
Two changes, and neither adds a check.
The message now names the cause
-------------------------------
The usual causes sentence named four things that leave a source property
unreadable and did not name the one the build cannot catch, being the
element that supplies the property added after this one rather than
before it. It does now, in the shared repository and here, kept word for
word the same because every language prints it.
The log says it once
--------------------
Where a request carries no element data at all under a key the pipeline
said some element writes, the element writes one error naming the script,
the property, the element data and what to do, and never writes it again.
That case and only that case is reported. The pipeline build refuses a
source property no element supplies, so reaching it means an element does
supply the key and did not run first. A property that is simply not there
on a request, such as one the 51Degrees JavaScript has not populated yet,
does not come through it at all, because the element data is present and
the property inside it is not. That is an ordinary thing to happen, it is
already reported in the value's own message, and an error in the log for
it would be a false alarm on the first request of every session.
Reported once rather than per request because a pipeline built the wrong
way round stays that way for the life of the process, and the flag is set
through Interlocked because requests run on many threads at once.
PipelineCheck_SupplierAfterTheElementLeavesNoValue now processes three
requests and asserts exactly one error was logged, so it would fail both
if the report were dropped and if it repeated.
150 tests pass. The submodule is pinned at the matching commit in the
shared repository, a9f5ef8.
* FIX: The packaged submodule commit could be a previous pin
BuiltInScripts.SubmoduleCommit is the answer to which scripts a package
holds, and the element now writes it to the log, so it being wrong is
worse than it being absent.
The target that generates it had Inputs and Outputs, so it ran only when
a file it named was newer than the generated file. Moving the submodule
to another commit does not always make any of those newer, because the
scripts themselves can be identical between two commits of the shared
repository, which is exactly what happened here: the commit that added a
sentence to the usual causes wording changed no script.
Found by packing the element, loading the packed assembly and reading the
field, which gave the commit of the pin before last. That is the whole
value of building the package locally rather than assuming it is right.
The target now runs on every build. It costs one git rev-parse and one
read of the template, and WriteOnlyWhenDifferent means an unchanged file
is not touched, so nothing downstream rebuilds.
Proved by packing again and reading the field back out of the assembly,
which now gives a9f5ef8, matching git submodule status. 150 tests pass.
* TEST: Give the held fetch test leeway for a busy runner
The nightly build of 3 September 2026 failed on macOS in
HeldFetchReadsTimeoutInsideBudgetAndIsFinishedLater, on both open pull
requests, at 314ms and 532ms against a 200ms budget with 100ms of
leeway. The proof the test exists for is untouched, because the fetch
is still held when the request comes back, so reading Timeout with
DirectoryPending already shows the request gave up rather than waited
for the agent. The elapsed bound only guards against waiting far past
the budget, and now allows two seconds, matching the other timing
assertions in this file, which use two, three and five second bounds.
* Take the script at 1ee9c16, whose category is Human Detection
common-metadata pull request 424 moves HumanConfidence out of the General
category and into Human Detection, shared with Location.HumanProbability,
and the script's Output block followed in derived-properties 1ee9c16. The
submodule pin moves so the package embeds the current definition, and the
JavaScript reference output is regenerated with the documented command,
which is the only other place the category appears in this repository.
All 150 tests pass, including the one that compares the canonical form of
the embedded script against that reference.
The cloud section said the element checks its own position and fails the build where it sits too late. That was written before the design settled and it contradicts "The Pipeline check" earlier in the same document, which states plainly that where an element sits is not judged, and it contradicts the shipped implementation, whose three build checks are collisions, source properties and override targets, with no position check among them. It also described behaviour that would refuse the very pipeline this section is about, because the 51Degrees cloud runs its device detection and IP intelligence engines in a parallel group, and a pipeline reports group members after every top level element. That defect was found by an independent review of pipeline-dotnet 376 and removed there before the element shipped. The replacement says what actually happens, being that an element placed too late reads its source properties as unavailable and the derived property has no value with a message naming them, which is the one rule answering the case rather than a second mechanism.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A specification page for a new Pipeline element,
pipeline-specification/pipeline-elements/derived-property-element.md,and its entry in
pipeline-specification/README.md.The element returns a property computed from properties already in the
flow data, following a script written in YAML or JSON that is shared
between every language rather than compiled into any of them. The script
format is format 1, and the shared scripts and the reference
implementation live in
51Degrees/derived-properties, which is privatetoday.
The behaviour the page specifies
A script names source properties inside its conditions. Where every one of
them is available, the checks run and the rules pick a value, first match
winning. Where any one of them is not available, the element writes a
value that has no value, and reading it raises the language's existing no
value error with a message naming every property that could not be read
and what its source element said about each.
That is the whole of the behaviour. There is one way to produce a value
and one way to produce no value.
The element data key is
derived, the evidence key filter is empty, andthe existing no value mechanism is reused, so no new
MissingPropertyReasonvalue is added and that enumeration does notchange.
Why the page reads the way it does
An earlier draft of this design carried more. It had an
Optionallistseparating source properties that had to be present from ones that did
not, a
Presentoperator, anEvaluatedaggregate counting the checksthat could be run, and a third "unknown" state running through every
truth table so that a script could carry on with some properties missing.
James Rosewell removed all of it on 1 September 2026, on the grounds that
the element is for simple derived properties whilst anything harder is
written as an ordinary flow element in code, the way such work is done
today. A property is either present or it is not. The page says so, and
also says plainly what the element is deliberately not for, so that
whoever implements it next does not rebuild the layer that was removed.
What is implemented already
The .NET implementation is open at 51Degrees/pipeline-dotnet as a
companion pull request, and this page was written against it and corrected
where the two disagreed. Its test suite passes 134 tests, and a
conformance runner reads 14 value cases and 29 rejection cases from the
shared repository so that every language proves the same behaviour against
the same cases rather than each asserting its own.
How this was checked
rather than against the design document, because the implementation is
what shipped.
paraphrase.
pipeline-specification/was searched for the removedfeatures. Nothing else refers to them. The other hits for the same words
are unrelated, being
MissingPropertyServicetest wording inautomated-testing.md, an "Optional" table heading inusage-sharing-element.md, "Optional fixed language" intranslation-engine.md, and the stringUnknownas a property value incloud-request-engine.md,set-headers-element.mdandfeatures/value-types.md.Added on 2 September 2026, after a verification review of the .NET work
Two commits, both needed before run book 03 starts the Node, Java, Python
and PHP work.
A cloud section on the element page (
d7a483c). The page describedthe element as a self-hosted element only and said nothing about how the
51Degrees cloud serves a derived property, which is what run book 04
depends on. The section names where the element sits in the cloud
pipeline, the response shape with and without a value, and four things a
cloud implementation owes.
The two JSON examples were taken from a test rather than written from
memory.
JsonBuilderIntegrationTestsin51Degrees/pipeline-dotnet#376 pins both the value
case and the no value case, so each example has code behind it.
A required example (
6d18fc3).required-examples.mdis what tellseach reference implementation which examples it owes, and it named
nothing for this element, so run book 03 had no definition of the example
each language must ship. The four points now written down were checked
against
Examples/DerivedPropertyin pipeline-dotnet first, and the .NETexample does all four today. The order is deliberately left to each
language, because the .NET example runs the configuration file case first
and the code case second, and imposing an order would have made the
reference implementation fail its own specification.
Both commits were made through the GitHub contents API rather than from
the local clone, because that clone was on another branch with
uncommitted work belonging to a different session.