Skip to content

Commit ab7bc8a

Browse files
committed
Declare graph inputs and outputs from the pipeline body (v0.1.23)
A pipeline's inputs come from its `In[T]` parameters and its outputs from the return annotation, which only covers graphs whose I/O is Python-shaped. A name that is not an identifier ("Pipeline Creation Time"), an exact Tangle type string, an input declared conditionally, or a per-input editor position has no spelling at all — which is why downstream code reaches into `current_builder().inputs` / `.output_values` behind private helpers. `graph_input()` / `graph_output()` declare the same entries on the active trace as a public, validated API: unique names checked against `In[...]` parameters and the return annotation, `default` implying `optional: true` the way a defaulted parameter already does, handles-not-constants for outputs, `when=` for conditional declaration, and `position=(x, y)` reusing the canonical editor.position serializer. Also fixes a latent bug: `graph_output()` beside a returned `Out[T]` was silently lossy, because emit prefers the multi-output map while the returned edge lived only in the legacy single-output shims. The returned output now joins the map, declared last.
1 parent a5dfe5e commit ab7bc8a

11 files changed

Lines changed: 818 additions & 6 deletions

File tree

‎README.md‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -603,6 +603,37 @@ The rules live in one place, `tangle_cli.schema_validation`: `check_annotations(
603603

604604
A distribution that reads these annotations from its own config file should call `check_annotations(mapping, policy=CALLER_ANNOTATION_POLICY, error_cls=...)` at config-parse time — passing its own error type and adding the config path and key to the message — so one user mistake produces one diagnostic instead of two competing ones. The compiler's own call is then the backstop for anything arriving by another route.
605605

606+
##### Declaring graph inputs and outputs from the body
607+
608+
A pipeline's inputs normally come from its `In[T]` parameters and its outputs from the return annotation. That only covers graphs whose I/O is Python-shaped. `graph_input()` / `graph_output()` declare the same entries directly on the active pipeline, for the shapes a signature cannot express:
609+
610+
```python
611+
from tangle_cli.python_pipeline import Out, graph_input, graph_output, pipeline
612+
613+
@pipeline("Daily Pulse")
614+
def daily_pulse() -> Out[str]:
615+
created = graph_input(
616+
"Pipeline Creation Time", "String", default="Use as a cache busting mechanism"
617+
)
618+
limit = graph_input("query_limit", "Integer", optional=True, when=has_limit)
619+
scrape = SCRAPE.named("Scrape").with_position(0, 0)(created=created, limit=limit)
620+
graph_output("scrape_date", scrape.scrape_date, "String")
621+
return scrape
622+
```
623+
624+
- **A name is a string, not an identifier.** `"Pipeline Creation Time"` is a legal Tangle input name and an illegal Python parameter name; same for a type that is a Tangle type string (`"Json"`) rather than a Python type.
625+
- **`when=False` declares nothing** and returns `None`, so a conditional input stays a single assignment instead of an `if`/`else` around the call site.
626+
- **`default` implies `optional: true`** unless `optional=False` is passed. This matches what a defaulted `In[T]` parameter emits; the corpus also contains defaulted-but-required inputs, which is why the implication is overridable.
627+
- **Declaration order is signature parameters first, then body order**, and it is the order the entries appear in the document.
628+
- **A name is declared once.** Duplicates are rejected, including a collision with an `In[...]` parameter or with the name the return annotation contributes (`Out[T]`'s `output_name`, or an `Outputs` field).
629+
- **Outputs wire to a handle, never a constant** — a task output or a graph input, the same rule the `Outputs` return path enforces. Returning a value *and* declaring outputs is supported: the returned output is appended after the declared ones.
630+
- **Layout comes along**: `position=(x, y)` writes the same canonical `editor.position` annotation as `.with_position(...)`, which is how a graph input gets a position in the editor. `annotations={...}` is validated under the same caller policy as `pipeline_annotations`.
631+
- **Validated up front, without echoing values.** A non-string `default` (the schema's `InputSpec.default` is a string), a non-string name/type/description, a non-boolean `optional`, a constant output, or a call outside a `@pipeline` body raises `InvalidGraphIoError` (a `CompileError`) naming the field, never the value.
632+
633+
The signature route remains the recommended default: it is typed, it is checked by your IDE, and it reads better. Reach for these when porting existing YAML or when the shape genuinely needs it.
634+
635+
Porting an existing helper that appended to the builder by hand (for example relevance-tools' `_gin` / `_gout`): the emitted document is identical except in two places. An input declared with a `default` and no explicit `optional` gains `optional: true`, because that is what a defaulted `In[T]` parameter already emits and what the majority of the corpus carries; pass `optional=False` to keep it required. An input declaring both keys emits `default` before `optional`, which is the order the corpus uses 28 times against 3 for the reverse.
636+
606637
##### Editor layout (`editor.*`)
607638

608639
The pipeline editor stores graph layout in ordinary annotations, so layout is authorable from Python. `.with_position(x, y)` on a task or subpipeline handle writes `editor.position`, and `@pipeline(flow_direction=...)` writes the root `editor.flow-direction`:
@@ -630,7 +661,7 @@ SCRAPE.with_annotations({"editor.position": '{"x": 0, "y": 0}'})
630661
- **Positions are descriptive.** They are task annotations, not part of any `componentRef`, so they never change component digests, compile identity, or cache behaviour. On a subpipeline handle the position applies to the parent task; the child sidecar is byte-identical either way.
631662
- **Auto-layout interaction.** At submit time the runner lays a graph out only when no task carries a *non-zero* position, so an explicit `.with_position(...)` suppresses auto-layout and `--force-layout` overrides that. A graph positioned entirely at `(0, 0)` still counts as unpositioned and is laid out. `tangle sdk pipelines layout` writes the same canonical value this sugar writes.
632663

633-
Graph **inputs** and **outputs** carry `editor.position` in the same way, and the editor reads it there, but there is no Python authoring surface for a graph input/output object yet — inputs come from the `In[T]` signature and outputs from the return annotation, neither of which has a place to hang layout. That is follow-up work tied to a public `graph_input()` / `graph_output()` API; until then, position inputs by editing the YAML or by using the editor.
664+
Graph **inputs** and **outputs** carry `editor.position` in the same way. An `In[T]` parameter has nowhere to hang layout, so positioning those means declaring them with `graph_input(..., position=(x, y))` instead — see the section above.
634665

635666
The key names, the serialized format, and the validation live in `tangle_cli.editor_layout` (`POSITION_ANNOTATION`, `FLOW_DIRECTION_ANNOTATION`, `FLOW_DIRECTIONS`, `position_annotation_value`, `validate_flow_direction`), which the CLI's own `pipelines layout` and the runner's auto-layout gate share, so a tool emitting layout cannot drift from what the editor reads. That module is stdlib-only and lives outside `python_pipeline` on purpose, so layout and submit paths do not pull in the authoring/codegen stack; it raises whatever `error_cls` the caller injects, and the authoring surfaces inject `InvalidEditorLayoutError`.
636667

‎packages/tangle-cli/src/tangle_cli/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,6 @@
1414
try:
1515
__version__ = metadata_version("tangle-cli")
1616
except PackageNotFoundError:
17-
__version__ = "0.1.22"
17+
__version__ = "0.1.23"
1818

1919
__all__ = ["TangleDynamicDiscoveryClient", "__version__"]

‎packages/tangle-cli/src/tangle_cli/python_pipeline/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
from __future__ import annotations
2222

2323
from .dynamic_data import dynamic_secret
24+
from .graph_io import graph_input, graph_output
2425
from .pipeline import pipeline
2526
from .raw import raw
2627
from .ref import ref
@@ -33,6 +34,8 @@
3334

3435
__all__ = [
3536
"pipeline",
37+
"graph_input",
38+
"graph_output",
3639
"task",
3740
"Publish",
3841
"registered",

‎packages/tangle-cli/src/tangle_cli/python_pipeline/errors.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,15 @@ class InvalidEditorLayoutError(CompileError):
3838
"""
3939

4040

41+
class InvalidGraphIoError(CompileError):
42+
"""Raised on an unusable ``graph_input()`` / ``graph_output()`` call.
43+
44+
Covers declaration outside a trace, duplicate names, fields the pipeline
45+
schema would reject, and constant graph outputs. Messages name the field,
46+
never the value.
47+
"""
48+
49+
4150
class InvalidPipelineAnnotationsError(CompileError):
4251
"""Raised on a malformed caller-supplied ``pipeline_annotations`` mapping.
4352
Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
"""``graph_input()`` / ``graph_output()`` — declare graph I/O from the body.
2+
3+
A pipeline's inputs normally come from its ``In[T]`` parameters and its
4+
outputs from the return annotation. That covers Python-shaped graphs only:
5+
it cannot express a name that is not an identifier (``"Pipeline Creation
6+
Time"``), an exact Tangle type string, an input declared conditionally, or
7+
a per-input ``editor.position``. These two functions declare the same
8+
entries directly on the active trace, so a YAML pipeline that uses those
9+
shapes stays expressible in Python.
10+
11+
Entry key order follows the tracer's signature path and the corpus:
12+
``name, type, description, default, optional, annotations``.
13+
"""
14+
from __future__ import annotations
15+
16+
from collections.abc import Mapping
17+
from typing import Any
18+
19+
from tangle_cli.editor_layout import POSITION_ANNOTATION, position_annotation_value
20+
from tangle_cli.schema_validation import CALLER_ANNOTATION_POLICY, check_annotations
21+
22+
from .errors import InvalidEditorLayoutError, InvalidGraphIoError
23+
from .graph import EdgeRef
24+
from .placeholders import GraphInputPlaceholder, TaskOutputProxy
25+
26+
# Distinguishes "no default" from an explicit ``default=None``, which is a
27+
# value the schema does not accept and must be reported, not dropped.
28+
_UNSET: Any = object()
29+
30+
__all__ = ["graph_input", "graph_output"]
31+
32+
33+
def graph_input(
34+
name: str,
35+
type: str | None = None,
36+
*,
37+
description: str | None = None,
38+
default: Any = _UNSET,
39+
optional: bool | None = None,
40+
annotations: Mapping[str, str] | None = None,
41+
position: tuple[float, float] | None = None,
42+
when: bool = True,
43+
) -> GraphInputPlaceholder | None:
44+
"""Declare a graph input on the active pipeline and return its handle.
45+
46+
The handle is passed to task arguments exactly like an ``In[T]``
47+
parameter. Supplying ``default`` implies ``optional: true`` unless
48+
``optional=False`` is passed explicitly.
49+
50+
``when=False`` declares nothing and returns ``None``, so a conditional
51+
input reads as one assignment instead of an ``if``/``else``.
52+
53+
Raises:
54+
InvalidGraphIoError: Outside a ``@pipeline`` body, on a duplicate
55+
name, or on an unusable field. Messages name the field, never
56+
the value.
57+
"""
58+
builder = _active_builder("graph_input")
59+
if not when:
60+
return None
61+
62+
_check_name(name, "graph_input")
63+
if any(entry.get("name") == name for entry in builder.inputs):
64+
# Also catches an ``In[...]`` parameter: signature inputs are
65+
# appended before the body runs.
66+
raise InvalidGraphIoError(
67+
f"graph input {name!r} is already declared on pipeline "
68+
f"{builder.name!r}. Input names must be unique."
69+
)
70+
71+
entry: dict[str, Any] = {"name": name}
72+
if type is not None:
73+
entry["type"] = _check_str("type", type)
74+
if description is not None:
75+
entry["description"] = _check_str("description", description)
76+
if default is not _UNSET:
77+
# InputSpec.default is a string in the schema; a non-string default
78+
# would only fail later, against a schema path instead of this call.
79+
entry["default"] = _check_str("default", default, hint="pass str(value)")
80+
entry["optional"] = True if optional is None else _check_bool(optional)
81+
elif optional is not None:
82+
entry["optional"] = _check_bool(optional)
83+
merged = _layout_annotations(annotations, position)
84+
if merged:
85+
entry["annotations"] = merged
86+
87+
builder.inputs.append(entry)
88+
return GraphInputPlaceholder(input_name=name)
89+
90+
91+
def graph_output(
92+
name: str,
93+
value: Any,
94+
type: str | None = None,
95+
*,
96+
description: str | None = None,
97+
annotations: Mapping[str, str] | None = None,
98+
position: tuple[float, float] | None = None,
99+
when: bool = True,
100+
) -> None:
101+
"""Declare a graph output on the active pipeline, wired to ``value``.
102+
103+
``value`` is a task output handle or a graph-input handle; a constant is
104+
refused, matching the ``Outputs`` return path.
105+
106+
Raises:
107+
InvalidGraphIoError: Outside a ``@pipeline`` body, on a duplicate
108+
name, on an unusable field, or on a constant value.
109+
"""
110+
builder = _active_builder("graph_output")
111+
if not when:
112+
return
113+
114+
_check_name(name, "graph_output")
115+
if any(entry.get("name") == name for entry in builder.outputs):
116+
raise InvalidGraphIoError(
117+
f"graph output {name!r} is already declared on pipeline "
118+
f"{builder.name!r}. Output names must be unique."
119+
)
120+
121+
entry: dict[str, Any] = {"name": name}
122+
if type is not None:
123+
entry["type"] = _check_str("type", type)
124+
if description is not None:
125+
entry["description"] = _check_str("description", description)
126+
merged = _layout_annotations(annotations, position)
127+
if merged:
128+
entry["annotations"] = merged
129+
130+
builder.outputs.append(entry)
131+
builder.output_values[name] = _edge(name, value)
132+
133+
134+
def _active_builder(caller: str) -> Any:
135+
from .trace import current_builder
136+
137+
builder = current_builder()
138+
if builder is None:
139+
raise InvalidGraphIoError(
140+
f"{caller}() must be called inside a @pipeline function body, "
141+
"where a trace is active."
142+
)
143+
return builder
144+
145+
146+
def _edge(name: str, value: Any) -> EdgeRef:
147+
if isinstance(value, TaskOutputProxy):
148+
return EdgeRef(
149+
kind="taskOutput",
150+
task_id=value._task_id,
151+
output=value._resolved_output_name(),
152+
)
153+
if isinstance(value, GraphInputPlaceholder):
154+
return EdgeRef(kind="graphInput", input_name=value.input_name)
155+
raise InvalidGraphIoError(
156+
f"graph output {name!r} must be wired to a task output or a graph "
157+
f"input; got {type(value).__name__}. Constant graph outputs are not "
158+
"supported — emit the value from a task."
159+
)
160+
161+
162+
def _layout_annotations(
163+
annotations: Mapping[str, str] | None,
164+
position: tuple[float, float] | None,
165+
) -> dict[str, str]:
166+
merged: dict[str, str] = {}
167+
if annotations is not None:
168+
if not isinstance(annotations, Mapping):
169+
raise InvalidGraphIoError(
170+
f"annotations must be a mapping; got {type(annotations).__name__}."
171+
)
172+
check_annotations(
173+
annotations,
174+
policy=CALLER_ANNOTATION_POLICY,
175+
error_cls=InvalidGraphIoError,
176+
)
177+
merged.update(annotations)
178+
if position is not None:
179+
# Applied after the mapping, matching ``@pipeline(flow_direction=...)``.
180+
if not isinstance(position, tuple) or len(position) != 2:
181+
raise InvalidEditorLayoutError(
182+
"position must be an (x, y) tuple; got "
183+
f"{type(position).__name__}."
184+
)
185+
merged[POSITION_ANNOTATION] = position_annotation_value(
186+
*position, error_cls=InvalidEditorLayoutError
187+
)
188+
return merged
189+
190+
191+
def _check_name(name: Any, caller: str) -> str:
192+
if not isinstance(name, str) or not name:
193+
raise InvalidGraphIoError(
194+
f"{caller}() name must be a non-empty string; got "
195+
f"{type(name).__name__}."
196+
)
197+
return name
198+
199+
200+
def _check_str(field: str, value: Any, *, hint: str | None = None) -> str:
201+
if not isinstance(value, str):
202+
suffix = f". {hint.capitalize()}." if hint else "."
203+
raise InvalidGraphIoError(
204+
f"{field} must be a string; got {type(value).__name__}{suffix}"
205+
)
206+
return value
207+
208+
209+
def _check_bool(value: Any) -> bool:
210+
if not isinstance(value, bool):
211+
raise InvalidGraphIoError(
212+
f"optional must be True or False; got {type(value).__name__}."
213+
)
214+
return value

‎packages/tangle-cli/src/tangle_cli/python_pipeline/trace.py‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,13 +285,20 @@ def trace_pipeline(
285285
inner = _annotation_inner_type(return_anno)
286286
type_str = _python_type_to_tangle_type(inner)
287287
output_name = pipeline_fn.output_name
288+
_ensure_no_duplicate_output(builder, output_name)
288289
builder.outputs.append({"name": output_name, "type": type_str})
289290
builder.output_name = output_name
290-
builder.output_taskref = EdgeRef(
291+
edge = EdgeRef(
291292
kind="taskOutput",
292293
task_id=result._task_id,
293294
output=result._resolved_output_name(),
294295
)
296+
builder.output_taskref = edge
297+
if builder.output_values:
298+
# graph_output() already populated the multi-output map, which
299+
# emit PREFERS; the returned output has to join it or it would be
300+
# silently dropped. Declared last, after the body's own outputs.
301+
builder.output_values[output_name] = edge
295302
elif return_anno is not inspect.Signature.empty and _is_outputs_class(return_anno):
296303
_trace_multi_output(builder, pipeline_fn, return_anno, result)
297304

@@ -355,9 +362,22 @@ def _trace_multi_output(
355362
"outputs."
356363
)
357364
type_str = _python_type_to_tangle_type(inner)
365+
_ensure_no_duplicate_output(builder, field_name)
358366
builder.outputs.append({"name": field_name, "type": type_str})
359367
builder.output_values[field_name] = edge
360368

361369

370+
def _ensure_no_duplicate_output(builder: GraphBuilder, name: str) -> None:
371+
"""Reject a return-annotation output that ``graph_output()`` already
372+
declared, instead of emitting the name twice."""
373+
if any(entry.get("name") == name for entry in builder.outputs):
374+
from .errors import InvalidGraphIoError
375+
376+
raise InvalidGraphIoError(
377+
f"graph output {name!r} is declared by graph_output() and by the "
378+
f"return annotation of pipeline {builder.name!r}. Declare it once."
379+
)
380+
381+
362382
# Re-export commonly-used names.
363383
__all__ = ["current_builder", "trace_pipeline"]

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "tangle-cli"
3-
version = "0.1.22"
3+
version = "0.1.23"
44
description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
55
readme = "README.md"
66
authors = [

0 commit comments

Comments
 (0)