Skip to content

Commit a60f3f8

Browse files
committed
Docs: Align execution observations with SQLMesh documentation conventions
Signed-off-by: mday-io <mdaytn@gmail.com>
1 parent e32060d commit a60f3f8

1 file changed

Lines changed: 49 additions & 49 deletions

File tree

‎docs/reference/execution_observations.md‎

Lines changed: 49 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,13 @@ transactions, or `Console` progress reporting.
77
## Console observer registration
88

99
`Console.get_execution_observer()` returns `None` by default. A `Console`
10-
subclass can return an `Observer` callback from
11-
`sqlmesh.core.execution_observation`. The callback receives a phase, a read-only
12-
mapping of facts, and the original exception or `None`.
10+
subclass can return an `Observer` callback from `sqlmesh.core.execution_observation`.
11+
The callback receives a
12+
phase, a read-only mapping of facts, and the original exception or `None`.
13+
For general Python integration details, see the [Python API reference](./python.md).
1314

14-
This console prints the kind, action, and finish error for each observed operation:
15+
The following console prints the kind, action, and finish error for each observed
16+
operation. The example uses `examples/sushi` relative to the SQLMesh repository root:
1517

1618
```python
1719
from typing import Any, Literal, Mapping, Optional
@@ -22,38 +24,36 @@ from sqlmesh.core.execution_observation import Observer
2224

2325

2426
def observe(
25-
phase: Literal["start", "finish"],
26-
facts: Mapping[str, Any],
27-
error: Optional[BaseException],
27+
phase: Literal["start", "finish"],
28+
facts: Mapping[str, Any],
29+
error: Optional[BaseException],
2830
) -> None:
29-
if phase == "finish":
30-
print(f"{facts['kind']}/{facts['action']}: error={error!r}")
31+
if phase == "finish":
32+
print(f"{facts['kind']}/{facts['action']}: error={error!r}")
3133

3234

3335
class ObservedConsole(TerminalConsole):
34-
def get_execution_observer(self) -> Optional[Observer]:
35-
return observe
36+
def get_execution_observer(self) -> Optional[Observer]:
37+
return observe
3638

3739

3840
set_console(ObservedConsole())
3941
context = Context(paths=["examples/sushi"])
4042
```
4143

42-
A `Context` captures the console returned by `get_console()` when the context is
43-
constructed. `BuiltInPlanEvaluator.evaluate()` and
44-
`Scheduler.run_merged_intervals()` enter `console_observer_scope()` for their
45-
observed work. Unless an `observer_scope()` is already active,
46-
`console_observer_scope()` calls `get_execution_observer()` once. Nested calls
47-
reuse the observer already active in the scope. If `get_execution_observer()`
48-
raises an ordinary `Exception`, SQLMesh disables observation for that scope and
44+
A `Context` uses the `Console` that was current when the context was created.
45+
Unless an explicit `observer_scope()` is active, SQLMesh resolves
46+
`get_execution_observer()` once for each outer plan evaluation or scheduler run.
47+
Nested calls reuse that observer. If the getter raises an ordinary `Exception`,
48+
SQLMesh disables observation for that plan evaluation or scheduler run and
4949
continues native work.
5050

5151
## Observer scopes
5252

5353
`observer_scope(observer)` from `sqlmesh.core.execution_observation` sets the
54-
active observer for the scope. A nested scope restores the previously active
55-
observer when it exits. `observer_scope(None)` disables observation for that
56-
scope, including automatic Console registration.
54+
observer for the duration of the scope. A nested scope restores the previously
55+
active observer when it exits. `observer_scope(None)` disables observation inside
56+
that scope, including automatic Console registration.
5757

5858
## Callback lifecycle
5959

@@ -79,13 +79,13 @@ is not an `Exception`, the work and `finish` callback do not run. If `finish`
7979
raises a `BaseException` that is not an `Exception`, that failure can replace
8080
the workload error.
8181

82-
A worker's `BaseException` can reach the caller after SQLMesh waits for its
83-
workers. A DAG run that records a terminal error stops scheduling nodes and
84-
raises that error after its running workers exit. In parallel execution,
85-
`concurrent_apply_to_values()` stores a worker's `BaseException` in a future and
86-
waits for the pool to finish before reading results. Blocking callbacks and
87-
process termination are outside the observer failure guarantee. SQLMesh does
88-
not log observer failures.
82+
If a worker raises a `BaseException`, the error can reach the caller after its
83+
worker pool exits. If a DAG run records a terminal error, SQLMesh stops
84+
scheduling nodes and waits for running workers to exit before it raises that
85+
error. Parallel value operations also wait for their workers to finish before
86+
raising a worker's `BaseException`. Blocking callbacks and process termination
87+
are outside the observer failure guarantee. SQLMesh does not log observer
88+
failures.
8989

9090
Nested workload calls produce nested observations. Callbacks for concurrent
9191
workers can interleave, and SQLMesh provides no global callback order. Observers
@@ -112,18 +112,21 @@ observed operation and can be absent when work exits early or fails.
112112
| `audit` | `audit` | `audit_name`; `audit_result` at finish if produced |
113113
| `query` | `execute` | `engine` dialect string |
114114

115-
`Snapshot` and `AuditResult` values are borrowed native objects. Consumers must
115+
[`Snapshot`](../concepts/architecture/snapshots.md) and `AuditResult` values are
116+
borrowed native objects. Consumers must
116117
treat these objects and other nested values as read-only and must not retain them
117118
beyond the matching `finish` callback. The mapping's read-only wrapper does not
118119
freeze nested objects. Facts are not a JSON event schema. SQLMesh does not
119120
interpret audit results as consumer statuses or inherit model facts into child
120121
mappings. Consumers that need facts after `finish` copy and normalize those
121122
values during the callback.
122123

123-
SQLMesh calls `finish` for an audit before it runs later audits or publishes WAP
124-
results. A later audit or publication can still fail. An audit query failure can
125-
produce a `finish` callback without an `AuditResult`. The physical materialization
126-
callback ends after its surrounding transaction and session exit.
124+
SQLMesh calls `finish` for an audit before it runs later audits or completes
125+
write-audit-publish (WAP). A later audit or WAP publication can still fail. An audit
126+
query failure can produce a `finish` callback without an `AuditResult`. The
127+
[audits guide](../concepts/audits.md) describes audit behavior in plans and
128+
runs. The physical materialization callback ends after its surrounding
129+
transaction and session exit.
127130

128131
An observed operation can do no work, issue multiple statements, or trigger
129132
internal driver retries. The observation does not predict how many statements
@@ -143,23 +146,20 @@ use that ordering to associate existing log entries with the invocation.
143146

144147
## Worker context
145148

146-
Outside callback delivery, an active observer makes
147-
`execution_context_factory()` return `contextvars.copy_context`.
148-
`Scheduler.run_merged_intervals()` passes that function to
149-
`concurrent_apply_to_dag()`. `SnapshotEvaluator` passes it to
150-
`concurrent_apply_to_snapshots()` and `concurrent_apply_to_values()`. Each node
151-
or value gets its own context copy, including during serial execution. Successor
152-
nodes start from the context captured for the DAG run. They do not inherit
153-
changes made by earlier workers. `ConcurrentDAGExecutor.run()` captures a fresh
154-
context for each run. Schema creation and data-object listing use
155-
`concurrent_apply_to_values()`, so each value gets its own context copy.
149+
When an observer is active outside callback delivery, SQLMesh copies the current
150+
Python context for each task in scheduler and snapshot DAG work, snapshot schema creation, and
151+
data-object listing. That context carries Python's [context variables](https://docs.python.org/3/library/contextvars.html).
152+
Each task gets its own context copy, including during serial execution. DAG
153+
successors use the context captured for the run. They do not inherit changes
154+
made by earlier workers. Reusing a DAG executor captures a fresh context for
155+
each run.
156156

157157
Context copies are shallow. Mutable objects stored in `ContextVar` values remain
158158
shared across copies. A parent identity stored in a consumer-owned `ContextVar`
159-
is available to each worker. These copies apply to the scheduler DAG and to the
160-
`SnapshotEvaluator` calls above. Other SQLMesh concurrency helpers and arbitrary
161-
consumer threads do not receive these copies.
159+
is available to each worker. SQLMesh uses these copies in the scheduler and
160+
snapshot DAG, schema creation, and data-object listing. Arbitrary consumer
161+
threads and other concurrency helpers do not receive these copies.
162162

163-
With no observer registered, `execution_context_factory()` returns `None`. The
164-
concurrency helpers retain their existing `ContextVar` behavior for serial and
165-
worker execution.
163+
With no observer registered, these operations do not copy contexts. The concurrency
164+
helpers retain their existing `ContextVar` behavior for serial and worker
165+
execution.

0 commit comments

Comments
 (0)