@@ -7,11 +7,13 @@ transactions, or `Console` progress reporting.
77## Console observer registration
88
99` Console.get_execution_observer() ` returns ` None ` by default. A ` Console `
10- subclass can return an ` Observer ` callback from
11- ` sqlmesh.core.execution_observation ` . The callback receives a phase, a read-only
12- mapping of facts, and the original exception or ` None ` .
10+ subclass can return an ` Observer ` callback from ` sqlmesh.core.execution_observation ` .
11+ The callback receives a
12+ phase, a read-only mapping of facts, and the original exception or ` None ` .
13+ For general Python integration details, see the [ Python API reference] ( ./python.md ) .
1314
14- This console prints the kind, action, and finish error for each observed operation:
15+ The following console prints the kind, action, and finish error for each observed
16+ operation. The example uses ` examples/sushi ` relative to the SQLMesh repository root:
1517
1618``` python
1719from typing import Any, Literal, Mapping, Optional
@@ -22,38 +24,36 @@ from sqlmesh.core.execution_observation import Observer
2224
2325
2426def observe (
25- phase : Literal[" start" , " finish" ],
26- facts : Mapping[str , Any],
27- error : Optional[BaseException ],
27+ phase : Literal[" start" , " finish" ],
28+ facts : Mapping[str , Any],
29+ error : Optional[BaseException ],
2830) -> None :
29- if phase == " finish" :
30- print (f " { facts[' kind' ]} / { facts[' action' ]} : error= { error!r } " )
31+ if phase == " finish" :
32+ print (f " { facts[' kind' ]} / { facts[' action' ]} : error= { error!r } " )
3133
3234
3335class ObservedConsole (TerminalConsole ):
34- def get_execution_observer (self ) -> Optional[Observer]:
35- return observe
36+ def get_execution_observer (self ) -> Optional[Observer]:
37+ return observe
3638
3739
3840set_console(ObservedConsole())
3941context = Context(paths = [" examples/sushi" ])
4042```
4143
42- A ` Context ` captures the console returned by ` get_console() ` when the context is
43- constructed. ` BuiltInPlanEvaluator.evaluate() ` and
44- ` Scheduler.run_merged_intervals() ` enter ` console_observer_scope() ` for their
45- observed work. Unless an ` observer_scope() ` is already active,
46- ` console_observer_scope() ` calls ` get_execution_observer() ` once. Nested calls
47- reuse the observer already active in the scope. If ` get_execution_observer() `
48- raises an ordinary ` Exception ` , SQLMesh disables observation for that scope and
44+ A ` Context ` uses the ` Console ` that was current when the context was created.
45+ Unless an explicit ` observer_scope() ` is active, SQLMesh resolves
46+ ` get_execution_observer() ` once for each outer plan evaluation or scheduler run.
47+ Nested calls reuse that observer. If the getter raises an ordinary ` Exception ` ,
48+ SQLMesh disables observation for that plan evaluation or scheduler run and
4949continues native work.
5050
5151## Observer scopes
5252
5353` observer_scope(observer) ` from ` sqlmesh.core.execution_observation ` sets the
54- active observer for the scope. A nested scope restores the previously active
55- observer when it exits. ` observer_scope(None) ` disables observation for that
56- scope, including automatic Console registration.
54+ observer for the duration of the scope. A nested scope restores the previously
55+ active observer when it exits. ` observer_scope(None) ` disables observation inside
56+ that scope, including automatic Console registration.
5757
5858## Callback lifecycle
5959
@@ -79,13 +79,13 @@ is not an `Exception`, the work and `finish` callback do not run. If `finish`
7979raises a ` BaseException ` that is not an ` Exception ` , that failure can replace
8080the workload error.
8181
82- A worker's ` BaseException ` can reach the caller after SQLMesh waits for its
83- workers. A DAG run that records a terminal error stops scheduling nodes and
84- raises that error after its running workers exit. In parallel execution,
85- ` concurrent_apply_to_values() ` stores a worker's ` BaseException ` in a future and
86- waits for the pool to finish before reading results . Blocking callbacks and
87- process termination are outside the observer failure guarantee. SQLMesh does
88- not log observer failures.
82+ If a worker raises a ` BaseException ` , the error can reach the caller after its
83+ worker pool exits. If a DAG run records a terminal error, SQLMesh stops
84+ scheduling nodes and waits for running workers to exit before it raises that
85+ error. Parallel value operations also wait for their workers to finish before
86+ raising a worker's ` BaseException ` . Blocking callbacks and process termination
87+ are outside the observer failure guarantee. SQLMesh does not log observer
88+ failures.
8989
9090Nested workload calls produce nested observations. Callbacks for concurrent
9191workers can interleave, and SQLMesh provides no global callback order. Observers
@@ -112,18 +112,21 @@ observed operation and can be absent when work exits early or fails.
112112| ` audit ` | ` audit ` | ` audit_name ` ; ` audit_result ` at finish if produced |
113113| ` query ` | ` execute ` | ` engine ` dialect string |
114114
115- ` Snapshot ` and ` AuditResult ` values are borrowed native objects. Consumers must
115+ [ ` Snapshot ` ] ( ../concepts/architecture/snapshots.md ) and ` AuditResult ` values are
116+ borrowed native objects. Consumers must
116117treat these objects and other nested values as read-only and must not retain them
117118beyond the matching ` finish ` callback. The mapping's read-only wrapper does not
118119freeze nested objects. Facts are not a JSON event schema. SQLMesh does not
119120interpret audit results as consumer statuses or inherit model facts into child
120121mappings. Consumers that need facts after ` finish ` copy and normalize those
121122values during the callback.
122123
123- SQLMesh calls ` finish ` for an audit before it runs later audits or publishes WAP
124- results. A later audit or publication can still fail. An audit query failure can
125- produce a ` finish ` callback without an ` AuditResult ` . The physical materialization
126- callback ends after its surrounding transaction and session exit.
124+ SQLMesh calls ` finish ` for an audit before it runs later audits or completes
125+ write-audit-publish (WAP). A later audit or WAP publication can still fail. An audit
126+ query failure can produce a ` finish ` callback without an ` AuditResult ` . The
127+ [ audits guide] ( ../concepts/audits.md ) describes audit behavior in plans and
128+ runs. The physical materialization callback ends after its surrounding
129+ transaction and session exit.
127130
128131An observed operation can do no work, issue multiple statements, or trigger
129132internal driver retries. The observation does not predict how many statements
@@ -143,23 +146,20 @@ use that ordering to associate existing log entries with the invocation.
143146
144147## Worker context
145148
146- Outside callback delivery, an active observer makes
147- ` execution_context_factory() ` return ` contextvars.copy_context ` .
148- ` Scheduler.run_merged_intervals() ` passes that function to
149- ` concurrent_apply_to_dag() ` . ` SnapshotEvaluator ` passes it to
150- ` concurrent_apply_to_snapshots() ` and ` concurrent_apply_to_values() ` . Each node
151- or value gets its own context copy, including during serial execution. Successor
152- nodes start from the context captured for the DAG run. They do not inherit
153- changes made by earlier workers. ` ConcurrentDAGExecutor.run() ` captures a fresh
154- context for each run. Schema creation and data-object listing use
155- ` concurrent_apply_to_values() ` , so each value gets its own context copy.
149+ When an observer is active outside callback delivery, SQLMesh copies the current
150+ Python context for each task in scheduler and snapshot DAG work, snapshot schema creation, and
151+ data-object listing. That context carries Python's [ context variables] ( https://docs.python.org/3/library/contextvars.html ) .
152+ Each task gets its own context copy, including during serial execution. DAG
153+ successors use the context captured for the run. They do not inherit changes
154+ made by earlier workers. Reusing a DAG executor captures a fresh context for
155+ each run.
156156
157157Context copies are shallow. Mutable objects stored in ` ContextVar ` values remain
158158shared across copies. A parent identity stored in a consumer-owned ` ContextVar `
159- is available to each worker. These copies apply to the scheduler DAG and to the
160- ` SnapshotEvaluator ` calls above. Other SQLMesh concurrency helpers and arbitrary
161- consumer threads do not receive these copies.
159+ is available to each worker. SQLMesh uses these copies in the scheduler and
160+ snapshot DAG, schema creation, and data-object listing. Arbitrary consumer
161+ threads and other concurrency helpers do not receive these copies.
162162
163- With no observer registered, ` execution_context_factory() ` returns ` None ` . The
164- concurrency helpers retain their existing ` ContextVar ` behavior for serial and
165- worker execution.
163+ With no observer registered, these operations do not copy contexts . The concurrency
164+ helpers retain their existing ` ContextVar ` behavior for serial and worker
165+ execution.
0 commit comments