Docs: Clarify execution observation contracts and examples

Signed-off-by: mday-io <mdaytn@gmail.com>

Author: mday-io

docs/reference/execution_observations.md

@@ -1,14 +1,17 @@
 # Execution observations
 
-Execution observations let Python integrations observe native SQLMesh work through
+Execution observations expose native SQLMesh work to Python integrations through
 an optional callback. They do not replace SQL generation, execution, retries,
-transactions, or Console progress reporting.
+transactions, or `Console` progress reporting.
 
-## Register an observer
+## Console observer registration
 
-Override `Console.get_execution_observer()` to return a callback accepting
-`phase`, `facts`, and `error`. Its default return value is `None`.
-Install the console before constructing the SQLMesh `Context`:
+`Console.get_execution_observer()` returns `None` by default. A `Console`
+subclass can return an `Observer` callback from
+`sqlmesh.core.execution_observation`. The callback receives a phase, a read-only
+mapping of facts, and the original exception or `None`.
+
+This console prints the kind, action, and finish error for each observed operation:
 
 ```python
 from typing import Any, Literal, Mapping, Optional
@@ -19,121 +22,144 @@ from sqlmesh.core.execution_observation import Observer
 
 
 def observe(
-    phase: Literal["start", "finish"],
-    facts: Mapping[str, Any],
-    error: Optional[BaseException],
+	phase: Literal["start", "finish"],
+	facts: Mapping[str, Any],
+	error: Optional[BaseException],
 ) -> None:
-    if phase == "finish":
-        print(f"{facts['kind']}/{facts['action']}: error={error!r}")
+	if phase == "finish":
+		print(f"{facts['kind']}/{facts['action']}: error={error!r}")
 
 
 class ObservedConsole(TerminalConsole):
-    def get_execution_observer(self) -> Optional[Observer]:
-        return observe
+	def get_execution_observer(self) -> Optional[Observer]:
+		return observe
 
 
 set_console(ObservedConsole())
-context = Context(paths=["path/to/project"])
+context = Context(paths=["examples/sushi"])
 ```
 
-Built-in plan-stage evaluation and scheduler dispatch resolve the getter at the
-outer observation boundary. Nested dispatch reuses the active registration.
-An ordinary `Exception` from the getter disables observation at that boundary;
-native work continues.
+A `Context` captures the console returned by `get_console()` when the context is
+constructed. `BuiltInPlanEvaluator.evaluate()` and
+`Scheduler.run_merged_intervals()` enter `console_observer_scope()` for their
+observed work. Unless an `observer_scope()` is already active,
+`console_observer_scope()` calls `get_execution_observer()` once. Nested calls
+reuse the observer already active in the scope. If `get_execution_observer()`
+raises an ordinary `Exception`, SQLMesh disables observation for that scope and
+continues native work.
+
+## Observer scopes
 
-For an explicitly bounded registration, use `observer_scope(observer)` from
-`sqlmesh.core.execution_observation`. Nested scopes restore the preceding
-registration on exit. `observer_scope(None)` explicitly disables observation
-inside the scope, including automatic Console registration.
+`observer_scope(observer)` from `sqlmesh.core.execution_observation` sets the
+active observer for the scope. A nested scope restores the previously active
+observer when it exits. `observer_scope(None)` disables observation for that
+scope, including automatic Console registration.
 
 ## Callback lifecycle
 
-`start` runs before the observed work, with `error=None`. Once the start callback
-returns, `finish` runs from `finally` after that work, with its original exception
-object or `None`. Both callbacks run synchronously on the invoking thread.
+SQLMesh calls `start` before observed work with `error=None`. When `start`
+returns or raises an ordinary `Exception`, SQLMesh continues the work and calls
+`finish` from a `finally` block afterward. The `finish` callback receives the
+original workload exception object or `None`. Both callbacks run synchronously
+on the thread that performs the observed work. Callbacks for work dispatched to
+a worker run on that worker's thread.
 
-SQLMesh passes the **same read-only mapping object** to start and finish for one
-invocation. Distinct invocations have distinct mappings. Consumers may retain the
-mapping until its matching finish callback to pair the calls by object identity;
-release it afterward. SQLMesh may add facts discovered during execution, so this
-mapping is a live view, not an immutable snapshot of the start facts.
+Every invocation that reaches `finish` passes the same read-only mapping object
+to `start` and `finish`. Distinct invocations use distinct mapping objects.
+Consumers can pair the phases by mapping identity. SQLMesh may add facts while
+the work runs, so values in the mapping at `finish` can differ from its values
+at `start`. The mapping's supported lifetime ends when the matching `finish`
+callback returns.
+
+Ordinary observer `Exception` failures are suppressed and do not replace the
+workload result or error. SQLMesh does not suppress observer-raised
+`BaseException` subclasses that are not `Exception` subclasses, such as
+`KeyboardInterrupt` and `SystemExit`. If `start` raises a `BaseException` that
+is not an `Exception`, the work and `finish` callback do not run. If `finish`
+raises a `BaseException` that is not an `Exception`, that failure can replace
+the workload error.
+
+A worker's `BaseException` can reach the caller after SQLMesh waits for its
+workers. A DAG run that records a terminal error stops scheduling nodes and
+raises that error after its running workers exit. In parallel execution,
+`concurrent_apply_to_values()` stores a worker's `BaseException` in a future and
+waits for the pool to finish before reading results. Blocking callbacks and
+process termination are outside the observer failure guarantee. SQLMesh does
+not log observer failures.
 
 Nested workload calls produce nested observations. Callbacks for concurrent
-workers can interleave, and there is no global callback ordering. Observers must
-support concurrent calls and return promptly. SQLMesh does not allocate execution
-IDs, timestamps, durations, or parent IDs; consumers can maintain that information.
+workers can interleave, and SQLMesh provides no global callback order. Observers
+must support concurrent calls and return promptly. SQLMesh allocates no
+execution IDs, timestamps, durations, or parent IDs. Consumers supply those
+values when needed.
 
-Ordinary observer `Exception` failures are suppressed and do not replace the
-workload result or error. This guarantee does **not** cover observer-raised
-`BaseException` subclasses such as `KeyboardInterrupt` or `SystemExit`: a failure
-in start can prevent work and finish, and a failure in finish can replace a
-workload error. A worker's `BaseException` can propagate to the caller instead
-of leaving parallel completion pending. Once a DAG run records a terminal error,
-it stops admitting further nodes and raises that error after already-running
-workers exit. The parallel value helper records a worker's `BaseException` in its
-result and joins the pool before resolving results. Blocking callbacks and
-process termination are also outside the guarantee. SQLMesh does not recursively
-log observer failures.
-
-While a callback is being delivered, observations triggered by that callback are
-suppressed, and internal metadata updates are ignored. This guard applies only
-during callback delivery: a normal model operation can still contain observed
-physical operations, audits, and queries.
-
-## Native facts and boundaries
-
-Every mapping contains `kind` and `action`. Other fields depend on the boundary
-and may be absent, particularly when work exits early or fails.
+While SQLMesh delivers a callback, it suppresses observations triggered by that
+callback and ignores calls to `update_execution()`. The guard covers callback
+delivery only. Normal model work can still contain observations for physical
+operations, audits, and queries.
+
+## Native facts and limits
+
+Every facts mapping contains `kind` and `action`. Other fields depend on the
+observed operation and can be absent when work exits early or fails.
 
 | Kind | Action | Native facts |
 | --- | --- | --- |
-| `stage` | Native plan-stage name | `native_plan_id` |
+| `stage` | Plan-stage class name | `native_plan_id` |
 | `model` | `evaluate`, `audit_only` | `snapshot`, `interval`, `batch_index`, `execution_time`, `audit_only` |
 | `physical` | `create`, `materialize`, `schema_migration` | `snapshot`; `target` when known; `creating` for materialization; `existed` for schema migration when known |
 | `virtual` | `promote`, `demote` | `snapshot`; `target` when known; `source_table` for promotion when known |
 | `audit` | `audit` | `audit_name`; `audit_result` at finish if produced |
 | `query` | `execute` | `engine` dialect string |
 
-`Snapshot` and `AuditResult` values are borrowed native objects. Treat them and
-other nested values as read-only, and do not retain them beyond finish. The
-mapping's read-only wrapper does not freeze these objects. Normalize any values
-needed for later processing during the callback; facts are not a JSON event
-schema. SQLMesh does not interpret audit results as consumer statuses or inherit
-model facts into child mappings.
-
-Audit observations finish before later audits or WAP publication can fail. An
-audit query failure can produce a finish callback without an `AuditResult`.
-Physical materialization includes the surrounding transaction/session exit.
-An invocation can perform no work, multiple statements, or internal driver
-retries; observations are not an expected-work manifest or a retry inventory.
-
-The query boundary surrounds the base adapter's existing SQL logging and
-`_execute` call. It does not include a surrounding transaction's commit or later
-fetch/result consumption. Adapter paths that bypass this boundary, including
-BigQuery session queries and dataframe loads and ClickHouse dataframe inserts,
-do not necessarily produce query observations. These hooks therefore do not
-provide a complete warehouse activity history or a transaction-success guarantee.
-
-SQL logs retain their existing behavior, including VALUES redaction for
-non-query expressions. The callback does not receive SQL text, adapter objects,
-or warehouse query IDs. Starting the observation before SQL logging lets a
-consumer associate existing logs with the invocation.
+`Snapshot` and `AuditResult` values are borrowed native objects. Consumers must
+treat these objects and other nested values as read-only and must not retain them
+beyond the matching `finish` callback. The mapping's read-only wrapper does not
+freeze nested objects. Facts are not a JSON event schema. SQLMesh does not
+interpret audit results as consumer statuses or inherit model facts into child
+mappings. Consumers that need facts after `finish` copy and normalize those
+values during the callback.
+
+SQLMesh calls `finish` for an audit before it runs later audits or publishes WAP
+results. A later audit or publication can still fail. An audit query failure can
+produce a `finish` callback without an `AuditResult`. The physical materialization
+callback ends after its surrounding transaction and session exit.
+
+An observed operation can do no work, issue multiple statements, or trigger
+internal driver retries. The observation does not predict how many statements
+the operation will issue or report each retry.
+
+The query observation surrounds the base adapter's existing SQL logging and
+`_execute` call. It ends before a surrounding transaction commits and before
+later fetch or result consumption. Some adapter operations do not call this
+query hook. BigQuery session queries, BigQuery dataframe loads, and ClickHouse
+dataframe inserts may not produce query observations. These hooks do not provide
+a complete warehouse activity history or guarantee transaction success.
+
+SQL logging retains its existing behavior, including VALUES redaction for
+non-query expressions. The callback receives no SQL text, adapter objects, or
+warehouse query IDs. The `start` callback runs before SQL logging. Consumers can
+use that ordering to associate existing log entries with the invocation.
 
 ## Worker context
 
-When observation is active, the instrumented scheduler and snapshot DAG dispatch
-paths copy the admitting context at execution time. Snapshot schema creation and
-data-object listing also copy the context for each admitted value. Each node or
-value runs in its own copy, including serial execution. Consumer-owned
-`ContextVar` values can therefore carry parent identities into workers;
-successors inherit the admission context rather than the preceding worker's
-changes. Reusing a DAG executor captures a fresh context for each run.
-
-Context copies are shallow. Mutable objects stored in ContextVars can still be
-shared; consumers should use immutable values or replace values instead of
-mutating shared state. This behavior applies to these instrumented scheduler,
-DAG, schema-creation, and data-object-listing paths, not to arbitrary consumer
-threads or every SQLMesh concurrency helper.
-
-With no observer registered, these call sites do not opt into context copying;
-ContextVars keep the concurrency helper's existing serial and worker behavior.
+Outside callback delivery, an active observer makes
+`execution_context_factory()` return `contextvars.copy_context`.
+`Scheduler.run_merged_intervals()` passes that function to
+`concurrent_apply_to_dag()`. `SnapshotEvaluator` passes it to
+`concurrent_apply_to_snapshots()` and `concurrent_apply_to_values()`. Each node
+or value gets its own context copy, including during serial execution. Successor
+nodes start from the context captured for the DAG run. They do not inherit
+changes made by earlier workers. `ConcurrentDAGExecutor.run()` captures a fresh
+context for each run. Schema creation and data-object listing use
+`concurrent_apply_to_values()`, so each value gets its own context copy.
+
+Context copies are shallow. Mutable objects stored in `ContextVar` values remain
+shared across copies. A parent identity stored in a consumer-owned `ContextVar`
+is available to each worker. These copies apply to the scheduler DAG and to the
+`SnapshotEvaluator` calls above. Other SQLMesh concurrency helpers and arbitrary
+consumer threads do not receive these copies.
+
+With no observer registered, `execution_context_factory()` returns `None`. The
+concurrency helpers retain their existing `ContextVar` behavior for serial and
+worker execution.