[Bug] ExecutionMode.WATCHER renders consumer tasks for nodes removed by RenderConfig.exclude #2981

Astronomer Cosmos Version

1.15.0

dbt Core or Fusion version

dbt-core 1.12.2

Versions of dbt adapters

dbt-bigquery 1.12.0
dbt-adapters 1.24.5

LoadMode

DBT_MANIFEST

(Filed via the API rather than the web form — note the form's LoadMode dropdown doesn't currently offer DBT_MANIFEST, only AUTOMATIC / CUSTOM / DBT_LS / DBT_LS_FILE / DBT_LS_MANIFEST.)

ExecutionMode

WATCHER

InvocationMode

DBT_RUNNER

airflow version

3.1.7

Operating System

Cloud Composer 3 base image (composer-3-airflow-3.1.7-build.7), Debian-based

Deployment

Google Cloud Composer

Deployment details

Cloud Composer 3 (Google's managed Airflow), image composer-3-airflow-3.1.7-build.7, ENVIRONMENT_SIZE_MEDIUM, CeleryExecutor.

Cosmos is installed as a pinned PyPI package (astronomer-cosmos==1.15.0) via the environment's managed dependency list, and DAGs are served from a GCS bucket mounted over gcsfuse.

What happened?

Under ExecutionMode.WATCHER with TestBehavior.AFTER_EACH, RenderConfig.exclude correctly keeps excluded test nodes out of the producer's dbt build command, but Cosmos still renders a consumer sensor task for each excluded test.

Those consumers poll for a node status the producer will never publish, because the producer legitimately excluded the node. They resolve inconsistently — some are skipped with ... was skipped by the dbt command, others are marked FAILED, which fails the DAG run even though the dbt build was completely green.

The producer's command is correct:

['--no-partial-parse', 'build', '--select', 'path:models/<folder>',
 '--exclude', 'tag:<excluded-tag> resource_type:unit_test', ...]

and its summary confirms none of the excluded tests ran:

Done. PASS=1041 WARN=0 ERROR=0 SKIP=0 NO-OP=3 REUSED=0 TOTAL=1044

Yet five excluded singular tests are rendered as tasks, across two different DAGs (duplicated wherever more than one DAG selects a parent of the same test). Over a 12 hour window that produced ~99 and ~97 task instances respectively, and one was marked FAILED on four separate scheduled runs.

Mechanism

1. DbtGraph._apply_manifest_node_selection correctly removes the excluded tests from filtered_nodes.

2. DbtGraph.update_node_dependency (cosmos/dbt/graph.py) then iterates self.nodes — the unfiltered set — and re-inserts any TEST node into filtered_nodes whenever one of its parents survived selection:

for _, node in list(self.nodes.items()):
    if node.resource_type == DbtResourceType.TEST:
        for node_id in node.depends_on:
            if node_id in self.filtered_nodes:
                self.filtered_nodes[node_id].has_test = True
                self.filtered_nodes[node.unique_id] = node   # re-added

This looks deliberate, added for #949 so that tests aren't lost entirely under DBT_MANIFEST + select (test nodes don't appear in depends_on). The re-add doesn't distinguish "was never selected" from "was explicitly excluded".

3. A task is therefore rendered for the excluded test.

4. fix(graph): forward RenderConfig.exclude to test tasks for all test behaviors #2850 addressed this for other execution modes by forwarding render_config.exclude into the test task's own dbt command. Under WATCHER a consumer never invokes dbt — it watches the producer — so that forwarding has no effect here, and the producer had already excluded the node.

Net effect: under WATCHER, exclude governs execution scope but not rendering scope, and the two disagree.

What I expected

No consumer task should be rendered for a node that RenderConfig.exclude removes from the producer's command. Rendering scope and execution scope should agree.

Possibly relevant: ExecutionMode.WATCHER was marked stable in the same release (1.15.0) that shipped #2850, so this interaction may simply not have been exercised yet.

Relevant log output

Tests for model 'test.<project>.<excluded_test_name>' was skipped by the dbt command.
This may happen if it is an ephemeral model or if the model sql file is empty.

Marking task as FAILED. dag_id=<dag>, task_id=<excluded_test_name>_test, run_id=scheduled__...

# No START test / PASS / FAIL line exists for any excluded test across 24h of logs,
# confirming dbt never ran them.

How to reproduce

1. A dbt project with a singular test that has multiple parent models and carries a tag, e.g. hybrid-test.

2. Render a DbtDag with TestBehavior.AFTER_EACH (the default), where at least one parent of the tagged test lives inside models/<folder>:

DbtDag(
    render_config=RenderConfig(
        load_method=LoadMode.DBT_MANIFEST,
        select=["path:models/<folder>"],
        exclude=["tag:hybrid-test"],
        should_detach_multiple_parents_tests=True,
    ),
    execution_config=ExecutionConfig(execution_mode=ExecutionMode.WATCHER),
    ...
)

3. Observe the producer's dbt build includes --exclude tag:hybrid-test and does not run the test — but a consumer task for that test is still present in the rendered DAG.

4. The consumer resolves as skipped, or fails, failing the DAG run.

Anything else :)?

Happens on every DAG run. Beyond the spurious failures, each phantom consumer task pays a full Cosmos graph build on the worker before discovering it has nothing to do.

I'm a Cloud Composer user, which constrains the workarounds available to me. Composer installs Cosmos as a pinned PyPI dependency of the managed environment, so I can't patch the library in place or carry a local fork — changing the pin triggers a full environment update. A released fix, or a documented configuration workaround, is what would be actionable at my end. Happy to test a fix against a real Composer 3 environment if that's useful.

Minor and unrelated: the LoadMode dropdown on the bug report form doesn't include DBT_MANIFEST (it lists DBT_LS_MANIFEST, which is a different mode). Might be worth adding.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

🤖 Drafted with Claude Code