Daemon, Sensors, Schedules

A Dagster 177,056 deployment has three long-running parts: the webserver (UI and GraphQL API), one code location server per project, which loads your definitions in its own process so a broken import cannot take the UI down, and the daemon, which evaluates schedules and sensors, dequeues runs and drives backfills. dagster dev starts all of them in one container (1.13 says it is superseded by dg dev); in production each part runs separately and the instance keeps its history in PostgreSQL 1,289 (dagster-postgres), not SQLite 4,756 . The processes in l2-dagster, about 615-680 MiB in total:

Output of 34

127 MiB  python3.13 /usr/local/bin/dagster dev -m booknest_dagster.definitions -h 0.0.0.0 ...
101 MiB  python3.13 -m dagster code-server start --socket /tmp/tmp44b2kt9w --heartbeat ...
219 MiB  python3.13 -m dagster api grpc --lazy-load-user-code --socket /tmp/tmp0m1o4424 ...
155 MiB  python3.13 -m dagster_webserver --port 3000 --host 0.0.0.0 ...
137 MiB  python3.13 -m dagster._daemon run --log-level info --log-format colored ...

An asset job selects assets; for a partitioned job, Dagster builds a schedule that runs at 02:00 UTC for the day that just closed. A sensor is a function the daemon calls every 30 seconds or more; it returns RunRequests, and a run_key makes a request idempotent:

definitions.py (excerpt): the job, its schedule and a sensorPython

booknest_daily = dg.define_asset_job(
    "booknest_daily", partitions_def=daily,
    selection=dg.AssetSelection.groups("ingest", "mart")
    | dg.AssetSelection.assets("fct_sales"))
daily_at_two = dg.build_schedule_from_partitioned_job(booknest_daily, hour_of_day=2)
@dg.sensor(job=booknest_daily, minimum_interval_seconds=30)
def export_ready(context: dg.SensorEvaluationContext):
    marker = f"{DATA}/source/_READY"           # the order service writes the finished day here
    if not os.path.exists(marker):
        return dg.SkipReason("no _READY marker")
    ds = open(marker, encoding="utf-8").read().strip()
    stamp = int(os.path.getmtime(marker))
    yield dg.RunRequest(run_key=f"{ds}@{stamp}", partition_key=ds)   # one run per marker write

After dagster sensor start export_ready and writing 2026-06-28 into the marker at 01:50:50, the daemon log showed one run, then a skip on the next tick:

Output of 35

01:51:21 SensorDaemon - Completed launch of run 3adb617f-e4b2-... for export_ready
01:51:25 QueuedRunCoordinatorDaemon - Launched 1 runs.
01:51:50 SensorDaemon - Skipping 1 run for sensor export_ready already completed with run keys:
         ["2026-06-28@1790905850"]

max_concurrent_runs: 1 in the instance's dagster.yaml queues runs like max_active_runs=1. Sensors poll, so keep them cheap; to run when upstream assets change, declarative automation conditions (AutomationCondition.eager()) let the daemon decide instead.

MENU