Chapter 8: Actions, Events, and Workflows
Chapter Introduction
Chapters 2–7 built a read-only picture of the ontology — object types, properties, links, persistence, queries. That picture is incomplete. Real systems do things: a trader submits an order; a clinician schedules a procedure; a central bank publishes a release; a grid operator dispatches a generator. These are actions, and they are the bridge from analytics to operations. An ontology without an action layer is a museum exhibit; an ontology with one is a living operational system.
This chapter teaches the discipline of designing, executing, and auditing actions at scale. Three concepts run through everything:
- Actions — typed operations with explicit pre-conditions, parameters, and effects. Each action is an atomic unit; each invocation produces a record.
- Events — immutable facts that something happened. An action’s invocation produces an event; external events (an exchange-published trade, a sensor reading, an inbound message) trigger downstream actions.
- Workflows — composed sequences of actions, often with branching, retries, and human-in-the-loop steps. Modern data engineering treats workflows as first-class objects, version-controlled and tested.
The chapter walks the action-design discipline, then surveys the three production-grade workflow orchestration tools an analyst will likely encounter — Dagster, Prefect, and Apache Airflow — and contrasts them with Palantir Foundry’s actions framework. Closing sections work the four domain case studies (a quant-trading order-management workflow, a hospital prior-authorisation workflow, a central-bank data-release pipeline, an energy-market clearing workflow) at the design level.
Table of Contents
- The Anatomy of an Action
- Events as Immutable Facts
- Idempotency, Retries, and Replay
- Workflow Orchestrators — Dagster, Prefect, Airflow
- Foundry Actions — the Palantir Reference Pattern
- Case Study: Quant — Order-Management Workflow
- Case Study: Healthcare — Prior-Authorisation Workflow
- Case Study: Macroeconomics — Data-Release Pipeline
- Case Study: Energy — Market-Clearing Workflow
The Anatomy of an Action
A well-designed action has six pieces of metadata. Skip any of them and the operational system gradually corrupts itself.
- Name — a verb.
ApproveLoan,SubmitOrder,DispatchGenerator,PublishMonthlyCPI. The business team should recognise it. - Pre-conditions — the state of the ontology that must hold for the action to be valid. “
Account.kyc_status == verifiedANDAccount.status == openANDprincipal <= officer.authority_limit.” - Parameters — typed inputs the actor provides. Each parameter has a name, type, and validation rule.
- Effects — the post-action state. “Create one
Loanobject; updateCustomer.outstanding_loanscount; emit aLoanApprovedevent.” - Actor / authorisation — who is allowed to invoke the action and under what authority. “Loan officer with
LOAN_APPROVEpermission and an authority limit of at leastprincipal.” - Audit metadata — actor identity, timestamp, parameters, before-state, after-state. Captured automatically; never optional.
Three rules that prevent the most common production errors:
- Pre-conditions live in the action, not in the caller. A UI button might enforce a limit; a batch script might bypass it; an internal API might too. Putting the check in the action means every caller is constrained the same way.
- The audit record is atomic with the effect. If the system creates a loan but then crashes before recording the audit event, the reconciliation is impossible. Use a database transaction; if the effect can’t be transactional, use the outbox pattern (write the effect and the event to the same DB in one transaction; a separate process publishes the event downstream).
- The actor is part of the metadata, always. “The system did it” is not an actor. A real human, a real service account, a real workflow run. Without this, regulators and incident-response teams can’t reconstruct who did what.
Events as Immutable Facts
Each action’s invocation produces an event — an immutable record describing what happened. Events are append-only and never edited; corrections produce new events that supersede the original. This is the same event-sourcing discipline introduced in Chapter 3 but applied at the action level.
A well-formed event has:
- A unique event ID.
- A topic / event type (
LoanApproved,OrderFilled,MeterReadingArrived). - A timestamp.
- A payload (the action’s parameters and effects in serialised form).
- A pointer to the actor that produced it.
- (Optional but recommended) a correlation ID linking related events across a workflow.
Modern event-sourcing infrastructure: Apache Kafka is the production default; Apache Pulsar, Redpanda, and EventStoreDB are alternatives. Inside Postgres, the Debezium and Outbox-pattern combinations get most of the way to event-sourcing without operating a separate event log.
The correlation ID is the most-undervalued piece of metadata in operational systems. It lets you reconstruct the full chain of events that arose from one initial trigger — a customer’s loan application produced four events spanning four services. Without the correlation ID you would join on timestamps and hope.
Idempotency, Retries, and Replay
Three operational disciplines that protect a workflow against the messy realities of distributed systems.
Idempotency — running an action twice with the same parameters produces the same result. A non-idempotent CreateLoan is dangerous: a network retry could create two loans. The standard fix is an idempotency key — a client-supplied unique key; the server records (key → result) and short-circuits subsequent invocations with the same key.
Retries with backoff — every action that crosses a network boundary will fail transiently. The pattern is exponential backoff with jitter, capped at a maximum retry count. Production frameworks implement this; you should configure it explicitly per action rather than accepting the default.
Replay — given the event log, re-execute the workflow from any historical point. Used for backfills, regulatory audits, and “what would have happened” counterfactuals.
The idempotency cache shown above is the operational core of every modern payments API (Stripe, Square, Adyen) and every “create resource” endpoint at every cloud provider (AWS, GCP, Azure all support an idempotency token in their write APIs).
Workflow Orchestrators — Dagster, Prefect, Airflow
A workflow composes actions into a directed acyclic graph of tasks with dependencies, retries, schedules, parallelism, and observability. Three orchestrators dominate the open-source space; each has its sweet spot.
Apache Airflow is the venerable default. Born at Airbnb (2014), now an Apache project. Mature ecosystem of operators; widely deployed (every major bank and tech company). Strong scheduling, weak typing of data passed between tasks. Production at: Airbnb, Lyft, Walmart, Twitter (pre-Musk), Adobe, GE Aerospace.
Prefect rebuilt the workflow concept around Python-native control flow. A workflow is just a Python function decorated with @flow; tasks are @task-decorated functions. Strong UX, dynamic workflows (the DAG can change at runtime), excellent for ML-style pipelines. Production at: LinkedIn, Coca-Cola, Microsoft Genomics.
Dagster is the most opinionated and the most data-aware. Workflows are typed; assets (the things produced by the workflow) are first-class; data quality and lineage are built-in. The right choice when the workflow is fundamentally about producing data assets (which it almost always is for analytics). Production at: GitHub, VMware, Mailchimp, Mozilla.
Three things every orchestrator does well:
- Define a task as a Python function with declared inputs and outputs.
- Schedule (cron, event-driven, manual).
- Retry on failure with backoff.
- Track execution history; surface logs and timing.
- Provide a UI for operators.
The illustrative example below uses pseudocode in Prefect-flavoured Python because none of the orchestrators run in Pyodide. The conceptual content is identical across the three.
The flow is a Python function; each @task is an action; retries and dependencies are declarative; the UI surfaces every run. The same workflow expressed in Airflow uses DAG/PythonOperator and is wordier; in Dagster it uses @asset/@op and gains typed asset lineage.
- Airflow — choose if you already have it, your org has Airflow ops expertise, your workflows are mostly schedule-driven ETL.
- Prefect — choose for Python-native ML / data-science workflows; dynamic DAGs; if your team values UX.
- Dagster — choose for data-platform engineering with strong lineage / asset semantics; if you treat datasets as first-class artefacts.
Foundry Actions — the Palantir Reference Pattern
Palantir Foundry’s actions framework is the most-polished commercial implementation of the action concept. Each action is a typed function with explicit parameters, pre-conditions (called submission criteria), and post-conditions, registered against an object type in the ontology. Foundry’s UI generates forms automatically; permissions are enforced; every invocation produces an audit event tied to the object it affected; lineage is automatic.
Conceptually, a Foundry action looks like:
Action: ApproveLoan
Object type: LoanApplication
Parameters: amount (Float, min 1000, max 100000)
rate (Float, min 0.03, max 0.20)
notes (String, optional)
Submission criteria:
$applicant.kyc_status == "verified"
$applicant.country in ["US", "CA"]
$action.amount <= $actor.authority_limit
Effects: Create Loan object; link to applicant.
Update LoanApplication.status to "approved".
Update Applicant.outstanding_loans_count by +1.
Side-effect: Emit LoanApproved event to "credit-decisions" topic.
Permissions: LOAN_APPROVE on application.regionThis is the same six-piece metadata every action should have, but enforced and exposed by the platform. The open-source path replicates the pattern: pick a workflow orchestrator (Dagster / Prefect / Airflow), wrap each action as a typed task, store the audit events in Postgres / Kafka, build the UI on top of GraphQL.
Case Study: Quant — Order-Management Workflow
A simplified buy-side order workflow:
[Portfolio Manager submits order intent]
│
▼
┌─────────────────────────┐
│ Action: SubmitOrder │ Pre: PM has trading authority for portfolio
│ Params: instrument, side,│ Risk pre-check passes
│ qty, limit_price, │ Effect: Create Order(status="pending")
│ urgency, broker │ Event: OrderSubmitted
└─────────────┬───────────┘
▼
┌─────────────────────────┐
│ Action: PreTradeRisk │ Pre: -
│ (RiskService task) │ Effect: Update Order.risk_check = ok/blocked
│ │ Event: PreTradeRiskCleared OR PreTradeRiskBreach
└─────────────┬───────────┘
│
────────┴────────
│ │
ok │ │ blocked
▼ ▼
[Route to broker] [Notify PM]
│
▼
┌─────────────────────────┐
│ Action: SendChildOrder │ Pre: Order.risk_check == "ok"
│ (per broker, per fill) │ Effect: Append ChildOrder to Order
│ │ Event: ChildOrderSent
└─────────────┬───────────┘
▼
[Fill events stream back from broker → Update Order with fills → eventually OrderFilled or OrderCancelled]Every node is an action; every transition is an event. The full graph is the trade lifecycle, which every modern OMS (Aladdin, Charles River, Eze, internal builds at Citadel/Two Sigma) models as roughly this structure with vendor-specific elaborations.
The audit value is enormous: a regulator asking “show me every action taken on this order from submission to settlement” gets one query that returns the full ordered event list — actor, timestamp, parameters, effect — with no manual reconciliation across systems.
Case Study: Macroeconomics — Data-Release Pipeline
A central bank or statistical agency operates a data-release pipeline on a strict schedule and an even stricter embargo:
[Internal data collection — already underway days before publication]
│
▼
┌─────────────────────────┐
│ Action: PrepareRelease │ Pre: All required components ingested
│ │ Effect: Create Release(status="draft")
│ │ Event: ReleaseDraftCreated
└─────────────┬───────────┘
▼
┌─────────────────────────┐
│ Action: QAReview │ Pre: Release in draft
│ │ Effect: Release.status ∈ {qa_passed, qa_failed}
│ │ Event: QAReviewCompleted
└─────────────┬───────────┘
▼
┌─────────────────────────┐
│ Action: SeniorSignoff │ Pre: qa_passed, by senior economist
│ │ Effect: Release.status = "approved"
│ │ Event: ReleaseApproved
└─────────────┬───────────┘
▼
[Scheduled publish at exact embargo time — automated; no human in the loop]
┌─────────────────────────┐
│ Action: PublishRelease │ Pre: Release.status == "approved",
│ │ current time >= embargo_time
│ │ Effect: Release.status = "published",
│ │ push SDMX bundle to public endpoint
│ │ Event: ReleasePublished
└─────────────────────────┘The embargo property is critical and unique to this domain. The action’s pre-condition includes a time check; the workflow scheduler triggers PublishRelease exactly at embargo. Any leak before the embargo is a regulatory incident.
Case Study: Energy — Market-Clearing Workflow
A wholesale electricity market (PJM, ERCOT, CAISO) runs a daily clearing workflow that determines who generates how much power, where, and at what price.
[Generators submit day-ahead bids]
│
▼
┌─────────────────────────┐
│ Action: SubmitBid │ Pre: Generator certified, bid window open
│ │ Effect: Bid stored
│ │ Event: BidReceived
└─────────────┬───────────┘
▼ (at gate-closure)
┌─────────────────────────┐
│ Action: RunDAClearing │ Pre: Bid window closed
│ │ Effect: Solve security-constrained
│ │ economic dispatch optimisation,
│ │ produce per-generator schedule
│ │ and locational marginal prices
│ │ Event: DAMarketCleared
└─────────────┬───────────┘
▼
┌─────────────────────────┐
│ Action: PublishLMP │ Pre: -
│ │ Effect: Push prices to all market participants
│ │ Event: LMPPublished
└─────────────────────────┘
[Real-time market then runs every 5 minutes with similar but faster cycle]The clearing action is the most-complex one in the workflow: it runs a mixed-integer optimisation over thousands of generators, transmission constraints, and reserve requirements. The optimisation is its own engineering specialty, but from the workflow perspective it is a single action with pre-conditions, parameters (the network state, the bid set, the reserve requirements), effects (the cleared schedule and prices), and a downstream event.
- Was the event written to the broker at all? Check the action’s audit log and the broker’s ingestion metrics for the timestamp. If the action wrote it, the broker has it. (2) If the broker has it, did the consumer subscribe correctly? Check the consumer-group offsets in Kafka, or the equivalent in your event bus. (3) If the consumer subscribed, did its handler crash silently? Check the consumer’s error log; modern consumers may move past a message without committing the offset if the handler fails. The fix in each layer is different — the diagnostic is to walk the layers in order. The outbox pattern from §2 prevents the case where the action ran but the event was never published.
Chapter Wrap-up
Actions are the verbs of the ontology; events are their immutable record; workflows compose them into operations a regulator can audit and a business can trust. Six metadata pieces per action (name, pre-conditions, parameters, effects, actor, audit), three operational disciplines (idempotency, retries, replay), three orchestrator choices (Dagster, Prefect, Airflow) plus Foundry as the commercial reference.
The four domain case studies share the same structural pattern: a sequence of actions, each with explicit pre-conditions and effects, joined by events with correlation IDs, scheduled by an orchestrator, audited end-to-end. The mechanical part is small; the design discipline is the entire game.
Chapter 9 turns from operations to intelligence on top of the ontology — RAG over the knowledge graph, GNNs that learn ontology-aware representations, and KG-embedding methods (TransE, DistMult, RotatE) that turn an ontology into a vector space.
← Chapter 7 · Contents · Chapter 9: AI on Top of the Ontology →