Understand Ansible execution

Follow a job from launch to output, choose where it executes, and understand why it may wait. The same execution model connects manual jobs, schedules, workflows, ad hoc commands and repository triggers. Use the diagrams below to locate the stage you need to investigate.

Where the work runs

Stackweaver's Ansible engine is built from a small number of cooperating services. The API owns all state and decisions; runners only execute. Two Redis lists connect them: ansible_jobs carries playbook runs and ansible_sync carries inventory and playbook syncs. Self-hosted agents do not use Redis at all - they poll the API over HTTPS.

flowchart LR
    UI["Web UI / API clients"] --> API["API server"]
    GH["GitHub / Azure DevOps webhooks"] --> API
    CB["Provisioned hosts (callbacks)"] --> API
    API --> PG[("PostgreSQL")]
    API -->|"ansible_jobs / ansible_sync"| Redis[("Redis")]
    Redis --> Runner["Platform ansible-runner"]
    Agent["Self-hosted agent"] -->|"heartbeat + artifacts over HTTPS"| API
    Runner --> PG
    API -.->|"scheduler tick (30s)"| API

Why a launched job can wait

A disabled template rejects a launch with 409; it does not create a held job. Inventory dependencies and concurrency limits can hold an otherwise valid job until the blocking condition clears. The scheduler rechecks held jobs periodically.

flowchart TD
    L1["UI / API launch"] --> GATE
    L2["Schedule"] --> GATE
    L3["Workflow node"] --> GATE
    L4["VCS push (launch on webhook)"] --> GATE
    L5["Provisioning callback"] --> GATE
    L6["Ad hoc command"] --> GATE
    GATE{"Template disabled?"} -->|yes| REJ["409 rejected"]
    GATE -->|no| VARS["Merge variables<br/>(platform → sets → template → launch)"]
    VARS --> DEP{"Inventory dependencies stale?<br/>(update-on-launch sources,<br/>constructed rebuild)"}
    DEP -->|yes| SYNC["Queue syncs + HOLD job"]
    DEP -->|no| CONC{"Concurrency gate?<br/>(allow_simultaneous off +<br/>active run exists)"}
    CONC -->|yes| HOLD["HOLD job"]
    CONC -->|no| DISPATCH
    SYNC --> TICK["Scheduler tick releases<br/>when syncs settle"]
    HOLD --> TICK
    TICK --> DISPATCH{"Agent pool set?"}
    DISPATCH -->|no| Q["Push to Redis ansible_jobs"]
    DISPATCH -->|yes| POLL["Wait for agent heartbeat pickup"]

Job slicing happens at launch: a template with a slice count above one fans out into N jobs, each receiving a deterministic subset of the inventory's hosts (sorted, distributed round-robin). Slices run simultaneously and are grouped under one slice group.

Configure and launch a template

Declare launch settings with Terraform

Configure the Stackweaver provider, then manage an existing template or create one from existing resources in the same organization:

variable "project_id" { type = string }
variable "playbook_id" { type = string }
variable "inventory_id" { type = string }

resource "stackweaver_ansible_job_template" "baseline" {
  organization       = "example-platform"
  name               = "Production baseline"
  project_id         = var.project_id
  playbook_id        = var.playbook_id
  inventory_id       = var.inventory_id
  enabled            = true
  timeout_seconds    = 1800
  allow_simultaneous = false
  job_slice_count    = 1
}

This manages the template, not an execution. Import an existing template before adopting it into a new Terraform configuration, review the plan and apply it. Credentials are attached through the provider's template-credential attachment resource shown below or the template's UI/API. For an existing stored credential, add this attachment to the same configuration:

variable "credential_id" { type = string }

resource "stackweaver_ansible_job_template_credential" "baseline" {
  job_template_id = stackweaver_ansible_job_template.baseline.id
  credential_id   = var.credential_id
}

The credential must belong to the same organization. Creating a credential alone does not attach it to a template. The API explorer provides the complete launch schema.

Launch once through the API

Use a token allowed to execute the template. Set the API origin, token and TEMPLATE_ID. Save this as launch.json:

{
  "data": {
    "type": "ansible-jobs",
    "attributes": {
      "extra-vars": {"deployment_environment": "production"},
      "limit": "web",
      "tags": "configure"
    }
  }
}

curl --fail-with-body --request POST \
  --header "Authorization: Bearer $STACKWEAVER_TOKEN" \
  --header 'Content-Type: application/vnd.api+json' \
  --data-binary @launch.json \
  "$STACKWEAVER_API_URL/api/v2/ansible/job-templates/$TEMPLATE_ID/launch"

With no body, or an empty object, this endpoint uses the template's launch values. Overrides belong under data.attributes, not top-level snake-case fields. Invalid documents return 400 and launch nothing. Check the returned job ID and status, then inspect its output.

Every slice receives the same overrides. Relaunching a job uses that original job's configuration and credentials, rather than silently substituting today's template values. A direct organization job launch can also carry a credentials relationship; each named credential must belong to that organization or the request is refused.

Inspect the queue

Set ORG to the organization name, then read its pending/running jobs, oldest first:

curl --fail-with-body \
  --header "Authorization: Bearer $STACKWEAVER_TOKEN" \
  "$STACKWEAVER_API_URL/api/v2/organizations/$ORG/ansible/jobs/queue"

For a sliced launch, list project jobs with filter[slice-group-id] set to the slice-group ID. That query returns the group's slices together in slice order. There is no management YAML object that replaces template registration or a launch request. Your playbook YAML still defines what Ansible executes.

Choose platform or self-hosted execution

Both paths run the same playbook with the same options - the difference is how the work and its inputs reach the executor. The platform runner is push-based and trusted with database access; the agent is pull-based and receives everything as a self-contained bundle, so it works from inside a private network with outbound-only connectivity and no credentials for Stackweaver's infrastructure.

Platform runner flow

sequenceDiagram
    participant API
    participant Redis
    participant Runner as Platform runner
    participant DB as PostgreSQL
    API->>Redis: enqueue job message (job id + pre-resolved clone URL)
    Runner->>Redis: dequeue
    Runner->>DB: load job, template, credentials
    Note over Runner: prepare playbook<br/>(cached snapshot / fresh clone /<br/>generated ad hoc playbook)
    Note over Runner: render inventory (+ slice filter),<br/>write SSH keys, vault files,<br/>cloud + OIDC environment
    Runner->>Runner: ansible-playbook (jsonl callback)
    Runner->>DB: stream events, final status + stats

Self-hosted agent flow

sequenceDiagram
    participant Agent
    participant API
    participant DB as PostgreSQL
    loop every heartbeat
        Agent->>API: heartbeat (pool, capacity)
        API->>DB: pending released jobs for this pool (bounded by free capacity)
        API-->>Agent: job id (if any)
    end
    Agent->>API: POST job start (atomic claim)
    API-->>Agent: 200 claimed / 409 already taken → skip
    Agent->>API: GET job artifacts<br/>(X-Stackweaver-Runner-Capabilities: playbook-snapshot)
    API-->>Agent: inventory content (sliced), credentials,<br/>vault passwords, env vars, playbook source mode
    alt cached playbook with a snapshot
        Agent->>API: GET playbook snapshot
        API-->>Agent: snapshot tarball (streamed from object storage)
    else fresh playbook, or cached with no snapshot yet
        Note over Agent: clone repo (VCS)
    else ad hoc
        Note over Agent: write shipped playbook content
    end
    Note over Agent: install requirements.yml<br/>(per-project Galaxy cache)
    Agent->>API: POST source banner + Galaxy install events
    Agent->>Agent: ansible-playbook
    Agent->>API: POST events + final status

Follow an inventory sync

Every way an inventory can refresh goes through the same queue and lands in the same sync history (the Syncs tab), each run with its captured output. Long syncs can be tailed live: the runner flushes output to the run's history row every couple of seconds while it executes, and the output dialog polls while a run is active. Use these read operations, with your inventory or sync ID:

flowchart TD
    T1["Manual sync / Rebuild button"] --> Q["ansible_sync queue"]
    T2["Schedule"] --> Q
    T3["VCS push webhook"] --> Q
    T4["Update-on-launch (job launch)"] --> Q
    T5["Workflow sync node"] --> ENG["runs in API"]
    Q --> R["ansible-runner"]
    R --> B1["VCS file inventory:<br/>clone repo, parse file"]
    R --> B2["Dynamic source:<br/>ansible-inventory with cloud plugin"]
    R --> B3["Constructed inventory:<br/>export inputs, apply rules"]
    B1 --> REC["Reconcile hosts/groups in DB"]
    B2 --> REC
    B3 --> REC
    REC --> HIST["Sync history row<br/>(status, trigger, counts, output)"]

Follow a workflow

Launching a workflow snapshots its graph and creates a run; from there the scheduler tick advances execution. Nodes start when their parents finish with a matching edge condition, approval nodes pause the run until a human decides, and the run rolls up once no node can advance.

flowchart TD
    LAUNCH["Launch (UI, API, or schedule)"] --> SNAP["Snapshot graph into a run"]
    SNAP --> ROOT["Start root nodes"]
    ROOT --> TICKLOOP["Scheduler tick"]
    TICKLOOP --> DONE{"Node finished?"}
    DONE -->|on_success / on_failure / always| NEXT["Evaluate child edges"]
    NEXT --> CONV{"Convergence: any parent<br/>or all parents?"}
    CONV -->|satisfied| START["Claim + start child node"]
    START --> KIND{"Node type"}
    KIND -->|job template| JOB["Launch job (full gates)"]
    KIND -->|approval| APPR["Wait for approve/deny<br/>(timeout denies)"]
    KIND -->|inventory sync| ISYNC["Sync the source"]
    JOB --> TICKLOOP
    APPR --> TICKLOOP
    ISYNC --> TICKLOOP
    TICKLOOP --> ROLLUP["All terminal → run successful/failed"]

Launching a workflow and approving/denying an approval node require job-execution permission in its project. A launch without a body uses the workflow's variables. To override keys, save this as workflow-launch.json:

{
  "data": {
    "type": "ansible-workflow-jobs",
    "attributes": {"extra-vars": {"deployment_environment": "production"}}
  }
}

curl --fail-with-body --request POST \
  --header "Authorization: Bearer $STACKWEAVER_TOKEN" \
  --header 'Content-Type: application/vnd.api+json' \
  --data-binary @workflow-launch.json \
  "$STACKWEAVER_API_URL/api/v2/ansible/workflows/$WORKFLOW_ID/launch"

Unlike a template launch, an empty object is not a valid supplied workflow body. A malformed document or non-object extra-vars returns 400 without a run. A workflow whose own project or inventory lies outside its organization is refused with 409. Check the returned run and each node's result rather than treating launch acceptance as completion.

Configure graph nodes and edges through the API; the UI creates, lists and launches workflows but has no drag-and-drop graph editor. Nested workflows and launch-time survey prompts are not supported. See the Ansible integration reference for graph fields and validation. The graph is not a playbook YAML file.

Run an ad hoc command

Run Command executes a single module against an inventory without a playbook or template. The platform generates a one-task playbook from the module and arguments and pushes it through the normal job pipeline, so ad hoc runs get live output, events, and statistics like any job - on a platform runner or any agent pool.

flowchart LR
    RC["Run Command dialog"] --> ALLOW{"Module in org allowlist?"}
    ALLOW -->|no| DENY["400 rejected"]
    ALLOW -->|yes| PERM{"ansible:adhoc:execute?"}
    PERM -->|no| FORBID["403 rejected"]
    PERM -->|yes| GEN["Generate transient playbook<br/>(action: module + args)"]
    GEN --> PIPE["Normal job pipeline<br/>(gates, dispatch, events)"]

The allowed modules are an organization setting (Settings → Ansible), defaulting to AWX's list, and the permission is distinct from template execution so you can grant ad hoc access separately.

The API command is:

POST /api/v2/ansible/inventories/{id}/actions/run-command

The job runs under the project named by the optional project-id attribute. Without it the job runs under the inventory's own project, or under the organization's default project when the inventory belongs to none. The credential, the agent pool and the project the request names must belong to the inventory's organization: one that does not exist there is rejected with 400, and the detail says which one was not found.

Webhook Triggers

Two unauthenticated-by-design entry points exist, both validated by their own secret:

flowchart TD
    PUSH["Git push (GitHub / Azure DevOps)"] --> WH["Webhook endpoint<br/>(signature verified)"]
    WH --> S1["Sync affected playbooks<br/>and VCS inventories"]
    WH --> S2["Launch templates with<br/>launch on webhook enabled"]
    HOST["Freshly provisioned host"] --> CBEP["Callback endpoint<br/>(host config key, constant-time)"]
    CBEP --> MATCH{"Caller IP in the<br/>template's inventory?"}
    MATCH -->|yes| CBJOB["Launch job limited to that host"]
    MATCH -->|no| CBDENY["400 rejected"]

For launch on webhook, fresh source mode fetches the selected branch's HEAD when the job executes; it does not pin the job to the triggering commit. In cached mode a launch can race the snapshot sync and run the previous captured commit. Provisioning callbacks require the template to allow callbacks and the calling host to already exist in the template's inventory.

Scheduled work and delayed transitions

A single 30-second loop in the API drives everything time-based. Knowing what it owns explains most "why did this happen a few seconds later" questions:

1. Schedules - due cron schedules launch their job template, inventory source sync, or workflow. A due playbook_sync schedule records a run and queues no sync.
2. Held-job release - re-evaluates jobs held by the concurrency or dependency gates and dispatches the ones whose conditions cleared.
3. Workflow progression - advances running workflow runs node by node.
4. Notification dispatch - finds job/workflow transitions and delivers configured webhook, email and Microsoft Teams messages. Treat the notification as a signal to inspect the run, not as a transaction proving a remote receiver acted exactly once.
5. Retention cleanup - once a day, deletes finished jobs older than the organization's retention window (template overrides apply; each template's most recent job is always kept).

Configure schedules

Use the dedicated inventory-source schedule. A source form's Sync Schedule field alone does not activate recurring execution. Invalid cron expressions, timezones or targets from another organization are rejected with 400; the error identifies the failed input.

A schedule update uses a schedules document. Supplied name, description, cron-expression, timezone and config fields change; omitted ones remain. Updated cron/timezone values are validated again.

• Enable recalculates the next run without launching immediately. A schedule can be enabled even if its template is disabled; each refused due launch records a failed run.
• Disable stops recurring launches.
• Run now executes without waiting for cron. A disabled job template returns 409 without a launch.
• A target missing from the schedule's organization at execution time records a failure.

The enable, disable and run-now actions return the resulting schedule. Use the API explorer for the schedule endpoints and the canonical Ansible catalogue to choose the task guide for each capability.