Skip to content

Guides

Prove a claim

Use a baseline, reproducible commands and actual execution results.

Measure before the change

Record the baseline before implementation. It distinguishes a regression from an existing failure. A claim links a file, an assertion and a command; registering it neither runs the command nor proves the assertion.

ork verify <thread> --baseline
ork claims add <thread> src/filtro.ts --claim "o filtro respeita o fuso" --verificar "node --test test/filtro.test.js"

Rerun against current state

ork verify reruns claims and manifest checks. Read the cause, exit code and interrupted commands. A timeout is incomplete execution, not a pass or an automatic explanation of the underlying defect.

When the runner names the failures, verify compares test by test within the same command: a name that appears only now is a regression and blocks with verify.regression, one present on both sides stays as earlier debt, and one that existed only in the baseline is recorded as fixed. A renamed test counts as the old one fixed and the new one regressed. Without names, the per-command comparison applies, and the output says so instead of claiming zero regressions.

ork verify <thread>
ork retry plan <thread>

Read the JUnit report from any runner

To read the structured result, replace only the command’s value with a map holding comando and junit, the report path relative to the worktree. Both forms live side by side in the same manifest, and each runner writes the report with its own option, as in the block below.

Verify deletes the report before the command and reads the new one afterwards. Only a regular file inside the worktree, produced by that run, counts. A missing, invalid, oversized or old report, or one reached through a link, is not evidence, and the ledger records the reason.

The exit code still judges the command: the XML never turns red into green. With a valid JUnit report, every failure and error goes to the ledger, with the counts and the summed duration, and the text summarizes the first ten. Without a report, the name comparison keeps the up to ten names verify extracts from the output.

In the object form, the junit key is required. The only exception is an Agile lane that declares recursos; even there, filtro and relogio require the report, and the error message says which key asked for it.

verify:
  build: "npm run build"
  test:
    comando: "pytest --junitxml reports/pytest.xml"
    junit: reports/pytest.xml

pytest --junitxml reports/pytest.xml
vitest run --reporter=junit --outputFile=reports/vitest.xml
JEST_JUNIT_OUTPUT_FILE=reports/jest.xml jest --reporters=default --reporters=jest-junit
PLAYWRIGHT_JUNIT_OUTPUT_NAME=reports/playwright.xml playwright test --reporter=junit
gotestsum --junitfile reports/go.xml -- ./...

Asynchronous verify over MCP

Over MCP, ork_verify starts the round in a server worker, outside the session sandbox, and returns a rodadaId at once. The worker waits in the same heavy queue as the CLI, uses the manifest timeouts and records the same verdict as ork verify. While the round is alive, calling ork_verify again returns the same id, without opening a second run.

ork_verify_status follows the round: na-fila, rodando, concluido or falhou. Poll every 30 to 60 seconds. A dead process, a worker that ends without a verdict and a lost lease are typed failures and never turn into a pass.

If the queue cannot prove that a lease owner is alive, the state is concurrency.lease-indeterminado: the session waits with --esperar, and the lease is never deleted or taken by inference. When the session needs an action outside its authority, ork_request_conduction records an enumerated reason and a one-line command, without running it, and the request stays with conduction.

Fix and verify again

A failure opens correction work with an explicit scope. Inspect the round before rerunning it. Behavior changes require full verification again; the core owns attempt limits and escalation.

ork fix open <thread>
ork fix list <thread>
ork fix reverify <thread>

On a fix round, rerun what failed first

A command with JUnit can also declare the filtro key: the runner’s own command with the {testes} marker exactly once. On a fix round, verify replaces the marker with the names that failed in the last verify_run, each as a separate, escaped argument.

The names arrive exactly as the report produced them, with no conversion to a framework id, so the declared command must know how to consume them. The round follows these rules:

  • Without filtro, without a valid JUnit report or without names from the previous round, the full command runs, as before.
  • If the full command changed since the previous round, the names are not reused.
  • A red filtered pass fails right there and marks the round as partial, with the names in the text and in the ledger.
  • A green filtered pass moves on to the full command, and only that result can turn the round green.
  • In Agile, each lane declares its own junit and its own filtro, and never receives another lane’s names.
  • ork ship refuses a partial receipt, even a green one: the full suite green at HEAD is still required.
verify:
  test:
    comando: "pytest --junitxml reports/pytest.xml"
    junit: reports/pytest.xml
    filtro: "pytest --junitxml reports/pytest.xml {testes}"

Agile Check, the default CHECK

In version 0.6.10, the default CHECK is the Agile Check. It runs the preparation once, the claims against its product with the hash guard, the test lanes and the scan in parallel. The Classic Check remains available through check.mode: classic in the manifest or through ork verify --check classic|agile, and the round's option overrides the manifest.

Both modes have the same coverage and the same verdict: Agile never reduces coverage silently, and an unavailable piece falls back to the Classic equivalent, with the piece, the reason and what ran instead recorded in agile.pecas. The preparation runs once, and a claim that starts with the verify.preparo steps reuses it while the tree and the product hash are those from the end of the preparation; any divergence runs the whole command. The lanes run in parallel, and the lane that starts containers waits for the scan to finish.

Every verify_run records the round's mode and where it came from (opcao, manifesto or padrao), the check line in ork verify's text shows the mode name, and ork thread status shows the last verify's mode. Without the check.agile block, Agile runs the same commands as Classic. A value other than agile or classic makes the manifest invalid.

Since 0.6.3, the hermetic lane gets the reserved CPUs back when the scan and the integration finish, and the expensive tests build once per process the repository templates that each case copies. Lanes can use tmpfs with ORK_CHECK_TMPFS=auto when memory, space and writes have headroom; the default stays on disk, and every fallback states the reason.

In 0.6.5, the Agile group keeps the sum of its reservations within the measured capacity. The scheduler decides by the median of several load and steal samples, without erasing the peak from the evidence. Steal above the threshold reduces each piece’s capacity by the measured fraction, and the scheduler piece reports degradado, with the capacity used, instead of sending the whole group back to the Classic order. That order only returns when the parallel group does not fit or the machine is busy.

The integration reserves the CPU it measures: with a compatible profile, the reservation follows the lane’s sustained peak and the hermetic lane gets the rest; without one, the reservation stays conservative. No test is removed, Classic does not change, and the ratio against Classic only counts with three valid bench pairs.

check:
  mode: classic

ork verify <thread> --check classic
ork verify <thread> --baseline --check agile

Native selection on the warm round

A check.agile.test lane can declare the selecao key: the runner’s native command that picks tests by the change, the paths it guarantees to map (mapeia) and the configuration files that void that guarantee (configuracao). Without the key, the lane runs in full, and verify.test and Classic do not accept the declaration.

The command carries exactly one marker: {base} receives the previous round’s commit, for a runner that computes the diff through Git, and {arquivos} receives each changed path as a separate argument.

verify_run records the selection and, for each lane, the mode, the reason, the files and the base. The text says SELECTED or FULL. A green selected round can feed the next one but stays partial: ork ship refuses it under the same contract as the name-based fix round.

Selection only applies when the last verify_run was green, with a recognized cache, a commit that is an ancestor of HEAD, the same merge-base and a clean tree. Otherwise the lane runs in full and records the reason:

  • a missing, invalid or red cache, or one from another base or outside the ancestry;
  • a change to orkastery.yaml, devmaster.yaml, a lockfile or a declared runner configuration;
  • a changed path outside mapeia, no verifiable path, or an empty mapeia;
  • a cold round, which always runs in full, or Classic Check.
check:
  mode: agile
  agile:
    test:
      vitest:
        comando: "vitest run --reporter=junit --outputFile=reports/vitest.xml"
        junit: reports/vitest.xml
        selecao:
          comando: "vitest run --changed {base} --reporter=junit --outputFile=reports/vitest.xml"
          mapeia: [src/**, test/**]
          configuracao: [vitest.config.*, vite.config.*]
      jest:
        comando: "jest --ci --reporters=default --reporters=jest-junit"
        junit: reports/jest.xml
        selecao:
          comando: "jest --findRelatedTests {arquivos} --ci --reporters=default --reporters=jest-junit"
          mapeia: [src/**, test/**]
          configuracao: [jest.config.*, package.json]
      playwright:
        comando: "playwright test --reporter=junit"
        junit: reports/playwright.xml
        selecao:
          comando: "playwright test --only-changed={base} --reporter=junit"
          mapeia: [src/**, tests/**]
          configuracao: [playwright.config.*]

Workers and resources per lane

When it starts a lane, Agile exports ORK_WORKERS with the current reservation the scheduler gave it, always at least one worker; a lane that starts after a release gets the new value. Classic exports the same variable with the full available capacity, so the same command works in both modes without changing the Classic order.

The text form of a lane stays valid. In the object form, the recursos key sits beside comando, junit, filtro, relogio and selecao. Two lanes that share a resource, such as a local database or a port, do not overlap, and different resources create no dependency. A lane acquires its whole set before taking a slot, so there is no circular wait.

Each runner reads the variable in its own command and keeps its fallback when it runs outside ork. ORK_WORKERS is the public contract; the round’s other variables stay internal.

check:
  agile:
    test:
      rapida: "npm run test:rapida"
      e2e:
        comando: "PLAYWRIGHT_JUNIT_OUTPUT_NAME=reports/playwright.xml npx playwright test --reporter=junit"
        junit: reports/playwright.xml
        recursos: [postgres-local, porta-http]

pytest -n "${ORK_WORKERS:-auto}"
playwright test --workers="${ORK_WORKERS:-50%}"
vitest --maxWorkers="${ORK_WORKERS:-50%}"
jest --maxWorkers="${ORK_WORKERS:-50%}"
go test -p "${ORK_WORKERS:-1}"

The security scan in CHECK

With verify.security_scan declared in the manifest, ork verify runs ork security-scan as the fourth command, after build, test and typecheck. The tool is the judge, and the agent only reads the result. Without the key, verify, CI, SHIP and skills behave as before; ork init writes the key on its own when it recognizes a JavaScript/TypeScript, Python or Go stack.

The scan covers secrets with Gitleaks, SAST with Semgrep, dependencies with Trivy and the stack's auditor, IaC with Trivy config, workflows with zizmor and actionlint, quality with ESLint, Ruff or golangci-lint, and licenses with Trivy license. The tools run from official images pinned by digest, with the repository mounted read-only; without Docker, the native binary applies, and without either the tool stays unavailable.

The scan compares HEAD with the base and classifies each finding as new, existing or dismissed. A new secret finding always enters the loop; a new dependency, SAST, IaC or workflow finding enters with high or critical; quality and license findings stay as warnings. A finding in the loop makes ork security-scan exit 1 and verify fail, and CHECK reads report.json with the security-auditor to give one verdict per finding.

Without Docker or a native binary, the local scan exits 0 with the tool unavailable, which does not become clean at SHIP: with verify.security_scan declared and no ci.required_for_ship, ork ship stops with human.pending and only proceeds with ork ship <thread> --para main --allow-unscanned "<who authorizes>".

verify:
  security_scan: ork security-scan

ork security-scan
ork security-scan --check agile

Failures, deadlines and focused commands

Verify records cause, deadline and duration. Timeout has a typed reason and uses the manifest deadline. Claims should run the smallest command that proves the change. Lint warns during ork claims add, and ork ci prepare rejects the full local suite: npm --prefix core test may need resources unavailable in CI. Instead of the full suite, run only the claim’s test (node --test on the test file) or a hermetic project script.

CI preparation compiles once, binds product identity and records executado to distinguish actual results from commands that never ran. Resource contention may cause a local timeout; it does not waive CI at the exact commit or turn failure into success.

node --test test/filtro.test.js
ork ci prepare <thread>

Clock-bound tests under CPU steal

On a VPS, the hypervisor can take CPU away (steal), and a test that measures the clock fails without any code change. ork verify, in the CLI, over MCP and in ork fix reverify, measures steal from /proc/stat, for the whole round and within each command’s window, and records it in verify_run (steal, plus stealPct on each failing command).

When a command fails only in clock-bound tests and the steal within its window exceeded 40%, the failure becomes verify.timeout: no verdict, retry runs it again, and it is never a regression. A clock-bound test is a case marked in the manifest and confirmed by JUnit, one that failed with the node --test timeout signature (the test timed out, or was cancelled by a parent that timed out) or one with a current entry in the instability registry; without JUnit, the names come from the ledger’s testeQueCaiu, which also reads the node --test spec reporter. An ordinary assertion under high steal is still a regression, a command with no named test is never reclassified, and without /proc/stat nothing is softened. ork ci run neither measures nor softens: CI remains the independent proof.

The registry is core/instabilidade.json, under the ork.instabilidade/v1 contract, read from the worktree being verified. It starts empty and only accepts an entry with a measured rate (failures over runs, the command that measured it and the date) and revalidation within 30 days. A missing file counts as an empty registry; with a contract error, no entry counts; an expired entry does not count until someone measures again. The registry never softens a failure on its own: it only says the test depends on the clock.

{
  "contrato": "ork.instabilidade/v1",
  "entradas": []
}

Mark clock-bound tests in the report

A command or lane with JUnit can declare the relogio key, with a single strategy: nome, a list of regular expressions over the JUnit identity; or marcador or tag, which state that the command itself already selects only that category. Without the declaration, Classic and Agile judge as before.

Under steal above the threshold, only a failure made up entirely of marked cases is left without a verdict. When the filtro key exists, verify reruns those names in isolation. That second round is a diagnosis and never turns the first result green.

An ordinary failure in the same command is still a regression, and so is a marked case that fails without high steal. The text and the ledger state the names, the measured steal and the second round’s result (passou, reprovou, sem-veredito or sem-filtro), in relogioSobSteal and retesteDeRelogio.

verify:
  test:
    comando: "pytest -m desempenho --junitxml reports/pytest.xml"
    junit: reports/pytest.xml
    relogio: { marcador: desempenho }

relogio: { nome: ["tempo$"] }      # Vitest
relogio: { nome: ["latencia"] }    # Jest
relogio: { tag: "@tempo" }         # Playwright
relogio: { nome: ["/Tempo$"] }     # Go

A missing optional dependency is a skip

The local suite includes integrations that need the sandbox codex at /usr/bin/codex, the OrkMind interpreter on PATH or PostgreSQL through Docker with pgvector/pgvector:pg16. Each test that depends on one of them probes first and, when it is missing, is skipped with the reason instead of failing. The full suite passes with 0 failures on a machine that has none of them, and the end of the report counts the skips; any failure left is real.

ORK_TESTE_EXIGE_AMBIENTE=1 skips nothing: a missing dependency fails. test:ci sets the variable on its own. node core/scripts/suite-local.cjs runs the suite, exits 0 only with 0 failures and lists each skip with its reason. A new test that depends on an external tool uses the helpers semCodexSandbox(), semOrkMind() and semPostgres().

npm --prefix core test
ORK_TESTE_EXIGE_AMBIENTE=1 npm --prefix core test
node core/scripts/suite-local.cjs

Unavailable models and session accounts

model_not_found produces runtime.model-unavailable. Retry tries authorized destinations with the same prompt, retains the profile for other models and records the change. Without a destination it escalates with a setup correction; ordinary rate limits wait for their window. ork sessions stop, logs and attach find the correct account among configured Claude profiles. Global inventory, ork doctor, ork pulse and native HITL control may still be limited to the process account. A profile requested with --perfil that does not exist or belongs to another runtime is refused with runtime.profile-invalid and escalated to a human, without switching profiles automatically.

When a session dies from quota after producing output, retry does not resend the original prompt over the changed worktree: it dispatches or queues a derived prompt with the phase, last commit, status, claims and round, preserving the original prompt and SHA-256. The checkpoint says to continue without repeating GOAL or PLAN, and a session that produced nothing receives the original prompt.

runtime.workspace-untrusted (the runtime refused the worktree directory) and runtime.consent-pending (the CLI is waiting for new terms to be accepted) become a wait for the owner, with the exact command; after acceptance, ork retry run dispatches the same phase again with the same prompt. runtime.consent-pending also covers the no-permissions mode warning and enabling auto mode for --bg, and the proof that the owner trusted the directory follows the rule Claude Code applies with --bg: the directory itself and those above it up to the root of the git repository count, as does the main repository of a linked worktree, but not a directory above that root. A directory that no longer exists and the home directory are not the owner’s to fix and stay as runtime.unavailable, and a repeated refusal for the same blocker uses no attempt and raises no effort.

ork retry plan <thread>
ork retry run <thread> --dry-run

Local proof when a claim is registered

The claim_sem_prova_local policy (alias claims_failed) is off by default and applies only when the manifest declares it. With it, ork claims add runs the claim’s commands once, in the thread worktree and within verify.timeout_ms. The claim is recorded either way: a failing command records policy_warn with claims.failed, and a timeout records it with verify.timeout. It never stops registration, even when declared as block. Because add becomes as slow as the command, the project owner decides whether to turn it on. Without the thread worktree (deleted), nothing runs and the warning says so. Through MCP (ork_claim_add), registration never runs the claim text, with or without the policy: there the evidence is ork_verify, in the sandbox.

The fx-pedido-colado behavior scenario proves that an owner request pasted in #Auto, with push and merge authorized, proceeds without stopping: doubt becomes an informed decision, and a free-text confirmation is refused without recording anything.

policies:
  claim_sem_prova_local: warn

ork eval --so-cenarios