Skip to content
k-msg

Migration Orchestrator

Question this page answers: How do I resume large backfills after interruption without breaking data consistency?

Core concept

Section titled “Core concept”
  • One-line definition: the orchestrator is an execution layer tracking backfill lifecycle with plan/apply/status/retry.
  • Why it matters: failed chunks are retried precisely without duplicate writes.
  • Configuration example (safe): persist progress in DB meta tables and local snapshots.
  • Common mistake: relying only on logs with no resumable state.

CLI flow

Section titled “CLI flow”
Terminal window
k-msg db tracking migrate plan --sqlite-file ./local.db
k-msg db tracking migrate apply --sqlite-file ./local.db
k-msg db tracking migrate status --sqlite-file ./local.db
k-msg db tracking migrate retry --sqlite-file ./local.db

apply and retry encrypt with the tracking store’s keys. Set KMSG_FIELD_CRYPTO_KEYS (a JSON object of base64url AES-256 keys by kid) and KMSG_ACTIVE_KID, plus KMSG_FIELD_CRYPTO_HASH_KEYS, KMSG_FIELD_CRYPTO_FIELDS, KMSG_FIELD_CRYPTO_TENANT_ID, and KMSG_FIELD_CRYPTO_AAD_FIELDS when the store uses them. See the runbook below.

Operational checkpoints

Section titled “Operational checkpoints”
  1. Keep the same planId for resumed execution.
  2. Retry only failed chunks with retry. A run that stopped on a read error has none; resume it with apply.
  3. Switch compatPlainColumns=false only after hash/cipher consistency checks.

Risk labels

Section titled “Risk labels”
  • safe: validate status before each stage transition
  • caution: force-stop during apply without reading status
  • unsafe: create a new plan while old plan is still active

Next

Section titled “Next”