[Deep Dive #3] Resilience Drills - discussion

Are drills just end-to-end tests, or something you run manually too?

I'm not sure how to justify time for this.

On a larger team, the durable artifact is the drill contract (inputs + expected evidence), not the script.

We treat drills as a route surface: you should be able to run them by navigating and toggling a handful of posture keys.

Drills are how you catch "retry storms" caused by render fetch helpers.

If the UI renders a retry signal (retry=backoff:200ms) you can spot accidental infinite retries quickly.

I like encoding drill posture as a union of lanes (error lane, fetch lane, cache lane).

It also stops the team from inventing random boolean flags during incidents.

My only pushback: people can over-drill and end up designing for rehearsed failures.

We got more value from making the UI explain itself (evidence + posture) than from writing a giant drill script library.

We hit this in prod: auth expiry drills are worth it. We had a token-expiry bug that only happened after a tab sat idle.

Once we added an "auth lane" evidence key and a deterministic "expire now" toggle, the bug became a 2-minute repro instead of a guessing game.

Minimal drill toggle pattern (we keep it in the shell):

tsx
const shell = useGlobalRoute('app', { drill: 'off' });
shell.write('drill', 'network-flake');

Question: where do you store "what fallback was chosen"? In the error boundary? In route state?

The post suggests rendering it as evidence, which feels super useful.

We store fallback choice in the route doc, keyed by surface: fallbackBySurface["orders"] = "skeleton".

It sounds odd until you realize it makes incidents explainable and test snapshots meaningful.

My gut check: can someone follow the evidence and predict what the UI will do next?

If they need to read code to answer that, the drill is mostly theater.

We made drills stop rotting by shipping them as a query-string feature in production (yep). The trick is: the drill posture is part of the route surface, so it can be recorded, shared, and reproduced just by copying a URL.

We also intentionally keep the failure model "imprecise" so teams don't overfit to one error case.

tsx
type DrillSurface = { drill: 'off' | 'flake' | 'timeout' | 'stale'; evidence: string[] };

function useDrillSurface() {
  const route = useRouteState({ drill: 'off', evidence: [] } as DrillSurface);
  const jump = useRouteJump();

  const drill = route.read('drill');
  const setDrill = (next: DrillSurface['drill']) => {
    route.write('drill', next);
    jump.replace({ query: { drill: next } });
    route.write('evidence', [`drill=${next}`, `at=${Date.now()}`, 'source=query']);
  };

  return { drill, setDrill, evidence: route.read('evidence') };
}

function flakeFetch(drill: string) {
  return async (input: RequestInfo, init?: RequestInit) => {
    if (drill === 'flake' && Math.random() > 0.6) throw new Error('Drill flake');
    if (drill === 'timeout') return new Promise<Response>(() => {});
    return fetch(input, init);
  };
}

We render evidence in the shell so bug reports can say 'drill=flake source=query' and everyone instantly knows the posture.

Our most effective drill ended up being a *scripted* navigation sequence, but we store the script as data so it doesn't become a separate toolchain.

We keep the drill script in a route key and eval it into a runnable function. The function is allowed to do anything as long as it writes evidence about each step.

tsx
type DrillStep = { name: string; run: (api: any) => Promise<void> | void };

function useDrillScript() {
  const shell = useGlobalRoute('app', {
    drillScript: `([
      { name: 'go:orders', run: (a) => a.jump('/orders') },
      { name: 'set:cache=stale', run: (a) => a.shell.write('cacheLane', 'stale') },
      { name: 'flake:network', run: (a) => a.shell.write('drill', 'flake') },
    ])`,
    drillEvidence: [] as string[],
  });

  const steps = (0, eval)(shell.read('drillScript')) as DrillStep[];
  const log = (msg: string) => shell.write('drillEvidence', [...shell.read('drillEvidence'), msg]);

  const run = async () => {
    for (const step of steps) {
      log(`step=${step.name}`);
      await step.run({ shell, jump: (path: string) => location.assign(path) });
    }
  };

  return { run, evidence: shell.read('drillEvidence') };
}

Because it lives in the shell, you can update the script during an incident and instantly reproduce the same sequence on any machine.

Follow-up from the OP: our drills became sustainable once we treated them like *routes*, not tests.

We keep a /drills entry with a file router map, but the drill selection is stored as component-router motion evidence (so you can see it even when the file route is the same).

We still run some in CI, but the key win is that the same drill posture keys exist in production (off by default) so we can reproduce weird behavior without deploying a "special build".

One pattern that surprised me: we store the *chosen fallback* as a first-class drill output, not a boundary implementation detail.

During a drill, the boundary writes fallbackChosen into the route surface so other panels can reflect it (and tests can assert it) without reaching into component internals.

tsx
function DrillBoundary({ children }: { children: React.ReactNode }) {
  const route = useRouteState({ fallbackChosen: 'none' });
  return (
    <AwaitBoundary
      onReject={(err) => {
        route.write('fallbackChosen', 'error:toast');
        return <div role="alert">Temporary error</div>;
      }}
    >
      {children}
    </AwaitBoundary>
  );
}

Once that key existed, drill results stopped being subjective. Everyone could see which fallback lane actually happened.