The boundaries of model trust in security automation

Language models are uniquely powerful at parsing unstructured security logs, mapping them to MITRE techniques, and writing the first draft of an incident report. But they are probabilistic engines, which means they will eventually hallucinate a critical piece of evidence.

In security automation, a model is only as useful as the boundaries placed around it. We call this boundary the harness.

What is a Harness?

A harness is the engineering layer that surrounds the model. It is responsible for:

1. Scoping the queue: Deciding which alerts the model is even allowed to see.
2. Tool constraints: Sandboxing what actions the model can take (e.g., read-only API access).
3. Verification loops: Using deterministic code to verify the probabilistic output of the model.

If the model says “this IP is malicious,” the harness should parse the model’s evidence and check the IP against a known threat feed before escalating it to a human.

Trust is earned through verification. The model generates hypotheses; the harness proves them.

Example: Detection Engineering

When writing detection rules as code, you don’t just ask the model for a YAML file and commit it. You run it through a CI/CD pipeline:

steps:
  - run: validate-syntax
  - run: check-against-historical-logs
  - run: require-human-approval

This is the essence of AV Threat Labs. We don’t just build prompts; we build the pipelines that make prompts safe to use in production.