Cleanup, evidence retention, and response
Cleanup changes what can still be proved. Deleting an unsigned test image can remove the ability to repeat its denial; deleting signature metadata can break fresh admission for an otherwise retained application digest. Decide retention before teardown and keep public screenshots separate from sensitive operational records.
Retain the artifact and decision chain
Section titled “Retain the artifact and decision chain”Preserve source revision, final image digest, selected Helm digest, verified signing identity, SBOM and provenance, CI logs, rendered desired state, admission responses, Argo CD observations, and runtime event timestamps. Record capture time and source revision independently. An image digest may describe an older release than the latest CI run.
Retain sensitive logs in controlled storage. Inspect screenshots and transcripts for tokens, kubeconfigs, personal account metadata, webhooks, and local state details before publishing. Never publish Terraform state to explain a resource. Public evidence derivatives should identify any redaction and keep their originals preserved privately.
Remove disposable tests first
Section titled “Remove disposable tests first”In the owned environment, review temporary Pods, labelled negative fixtures, the test-only ClusterPolicy, and any negative-test Argo Application. The historical negative Application has no automated sync; check whether it was actually applied before removing it. Do not assume an absent rejected Pod means its request was never made: denied requests may only exist in the saved server response.
After evidence review, delete the disposable unsigned registry version separately from the trusted artifact. Remove test policies so intentionally incorrect expectations do not affect later labelled requests. Verify that ordinary protected workloads still admit under the intended policy.
Destructive teardown requires an owned, reviewed plan
Section titled “Destructive teardown requires an owned, reviewed plan”The source guide orders application and policy removal, separately installed Argo CD/Kyverno Helm releases, and finally the Terraform-managed foundation. Review dependencies and existing namespaces before proceeding. A Terraform destroy does not automatically uninstall manually installed releases with the same care, and deleting the cluster makes their later inspection impossible.
The following command prepares a destructive plan. It contacts the owned environment and writes sensitive operational state to a local plan file; it does not destroy resources by itself. Do not publish that file.
terraform -chdir=infrastructure/environments/prod plan -destroy -out=cleanup.tfplanReview project, resource list, registry contents, state access, and retention explicitly before the separately approved destruction. The root config sets API disable_on_destroy = false; destroying its tracked resources does not promise a completely empty project. Decide whether to retain the independently created versioned state bucket and final artifact. No destruction or cleanup rehearsal was executed while authoring this handbook.
Respond to a suspicious runtime event
Section titled “Respond to a suspicious runtime event”The repository demonstrates a controlled shell detection and contains troubleshooting and cleanup notes. It does not demonstrate a complete incident-response or recovery exercise. The guidance below is added conceptual practice, not a recorded baseline procedure.
First preserve the event, workload digest, Pod UID, selected Git revision, relevant authentication records, and time window. Determine whether the action was a planned operator test, a compromised application process, or unauthorized exec access. Containment may involve removing workload authority or changing reviewed Git desired state, but choose it with the owner: Argo self-heal can undo an ad hoc live patch, and deleting a Pod can erase volatile evidence.
If signing-workflow authority is suspected, suspend further promotion and examine workflow changes, repository access, and affected digests. A valid signature from a compromised trusted workflow remains valid cryptographically. Rebuild and revalidate under restored authority; changing only a mutable tag does not remediate a selected digest. Background policy reports do not retroactively evict already running Pods.