microsegment.io

segment all the things

The incident responder clicks quarantine. A fresh connection test fails. The ticket says contained.

What happened to the connection that was already open?

That is the next test I would add to a microsegmentation response exercise. Not another port scan. A live, harmless application exchange that starts before isolation and is observed throughout the change.

In the earlier mTLS article, I touched on existing sessions. They deserve their own runbook. A policy change and a session termination are different events.

Two Documented Behaviors, One Operational Question

AWS documents that changing a security group rule does not immediately interrupt tracked connections. Those connections can continue until they time out. Untracked flows behave differently: removing their enabling rule can interrupt them immediately.

That distinction makes a single successful demonstration insufficient. Record whether the tested path is tracked and how it is routed. Do not turn one observed disconnect into a general claim about every security group change.

Kubernetes documents that the effect of a NetworkPolicy change on an existing connection is implementation-defined. The API object alone does not promise termination. Policy processing is also asynchronous; the API does not tell you exactly when the network plugin has applied it.

My operational conclusion: define quarantine by observed traffic behavior, with a deadline, rather than by successful submission of a configuration change.

Write The Quarantine Contract First

For a hypothetical application worker, I would define the following response contract before choosing an enforcement mechanism:

Path Required behavior after quarantine
Worker to business database New connections blocked; existing application exchanges stopped
Worker to unrelated workloads No connectivity
Worker to approved evidence collector Only the explicitly retained evidence path
Authorized response system to worker Only the access needed for the agreed investigation method
Worker to public destinations Blocked unless a specific response dependency is approved

This is a proposed design, not a universal production policy. An OT process, a payment system and a disposable build worker need different interruption decisions. Name the service owner and agree on the operational consequence before the exercise.

Avoid the vague requirement “keep management working.” List the caller, destination, protocol and purpose of each retained flow. Otherwise the exception is harder to test than the quarantine itself.

Run A Warm-Connection Test

Use a representative test environment with synthetic data. My proposed drill has two clients and one controlled destination:

  1. Establish a persistent connection. Send numbered, harmless messages and record acknowledgments at the destination. Keep this client running through the policy change.
  2. Start a separate fresh-connection probe. Make it open a new connection for each attempt. Confirm it is not reusing a pool or an existing socket.
  3. Apply the approved quarantine action. Record the request time and any enforcement acknowledgment separately.
  4. Observe both clients. Record the last acknowledged message on the original connection and the first sustained rejection of new attempts.
  5. Check the retained response path. Confirm the approved evidence channel still works, without restoring general application access.
  6. Attempt reconnects for the agreed observation window. A transient failure is not the same as sustained isolation.

The destination log matters. A client reporting that it wrote bytes is not my acceptance criterion. I want evidence of whether the receiving application continued to process them.

Include a quiet established connection as a second test case, then try to use it after quarantine. Record it separately from the continuously active connection. Do not assume an idle-session result describes an active exchange.

Pick A Termination Mechanism With A Known Scope

For EC2, AWS identifies a network ACL as a way to interrupt traffic regardless of security group tracking state. Network ACLs operate at the subnet boundary, are stateless, and evaluate numbered rules in order. They are not a drop-in workload-level quarantine control.

Before using one in a response design, map the affected subnet traffic, rule ordering and return paths. Test the actual route. A subnet control is not evidence that every path between workloads inside that subnet is covered.

For Kubernetes, ask the networking implementation owner for the supported way to handle established sessions. Then validate that procedure on the deployed version. Do not improvise cluster-wide connection-state deletion during an incident.

My preference is the narrowest supported action that meets the response contract. Depending on the platform, that could involve an enforcement control with session interruption, a scoped application-session termination, or stopping an isolated workload. These are alternatives to evaluate, not interchangeable commands. Document evidence-preservation and availability consequences before choosing one.

Also verify that a broader applicable allow policy is not preserving access. Kubernetes NetworkPolicies are additive; adding an empty allow set does not cancel permissions granted by another applicable policy. NetworkPolicy semantics

Measure The Result, Not The Button

I would record four timestamps for every exercise:

  • Quarantine requested.
  • Enforcement acknowledgment received, if available.
  • Last successful exchange on the pre-existing connection.
  • Start of sustained rejection of fresh connections.

Add the workload identity, policy revision, enforcement version, tested route and retained exceptions. Set the acceptable interval with the incident-response and application owners. There is no universal number in this article.

A useful result might read: “New connections were rejected within our target, but the original session continued. The quarantine procedure failed the established-session requirement.”

That is better evidence than a green isolation badge.

Repeat the drill after changing the enforcement implementation or network path. Keep recovery in scope too: restoration should return the workload to its approved baseline, not to a temporary broad allow rule.

Microsegmentation defines who may connect. A containment runbook must also prove what happens to whoever is already connected.

Sources