Zero Trust Architecture for Microservices

Thu, 16 Apr 2026 14:41:59 +0200

Introduction

“All of ARPA’s protection has, by design, left the internal AT&T machines untested. A sort of crunchy shell around a soft, chewy center.”

William R. Cheswick, The Design of a Secure Internet Gateway (1990)¹

That was 1990. We’re still dealing with the fallout.

For decades, network security operated on this exact premise: everything inside the corporate perimeter could be trusted. Firewalls guarded the castle gates, and once you were inside, you had access to virtually everything. Bob Blakley, a Security Architect at IBM, famously summarized the three myths of firewalls²:

“We’ve got the place surrounded”: assuming there are no back doors
“Nobody here but us chickens”: assuming all threats are outside the perimeter
“Sticks and stones may break my bones”: assuming data can’t execute code

We built an entire industry on these myths. Then came cloud-native architectures.

Modern microservices don’t live behind a single perimeter-they span multiple clouds, edge locations, data centers, and third-party services. A typical enterprise might have hundreds of services communicating across Kubernetes³ clusters, with workloads scaling dynamically and developers deploying code multiple times per day. The traditional perimeter has dissolved into a complex mesh of ephemeral connections.

Where is Zero Trust ** Actually Useful**? Let me give you some real examples:

E-commerce platforms: Payment services, inventory systems, recommendation engines, and third-party analytics all need to talk to each other-and to external payment gateways, shipping APIs, and fraud detection services. Each service should prove its identity, not just rely on network location.
Healthcare systems: Patient data flows between EHR systems, billing services, diagnostic APIs, and insurance verification-subject to HIPAA. A breach in one service shouldn’t grant access to everything.
IoT device management: Thousands of devices connecting to backend services, often from untrusted networks. Each device Needs certificate-based identity, not shared API keys.
Multi-cloud deployments: Services running on AWS, GCP, and on-premises, needing secure cross-cloud communication without exposing public endpoints.

This post explores how Zero Trust Architecture addresses these challenges, with a deep dive into mutual TLS (mTLS) as the cornerstone of service-to-service authentication, and service mesh as the Infrastructure layer that makes Zero Trust practical at scale. We’ll connect these concepts to the distributed tracing patterns discussed in my previous post, where we saw how OpenTelemetry enables end-to-end observability-the same infrastructure that carries trace context (W3C TraceContext) can simultaneously carry security identity.

1. The Traditional Perimeter Security Problem

1.1 Castle-and-Moat Architecture

The traditional network security model assumed that the local network was safe. Security controls focused on the perimeter:

Firewalls blocked unauthorized external access
VPNs provided “trusted” remote access
Internal networks operated with implicit trust

In this model, if an attacker breached the perimeter or compromised an insider’s credentials, they gained lateral access to everything inside.

1.2 Lateral Movement Attacks

Once inside an untrusted perimeter, attackers could move freely:

Compromise one service: through a vulnerability, insider threat, or supply chain attack
Escalate privileges: use the compromised service’s credentials to access others
Exfiltrate data: hop between services until reaching the target
Establish persistence: plant backdoors while moving laterally

The 2013 Target breach exemplifies this⁴. Attackers compromised an HVAC vendor’s credentials, then moved laterally through Target’s network until reaching the point-of-sale systems-accessing 40 million credit card numbers. The HVAC vendor had legitimate network access that should never have reached payment systems.

1.3 Why Internal Traffic Was Considered “Trusted”

The assumption that internal traffic is safe relied on several flawed premises:

Physical security: Only authorized personnel could access the data center
Network isolation: Firewalls separated internal from external
Static infrastructure: Services didn’t change frequently, so monitoring was manageable

These assumptions broke down with:

Cloud workloads: Applications running outside corporate infrastructure
Remote work: Employees accessing internal services from home networks
Dynamic scaling: Services appearing and disappearing as load demands
Supply chain complexity: Third-party services and libraries with their own access

1.4 Real-World Breaches from Internal Threats

The Verizon 2024 Data Breach Investigations Report found that 68% of breaches involved a human element-insiders, credentials, or social engineering⁵. A significant portion of these exploited implicit trust within the perimeter.

High-profile examples include:

SolarWinds (2020): Attackers inserted malicious code into software updates, allowing them to move laterally through customer networks once inside⁶
Capital One (2019): A misconfigured web application firewall allowed access to AWS metadata, then to customer data⁷
NPM supply chain attacks: Compromised packages granted access to internal build systems⁸

The pattern is clear: trusting internal traffic is the vulnerability.

2. Zero Trust Architecture: Core Principles

Zero Trust Architecture, as formalized by NIST SP 800-207⁹, isn’t just a different kind of firewall: it’s a fundamental rethinking of access control. Instead of granting broad network access based on location, Zero Trust grants fine-grained access to specific resources based on verified identity. Every request is authenticated, every path is encrypted, and every decision is explicit.

2.1 The Paradigm Shift

Zero Trust inverts the fundamental assumption. Instead of “trust inside, verify outside,” Zero Trust operates on:

“Never trust, always verify.”

Every request, regardless of origin, must be authenticated and authorized. The location (inside or outside the perimeter) becomes irrelevant-what matters is whether the requester can prove their identity and is authorized for the specific resource.

sequenceDiagram participant A as Service A participant Auth as Identity & AuthZ participant B as Service B A->>Auth: "Verify me!" Auth->>B: "Verify me!" B->>Auth: "Identity confirmed" Auth->>A: "Both verified" A->>B: Encrypted, authenticated request

This diagram illustrates the Zero Trust flow:

Service A initiates contact: but has no implicit trust. It must present its identity to the authorization service.
The identity service verifies Service A: checks its certificate, workload metadata, and policies.
The identity service verifies Service B: confirms the target is legitimate and accepts requests.
Both services are confirmed: identity and authorization are established.
Service A can now communicate with Service B: over an encrypted, mutually authenticated channel.

Compare this to the castle-and-moat model where Service A would simply connect directly to Service B on the internal network: no verification needed. The key difference: in Zero Trust, there’s no “inside” that bypasses authentication.

2.2 NIST Zero Trust Principles

While the conceptual shift from perimeter to identity provides a guiding philosophy, practitioners need concrete principles to implement. NIST SP 800-207 (Zero Trust Architecture)⁹ offers exactly that: a formal framework that translates the “never trust, always verify” mantra into actionable design principles.

NIST SP 800-207 defines Zero Trust Architecture around these tenets⁹:

Verify Explicitly

Always authenticate and authorize based on all available data points:

Identity (user and workload): Verifies who or what is making the request through certificates, tokens, or workload metadata
Location: Uses network position as a risk factor (e.g., request from unexpected region triggers additional verification)
Device health: Confirms the endpoint meets security posture requirements (patched, encrypted disk, enabled firewall)
Service or workload: Validates the calling service’s identity and runtime properties
Data classification: Considers sensitivity of the requested data resource
Abnormalities: Detects behavioral anomalies through continuous monitoring

No single factor grants access; multiple signals must align.

Least Privilege Access

Just-in-time (JIT) and just-enough-access (JEA):

Just-in-time: Access is granted only when needed, for the duration needed
Just-enough-access: Grant only the minimum permissions required

This limits the blast radius when credentials are compromised.

Assume Breach

Design systems as if an attacker is already inside:

Minimize blast radius through microsegmentation
Verify end-to-end encryption
Continuous monitoring and detection
Assume any credential could be compromised

2.3 The Identity Plane

In Zero Trust, identity becomes the new perimeter. Every workload (service, container, VM) has a cryptographic identity that persists regardless of where it runs.

graph TB subgraph IP["IDENTITY PLANE"] WA["Workload A"] WB["Workload B"] WC["Workload C"] Registry["Identity Registry"] end WA --> Registry WB --> Registry WC --> Registry

This identity is used for:

Authentication: Proving “I am service A”
Authorization: Determining “service A can access endpoint X”
Auditing: Logging “who accessed what and when”
Encryption: Establishing secure channels between verified identities

2.4 SPIFFE and Workload Identity

[SPIFFE (Secure Production Identity Framework for Everyone)]¹⁰ provides a standardized framework for workload identity. The key concepts:

SPIFFE ID: A URI that uniquely identifies a workload. The format is spiffe://{trust_domain}/{path}, where each component carries semantic meaning:

Trust domain: The administrative boundary (e.g., example.org) - represents a cluster or organization that manages its own CA hierarchy
Path: An hierarchical identifier that encodes namespace, service account, or application details (e.g., /ns/default/sa/payment means: namespace default, service account sa, workload payment)

This hierarchical design enables decentralized issuance: each trust domain operates its own CA, and the SPIFFE ID makes it explicit which domain issued which identity. Authorization policies can match on any component: for example, allow only workloads in the payment path to access the billing service.

graph TD A["spiffe://example.org/ns/default/sa/payment"] S["spiffe://"] T["example.org"] P["/ns/default/sa/payment"] A --- S A --- T A --- P

SVID (SPIFFE Verifiable Identity Document): A signed document containing the SPIFFE ID and cryptographic material (certificate and private key) that the workload uses to prove its identity.

Trust Bundle: A set of certificates that a workload uses to verify the SVIDs of other workloads. Only workloads with identities signed by the same CA (or a federated CA) can communicate.

3. Mutual TLS (mTLS) Deep Dive

Identity without authentication is incomplete. In Zero Trust, mutual TLS (mTLS) is the mechanism that makes identity concrete: both client and server prove they hold the private key corresponding to their asserted identity before any data flows. This section walks through how TLS handshakes work, how mTLS extends one-way TLS, and how to implement it in practice.

3.1 How TLS Works (Traditional One-Way TLS)

Before understanding mTLS, we need to understand standard TLS.

In one-way TLS (HTTPS in browsers), the flow is:

sequenceDiagram participant C as Client participant S as Server rect rgb(40, 42, 54) Note over C: 1. ClientHello C->>S: ClientHello end rect rgb(40, 42, 54) Note over S: 2. ServerHello, Certificate S->>C: ServerHello, Certificate, ServerHelloDone end rect rgb(40, 42, 54) Note over C: 3. Verify certificate Note over C: 4. ClientKeyExchange C->>S: ClientKeyExchange end rect rgb(40, 42, 54) Note over C,S: 5. Both derive session keys end rect rgb(40, 42, 54) C->>S: Finished (encrypted) S->>C: Finished (encrypted) end rect rgb(40, 42, 54) Note over C,S: 7. Application data C->>S: Encrypted request S->>C: Encrypted response end

ClientHello: The client initiates the handshake by sending a random value and supported cipher suites.
ServerHello, Certificate: The server responds with its random value, picks a cipher suite, and sends its certificate containing the public key.
ClientKeyExchange: The client verifies the server’s certificate against trusted CAs, then generates a pre-master secret encrypted with the server’s public key.
Both derive session keys: Both client and server derive the same master key from the pre-master secret.
Finished: Both parties send encrypted “Finished” messages to verify the handshake succeeded.
Application data: The encrypted tunnel is established; data flows securely.

The server proves its identity to the client via its certificate, but the server has no idea who the client is.

This works for human-to-website interaction (you need to trust your bank’s website), but it fails for service-to-service communication where both parties need to verify each other.

3.2 Mutual TLS: Both Ways

mTLS extends TLS so that both the client and server present certificates and both authenticate each other.

sequenceDiagram participant C as Client participant S as Server rect rgb(40, 42, 54) Note over C: 1. ClientHello C->>S: ClientHello end rect rgb(40, 42, 54) Note over S: 2. ServerHello + CertRequest S->>C: ServerHello, Certificate, CertificateRequest, ServerHelloDone end rect rgb(40, 42, 54) Note over C: 3. Verify server cert Note over C: 4. Send cert + CertVerify C->>S: Certificate, ClientKeyExchange, CertificateVerify end rect rgb(40, 42, 54) Note over S: 5. Verify client cert end rect rgb(40, 42, 54) Note over C,S: 6. Both derive session keys end rect rgb(40, 42, 54) C->>S: Finished (encrypted) S->>C: Finished (encrypted) end rect rgb(40, 42, 54) Note over S: 8. Both parties authenticated C->>S: Encrypted, authenticated request S->>C: Encrypted response end

ClientHello: Client initiates with random value and cipher suites.
ServerHello + CertRequest: Server responds but also requests a client certificate (CertificateRequest message).
Verify server cert + Send cert + CertVerify: Client verifies the server’s certificate, then sends its own certificate plus a CertificateVerify (signed data proving the client owns the private key).
Verify client cert: Server validates the client’s certificate and signature.
Both derive session keys: Key derivation proceeds as in standard TLS.
Finished: Both confirm handshake success.
Application data: Both sides are authenticated; encrypted tunnel established.

Key Differences from Standard TLS

Aspect	Standard TLS (One-Way)	mTLS (Mutual TLS)
Client certificate	Not required	Required - client must present certificate
Server knows client identity	No - server accepts any client	Yes - verifies client certificate
CertificateRequest message	Not sent	Sent by server requesting client cert
CertificateVerify	Not sent	Sent by client proving key ownership
Authentication direction	One-way (server only)	Two-way (both parties)

The critical addition in mTLS: Steps 2-4 are new. The server explicitly requests the client’s certificate (CertificateRequest), the client responds with its certificate plus a signed proof (CertificateVerify), and the server verifies both. This is what enables service-to-service authentication.

3.3 What mTLS Achieves

With mTLS, we gain:

Server authentication: The client knows it’s talking to the real service (not an impostor)
Client authentication: The server knows which client is calling (not an anonymous request)
Encryption: All traffic is encrypted in transit
Integrity: Tampering with messages in transit is detected

This prevents:

Man-in-the-middle attacks: An attacker can’t intercept traffic without a valid certificate
Service impersonation: A compromised service can’t pretend to be another service
Unauthenticated access: Requests without valid certificates are rejected

3.4 Certificate Management Challenges

mTLS requires each service to have a certificate, which introduces operational complexity:

Certificate Lifecycle

stateDiagram-v2 [*] --> Issue: CA signs certificate Issue --> Distribute: Load into workload Distribute --> Validated: TLS handshake Validated --> Rotate: Near expiration Rotate --> Issue: New certificate Validated --> Revoke: Compromise detected Revoke --> [*]: CRL/OCSP updated

The certificate lifecycle has four phases:

Issuance: The workload requests a certificate from the CA, presenting its identity (SPIFFE ID or other identifier). The CA validates this identity and signs a certificate binding the workload’s public key to its identity.
Distribution: The certificate (and private key) is delivered to the workload. This typically happens at startup or is mounted as a secret. The workload can now present this certificate during TLS handshakes.
Validation: During each TLS handshake, the peer verifies the certificate against the trust bundle (root CA or intermediate CA certificates). The peer also checks the certificate hasn’t expired or been revoked via CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol).
Rotation: Certificates near expiration are replaced with new ones. Short-lived certificates (hours) require frequent rotation; long-lived certificates (months) need fewer rotations but carry higher risk.

How It’s Implemented in Practice

Most production systems automate the entire lifecycle:

Certificate request: The workload contacts the CA (e.g., SPIRE, Vault, cloud CA) via the ACME protocol or a custom API, presenting its workload identity.
Automated distribution: The CA pushes certificates to the workload (via Secret Manager, Vault agent injector, or SPIRE’s node plugin). A common pattern: certificates are stored in a secrets mount and automatically reloaded when renewed.
Automated rotation: A background process monitors certificate expiration and triggers renewal before expiry. Kubernetes secrets with cert-manager, Vault’s agent, or SPIRE’s node agent handle this automatically.
Revocation handling: If a key is compromised, the CA adds the certificate to a CRL or marks it revoked in OCSP. Peers check revocation status during handshake; revoked certs are rejected.

This automation is critical: manual certificate management at scale leads to outages (expired certs) or security gaps (revoked certs still accepted).

Short-Lived Certificates vs Long-Lived

Aspect	Short-Lived (< 24h)	Long-Lived (> 30 days)
Rotation frequency	High	Low
Rotation automation	Required	Often manual
Key compromise window	Small	Large
Operational complexity	High	Low
Certificate issuance load	High	Low

Best practice: Use short-lived certificates (hours to days) and automate rotation. This limits the window of damage if a key is compromised.

Certificate Authority Options

Public CAs (Let’s Encrypt, DigiCert): Good for external-facing services
Private CAs (Cloud providers, HashiCorp Vault, Step): For internal service mesh
Intermediate CAs: Sign workload certificates, keep root CA offline

3.5 SPIFFE and SPIRE

[SPIRE]¹¹ (SPIFFE Runtime Environment) automates workload identity management:

graph TB Server["SPIRE SERVER"] Agent["SPIRE AGENT"] CSI["Workload API"] W1["Container A"] W2["Container B"] W3["Container C"] Server --> Agent Agent --> CSI W1 -.-> CSI W2 -.-> CSI W3 -.-> CSI

How it works:

Registration: An operator registers workload identity with the SPIRE server (e.g., “Kubernetes pod with selector namespace=default, serviceAccount=payment should receive identity spiffe://example.org/payment”)
Agent attestation: The SPIRE agent running on the node verifies the workload’s identity (using Kubernetes token review or node attestation)
SVID issuance: The SPIRE server issues a short-lived X.509 SVID to the workload via the agent
Workload uses SVID: The workload uses its SVID certificate to authenticate mTLS connections

Federation: SPIRE supports federating trust between organizations. Two SPIRE servers can exchange trust bundles, allowing workloads in one trust domain to authenticate workloads in another.

4. Service Mesh: Infrastructure for Zero Trust

So far, we’ve discussed the conceptual foundations of Zero Trust and the cryptographic mechanisms that make it possible. But here’s the practical reality: implementing mTLS manually between every service pair, managing certificate rotation, enforcing authorization policies, and collecting observability data across hundreds of microservices quickly becomes unmanageable. You need infrastructure that automates these concerns at scale.

This is where a service mesh comes in. A service mesh is a dedicated infrastructure layer that handles service-to-service communication, offloading security, reliability, and observability concerns from application code to the platform. Rather than each team implementing mTLS, writing custom interceptors, or instrumenting their own metrics, services simply communicate through the mesh and let it handle the rest.

In this section, we’ll explore what a service mesh is, how it implements Zero Trust principles at the network layer, and compare the two most popular implementations: Istio¹² and Linkerd¹³.

4.1 What is a Service Mesh?

A service mesh is a dedicated infrastructure layer that handles service-to-service communication. It provides:

Automatic mTLS: Encryption and authentication without application changes
Traffic management: Load balancing, retries, circuit breaking
Observability: Metrics, traces, and logs for all traffic
Security policies: Authorization rules, rate limiting

The key architectural pattern is the sidecar proxy:

graph LR Ext1["External Service A"] ==mTLS==> E subgraph Pod["POD / CONTAINER"] subgraph App["Application"] SC["Service Code"] L1["localhost:8080"] end subgraph Sidecar["Sidecar Proxy"] E["Envoy Proxy"] L2["localhost:15001"] end SC --> L1 L1 --> L2 end E ==mTLS==> Ext2["External Service B"] subgraph Features["What the Proxy Handles"] direction LR TLS["TLS: Encryption"] POL["AuthZ: Policy"] MET["Telemetry"] end E -.->|terminates| TLS E -.->|enforces| POL E -.->|collects| MET

The application doesn’t handle TLS, networking, or security policies-these are handled by the sidecar proxy running alongside it.

4.2 Envoy Proxy Deep Dive

When a network packet arrives at the Envoy sidecar, it goes through a well-defined processing pipeline. Understanding this path is essential for debugging issues and designing effective service mesh deployments.

Inbound Request Path (when another service calls this one):

graph LR subgraph Inbound["INBOUND PATH (Service A → Service B)"] direction TB P1["Packet arrives at
sidecar port"] TLSI1["TLS Inspector
detect TLS/SNI"] HTTP1["HTTP Codec
parse request"] ROUT1["Router Filter
match route"] UP["Forward to
upstream pod"] P1 --> TLSI1 --> HTTP1 --> ROUT1 --> UP end

A packet arrives at the container’s network namespace on the listener port (e.g., 443)
The TLS Inspector filter examines the incoming bytes to detect whether this is TLS and extract SNI (Server Name Indication)
The HTTP Codec parses the HTTP request: method, path, headers, body
The Router Filter evaluates the request against configured routes to determine the upstream cluster
The request is forwarded to an endpoint in that cluster (another pod’s sidecar)

Outbound Request Path (when this service calls another):

The application makes a plain HTTP request to localhost (the sidecar listener, e.g., port 15001)
Envoy accepts the connection and applies its own processing pipeline
The router filter looks up the destination cluster based on the Host header or IP
Envoy establishes a new mTLS connection to the upstream sidecar
The response travels back through the same chain in reverse

graph LR subgraph Outbound["OUTBOUND PATH (Service B → Service C)"] direction TB APP["App calls
localhost:15001"] ROUT2["Router Filter
lookup cluster"] TLS2["mTLS
establish connection"] UP2["Forward to
upstream pod"] APP --> ROUT2 --> TLS2 --> UP2 end

This bidirectional interception is what makes the service mesh so powerful: every network flow is visible, every connection is encrypted, and every policy is enforced consistently across the entire mesh.

Envoy¹⁴ is the de facto standard sidecar proxy for service meshes. Its architecture consists of:

Listeners: Network listeners that accept incoming connections (one per port)
Filter Chains: Ordered list of filters applied to incoming requests (TLS inspector, HTTP router, etc.)
Routes: How requests are routed to upstream clusters
Clusters: Groups of upstream endpoints (your services)
Endpoints: Individual IP:port combinations for upstream services

graph TB subgraph Listener["LISTENER"] TLSI["TLS Inspector"] HTTPC["HTTP Codec"] ROUT["Router Filter"] TLSI --> HTTPC HTTPC --> ROUT end subgraph Routes["ROUTE CONFIG"] R1["/api/*"] R2["/auth/*"] R3["/health"] ROUT --> R1 ROUT --> R2 ROUT --> R3 end subgraph Clusters["CLUSTERS"] C1["pod-1"] C2["pod-2"] C3["pod-3"] end R1 --> C1 R1 --> C2 R1 --> C3

The xDS Protocol¹⁵: Envoy discovers its configuration dynamically via the xDS APIs:

LDS (Listener Discovery Service): What listeners to create
RDS (Route Discovery Service): How to route traffic
CDS (Cluster Discovery Service): What upstream clusters exist
EDS (Endpoint Discovery Service): What endpoints exist in each cluster
SDS (Secret Discovery Service): TLS certificates and keys
ADs (Aggregated Discovery Service): Combines multiple discovery services

This allows the control plane to push configuration updates without restarting proxies.

4.3 Istio Architecture

Istio¹² is the most feature-rich service mesh, built on top of Envoy¹⁴.

Key Istio Resources:

# PeerAuthentication: Enforce mTLS mode
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
 namespace: istio-system
spec:
 mtls:
 mode: STRICT # or PERMISSIVE (allows plaintext for migration)
---
# AuthorizationPolicy: Define who can talk to whom
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: payment-policy
 namespace: default
spec:
 selector:
 matchLabels:
 app: payment
 rules:
 - from:
 - source:
 principals: ["cluster.local/ns/default/sa/checkout"]
 to:
 - operation:
 methods: ["POST"]
 paths: ["/api/v1/charge"]
---
# DestinationRule: Configure mTLS and load balancing
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: payment-destination
 namespace: default
spec:
 host: payment.default.svc.cluster.local
 trafficPolicy:
 tls:
 mode: ISTIO_MUTUAL # Use Istio-managed certificates
 loadBalancer:
 simple: LEAST_REQUEST

4.4 Linkerd Architecture

Linkerd¹³ takes a different approach, prioritizing simplicity and security defaults. While Istio gives you fine-grained control over every aspect of the mesh, Linkerd focuses on doing the essentials really well with minimal configuration.

4.4.1 Architecture Overview

Linkerd uses the same sidecar pattern as Istio: each pod gets a proxy container injected alongside the application container. However, the implementation is simpler and the proxy is more specialized.

Here’s how it works:

Injection: When you annotate a namespace or pod with linkerd.io/inject: enabled, Kubernetes adds a linkerd2-proxy container to the pod
Inbound traffic: External traffic arrives at the pod’s IP on the service port. The sidecar intercepts it, terminates mTLS, and forwards plain HTTP to the application on localhost
Outbound traffic: The application makes plain HTTP requests to localhost. The sidecar intercepts these, looks up the destination via the destination service, and establishes mTLS to the upstream
Certificate management: At startup, the sidecar requests a certificate from the identity service. This certificate is automatically rotated before expiration

graph LR subgraph Pod["POD"] subgraph App["Application"] SC["Service Code"] L1["localhost:8080"] end subgraph Sidecar["linkerd2-proxy"] P["linkerd2-proxy"] L2["localhost:4143"] end SC --> L1 L1 <--> L2 end Ext1["External"] ==mTLS==> P P ==mTLS==> Ext2["Upstream"] P -.->|"service discovery"| DEST["destination"] P -.->|"get cert"| ID["identity"] P -.->|"check policy"| POL["policy"]

The control plane consists of just four core components:

destination: Handles service discovery and routing decisions. It provides the proxy with information about where to send requests (which pods, which ports)
identity: Issues and manages mTLS certificates. Each pod gets a certificate signed by the Linkerd CA, with a 24-hour validity period
proxy-api: A thin wrapper that translates Kubernetes concepts into what the proxy understands
policy: Stores and serves authorization policies

The data plane uses linkerd2-proxy, a Rust-based reverse proxy designed specifically for Kubernetes. Unlike Envoy (which is general-purpose), linkerd2-proxy is optimized for the specific needs of service mesh: mTLS, observability, and basic traffic management.

4.4.2 How Linkerd Differs from Istio

Key differences from Istio¹²:

Rust-based proxy: The data plane proxy (linkerd2-proxy) is written in Rust for memory safety and performance. No C++ means reduced risk of memory safety vulnerabilities in the data plane
No configuration required for basics: Default install enables mTLS, retries, timeouts, and observability out of the box
Simpler control plane: Written in Go, with fewer moving parts. No complex istiod split - just four single-purpose services
Purpose-built for Kubernetes: Less general-purpose than Istio. It assumes Kubernetes and focuses on making that work really well

4.4.3 Automatic mTLS

One of Linkerd’s standout features is automatic mTLS that requires zero configuration. When you install Linkerd, every pod gets a sidecar proxy that:

On startup, the proxy contacts the identity service and receives a short-lived certificate
This certificate is automatically rotated before expiration (Linkerd uses 24-hour certificates by default)
For every outgoing connection, the proxy presents its certificate
For every incoming connection, the proxy verifies the client’s certificate
The application code is completely unaware - it just listens on localhost

This is fundamentally different from Istio, where you need to explicitly enable auto-mTLS and configure PeerAuthentication policies.

# Install Linkerd (mTLS enabled by default - no flags needed)
linkerd install | kubectl apply -f -

# Check mTLS status across your mesh
linkerd viz auth policy

# View certificate details for a specific pod
linkerd viz authz -n payments deploy/payment-api

4.4.4 Authorization Model

Linkerd uses a simpler authorization model than Istio. Instead of complex YAML with multiple resource types, you primarily work with two concepts:

Server: Defines a port on a pod that accepts inbound connections
ServerAuthorization: Defines which clients (by service account or meshTLS identity) can access which servers

apiVersion: policy.linkerd.io/v1beta3
kind: ServerAuthorization
metadata:
 namespace: payments
 name: checkout-access
spec:
 server:
 name: payments-api
 client:
 meshTLS:
 serviceAccounts:
 - name: checkout

This says: “allow the checkout service account to connect to the payments-api server, but only if mTLS is verified.” The meshTLS selector means the client must have a valid Linkerd-issued certificate - exactly what you want in Zero Trust.

4.5 Istio vs Linkerd Comparison

Aspect	Istio	Linkerd
Proxy	Envoy (C++)	linkerd2-proxy (Rust)
Complexity	High (many CRDs)	Low (minimal config)
mTLS	Configurable	Automatic by default
Traffic management	Fullfeatured	Core features only
Performance	Good	Excellent
Learning curve	Steep	Gentle
Extensibility	Very high	Limited
L7 features	Full HTTP/gRPC, TCP	HTTP/gRPC focus
Best for	Complex enterprise	Simpler deployments

5. Implementing mTLS in Rust

Before diving into code: in production, you rarely implement mTLS manually. When you use a service mesh like Istio or Linkerd, the sidecar proxies handle all TLS termination, certificate rotation, and identity verification. Your application code speaks plain HTTP to localhost, and the mesh handles everything else.

This section exists to show you what actually happens under the hood. Understanding these internals helps when:

Debugging mTLS issues in production
Building custom sidecar proxies
Implementing workload identity outside Kubernetes
Learning how Zero Trust actually works at the protocol level

5.1 TLS with rustls

rustls is a modern TLS library written in Rust, offering memory safety without garbage collection overhead:

use rustls::{Certificate, PrivateKey, ServerConfig, ClientConfig};
use rustls::pki_types::{UnixTime, CertificateRevocationListParams};
use std::sync::Arc;
use std::time::{Duration, SystemTime};

/// Load certificate chain and private key from PEM files
/// In production, these would be fetched automatically from SPIRE or similar
fn load_certs_and_key(
 cert_path: &str,
 key_path: &str,
) -> Result<(Vec<Certificate>, PrivateKey), std::io::Error> {
 // Read certificate file (PEM format)
 let cert_file = std::fs::File::open(cert_path)?;
 let mut cert_reader = std::io::BufReader::new(cert_file);
 // Parse PEM-encoded certificates into rustls Certificate type
 let certs = rustls_pemfile::certs(&mut cert_reader)?
 .into_iter()
 .map(Certificate)
 .collect();

 // Read private key file
 let key_file = std::fs::File::open(key_path)?;
 let mut key_reader = std::io::BufReader::new(key_file);
 // Parse PKCS#8 formatted private key
 let keys = rustls_pemfile::pkcs8_private_keys(&mut key_reader)?;
 let key = PrivateKey(keys.into_iter().next().unwrap());

 Ok((certs, key))
}

5.2 Building a Secure Server with mTLS

This example shows how to configure a server that requires client certificates. In Zero Trust, this is essential: the server verifies the client’s identity via their certificate, not just their IP address.

use rustls::{
 server::{ClientCertVerified, ClientCertVerifier, ResolvesServerCertUsingSni},
 Certificate, DistinguishedName, PrivateKey, RootCertStore, ServerConfig,
};
use std::sync::Arc;
use tokio::net::TcpListener;
use tokio_rustls::TlsAcceptor;

/// Configuration for mutual TLS (server authentication + client authentication)
pub struct MutualTlsConfig {
 /// CA certificate used to verify client certificates (trust anchor)
 ca_cert: Certificate,
 /// Server's own certificate (presented to clients)
 server_cert: Certificate,
 /// Server's private key (used to prove server identity)
 server_key: PrivateKey,
}

impl MutualTlsConfig {
 /// Load certificates from files
 pub fn new(
 ca_cert_path: &str,
 server_cert_path: &str,
 server_key_path: &str,
 ) -> Result<Self, Box<dyn std::error::Error>> {
 // Load CA certificate that will verify client certificates
 let ca_cert = std::fs::read(ca_cert_path)?;
 // Load server certificate and key
 let (server_cert, server_key) = load_certs_and_key(server_cert_path, server_key_path)?;

 Ok(Self {
 ca_cert: Certificate(ca_cert),
 server_cert: server_cert.into_iter().next().unwrap(),
 server_key,
 })
 }

 /// Build the rustls ServerConfig with mTLS enabled
 /// This is where the magic happens: we require client certificates
 pub fn build_server_config(&self) -> Result<Arc<ServerConfig>, rustls::Error> {
 // Create a root certificate store and add our CA
 // This trust store validates incoming client certificates
 let mut root_store = RootCertStore::empty();
 root_store.add(&self.ca_cert)?;

 // Build a client certificate verifier that requires valid client certs
 let client_cert_verifier = ClientCertVerifier::builder(
 std::time::Duration::from_secs(300), // clock skew allowance
 root_store,
 None::<DistinguishedName>,
 )?
 // Reject requests without a valid client certificate (mTLS enforcement)
 .allow_unauthenticated(false)
 .build()?;

 // Configure server with both server cert and client cert requirement
 let mut config = ServerConfig::builder()
 .with_client_cert_verifier(Arc::new(client_cert_verifier))
 .with_single_cert(
 vec![self.server_cert.clone()],
 self.server_key.clone(),
 )?;

 // Enable HTTP/2 and HTTP/1.1 via ALPN
 config.alpn_protocols = vec![b"h2".to_vec(), b"http/1.1".to_vec()];

 Ok(Arc::new(config))
 }
}

pub async fn start_mtls_server(
 config: Arc<ServerConfig>,
 addr: &str,
) -> Result<(), Box<dyn std::error::Error>> {
 let listener = TcpListener::bind(addr).await?;
 let tls_acceptor = TlsAcceptor::from(config);

 loop {
 let (stream, addr) = listener.accept().await?;

 tokio::spawn(async move {
 match tls_acceptor.accept(stream).await {
 Ok(mut tls_stream) => {
 // Client certificate verified - get peer identity
 if let Some(certs) = tls_stream.peer_certificates() {
 if let Some(cert) = certs.first() {
 let identity = extract_san_from_cert(cert);
 eprintln!("Authenticated connection from: {:?}", identity);
 }
 }
 // Handle request...
 }
 Err(e) => {
 eprintln!("TLS handshake failed: {}", e);
 }
 }
 });
 }
}

5.3 Extracting Identity from Client Certificates

Once the TLS handshake completes with mTLS, we have verified that the client owns a certificate signed by our trusted CA. But we still need to extract the identity from that certificate to make authorization decisions.

use x509_parser::prelude::*;

/// Extract identity (SPIFFE ID, DNS, or CN) from a client certificate
/// This is how we move from "client has valid cert" to "client is payment-service"
fn extract_san_from_cert(cert_der: &[u8]) -> Option<String> {
 // Parse the DER-encoded X.509 certificate
 let (_, cert) = X509Certificate::from_der(cert_der).ok()?;

 // First, check Subject Alternative Names (SAN) - this is the preferred method
 // SANs can contain URI, DNS, IP, or email identities
 for san in cert.subject_alternative_name().ok()?.value.general_names {
 match san {
 // SPIFFE IDs are stored as URIs starting with "spiffe://"
 GeneralName::URI(uri) => {
 if uri.starts_with("spiffe://") {
 return Some(uri.clone());
 }
 }
 // DNS names are common for service identities
 GeneralName::DNSName(dns) => {
 return Some(format!("dns:{}", dns));
 }
 _ => {}
 }
 }

 // Fall back to Common Name (CN) - less preferred but commonly used
 cert.subject().iter_common_name()
 .next()
 .and_then(|cn| cn.as_str().ok())
 .map(|s| format!("cn:{}", s))
}

/// Simple authorization policy based on extracted identity
fn authorize_peer(identity: &str, policy: &AuthorizationPolicy) -> bool {
 match policy {
 AuthorizationPolicy::AllowAll => true,
 AuthorizationPolicy::SpiffeAllow(spiffe_ids) => {
 spiffe_ids.iter().any(|allowed| {
 identity.starts_with(allowed)
 })
 }
 AuthorizationPolicy::DenyAll => false,
 }
}

5.4 Integrating with SPIRE via the Workload API

In production, you rarely manage certificates manually. Instead, you use something like SPIRE to automatically issue and rotate certificates. Here’s how a workload fetches its identity from SPIRE:

use tonic::transport::Endpoint;
use api::workload::workload_client::WorkloadClient;
use api::workload::X509SVIDRequest;

/// Client for communicating with the SPIRE agent via the Workload API
pub struct SpireClient {
 socket_path: String,
}

impl SpireClient {
 pub fn new(socket_path: &str) -> Self {
 Self {
 socket_path: socket_path.to_string(),
 }
 }

 /// Fetch the workload's SVID (SPIFFE Verifiable Identity Document) from SPIRE
 /// This includes the certificate, private key, and trust bundle
 pub async fn fetch_svids(&self) -> Result<SvidBundle, Box<dyn std::error::Error>> {
 // Connect to SPIRE's Unix socket (typically at /run/spire/sockets/agent/spire-agent.sock)
 let channel = Endpoint::from_static("http://[::]:50051")
 .connect_with_connector(service::unix_connect(&self.socket_path))
 .await?;

 let mut client = WorkloadClient::new(channel);

 // Request X.509 SVID - SPIRE will attest the workload's identity
 // based on its node attestor and workload attestor
 let request = tonic::Request::new(X509SVIDRequest {
 ..Default::default()
 });

 // The response contains:
 // - svids: the workload's certificates + private keys
 // - federation_trust_bundle: for cross-trust-domain communication
 let response = client.fetch_x509_svid(request).await?;
 let svids = response.into_inner();

 Ok(SvidBundle {
 svid: svids.svids.first().cloned(),
 bundle: svids.federation_trust_bundle,
 })
 }
}

struct SvidBundle {
 svid: Option<Svid>,
 bundle: Option<TrustBundle>,
}

6. Authorization Policies Beyond mTLS

mTLS gives you strong guarantees: the remote party has a certificate signed by your trusted CA, and all traffic is encrypted. But here’s the gap: mTLS only answers “who is calling?” It doesn’t answer “are they allowed to do this?”

Consider a scenario: your payment service has a valid mTLS certificate from your internal CA. Can it call your user database? Can it write to the audit log? Can it access the analytics service? mTLS can’t answer these questions - it only proves identity, not authorization. This is where authorization policies come in.

In this section, we’ll explore the difference between transport-layer (L4) and application-layer (L7) authorization, and how service meshes implement fine-grained access control.

6.1 L4 vs L7 Authorization

mTLS provides transport-layer security-verifying identity and encrypting bytes. But we often need application-layer controls:

Layer	What it controls	Example
L4 (Transport)	Who can connect	mTLS, IP allowlists
L7 (Application)	What they can do	HTTP method, path, headers, JWT claims

Istio AuthorizationPolicy combines both:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: inventory-access
 namespace: production
spec:
 # Apply this policy to pods labeled with app: inventory
 selector:
 matchLabels:
 app: inventory
 # Action can be ALLOW, DENY, or AUDIT (log but don't block)
 action: ALLOW
 rules:
 # Rule 1: Any service with a valid JWT can read inventory
 # This handles external API consumers with JWT tokens
 - from:
 - source:
 # Wildcard means any JWT-validated principal
 requestPrincipals: ["*"]
 to:
 - operation:
 methods: ["GET"]
 paths: ["/api/v1/inventory/*"]
 # Rule 2: Internal payment service (mTLS identity) can write
 # This is mTLS-based, not JWT - for service-to-service within mesh
 - from:
 - source:
 # Istio extracts SPIFFE ID from mTLS certificate
 principals: ["cluster.local/ns/default/sa/payment"]
 to:
 - operation:
 methods: ["POST", "PUT", "DELETE"]
 paths: ["/api/v1/inventory/*"]
 # Rule 3: Explicit deny for everything not matching above
 # Istio has an implicit ALLOW at the end, so explicit rules catch the rest
 - to:
 - operation:
 methods: ["*"]

6.2 JWT Validation at the Mesh Layer

For external clients (mobile apps, SPAs, third-party APIs), you often use JWTs instead of mTLS. Rather than validating JWTs in your application code, the mesh can validate them before requests reach your service:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: jwt-auth
 namespace: production
spec:
 # Apply JWT validation to the API gateway
 selector:
 matchLabels:
 app: api-gateway
 jwtRules:
 # Configure JWT validation parameters
 - issuer: "https://auth.example.com"
 # Expected audience - JWT must contain this aud claim
 audiences:
 - "api.example.com"
 # Forward the original JWT to the backend (for authorization logging)
 forwardOriginalToken: true
 # Where to fetch the public keys for JWT verification
 jwksUri: "https://auth.example.com/.well-known/jwks.json"
 # Don't require JWT for health/metrics endpoints
 triggerRules:
 - excludedPaths:
 - exact: /health
 - prefix: /metrics

6.2.1 Integration with OIDC/OAuth2

JWTs don’t exist in a vacuum: they are issued by an OpenID Connect (OIDC) provider after an OAuth 2.0 authorization flow. Here’s how this fits together:

The OAuth 2.0 Flow (how clients get tokens):

A client (mobile app, SPA, service) redirects users to your identity provider (e.g., Keycloak, Auth0, Okta)
The user authenticates with their credentials
The identity provider issues an authorization code
The client exchanges the code for an access token (JWT) + refresh token
The client uses the access token in API requests

The OIDC Flow (how tokens are verified):

Your API gateway receives a request with a JWT in the Authorization header
The gateway fetches public keys (JWKS) from the identity provider’s well-known endpoint
The gateway verifies the JWT signature using the public key
The gateway validates claims: iss (issuer), aud (audience), exp (expiration)
If valid, the request proceeds with the JWT claims available for authorization

sequenceDiagram participant Client as Client App
(Mobile/SPA/Service) participant IdP as Identity Provider
(Keycloak/Auth0/Okta) participant Gateway as Service Mesh
(Istio/Linkerd) participant Backend as Your Service Client->>IdP: 1. Authorization request
(login, scopes) IdP-->>Client: 2. Authorization code Client->>IdP: 3. Exchange code for tokens IdP-->>Client: 4. Access token (JWT)
+ Refresh token Client->>Gateway: 5. Request + JWT
(Authorization: Bearer ...) Gateway->>IdP: 6. Fetch JWKS
(/.well-known/jwks.json) IdP-->>Gateway: 7. Public keys Gateway->>Gateway: 8. Verify JWT signature
+ claims Gateway->>Backend: 9. mTLS to backend
(身份验证完成) Backend-->>Client: 10. Response

How Service Meshes Integrate:

Istio’s RequestAuthentication CRD connects to this flow by:

Configuring the JWKS URI: Pointing to your identity provider’s JWKS endpoint
Validating issuer and audience: Ensuring tokens came from your IdP
Extracting claims: Making JWT claims available in AuthorizationPolicy rules
Forwarding the token: Optionally passing the original token to backend services

# Full example: OIDC integration with Istio
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
 name: oidc-auth
 namespace: production
spec:
 selector:
 matchLabels:
 app: api-gateway
 jwtRules:
 # Connect to your OIDC provider (Keycloak example)
 - issuer: "https://auth.example.com/realms/internal"
 audiences:
 - "api-service"
 # Forward JWT to backend for audit logging
 forwardOriginalToken: true
 # JWKS endpoint - IdP publishes keys here
 jwksUri: "https://auth.example.com/realms/internal/protocol/openid-connect/certs"
 # Don't require JWT for public endpoints
 triggerRules:
 - excludedPaths:
 - exact: /health
 - exact: /public/*

This is how you extend Zero Trust beyond the service mesh to external consumers: they authenticate via OIDC, receive a JWT, and the mesh validates that JWT at the edge before any mTLS traffic flows internally.

6.3 Rate Limiting

Even with perfect authentication and authorization, a compromised or malicious client can overwhelm a service with requests. Rate limiting protects services from abuse by throttling based on various dimensions:

Per-client limits: Prevent a single client from consuming too many resources
Per-service limits: Prevent downstream services from being overwhelmed
Global limits: Protect the entire system from traffic spikes

Rate limiting happens at L7, after authentication but before authorization.

# Example: Limit API requests per client identity
apiVersion: telemetrization.io/v1alpha1
kind: RateLimiting
metadata:
 name: global-rate-limit
spec:
 rules:
 - dimensions:
 # Limit based on the authenticated identity (from mTLS or JWT)
 - header:
 name: ":path"
 value: "/api/*"
 limit:
 requests: 100
 unit: minute
 enforced: true

7. Observability Integration

Zero Trust is not a “set it and forget it” architecture. Every authentication decision, authorization check, and certificate rotation needs to be visible to your security and operations teams. Without observability, you can’t detect attacks, debug issues, or prove compliance.

The good news: service meshes already intercept every network flow. This means you get security telemetry “for free” - you just need to collect and correlate it properly.

In this section, we’ll explore how to:

Record authentication and authorization decisions in distributed traces
Correlate security events with request traces for incident investigation
Generate audit logs that satisfy compliance requirements

7.1 Security Signals in Distributed Tracing

Your distributed tracing infrastructure carries more than latency data—it can carry security context. By recording authentication (authN) and authorization (authZ) decisions as spans in your traces, you can:

See which identities attempted to access which services
Detect patterns like repeated authentication failures from the same source
Trace a request through multiple services while preserving security context

use opentelemetry::trace::{Span, SpanKind, Tracer};
use opentelemetry::global;

/// Record security events as spans in distributed traces
/// This enables security analysts to see authZ decisions in the trace timeline
fn record_security_event(
 tracer: &dyn Tracer,
 event: SecurityEvent,
 trace_context: TraceContext,
) {
 // Create a span to represent this security event
 // Use SpanKind::Internal so it's visible in traces
 // but not as a user-facing operation
 let span = tracer.start("security.event", SpanKind::Internal);
 span.set_parent(trace_context);

 // Add security-specific attributes for querying and alerting
 // These become searchable fields in your tracing backend (Jaeger, Zipkin, etc.)
 span.set_attribute("security.event_type", event.event_type());
 span.set_attribute("security.severity", event.severity());
 span.set_attribute("security.source_identity", event.source());
 span.set_attribute("security.target_resource", event.target());
 span.set_attribute("security.decision", event.decision());

 // Add denial reason if access was denied
 if let Some(reason) = event.denial_reason() {
 span.set_attribute("security.denial_reason", reason);
 }

 // Add the authenticated principal for allow decisions
 if let Some(principal) = event.principal() {
 span.set_attribute("security.principal", principal);
 }

 span.end();
}

/// Security events that can be recorded in traces
/// These map to common security operations that should be visible
pub enum SecurityEvent {
 AuthenticationSuccess { principal: String, method: String },
 AuthenticationFailure { reason: String, source_ip: String },
 AuthorizationDenied {
 principal: String,
 resource: String,
 action: String,
 reason: String,
 },
 CertificateExpired { identity: String },
 MutualTlsHandshakeFailure { error: String },
}

7.2 Correlating Security Events with Traces

When investigating an incident, security events should appear in the trace timeline. This lets you answer questions like:

“Why did this request fail?” → Look at the authorization span for the denial reason
“Who was this user?” → Look at the authentication span for the identity
“What happened after the deny?” → See if the trace stops or request gets rerouted

gantt title Trace: POST /api/v1/checkout dateFormat X axisFormat %sms section Gateway validate_request :0, 15 section Auth Service verify_token :15, 45 section Inventory Service reserve :45, 200 section Payment Service charge :200, 800

7.3 Security Metrics and Alerting

Service meshes expose metrics that let you build dashboards and alerts for security posture. Here are the key metrics to track:

mTLS Handshake Success Rate - If this drops, something is wrong with certificate issuance or rotation
Authorization Denials - Who is being denied access? Is it expected or an attack?
Certificate Expiration - Alert when any certificate has < 7 days remaining
Authentication Failures - Track by source principal to detect brute force or compromised credentials

# mTLS handshake success rate (should be near 100%)
sum(rate(istio_tls_handshake_success_total[5m]))
 /
sum(rate(istio_tls_handshake_total[5m]))

# Authorization denials by service
sum by (destination_service) (
 rate(istio_request_total{
 response_code="403",
 reporter="destination"
 }[5m])
)

# Certificate expiration (alert when < 7 days)
istio_certificate_expiry_seconds{namespace!="istio-system"}

# Authentication failures by source
sum by (source_principal) (
 rate(envoy_auth_failure[5m])
)

8. Common Pitfalls and Best Practices

Zero Trust sounds simple in theory—verify identity, enforce least privilege, encrypt everything. But in practice, there are traps that can undermine your entire architecture. This section covers the most common mistakes and how to avoid them.

8.1 Common Mistakes

1. PERMISSIVE mTLS Mode in Production PERMISSIVE mode accepts both mTLS and plaintext connections. This defeats the entire purpose of Zero Trust—an attacker who gains network access can bypass authentication entirely.

# WRONG: PERMISSIVE allows plaintext traffic
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
spec:
 mtls:
 mode: PERMISSIVE # Attackers can bypass mTLS!
---
# RIGHT: STRICT blocks plaintext
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
spec:
 mtls:
 mode: STRICT

2. Allow-All AuthorizationPolicies An empty AuthorizationPolicy {} allows all traffic regardless of identity. This is the Zero Trust equivalent of “allow anywhere.”

# WRONG: Wide open
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-all
spec: {} # Empty = allow everything!
---
# RIGHT: Explicit allow with minimal privileges
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: inventory-policy
spec:
 action: ALLOW
 rules:
 - from:
 - source:
 principals: ["cluster.local/ns/default/sa/order-service"]
 to:
 - operation:
 methods: ["GET"]
 paths: ["/api/v1/inventory/*"]

3. Long-Lived Certificates Without Automation Certificates expiring silently cause production incidents. Always automate rotation with short-lived certificates (~24 hours).

4. Trusting the Entire Cluster Granting access to all namespaces defeats fine-grained authorization. Attackers who compromise one namespace gain access to everything.

# WRONG: Any workload in the cluster can access
source:
 namespaces: ["production"]

# RIGHT: Only specific service accounts
source:
 principals: ["cluster.local/ns/default/sa/order-service"]

8.2 Hardening Checklist

Use this checklist to verify your Zero Trust deployment. These are all essential—if any item is unchecked, you have a gap.

Enable STRICT mTLS mode cluster-wide
Rotate certificates automatically (≤ 24 hour lifetime)
Use workload identity (SPIFFE/SPIRE)
Apply least-privilege AuthorizationPolicies
Enable egress control (deny-by-default outbound)
Validate JWTs at the mesh layer
Monitor security events with alerting
Enable audit logging for all authorization decisions
Test policies before production deployment
Regularly review policy effectiveness

9. Service Mesh Trade-offs

Service meshes add significant capabilities, but they come with costs. Before adopting a service mesh, understand what you’re trading:

Complexity: More components to deploy, monitor, and upgrade
Latency: Every hop goes through a proxy
Resource usage: CPU and memory for sidecars and control plane

This section helps you decide if a service mesh makes sense for your deployment and choose between Istio and Linkerd.

9.1 Performance Overhead

Every network hop adds latency through the sidecar proxy:

Configuration	Latency Impact	Throughput Impact
No mesh	Baseline	Baseline
Istio (Envoy)	+1-3ms	-5-10%
Linkerd (Rust proxy)	+0.2-0.5ms	-1-3%

For most applications, this overhead is acceptable. For ultra-low-latency requirements (high-frequency trading, real-time control), consider:

eBPF-based approaches: Move security enforcement to the kernel
Native integration: Compile security libraries directly into applications
Selective mesh: Only mesh critical paths

9.2 Complexity and Operational Overhead

Service meshes add significant operational complexity:

Concern	Without Mesh	With Mesh
Configuration	Simple	Complex CRDs
Debugging	Application logs	Logs + mesh metrics + traces
Upgrade path	Standard K8s upgrade	Mesh + workload upgrade
Troubleshooting	Direct	Must check sidecar first

9.3 When NOT to Use a Service Mesh

Consider alternatives when:

Small number of services: A few services might not justify the overhead
Latency-critical paths: HFT, real-time control systems
Resource-constrained environments: Edge devices with limited CPU/memory
Greenfield projects: Simpler alternatives might suffice initially
Team expertise: If the team lacks mesh operational experience

Alternatives to service mesh:

Sidecar-less approaches: eBPF-based security (Cilium, Octarine)
Application-level mTLS: Direct TLS in application code
API gateways: Centralized security at ingress points

10. Real-World Deployment Patterns

Zero Trust doesn’t happen overnight. Most organizations have existing systems that weren’t designed with Zero Trust in mind. This section covers practical patterns for migrating brownfield systems, multi-cluster strategies, and how to connect Zero Trust to external services.

10.1 Brownfield Migration Strategy

Migrating existing services to Zero Trust requires careful sequencing:

graph LR P1["PERMISSIVE"] --> P2["STRICT"] P2 --> P3["MIGRATE"] P3 --> P4["COMPLETE"]

Phase 1: Observability without enforcement

# Install mesh in PERMISSIVE mode (log-only)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
 namespace: istio-system
spec:
 mtls:
 mode: PERMISSIVE
# Monitor what traffic is plaintext
# Identify which services need mesh, which don't

Phase 2: Namespace isolation

# Isolate new services with STRICT
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: strict-namespace
spec:
 mtls:
 mode: STRICT

Phase 3: Gradual migration of legacy services

# Legacy namespace stays PERMISSIVE (until upgraded)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: legacy-namespace
spec:
 mtls:
 mode: PERMISSIVE

Phase 4: Full enforcement

# Cluster-wide STRICT once all services support mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
 name: default
spec:
 mtls:
 mode: STRICT

10.2 Multi-Cluster Federation

For high availability or geo-distribution:

10.3 Edge and IoT Deployments

Resource-constrained edge nodes benefit from lightweight approaches:

Linkerd on edge: Minimal resource footprint
SPIRE leaf agents: Lightweight attestation
Simplified policies: Narrower trust domains
Offline operation: Certificate rotation without central connectivity

Conclusion

Zero Trust Architecture represents a fundamental shift in how we think about security: from protecting perimeters to protecting workloads, from implicit trust to continuous verification, from static policies to dynamic, risk-based decisions.

Key takeaways:

mTLS is foundational: It provides the cryptographic basis for service-to-service identity and encryption. Without it, Zero Trust is just policy on paper.
Service mesh provides operational leverage: Automatic mTLS, policy enforcement, and observability without application changes—but adds complexity
Identity is the new perimeter: Workload identity (SPIFFE/SPIRE) enables fine-grained authorization that follows workloads across clusters
L4 alone isn’t enough: Authorization policies (JWT validation, L7 controls) layer on top of mTLS for complete Zero Trust
Observability enables confidence: You can’t secure what you can’t see—security metrics and traces are as essential as the policies themselves

Practical starting points:

Start with STRICT mTLS (not PERMISSIVE)
Enable automatic certificate rotation (≤ 24 hours)
Use AuthorizationPolicies with explicit rules, not allow-all
Monitor mTLS handshake success rate (should be 100%)
Track authorization denials by principal

When service mesh isn’t the answer: For latency-critical paths, small deployments, or resource-constrained edge devices, consider alternatives like eBPF-based networking or application-level mTLS.

The journey, not the destination: Zero Trust isn’t a product you install—it’s a discipline you practice. Start with mTLS, layer in observability, then progressively add authorization policies. Each step reduces your attack surface and increases your confidence in the system’s security posture.

As we explored in the distributed tracing post, modern observability infrastructure already carries the context needed for security. W3C TraceContext propagation happens over the same network paths that carry mTLS certificates. The convergence of observability and security isn’t a future vision—it’s today’s service mesh.

References

Cheswick, W.R. (1990). “The Design of a Secure Internet Gateway”. AT&T Bell Laboratories. ↩︎
Blakley, B. “The Three Myths of Firewalls”. IBM Security Architecture. ↩︎
Kubernetes Documentation. https://kubernetes.io/docs/ ↩︎
Krebs on Security. (2014). “Target Hackers Broke in Via HVAC Company”. https://krebsonsecurity.com/2014/02/target-hackers-broke-in-via-hvac-company/ ↩︎
Verizon. (2024). “Data Breach Investigations Report”. https://www.verizon.com/business/resources/reports/dbir/ ↩︎
CISA. (2020). “Supply Chain Compromise”. https://www.cisa.gov/news-events/alerts/2020/12/13/advanced-persistent-threat-compromise-of-government-corporations-it ↩︎
DOJ. (2019). “Capital One Data Breach Defendant Sentenced”. https://www.justice.gov/opa/pr/capital-one-data-breach-defendant-sentenced-federal-prison ↩︎
GitHub Security Lab. (2022). “Typosquatting and Masquerading in npm”. https://securitylab.github.com/research/npm-packages-malicious/ ↩︎
NIST SP 800-207 - Zero Trust Architecture. National Institute of Standards and Technology. https://csrc.nist.gov/publications/detail/sp/800-207/final ↩︎ ↩︎ ↩︎
SPIFFE Specification - Secure Production Identity Framework for Everyone. The SPIFFE Project. https://spiffe.io/docs/latest/spiffe-about/overview/ ↩︎
SPIRE - SPIFFE Runtime Environment. The SPIFFE Project. https://spiffe.io/docs/latest/spire-about/ ↩︎
Istio Documentation. https://istio.io/latest/docs/ ↩︎ ↩︎ ↩︎
Linkerd Documentation. https://linkerd.io/2.14/overview/ ↩︎ ↩︎
Envoy Proxy Documentation. https://www.envoyproxy.io/docs/envoy/latest/ ↩︎ ↩︎
xDS Protocol. Envoy Proxy. https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol ↩︎

MTLS | Filippo Berto