Zero Trust Architecture for Microservices

Introduction

“All of ARPA’s protection has, by design, left the internal AT&T machines untested. A sort of crunchy shell around a soft, chewy center.”

• William R. Cheswick, The Design of a Secure Internet Gateway (1990)¹

That was 1990. We’re still dealing with the fallout.

For decades, network security operated on this exact premise: everything inside the corporate perimeter could be trusted. Firewalls guarded the castle gates, and once you were inside, you had access to virtually everything. Bob Blakley, a Security Architect at IBM, famously summarized the three myths of firewalls²:

1. “We’ve got the place surrounded”: assuming there are no back doors
2. “Nobody here but us chickens”: assuming all threats are outside the perimeter
3. “Sticks and stones may break my bones”: assuming data can’t execute code

We built an entire industry on these myths. Then came cloud-native architectures.

Modern microservices don’t live behind a single perimeter-they span multiple clouds, edge locations, data centers, and third-party services. A typical enterprise might have hundreds of services communicating across Kubernetes³ clusters, with workloads scaling dynamically and developers deploying code multiple times per day. The traditional perimeter has dissolved into a complex mesh of ephemeral connections.

Where is Zero Trust ** Actually Useful**? Let me give you some real examples:

• E-commerce platforms: Payment services, inventory systems, recommendation engines, and third-party analytics all need to talk to each other-and to external payment gateways, shipping APIs, and fraud detection services. Each service should prove its identity, not just rely on network location.

• Healthcare systems: Patient data flows between EHR systems, billing services, diagnostic APIs, and insurance verification-subject to HIPAA. A breach in one service shouldn’t grant access to everything.

• IoT device management: Thousands of devices connecting to backend services, often from untrusted networks. Each device Needs certificate-based identity, not shared API keys.

• Multi-cloud deployments: Services running on AWS, GCP, and on-premises, needing secure cross-cloud communication without exposing public endpoints.

This post explores how Zero Trust Architecture addresses these challenges, with a deep dive into mutual TLS (mTLS) as the cornerstone of service-to-service authentication, and service mesh as the Infrastructure layer that makes Zero Trust practical at scale. We’ll connect these concepts to the distributed tracing patterns discussed in my previous post, where we saw how OpenTelemetry enables end-to-end observability-the same infrastructure that carries trace context (W3C TraceContext) can simultaneously carry security identity.

1. The Traditional Perimeter Security Problem

1.1 Castle-and-Moat Architecture

The traditional network security model assumed that the local network was safe. Security controls focused on the perimeter:

• Firewalls blocked unauthorized external access
• VPNs provided “trusted” remote access
• Internal networks operated with implicit trust

In this model, if an attacker breached the perimeter or compromised an insider’s credentials, they gained lateral access to everything inside.

1.2 Lateral Movement Attacks

Once inside an untrusted perimeter, attackers could move freely:

1. Compromise one service: through a vulnerability, insider threat, or supply chain attack
2. Escalate privileges: use the compromised service’s credentials to access others
3. Exfiltrate data: hop between services until reaching the target
4. Establish persistence: plant backdoors while moving laterally

The 2013 Target breach exemplifies this⁴. Attackers compromised an HVAC vendor’s credentials, then moved laterally through Target’s network until reaching the point-of-sale systems-accessing 40 million credit card numbers. The HVAC vendor had legitimate network access that should never have reached payment systems.

1.3 Why Internal Traffic Was Considered “Trusted”

The assumption that internal traffic is safe relied on several flawed premises:

• Physical security: Only authorized personnel could access the data center
• Network isolation: Firewalls separated internal from external
• Static infrastructure: Services didn’t change frequently, so monitoring was manageable

These assumptions broke down with:

• Cloud workloads: Applications running outside corporate infrastructure
• Remote work: Employees accessing internal services from home networks
• Dynamic scaling: Services appearing and disappearing as load demands
• Supply chain complexity: Third-party services and libraries with their own access

1.4 Real-World Breaches from Internal Threats

The Verizon 2024 Data Breach Investigations Report found that 68% of breaches involved a human element-insiders, credentials, or social engineering⁵. A significant portion of these exploited implicit trust within the perimeter.

High-profile examples include:

• SolarWinds (2020): Attackers inserted malicious code into software updates, allowing them to move laterally through customer networks once inside⁶
• Capital One (2019): A misconfigured web application firewall allowed access to AWS metadata, then to customer data⁷
• NPM supply chain attacks: Compromised packages granted access to internal build systems⁸

The pattern is clear: trusting internal traffic is the vulnerability.

2. Zero Trust Architecture: Core Principles

Zero Trust Architecture, as formalized by NIST SP 800-207⁹, isn’t just a different kind of firewall: it’s a fundamental rethinking of access control. Instead of granting broad network access based on location, Zero Trust grants fine-grained access to specific resources based on verified identity. Every request is authenticated, every path is encrypted, and every decision is explicit.

2.1 The Paradigm Shift

Zero Trust inverts the fundamental assumption. Instead of “trust inside, verify outside,” Zero Trust operates on:

“Never trust, always verify.”

Every request, regardless of origin, must be authenticated and authorized. The location (inside or outside the perimeter) becomes irrelevant-what matters is whether the requester can prove their identity and is authorized for the specific resource.

This diagram illustrates the Zero Trust flow:

1. Service A initiates contact: but has no implicit trust. It must present its identity to the authorization service.
2. The identity service verifies Service A: checks its certificate, workload metadata, and policies.
3. The identity service verifies Service B: confirms the target is legitimate and accepts requests.
4. Both services are confirmed: identity and authorization are established.
5. Service A can now communicate with Service B: over an encrypted, mutually authenticated channel.

Compare this to the castle-and-moat model where Service A would simply connect directly to Service B on the internal network: no verification needed. The key difference: in Zero Trust, there’s no “inside” that bypasses authentication.

2.2 NIST Zero Trust Principles

While the conceptual shift from perimeter to identity provides a guiding philosophy, practitioners need concrete principles to implement. NIST SP 800-207 (Zero Trust Architecture)⁹ offers exactly that: a formal framework that translates the “never trust, always verify” mantra into actionable design principles.

NIST SP 800-207 defines Zero Trust Architecture around these tenets⁹:

Verify Explicitly

Always authenticate and authorize based on all available data points:

• Identity (user and workload): Verifies who or what is making the request through certificates, tokens, or workload metadata
• Location: Uses network position as a risk factor (e.g., request from unexpected region triggers additional verification)
• Device health: Confirms the endpoint meets security posture requirements (patched, encrypted disk, enabled firewall)
• Service or workload: Validates the calling service’s identity and runtime properties
• Data classification: Considers sensitivity of the requested data resource
• Abnormalities: Detects behavioral anomalies through continuous monitoring

No single factor grants access; multiple signals must align.

Least Privilege Access

Just-in-time (JIT) and just-enough-access (JEA):

• Just-in-time: Access is granted only when needed, for the duration needed
• Just-enough-access: Grant only the minimum permissions required

This limits the blast radius when credentials are compromised.

Assume Breach

Design systems as if an attacker is already inside:

• Minimize blast radius through microsegmentation
• Verify end-to-end encryption
• Continuous monitoring and detection
• Assume any credential could be compromised

2.3 The Identity Plane

In Zero Trust, identity becomes the new perimeter. Every workload (service, container, VM) has a cryptographic identity that persists regardless of where it runs.

This identity is used for:

• Authentication: Proving “I am service A”
• Authorization: Determining “service A can access endpoint X”
• Auditing: Logging “who accessed what and when”
• Encryption: Establishing secure channels between verified identities

2.4 SPIFFE and Workload Identity

[SPIFFE (Secure Production Identity Framework for Everyone)]¹⁰ provides a standardized framework for workload identity. The key concepts:

SPIFFE ID: A URI that uniquely identifies a workload. The format is spiffe://{trust_domain}/{path}, where each component carries semantic meaning:

• Trust domain: The administrative boundary (e.g., example.org) - represents a cluster or organization that manages its own CA hierarchy
• Path: An hierarchical identifier that encodes namespace, service account, or application details (e.g., /ns/default/sa/payment means: namespace default, service account sa, workload payment)

This hierarchical design enables decentralized issuance: each trust domain operates its own CA, and the SPIFFE ID makes it explicit which domain issued which identity. Authorization policies can match on any component: for example, allow only workloads in the payment path to access the billing service.

SVID (SPIFFE Verifiable Identity Document): A signed document containing the SPIFFE ID and cryptographic material (certificate and private key) that the workload uses to prove its identity.

Trust Bundle: A set of certificates that a workload uses to verify the SVIDs of other workloads. Only workloads with identities signed by the same CA (or a federated CA) can communicate.

3. Mutual TLS (mTLS) Deep Dive

Identity without authentication is incomplete. In Zero Trust, mutual TLS (mTLS) is the mechanism that makes identity concrete: both client and server prove they hold the private key corresponding to their asserted identity before any data flows. This section walks through how TLS handshakes work, how mTLS extends one-way TLS, and how to implement it in practice.

3.1 How TLS Works (Traditional One-Way TLS)

Before understanding mTLS, we need to understand standard TLS.

In one-way TLS (HTTPS in browsers), the flow is:

1. ClientHello: The client initiates the handshake by sending a random value and supported cipher suites.
2. ServerHello, Certificate: The server responds with its random value, picks a cipher suite, and sends its certificate containing the public key.
3. ClientKeyExchange: The client verifies the server’s certificate against trusted CAs, then generates a pre-master secret encrypted with the server’s public key.
4. Both derive session keys: Both client and server derive the same master key from the pre-master secret.
5. Finished: Both parties send encrypted “Finished” messages to verify the handshake succeeded.
6. Application data: The encrypted tunnel is established; data flows securely.

The server proves its identity to the client via its certificate, but the server has no idea who the client is.

This works for human-to-website interaction (you need to trust your bank’s website), but it fails for service-to-service communication where both parties need to verify each other.

3.2 Mutual TLS: Both Ways

mTLS extends TLS so that both the client and server present certificates and both authenticate each other.

1. ClientHello: Client initiates with random value and cipher suites.
2. ServerHello + CertRequest: Server responds but also requests a client certificate (CertificateRequest message).
3. Verify server cert + Send cert + CertVerify: Client verifies the server’s certificate, then sends its own certificate plus a CertificateVerify (signed data proving the client owns the private key).
4. Verify client cert: Server validates the client’s certificate and signature.
5. Both derive session keys: Key derivation proceeds as in standard TLS.
6. Finished: Both confirm handshake success.
7. Application data: Both sides are authenticated; encrypted tunnel established.

Key Differences from Standard TLS

The critical addition in mTLS: Steps 2-4 are new. The server explicitly requests the client’s certificate (CertificateRequest), the client responds with its certificate plus a signed proof (CertificateVerify), and the server verifies both. This is what enables service-to-service authentication.

3.3 What mTLS Achieves

With mTLS, we gain:

1. Server authentication: The client knows it’s talking to the real service (not an impostor)
2. Client authentication: The server knows which client is calling (not an anonymous request)
3. Encryption: All traffic is encrypted in transit
4. Integrity: Tampering with messages in transit is detected

This prevents:

• Man-in-the-middle attacks: An attacker can’t intercept traffic without a valid certificate
• Service impersonation: A compromised service can’t pretend to be another service
• Unauthenticated access: Requests without valid certificates are rejected

3.4 Certificate Management Challenges

mTLS requires each service to have a certificate, which introduces operational complexity:

Certificate Lifecycle

The certificate lifecycle has four phases:

1. Issuance: The workload requests a certificate from the CA, presenting its identity (SPIFFE ID or other identifier). The CA validates this identity and signs a certificate binding the workload’s public key to its identity.
2. Distribution: The certificate (and private key) is delivered to the workload. This typically happens at startup or is mounted as a secret. The workload can now present this certificate during TLS handshakes.
3. Validation: During each TLS handshake, the peer verifies the certificate against the trust bundle (root CA or intermediate CA certificates). The peer also checks the certificate hasn’t expired or been revoked via CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol).
4. Rotation: Certificates near expiration are replaced with new ones. Short-lived certificates (hours) require frequent rotation; long-lived certificates (months) need fewer rotations but carry higher risk.

How It’s Implemented in Practice

Most production systems automate the entire lifecycle:

• Certificate request: The workload contacts the CA (e.g., SPIRE, Vault, cloud CA) via the ACME protocol or a custom API, presenting its workload identity.
• Automated distribution: The CA pushes certificates to the workload (via Secret Manager, Vault agent injector, or SPIRE’s node plugin). A common pattern: certificates are stored in a secrets mount and automatically reloaded when renewed.
• Automated rotation: A background process monitors certificate expiration and triggers renewal before expiry. Kubernetes secrets with cert-manager, Vault’s agent, or SPIRE’s node agent handle this automatically.
• Revocation handling: If a key is compromised, the CA adds the certificate to a CRL or marks it revoked in OCSP. Peers check revocation status during handshake; revoked certs are rejected.

This automation is critical: manual certificate management at scale leads to outages (expired certs) or security gaps (revoked certs still accepted).

Short-Lived Certificates vs Long-Lived

Best practice: Use short-lived certificates (hours to days) and automate rotation. This limits the window of damage if a key is compromised.

Certificate Authority Options

1. Public CAs (Let’s Encrypt, DigiCert): Good for external-facing services
2. Private CAs (Cloud providers, HashiCorp Vault, Step): For internal service mesh
3. Intermediate CAs: Sign workload certificates, keep root CA offline

3.5 SPIFFE and SPIRE

[SPIRE]¹¹ (SPIFFE Runtime Environment) automates workload identity management:

How it works:

1. Registration: An operator registers workload identity with the SPIRE server (e.g., “Kubernetes pod with selector namespace=default, serviceAccount=payment should receive identity spiffe://example.org/payment”)
2. Agent attestation: The SPIRE agent running on the node verifies the workload’s identity (using Kubernetes token review or node attestation)
3. SVID issuance: The SPIRE server issues a short-lived X.509 SVID to the workload via the agent
4. Workload uses SVID: The workload uses its SVID certificate to authenticate mTLS connections

Federation: SPIRE supports federating trust between organizations. Two SPIRE servers can exchange trust bundles, allowing workloads in one trust domain to authenticate workloads in another.

4. Service Mesh: Infrastructure for Zero Trust

So far, we’ve discussed the conceptual foundations of Zero Trust and the cryptographic mechanisms that make it possible. But here’s the practical reality: implementing mTLS manually between every service pair, managing certificate rotation, enforcing authorization policies, and collecting observability data across hundreds of microservices quickly becomes unmanageable. You need infrastructure that automates these concerns at scale.

This is where a service mesh comes in. A service mesh is a dedicated infrastructure layer that handles service-to-service communication, offloading security, reliability, and observability concerns from application code to the platform. Rather than each team implementing mTLS, writing custom interceptors, or instrumenting their own metrics, services simply communicate through the mesh and let it handle the rest.

In this section, we’ll explore what a service mesh is, how it implements Zero Trust principles at the network layer, and compare the two most popular implementations: Istio¹² and Linkerd¹³.

4.1 What is a Service Mesh?

A service mesh is a dedicated infrastructure layer that handles service-to-service communication. It provides:

• Automatic mTLS: Encryption and authentication without application changes
• Traffic management: Load balancing, retries, circuit breaking
• Observability: Metrics, traces, and logs for all traffic
• Security policies: Authorization rules, rate limiting

The key architectural pattern is the sidecar proxy:

The application doesn’t handle TLS, networking, or security policies-these are handled by the sidecar proxy running alongside it.

4.2 Envoy Proxy Deep Dive

When a network packet arrives at the Envoy sidecar, it goes through a well-defined processing pipeline. Understanding this path is essential for debugging issues and designing effective service mesh deployments.

Inbound Request Path (when another service calls this one):

1. A packet arrives at the container’s network namespace on the listener port (e.g., 443)
2. The TLS Inspector filter examines the incoming bytes to detect whether this is TLS and extract SNI (Server Name Indication)
3. The HTTP Codec parses the HTTP request: method, path, headers, body
4. The Router Filter evaluates the request against configured routes to determine the upstream cluster
5. The request is forwarded to an endpoint in that cluster (another pod’s sidecar)

Outbound Request Path (when this service calls another):

1. The application makes a plain HTTP request to localhost (the sidecar listener, e.g., port 15001)
2. Envoy accepts the connection and applies its own processing pipeline
3. The router filter looks up the destination cluster based on the Host header or IP
4. Envoy establishes a new mTLS connection to the upstream sidecar
5. The response travels back through the same chain in reverse

This bidirectional interception is what makes the service mesh so powerful: every network flow is visible, every connection is encrypted, and every policy is enforced consistently across the entire mesh.

Envoy¹⁴ is the de facto standard sidecar proxy for service meshes. Its architecture consists of:

• Listeners: Network listeners that accept incoming connections (one per port)
• Filter Chains: Ordered list of filters applied to incoming requests (TLS inspector, HTTP router, etc.)
• Routes: How requests are routed to upstream clusters
• Clusters: Groups of upstream endpoints (your services)
• Endpoints: Individual IP:port combinations for upstream services

The xDS Protocol¹⁵: Envoy discovers its configuration dynamically via the xDS APIs:

• LDS (Listener Discovery Service): What listeners to create
• RDS (Route Discovery Service): How to route traffic
• CDS (Cluster Discovery Service): What upstream clusters exist
• EDS (Endpoint Discovery Service): What endpoints exist in each cluster
• SDS (Secret Discovery Service): TLS certificates and keys
• ADs (Aggregated Discovery Service): Combines multiple discovery services

This allows the control plane to push configuration updates without restarting proxies.

4.3 Istio Architecture

Istio¹² is the most feature-rich service mesh, built on top of Envoy¹⁴.

Key Istio Resources:

# PeerAuthentication: Enforce mTLS mode
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT # or PERMISSIVE (allows plaintext for migration)
---
# AuthorizationPolicy: Define who can talk to whom
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-policy
  namespace: default
spec:
  selector:
    matchLabels:
      app: payment
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/default/sa/checkout"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/api/v1/charge"]
---
# DestinationRule: Configure mTLS and load balancing
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: payment-destination
  namespace: default
spec:
  host: payment.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL # Use Istio-managed certificates
    loadBalancer:
      simple: LEAST_REQUEST

4.4 Linkerd Architecture

Linkerd¹³ takes a different approach, prioritizing simplicity and security defaults. While Istio gives you fine-grained control over every aspect of the mesh, Linkerd focuses on doing the essentials really well with minimal configuration.

4.4.1 Architecture Overview

Linkerd uses the same sidecar pattern as Istio: each pod gets a proxy container injected alongside the application container. However, the implementation is simpler and the proxy is more specialized.

Here’s how it works:

1. Injection: When you annotate a namespace or pod with linkerd.io/inject: enabled, Kubernetes adds a linkerd2-proxy container to the pod
2. Inbound traffic: External traffic arrives at the pod’s IP on the service port. The sidecar intercepts it, terminates mTLS, and forwards plain HTTP to the application on localhost
3. Outbound traffic: The application makes plain HTTP requests to localhost. The sidecar intercepts these, looks up the destination via the destination service, and establishes mTLS to the upstream
4. Certificate management: At startup, the sidecar requests a certificate from the identity service. This certificate is automatically rotated before expiration

The control plane consists of just four core components:

• destination: Handles service discovery and routing decisions. It provides the proxy with information about where to send requests (which pods, which ports)
• identity: Issues and manages mTLS certificates. Each pod gets a certificate signed by the Linkerd CA, with a 24-hour validity period
• proxy-api: A thin wrapper that translates Kubernetes concepts into what the proxy understands
• policy: Stores and serves authorization policies

The data plane uses linkerd2-proxy, a Rust-based reverse proxy designed specifically for Kubernetes. Unlike Envoy (which is general-purpose), linkerd2-proxy is optimized for the specific needs of service mesh: mTLS, observability, and basic traffic management.

4.4.2 How Linkerd Differs from Istio

Key differences from Istio¹²:

1. Rust-based proxy: The data plane proxy (linkerd2-proxy) is written in Rust for memory safety and performance. No C++ means reduced risk of memory safety vulnerabilities in the data plane
2. No configuration required for basics: Default install enables mTLS, retries, timeouts, and observability out of the box
3. Simpler control plane: Written in Go, with fewer moving parts. No complex istiod split - just four single-purpose services
4. Purpose-built for Kubernetes: Less general-purpose than Istio. It assumes Kubernetes and focuses on making that work really well

4.4.3 Automatic mTLS

One of Linkerd’s standout features is automatic mTLS that requires zero configuration. When you install Linkerd, every pod gets a sidecar proxy that:

1. On startup, the proxy contacts the identity service and receives a short-lived certificate
2. This certificate is automatically rotated before expiration (Linkerd uses 24-hour certificates by default)
3. For every outgoing connection, the proxy presents its certificate
4. For every incoming connection, the proxy verifies the client’s certificate
5. The application code is completely unaware - it just listens on localhost

This is fundamentally different from Istio, where you need to explicitly enable auto-mTLS and configure PeerAuthentication policies.

# Install Linkerd (mTLS enabled by default - no flags needed)
linkerd install | kubectl apply -f -

# Check mTLS status across your mesh
linkerd viz auth policy

# View certificate details for a specific pod
linkerd viz authz -n payments deploy/payment-api

4.4.4 Authorization Model

Linkerd uses a simpler authorization model than Istio. Instead of complex YAML with multiple resource types, you primarily work with two concepts:

• Server: Defines a port on a pod that accepts inbound connections
• ServerAuthorization: Defines which clients (by service account or meshTLS identity) can access which servers

apiVersion: policy.linkerd.io/v1beta3
kind: ServerAuthorization
metadata:
  namespace: payments
  name: checkout-access
spec:
  server:
    name: payments-api
  client:
    meshTLS:
      serviceAccounts:
        - name: checkout

This says: “allow the checkout service account to connect to the payments-api server, but only if mTLS is verified.” The meshTLS selector means the client must have a valid Linkerd-issued certificate - exactly what you want in Zero Trust.

4.5 Istio vs Linkerd Comparison

5. Implementing mTLS in Rust

Before diving into code: in production, you rarely implement mTLS manually. When you use a service mesh like Istio or Linkerd, the sidecar proxies handle all TLS termination, certificate rotation, and identity verification. Your application code speaks plain HTTP to localhost, and the mesh handles everything else.

This section exists to show you what actually happens under the hood. Understanding these internals helps when:

• Debugging mTLS issues in production
• Building custom sidecar proxies
• Implementing workload identity outside Kubernetes
• Learning how Zero Trust actually works at the protocol level

5.1 TLS with rustls

rustls is a modern TLS library written in Rust, offering memory safety without garbage collection overhead:

use rustls::{Certificate, PrivateKey, ServerConfig, ClientConfig};
use rustls::pki_types::{UnixTime, CertificateRevocationListParams};
use std::sync::Arc;
use std::time::{Duration, SystemTime};

/// Load certificate chain and private key from PEM files
/// In production, these would be fetched automatically from SPIRE or similar
fn load_certs_and_key(
    cert_path: &str,
    key_path: &str,
) -> Result<(Vec<Certificate>, PrivateKey), std::io::Error> {
    // Read certificate file (PEM format)
    let cert_file = std::fs::File::open(cert_path)?;
    let mut cert_reader = std::io::BufReader::new(cert_file);
    // Parse PEM-encoded certificates into rustls Certificate type
    let certs = rustls_pemfile::certs(&mut cert_reader)?
        .into_iter()
        .map(Certificate)
        .collect();

    // Read private key file
    let key_file = std::fs::File::open(key_path)?;
    let mut key_reader = std::io::BufReader::new(key_file);
    // Parse PKCS#8 formatted private key
    let keys = rustls_pemfile::pkcs8_private_keys(&mut key_reader)?;
    let key = PrivateKey(keys.into_iter().next().unwrap());

    Ok((certs, key))
}

5.2 Building a Secure Server with mTLS

This example shows how to configure a server that requires client certificates. In Zero Trust, this is essential: the server verifies the client’s identity via their certificate, not just their IP address.

use rustls::{
    server::{ClientCertVerified, ClientCertVerifier, ResolvesServerCertUsingSni},
    Certificate, DistinguishedName, PrivateKey, RootCertStore, ServerConfig,
};
use std::sync::Arc;
use tokio::net::TcpListener;
use tokio_rustls::TlsAcceptor;

/// Configuration for mutual TLS (server authentication + client authentication)
pub struct MutualTlsConfig {
    /// CA certificate used to verify client certificates (trust anchor)
    ca_cert: Certificate,
    /// Server's own certificate (presented to clients)
    server_cert: Certificate,
    /// Server's private key (used to prove server identity)
    server_key: PrivateKey,
}

impl MutualTlsConfig {
    /// Load certificates from files
    pub fn new(
        ca_cert_path: &str,
        server_cert_path: &str,
        server_key_path: &str,
    ) -> Result<Self, Box<dyn std::error::Error>> {
        // Load CA certificate that will verify client certificates
        let ca_cert = std::fs::read(ca_cert_path)?;
        // Load server certificate and key
        let (server_cert, server_key) = load_certs_and_key(server_cert_path, server_key_path)?;

        Ok(Self {
            ca_cert: Certificate(ca_cert),
            server_cert: server_cert.into_iter().next().unwrap(),
            server_key,
        })
    }

    /// Build the rustls ServerConfig with mTLS enabled
    /// This is where the magic happens: we require client certificates
    pub fn build_server_config(&self) -> Result<Arc<ServerConfig>, rustls::Error> {
        // Create a root certificate store and add our CA
        // This trust store validates incoming client certificates
        let mut root_store = RootCertStore::empty();
        root_store.add(&self.ca_cert)?;

        // Build a client certificate verifier that requires valid client certs
        let client_cert_verifier = ClientCertVerifier::builder(
            std::time::Duration::from_secs(300), // clock skew allowance
            root_store,
            None::<DistinguishedName>,
        )?
        // Reject requests without a valid client certificate (mTLS enforcement)
        .allow_unauthenticated(false)
        .build()?;

        // Configure server with both server cert and client cert requirement
        let mut config = ServerConfig::builder()
            .with_client_cert_verifier(Arc::new(client_cert_verifier))
            .with_single_cert(
                vec![self.server_cert.clone()],
                self.server_key.clone(),
            )?;

        // Enable HTTP/2 and HTTP/1.1 via ALPN
        config.alpn_protocols = vec![b"h2".to_vec(), b"http/1.1".to_vec()];

        Ok(Arc::new(config))
    }
}

pub async fn start_mtls_server(
    config: Arc<ServerConfig>,
    addr: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind(addr).await?;
    let tls_acceptor = TlsAcceptor::from(config);

    loop {
        let (stream, addr) = listener.accept().await?;

        tokio::spawn(async move {
            match tls_acceptor.accept(stream).await {
                Ok(mut tls_stream) => {
                    // Client certificate verified - get peer identity
                    if let Some(certs) = tls_stream.peer_certificates() {
                        if let Some(cert) = certs.first() {
                            let identity = extract_san_from_cert(cert);
                            eprintln!("Authenticated connection from: {:?}", identity);
                        }
                    }
                    // Handle request...
                }
                Err(e) => {
                    eprintln!("TLS handshake failed: {}", e);
                }
            }
        });
    }
}

5.3 Extracting Identity from Client Certificates

Once the TLS handshake completes with mTLS, we have verified that the client owns a certificate signed by our trusted CA. But we still need to extract the identity from that certificate to make authorization decisions.

use x509_parser::prelude::*;

/// Extract identity (SPIFFE ID, DNS, or CN) from a client certificate
/// This is how we move from "client has valid cert" to "client is payment-service"
fn extract_san_from_cert(cert_der: &[u8]) -> Option<String> {
    // Parse the DER-encoded X.509 certificate
    let (_, cert) = X509Certificate::from_der(cert_der).ok()?;

    // First, check Subject Alternative Names (SAN) - this is the preferred method
    // SANs can contain URI, DNS, IP, or email identities
    for san in cert.subject_alternative_name().ok()?.value.general_names {
        match san {
            // SPIFFE IDs are stored as URIs starting with "spiffe://"
            GeneralName::URI(uri) => {
                if uri.starts_with("spiffe://") {
                    return Some(uri.clone());
                }
            }
            // DNS names are common for service identities
            GeneralName::DNSName(dns) => {
                return Some(format!("dns:{}", dns));
            }
            _ => {}
        }
    }

    // Fall back to Common Name (CN) - less preferred but commonly used
    cert.subject().iter_common_name()
        .next()
        .and_then(|cn| cn.as_str().ok())
        .map(|s| format!("cn:{}", s))
}

/// Simple authorization policy based on extracted identity
fn authorize_peer(identity: &str, policy: &AuthorizationPolicy) -> bool {
    match policy {
        AuthorizationPolicy::AllowAll => true,
        AuthorizationPolicy::SpiffeAllow(spiffe_ids) => {
            spiffe_ids.iter().any(|allowed| {
                identity.starts_with(allowed)
            })
        }
        AuthorizationPolicy::DenyAll => false,
    }
}

5.4 Integrating with SPIRE via the Workload API

In production, you rarely manage certificates manually. Instead, you use something like SPIRE to automatically issue and rotate certificates. Here’s how a workload fetches its identity from SPIRE:

use tonic::transport::Endpoint;
use api::workload::workload_client::WorkloadClient;
use api::workload::X509SVIDRequest;

/// Client for communicating with the SPIRE agent via the Workload API
pub struct SpireClient {
    socket_path: String,
}

impl SpireClient {
    pub fn new(socket_path: &str) -> Self {
        Self {
            socket_path: socket_path.to_string(),
        }
    }

    /// Fetch the workload's SVID (SPIFFE Verifiable Identity Document) from SPIRE
    /// This includes the certificate, private key, and trust bundle
    pub async fn fetch_svids(&self) -> Result<SvidBundle, Box<dyn std::error::Error>> {
        // Connect to SPIRE's Unix socket (typically at /run/spire/sockets/agent/spire-agent.sock)
        let channel = Endpoint::from_static("http://[::]:50051")
            .connect_with_connector(service::unix_connect(&self.socket_path))
            .await?;

        let mut client = WorkloadClient::new(channel);

        // Request X.509 SVID - SPIRE will attest the workload's identity
        // based on its node attestor and workload attestor
        let request = tonic::Request::new(X509SVIDRequest {
            ..Default::default()
        });

        // The response contains:
        // - svids: the workload's certificates + private keys
        // - federation_trust_bundle: for cross-trust-domain communication
        let response = client.fetch_x509_svid(request).await?;
        let svids = response.into_inner();

        Ok(SvidBundle {
            svid: svids.svids.first().cloned(),
            bundle: svids.federation_trust_bundle,
        })
    }
}

struct SvidBundle {
    svid: Option<Svid>,
    bundle: Option<TrustBundle>,
}

6. Authorization Policies Beyond mTLS

mTLS gives you strong guarantees: the remote party has a certificate signed by your trusted CA, and all traffic is encrypted. But here’s the gap: mTLS only answers “who is calling?” It doesn’t answer “are they allowed to do this?”

Consider a scenario: your payment service has a valid mTLS certificate from your internal CA. Can it call your user database? Can it write to the audit log? Can it access the analytics service? mTLS can’t answer these questions - it only proves identity, not authorization. This is where authorization policies come in.

In this section, we’ll explore the difference between transport-layer (L4) and application-layer (L7) authorization, and how service meshes implement fine-grained access control.

6.1 L4 vs L7 Authorization

mTLS provides transport-layer security-verifying identity and encrypting bytes. But we often need application-layer controls:

Istio AuthorizationPolicy combines both:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: inventory-access
  namespace: production
spec:
  # Apply this policy to pods labeled with app: inventory
  selector:
    matchLabels:
      app: inventory
  # Action can be ALLOW, DENY, or AUDIT (log but don't block)
  action: ALLOW
  rules:
    # Rule 1: Any service with a valid JWT can read inventory
    # This handles external API consumers with JWT tokens
    - from:
        - source:
            # Wildcard means any JWT-validated principal
            requestPrincipals: ["*"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/api/v1/inventory/*"]
    # Rule 2: Internal payment service (mTLS identity) can write
    # This is mTLS-based, not JWT - for service-to-service within mesh
    - from:
        - source:
            # Istio extracts SPIFFE ID from mTLS certificate
            principals: ["cluster.local/ns/default/sa/payment"]
      to:
        - operation:
            methods: ["POST", "PUT", "DELETE"]
            paths: ["/api/v1/inventory/*"]
    # Rule 3: Explicit deny for everything not matching above
    # Istio has an implicit ALLOW at the end, so explicit rules catch the rest
    - to:
        - operation:
            methods: ["*"]

6.2 JWT Validation at the Mesh Layer

For external clients (mobile apps, SPAs, third-party APIs), you often use JWTs instead of mTLS. Rather than validating JWTs in your application code, the mesh can validate them before requests reach your service:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: production
spec:
  # Apply JWT validation to the API gateway
  selector:
    matchLabels:
      app: api-gateway
  jwtRules:
    # Configure JWT validation parameters
    - issuer: "https://auth.example.com"
      # Expected audience - JWT must contain this aud claim
      audiences:
        - "api.example.com"
      # Forward the original JWT to the backend (for authorization logging)
      forwardOriginalToken: true
      # Where to fetch the public keys for JWT verification
      jwksUri: "https://auth.example.com/.well-known/jwks.json"
      # Don't require JWT for health/metrics endpoints
      triggerRules:
        - excludedPaths:
            - exact: /health
            - prefix: /metrics

6.2.1 Integration with OIDC/OAuth2

JWTs don’t exist in a vacuum: they are issued by an OpenID Connect (OIDC) provider after an OAuth 2.0 authorization flow. Here’s how this fits together:

The OAuth 2.0 Flow (how clients get tokens):

1. A client (mobile app, SPA, service) redirects users to your identity provider (e.g., Keycloak, Auth0, Okta)
2. The user authenticates with their credentials
3. The identity provider issues an authorization code
4. The client exchanges the code for an access token (JWT) + refresh token
5. The client uses the access token in API requests

The OIDC Flow (how tokens are verified):

1. Your API gateway receives a request with a JWT in the Authorization header
2. The gateway fetches public keys (JWKS) from the identity provider’s well-known endpoint
3. The gateway verifies the JWT signature using the public key
4. The gateway validates claims: iss (issuer), aud (audience), exp (expiration)
5. If valid, the request proceeds with the JWT claims available for authorization

How Service Meshes Integrate:

Istio’s RequestAuthentication CRD connects to this flow by:

• Configuring the JWKS URI: Pointing to your identity provider’s JWKS endpoint
• Validating issuer and audience: Ensuring tokens came from your IdP
• Extracting claims: Making JWT claims available in AuthorizationPolicy rules
• Forwarding the token: Optionally passing the original token to backend services

# Full example: OIDC integration with Istio
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: oidc-auth
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-gateway
  jwtRules:
    # Connect to your OIDC provider (Keycloak example)
    - issuer: "https://auth.example.com/realms/internal"
      audiences:
        - "api-service"
      # Forward JWT to backend for audit logging
      forwardOriginalToken: true
      # JWKS endpoint - IdP publishes keys here
      jwksUri: "https://auth.example.com/realms/internal/protocol/openid-connect/certs"
      # Don't require JWT for public endpoints
      triggerRules:
        - excludedPaths:
            - exact: /health
            - exact: /public/*

This is how you extend Zero Trust beyond the service mesh to external consumers: they authenticate via OIDC, receive a JWT, and the mesh validates that JWT at the edge before any mTLS traffic flows internally.

6.3 Rate Limiting

Even with perfect authentication and authorization, a compromised or malicious client can overwhelm a service with requests. Rate limiting protects services from abuse by throttling based on various dimensions:

• Per-client limits: Prevent a single client from consuming too many resources
• Per-service limits: Prevent downstream services from being overwhelmed
• Global limits: Protect the entire system from traffic spikes

Rate limiting happens at L7, after authentication but before authorization.

# Example: Limit API requests per client identity
apiVersion: telemetrization.io/v1alpha1
kind: RateLimiting
metadata:
  name: global-rate-limit
spec:
  rules:
    - dimensions:
        # Limit based on the authenticated identity (from mTLS or JWT)
        - header:
            name: ":path"
            value: "/api/*"
      limit:
        requests: 100
        unit: minute
      enforced: true

7. Observability Integration

Zero Trust is not a “set it and forget it” architecture. Every authentication decision, authorization check, and certificate rotation needs to be visible to your security and operations teams. Without observability, you can’t detect attacks, debug issues, or prove compliance.

The good news: service meshes already intercept every network flow. This means you get security telemetry “for free” - you just need to collect and correlate it properly.

In this section, we’ll explore how to:

• Record authentication and authorization decisions in distributed traces
• Correlate security events with request traces for incident investigation
• Generate audit logs that satisfy compliance requirements

7.1 Security Signals in Distributed Tracing

Your distributed tracing infrastructure carries more than latency data—it can carry security context. By recording authentication (authN) and authorization (authZ) decisions as spans in your traces, you can:

• See which identities attempted to access which services
• Detect patterns like repeated authentication failures from the same source
• Trace a request through multiple services while preserving security context

use opentelemetry::trace::{Span, SpanKind, Tracer};
use opentelemetry::global;

/// Record security events as spans in distributed traces
/// This enables security analysts to see authZ decisions in the trace timeline
fn record_security_event(
    tracer: &dyn Tracer,
    event: SecurityEvent,
    trace_context: TraceContext,
) {
    // Create a span to represent this security event
    // Use SpanKind::Internal so it's visible in traces
    // but not as a user-facing operation
    let span = tracer.start("security.event", SpanKind::Internal);
    span.set_parent(trace_context);

    // Add security-specific attributes for querying and alerting
    // These become searchable fields in your tracing backend (Jaeger, Zipkin, etc.)
    span.set_attribute("security.event_type", event.event_type());
    span.set_attribute("security.severity", event.severity());
    span.set_attribute("security.source_identity", event.source());
    span.set_attribute("security.target_resource", event.target());
    span.set_attribute("security.decision", event.decision());

    // Add denial reason if access was denied
    if let Some(reason) = event.denial_reason() {
        span.set_attribute("security.denial_reason", reason);
    }

    // Add the authenticated principal for allow decisions
    if let Some(principal) = event.principal() {
        span.set_attribute("security.principal", principal);
    }

    span.end();
}

/// Security events that can be recorded in traces
/// These map to common security operations that should be visible
pub enum SecurityEvent {
    AuthenticationSuccess { principal: String, method: String },
    AuthenticationFailure { reason: String, source_ip: String },
    AuthorizationDenied {
        principal: String,
        resource: String,
        action: String,
        reason: String,
    },
    CertificateExpired { identity: String },
    MutualTlsHandshakeFailure { error: String },
}

7.2 Correlating Security Events with Traces

When investigating an incident, security events should appear in the trace timeline. This lets you answer questions like:

• “Why did this request fail?” → Look at the authorization span for the denial reason
• “Who was this user?” → Look at the authentication span for the identity
• “What happened after the deny?” → See if the trace stops or request gets rerouted

7.3 Security Metrics and Alerting

Service meshes expose metrics that let you build dashboards and alerts for security posture. Here are the key metrics to track:

• mTLS Handshake Success Rate - If this drops, something is wrong with certificate issuance or rotation
• Authorization Denials - Who is being denied access? Is it expected or an attack?
• Certificate Expiration - Alert when any certificate has < 7 days remaining
• Authentication Failures - Track by source principal to detect brute force or compromised credentials

# mTLS handshake success rate (should be near 100%)
sum(rate(istio_tls_handshake_success_total[5m]))
  /
sum(rate(istio_tls_handshake_total[5m]))

# Authorization denials by service
sum by (destination_service) (
  rate(istio_request_total{
    response_code="403",
    reporter="destination"
  }[5m])
)

# Certificate expiration (alert when < 7 days)
istio_certificate_expiry_seconds{namespace!="istio-system"}

# Authentication failures by source
sum by (source_principal) (
  rate(envoy_auth_failure[5m])
)

8. Common Pitfalls and Best Practices

Zero Trust sounds simple in theory—verify identity, enforce least privilege, encrypt everything. But in practice, there are traps that can undermine your entire architecture. This section covers the most common mistakes and how to avoid them.

8.1 Common Mistakes

1. PERMISSIVE mTLS Mode in Production PERMISSIVE mode accepts both mTLS and plaintext connections. This defeats the entire purpose of Zero Trust—an attacker who gains network access can bypass authentication entirely.

# WRONG: PERMISSIVE allows plaintext traffic
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: PERMISSIVE # Attackers can bypass mTLS!
---
# RIGHT: STRICT blocks plaintext
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

2. Allow-All AuthorizationPolicies An empty AuthorizationPolicy {} allows all traffic regardless of identity. This is the Zero Trust equivalent of “allow anywhere.”

# WRONG: Wide open
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all
spec: {} # Empty = allow everything!
---
# RIGHT: Explicit allow with minimal privileges
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: inventory-policy
spec:
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/default/sa/order-service"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/api/v1/inventory/*"]

3. Long-Lived Certificates Without Automation Certificates expiring silently cause production incidents. Always automate rotation with short-lived certificates (~24 hours).

4. Trusting the Entire Cluster Granting access to all namespaces defeats fine-grained authorization. Attackers who compromise one namespace gain access to everything.

# WRONG: Any workload in the cluster can access
source:
  namespaces: ["production"]

# RIGHT: Only specific service accounts
source:
  principals: ["cluster.local/ns/default/sa/order-service"]

8.2 Hardening Checklist

Use this checklist to verify your Zero Trust deployment. These are all essential—if any item is unchecked, you have a gap.

• Enable STRICT mTLS mode cluster-wide
• Rotate certificates automatically (≤ 24 hour lifetime)
• Use workload identity (SPIFFE/SPIRE)
• Apply least-privilege AuthorizationPolicies
• Enable egress control (deny-by-default outbound)
• Validate JWTs at the mesh layer
• Monitor security events with alerting
• Enable audit logging for all authorization decisions
• Test policies before production deployment
• Regularly review policy effectiveness

9. Service Mesh Trade-offs

Service meshes add significant capabilities, but they come with costs. Before adopting a service mesh, understand what you’re trading:

• Complexity: More components to deploy, monitor, and upgrade
• Latency: Every hop goes through a proxy
• Resource usage: CPU and memory for sidecars and control plane

This section helps you decide if a service mesh makes sense for your deployment and choose between Istio and Linkerd.

9.1 Performance Overhead

Every network hop adds latency through the sidecar proxy:

For most applications, this overhead is acceptable. For ultra-low-latency requirements (high-frequency trading, real-time control), consider:

• eBPF-based approaches: Move security enforcement to the kernel
• Native integration: Compile security libraries directly into applications
• Selective mesh: Only mesh critical paths

9.2 Complexity and Operational Overhead

Service meshes add significant operational complexity:

9.3 When NOT to Use a Service Mesh

Consider alternatives when:

1. Small number of services: A few services might not justify the overhead
2. Latency-critical paths: HFT, real-time control systems
3. Resource-constrained environments: Edge devices with limited CPU/memory
4. Greenfield projects: Simpler alternatives might suffice initially
5. Team expertise: If the team lacks mesh operational experience

Alternatives to service mesh:

• Sidecar-less approaches: eBPF-based security (Cilium, Octarine)
• Application-level mTLS: Direct TLS in application code
• API gateways: Centralized security at ingress points

10. Real-World Deployment Patterns

Zero Trust doesn’t happen overnight. Most organizations have existing systems that weren’t designed with Zero Trust in mind. This section covers practical patterns for migrating brownfield systems, multi-cluster strategies, and how to connect Zero Trust to external services.

10.1 Brownfield Migration Strategy

Migrating existing services to Zero Trust requires careful sequencing:

Phase 1: Observability without enforcement

# Install mesh in PERMISSIVE mode (log-only)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: PERMISSIVE
# Monitor what traffic is plaintext
# Identify which services need mesh, which don't

Phase 2: Namespace isolation

# Isolate new services with STRICT
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-namespace
spec:
  mtls:
    mode: STRICT

Phase 3: Gradual migration of legacy services

# Legacy namespace stays PERMISSIVE (until upgraded)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: legacy-namespace
spec:
  mtls:
    mode: PERMISSIVE

Phase 4: Full enforcement

# Cluster-wide STRICT once all services support mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

10.2 Multi-Cluster Federation

For high availability or geo-distribution:

10.3 Edge and IoT Deployments

Resource-constrained edge nodes benefit from lightweight approaches:

• Linkerd on edge: Minimal resource footprint
• SPIRE leaf agents: Lightweight attestation
• Simplified policies: Narrower trust domains
• Offline operation: Certificate rotation without central connectivity

Conclusion

Zero Trust Architecture represents a fundamental shift in how we think about security: from protecting perimeters to protecting workloads, from implicit trust to continuous verification, from static policies to dynamic, risk-based decisions.

Key takeaways:

1. mTLS is foundational: It provides the cryptographic basis for service-to-service identity and encryption. Without it, Zero Trust is just policy on paper.
2. Service mesh provides operational leverage: Automatic mTLS, policy enforcement, and observability without application changes—but adds complexity
3. Identity is the new perimeter: Workload identity (SPIFFE/SPIRE) enables fine-grained authorization that follows workloads across clusters
4. L4 alone isn’t enough: Authorization policies (JWT validation, L7 controls) layer on top of mTLS for complete Zero Trust
5. Observability enables confidence: You can’t secure what you can’t see—security metrics and traces are as essential as the policies themselves

Practical starting points:

• Start with STRICT mTLS (not PERMISSIVE)
• Enable automatic certificate rotation (≤ 24 hours)
• Use AuthorizationPolicies with explicit rules, not allow-all
• Monitor mTLS handshake success rate (should be 100%)
• Track authorization denials by principal

When service mesh isn’t the answer: For latency-critical paths, small deployments, or resource-constrained edge devices, consider alternatives like eBPF-based networking or application-level mTLS.

The journey, not the destination: Zero Trust isn’t a product you install—it’s a discipline you practice. Start with mTLS, layer in observability, then progressively add authorization policies. Each step reduces your attack surface and increases your confidence in the system’s security posture.

As we explored in the distributed tracing post, modern observability infrastructure already carries the context needed for security. W3C TraceContext propagation happens over the same network paths that carry mTLS certificates. The convergence of observability and security isn’t a future vision—it’s today’s service mesh.

References