Key Management Critical

The service uses two independent secrets: the AES encryption key (TOKENIZATION_KEY) and the API authentication key (API_KEY). Both must be treated as equally sensitive.

Generating secure keys

# AES-256 tokenization key (base64-encoded, 32 bytes)
export TOKENIZATION_KEY=$(python3 -c \
  "import os,base64; print(base64.b64encode(os.urandom(32)).decode())")

# API authentication key (hex, 32 bytes)
export API_KEY=$(openssl rand -hex 32)

# Store them in AWS Secrets Manager
aws secretsmanager create-secret \
  --name "tokenization/prod/tokenization-key" \
  --secret-string "$TOKENIZATION_KEY"

aws secretsmanager create-secret \
  --name "tokenization/prod/api-key" \
  --secret-string "$API_KEY"

Injecting keys at runtime

TOK_KEY=$(aws secretsmanager get-secret-value \
  --secret-id tokenization/prod/tokenization-key \
  --query SecretString --output text)

API_KEY=$(aws secretsmanager get-secret-value \
  --secret-id tokenization/prod/api-key \
  --query SecretString --output text)

docker run -d \
  --name tokenizer \
  -p 8000:8000 \
  -e TOKENIZATION_KEY="$TOK_KEY" \
  -e API_KEY="$API_KEY" \
  --read-only \
  --security-opt no-new-privileges \
  agiletrust/tokenization:0.2

Network Isolation Critical

The tokenization service should never be reachable from the public internet. It is an internal cryptographic primitive — treat it like a database.

Kubernetes NetworkPolicy example

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tokenizer-ingress
  namespace: tokenization
spec:
  podSelector:
    matchLabels:
      app: tokenizer
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: tokenization-client
      ports:
        - protocol: TCP
          port: 8000

Access Control High

• Principle of least privilege: only services that legitimately need to tokenize or detokenize data should have network access to the service.
• Separate tokenize/detokenize: if some services only need to tokenize (write path) but should never detokenize (read path), implement a network-level or application-level split.
• Audit callers: log the source IP and/or service identity for every tokenize and detokenize call. This is your primary data access audit trail.
• Rate limiting: apply rate limits per client IP or service account to prevent brute-force enumeration attacks.

Logging & Audit High

Structured log example

{
  "timestamp": "2026-03-30T14:23:01.842Z",
  "service": "tokenization",
  "endpoint": "/tokenize",
  "encoding": "numeric",
  "input_length": 10,
  "status": 200,
  "duration_ms": 1.4,
  "caller_ip": "10.0.1.45",
  "request_id": "req_7f3a9b2c"
}

Encoding Selection Medium

Choosing the correct encoding for each field is critical. A mismatch between tokenization and detokenization encoding silently returns wrong data.

Tweak Strategy Medium

Tweaks add field-level context without requiring additional keys. A consistent tweak strategy prevents cross-field token correlation attacks.

When to use tweaks

• Same value in multiple fields: a person's name appears in both first_name and emergency_contact_name. Use different tweaks so the tokens are distinct.
• Multi-tenant systems: use a tenant ID-derived tweak so the same plaintext produces different tokens per tenant.
• Temporal separation: use a date-derived tweak if tokens should be unlinkable across time periods.

Deriving consistent tweaks

def field_tweak(field_name: str) -> str:
    """Convert a field name to a deterministic 14-hex-char tweak."""
    raw = field_name.encode("ascii")[:7]   # take first 7 bytes
    padded = raw.ljust(7, b"\x00")          # zero-pad to 7 bytes
    return padded.hex()                     # → 14 hex chars

print(field_tweak("name"))    # 6e616d65000000
print(field_tweak("rut"))     # 72757400000000
print(field_tweak("email"))   # 656d61696c0000

Error Handling Medium

Handling 422 validation errors

Validation errors always include a descriptive message. Parse the error field to surface actionable feedback:

import requests

def safe_tokenize(plaintext, encoding="utf8"):
    try:
        r = requests.post("http://localhost:8000/tokenize",
                          json={"plaintext": plaintext, "encoding": encoding},
                          timeout=5)
        if r.status_code == 422:
            raise ValueError(f"Validation error: {r.json()['error']}")
        r.raise_for_status()
        return r.json()["token"]
    except requests.Timeout:
        raise RuntimeError("Tokenization service timeout — check health endpoint")
    except requests.ConnectionError:
        raise RuntimeError("Cannot reach tokenization service")

The silent detokenization failure

If you detokenize with the wrong encoding or key, you receive HTTP 200 with garbled plaintext. There is no exception to catch. If data integrity is critical, store a keyed HMAC of the original plaintext alongside the token and verify after detokenization.

import hmac, hashlib

HMAC_KEY = b"separate-integrity-key"   # not the tokenization key

def store_with_integrity(plaintext, token, encoding):
    mac = hmac.new(HMAC_KEY, plaintext.encode(), hashlib.sha256).hexdigest()
    return {"token": token, "encoding": encoding, "mac": mac}

def verified_detokenize(record):
    recovered = detokenize(record["token"], record["encoding"])
    expected_mac = hmac.new(HMAC_KEY, recovered.encode(), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected_mac, record["mac"]):
        raise ValueError("Detokenization integrity check failed — wrong key or encoding?")
    return recovered

Idempotency & Caching Low

Because tokenization is deterministic, the same plaintext + encoding + tweak always produces the same token. You can safely cache tokens or call /tokenize multiple times without side effects.

Key Rotation High

Key rotation requires re-tokenizing all stored tokens under the new key. This is an operational decision — plan for it before going to production.

1. Generate a new key and store it in your secrets manager alongside the old key. Keep the old key — you need it to detokenize existing data.
2. Deploy a second container instance (or use a separate endpoint) with the new key. Do not yet point production traffic at it.
3. Re-tokenize in batches: for each stored token, detokenize with the old container, then tokenize with the new container. Write the new token back atomically.
4. Verify a sample of re-tokenized records by detokenizing with the new key and comparing to the expected plaintext.
5. Cut over production traffic to the new container. Decommission the old key after a retention window.

High Availability Medium

The tokenization service is stateless — scale horizontally as needed.

• Run at least 2 replicas in production, on different availability zones.
• Use a health check on GET /health with a 2-second timeout and 3-failure threshold.
• Set resource limits: the service is CPU-light but allocate at least 128 MB RAM per replica.
• Set readinessProbe to /health so traffic is not routed until the container is ready.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tokenizer
  namespace: tokenization
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tokenizer
  template:
    metadata:
      labels:
        app: tokenizer
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
        - name: tokenizer
          image: agiletrust/tokenization:0.2
          ports:
            - containerPort: 8000
          env:
            - name: TOKENIZATION_KEY
              valueFrom:
                secretKeyRef:
                  name: tokenization-key
                  key: key
          resources:
            requests: { cpu: "100m", memory: "128Mi" }
            limits:   { cpu: "500m", memory: "256Mi" }
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 5
            periodSeconds: 5
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true

Monitoring Medium

Key metrics to track

Health check script

#!/bin/bash
STATUS=$(curl -sf http://localhost:8000/health | jq -r '.status' 2>/dev/null)
if [ "$STATUS" = "ok" ]; then
  echo "Tokenization service: healthy"
  exit 0
else
  echo "Tokenization service: UNHEALTHY (got: $STATUS)"
  exit 1
fi

PCI DSS Guidance

AgileTrust Tokenization implements Format-Preserving Encryption as defined by NIST SP 800-38G Rev 1. FPE is recognized by PCI SSC as a tokenization technology when deployed correctly.

Relevant PCI DSS controls

• Requirement 3.5: Primary Account Numbers (PANs) must be rendered unreadable wherever stored. Using numeric encoding tokenization satisfies this when the tokenization system is properly isolated.
• Requirement 3.6 / 3.7: Key management procedures must cover generation, distribution, storage, access, retirement, and destruction. Map your secrets manager workflows to these requirements.
• Requirement 7: Restrict access to tokenization services to only those with a business need.
• Requirement 10: Log all access to tokenized data (detokenize calls) with timestamps and caller identity.

Scope reduction

When PANs are tokenized before leaving the point of interaction and the tokenization service itself is isolated, systems that only store or process tokens (not PANs) may be eligible for a reduced PCI DSS scope. The tokenization service itself remains in scope.

GDPR & Privacy

Under GDPR, tokenized data is still personal data if the original can be recovered — which is the case here. However, pseudonymization (Art. 4(5) GDPR) is explicitly recognized as a risk-reduction measure that can reduce obligations for the pseudonymized data.

Key considerations

• Pseudonymization, not anonymization: tokens are reversible. Your DPA must document this distinction.
• Right to erasure: destroying the encryption key effectively anonymizes all tokens derived from it — a practical path to "erasure by key deletion."
• Data minimization: only tokenize fields that require reversibility. For fields that never need to be recovered, use one-way hashing instead.
• Data processing agreements: if the tokenization service runs in a third-party cloud, ensure appropriate DPAs are in place.
• Transfer restrictions: tokens reduce risk in cross-border transfers but do not eliminate the need for adequate transfer mechanisms (SCCs, adequacy decisions).

Production Deployment Checklist

Review each item before going live.

Key management

• AES-256 TOKENIZATION_KEY generated with CSPRNG
• API_KEY generated with CSPRNG (openssl rand -hex 32)
• Both keys stored in secrets manager (not in code or config files)
• Separate keys for dev / staging / production
• Key rotation procedure documented and tested
• Key access audit logging enabled in secrets manager

Network security

• Service deployed in private subnet with no public IP
• Security groups restrict access to authorized callers only
• Health endpoint not reachable from external networks
• TLS / mTLS enforced between callers and service

Container hardening

• Container runs as non-root user (UID 1000)
• readOnlyRootFilesystem: true set
• allowPrivilegeEscalation: false set
• Resource limits (CPU + memory) configured
• Image pulled from trusted registry, digest pinned

Operations

• Liveness and readiness probes configured on /health
• At least 2 replicas across availability zones
• Structured logging enabled (no PII in logs)
• Alerts configured for HTTP 500 rate and service unavailability
• Re-tokenization runbook documented for key rotation

Integration

• Encoding, tweak, and algorithm stored alongside each token
• Encoding selection reviewed for each field type
• Error handling implemented for 422 and 500 responses
• Detokenization integrity check in place for critical fields
• Token cache invalidation on key rotation