Overview

AgileTrust Tokenization replaces sensitive data with format-preserving tokens — values that look identical to the original (same length, same structure, same character set) but are cryptographically secure substitutions.

Unlike traditional masking or hashing, tokenization is fully reversible. Given the same key, the original value can be recovered at any time without storing a lookup table.

Key properties

• Format-preserving: a 16-digit credit card number tokenizes to another 16-digit number; a Chilean RUT like 13301430-6 tokenizes to 35240589-5.
• Deterministic: same input + same key + same encoding + same tweak always produces the same token.
• Reversible: /detokenize recovers the original value with 100% fidelity.
• Symbol-preserving: spaces, hyphens, @, ., (), and all non-alphanumeric characters stay at their exact positions.
• Schema-transparent: no database schema changes, no validation rule updates, no regex rewrites.

How It Works

Each tokenization request goes through three stages:

1. Normalize — The input is NFC-normalized (Unicode Normalization Form C) to ensure consistent byte representation across platforms and locales.
2. Split — The string is split into two lists: tokenizable characters (letters and digits from the chosen encoding's alphabet) and passthrough characters (everything else: spaces, hyphens, punctuation, symbols). Passthrough positions are recorded.
3. Encrypt — The tokenizable characters are encrypted using FF3-1/AES. For inputs shorter than FF3's minimum (6 for numeric, 3 for text modes), a keyed SHA-256 substitution is used instead. For very long inputs, the tokenizable characters are processed in segments. The passthrough characters are then re-inserted at their original positions.

Detokenization runs the exact same pipeline in reverse — the cipher operates identically in both directions.

Encoding Modes

The encoding parameter controls which characters are considered "tokenizable" and sets the encryption alphabet (radix).

Choosing the right mode

• Use numeric for any value composed primarily of digits — even if it contains separators like hyphens or parentheses (they are preserved, not encrypted).
• Use utf8 (default) for names, free-text fields, email addresses, and any string containing non-Latin characters.
• Use latin1 for Western European names when you specifically need tokens that stay within the Latin-1 code page (e.g., legacy systems that cannot handle characters above U+00FF).

Quick Start

1. Run the container

export TOKENIZATION_KEY="$(python3 -c 'import os,base64; print(base64.b64encode(os.urandom(32)).decode())')"
export API_KEY="$(openssl rand -hex 24)"

docker run -d \
  --name tokenizer \
  -p 8000:8000 \
  -e TOKENIZATION_KEY="$TOKENIZATION_KEY" \
  -e API_KEY="$API_KEY" \
  agiletrust/tokenization:0.3

The container starts a FastAPI server on port 8000. TOKENIZATION_KEY must be a base64-encoded AES key — 16, 24, or 32 bytes (AES-128/192/256). API_KEY is the shared secret required in the X-API-Key header.

2. Tokenize a value

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "13301430-6", "encoding": "numeric"}' | jq .

{
  "token": "35240589-5",
  "algorithm": "FF3-1/AES",
  "encoding": "numeric"
}

3. Detokenize to recover the original

curl -s -X POST http://localhost:8000/detokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"token": "35240589-5", "encoding": "numeric"}' | jq .

{
  "plaintext": "13301430-6"
}

Authentication

All requests to /tokenize and /detokenize require an X-API-Key header containing the shared API key. The /health endpoint is open (required for liveness probes).

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "13301430-6", "encoding": "numeric"}' | jq .

In single-tenant mode, the API key is set via the API_KEY environment variable and is shared across all callers. The AES encryption key is set via TOKENIZATION_KEY. Neither key ever appears in logs or responses — clients never send key material in request bodies.

In multi-tenant mode (MULTI_TENANT=true), API keys are generated per application in the management console. Each key is stored as a SHA-256 hash and resolves its own application-specific AES key from the database on every request. See the Multi-Tenant & Console section for setup details.

GET /health

Returns service status and version. Use for liveness probes and uptime monitoring.

Response — 200 OK

{
  "status": "ok",
  "version": "0.3"
}

POST /tokenize

Converts a plaintext string into a format-preserving token.

Request body

Response — 200 OK

Response — 422 Unprocessable Entity

Returned for validation errors. The error field describes the issue:

POST /detokenize

Reverses a token produced by /tokenize, recovering the original plaintext.

Request body

Response — 200 OK

Numeric — RUT, Phone, Credit Card

Use "encoding": "numeric" for any digit-heavy identifier. Separators are preserved.

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "13301430-6", "encoding": "numeric"}' | jq .

{
  "token": "35240589-5",
  "algorithm": "FF3-1/AES",
  "encoding": "numeric"
}

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "(555) 123-4567", "encoding": "numeric"}' | jq .

{
  "token": "(555) 089-2341",
  "algorithm": "FF3-1/AES",
  "encoding": "numeric"
}

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "4532015112830366", "encoding": "numeric"}' | jq .

{
  "token": "7841392065194827",
  "algorithm": "FF3-1/AES",
  "encoding": "numeric"
}

Names and Free Text

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "Juan Pérez-García"}' | jq .

{
  "token": "Ñkrp Çmevq-Ĥapcw",
  "algorithm": "FF3-1/AES",
  "encoding": "utf8"
}

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "Juan Pérez-García", "encoding": "latin1"}' | jq .

{
  "token": "YrfãröxÍ-Ý",
  "algorithm": "FF3-1/AES",
  "encoding": "latin1"
}

import requests

BASE = "http://localhost:8000"
HEADERS = {"X-API-Key": "your-api-key-here"}

def tokenize(plaintext, encoding="utf8", tweak=None):
    payload = {"plaintext": plaintext, "encoding": encoding}
    if tweak:
        payload["tweak"] = tweak
    r = requests.post(f"{BASE}/tokenize", json=payload, headers=HEADERS)
    r.raise_for_status()
    return r.json()["token"]

def detokenize(token, encoding="utf8", tweak=None):
    payload = {"token": token, "encoding": encoding}
    if tweak:
        payload["tweak"] = tweak
    r = requests.post(f"{BASE}/detokenize", json=payload, headers=HEADERS)
    r.raise_for_status()
    return r.json()["plaintext"]

# Usage
token = tokenize("Juan Pérez-García")
print(token)          # Ñkrp Çmevq-Ĥapcw

original = detokenize(token)
print(original)       # Juan Pérez-García

# Numeric
rut_token = tokenize("13301430-6", encoding="numeric")
print(rut_token)      # 35240589-5

const BASE = 'http://localhost:8000';
const HEADERS = { 'Content-Type': 'application/json', 'X-API-Key': 'your-api-key-here' };

async function tokenize(plaintext, encoding = 'utf8', tweak = null) {
  const body = { plaintext, encoding };
  if (tweak) body.tweak = tweak;
  const res = await fetch(`${BASE}/tokenize`, {
    method: 'POST',
    headers: HEADERS,
    body: JSON.stringify(body),
  });
  if (!res.ok) throw new Error(await res.text());
  return (await res.json()).token;
}

async function detokenize(token, encoding = 'utf8', tweak = null) {
  const body = { token, encoding };
  if (tweak) body.tweak = tweak;
  const res = await fetch(`${BASE}/detokenize`, {
    method: 'POST',
    headers: HEADERS,
    body: JSON.stringify(body),
  });
  if (!res.ok) throw new Error(await res.text());
  return (await res.json()).plaintext;
}

// Usage
const token = await tokenize('Juan Pérez-García');
console.log(token);                        // Ñkrp Çmevq-Ĥapcw
console.log(await detokenize(token));      // Juan Pérez-García

Email Addresses

The @ symbol and . are always preserved, making tokenized emails syntactically valid in most validators.

curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "user@example.com", "encoding": "utf8"}' | jq .

{
  "token": "ŋĭĕĭ@ĩźĝŋĪĮĩ.ĕĝĬ",
  "algorithm": "FF3-1/AES",
  "encoding": "utf8"
}

Using Tweaks

A tweak is a 14-character hex string (7 bytes) that provides field-level context. The same plaintext tokenized with different tweaks produces completely different tokens, preventing an attacker from correlating tokens across fields.

# Tokenize as "name" field
curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "Maria Lopez", "tweak": "6e616d65000000"}' | jq .token

# Tokenize as "alias" field
curl -s -X POST http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key-here" \
  -d '{"plaintext": "Maria Lopez", "tweak": "616c696173000000"}' | jq .token

Limits & Constraints

• Empty strings are accepted and return an empty token.
• Input is NFC-normalized before length is checked. Pre-normalize if exact limits matter.
• Tweak must be exactly 14 hexadecimal characters (case-insensitive). Defaults to 00000000000000.

Error Responses

All errors return JSON with a single error field:

{ "error": "plaintext: Value error, 'plaintext' exceeds maximum length of 16 characters." }

FAQ

Can two different plaintexts produce the same token?

No. FF3-1 is a pseudorandom permutation (PRP), so tokenization is a bijection — every plaintext maps to a unique token and vice versa, for a given key + tweak + encoding.

Does the token length always equal the input length?

Yes — the token length equals the length of the NFC-normalized input. If your input is already NFC, the lengths are identical.

What happens if I tokenize the same value twice?

You get the same token. Tokenization is deterministic. If you need different tokens for the same value in different contexts, use distinct tweaks.

Can I use the numeric encoding for credit card numbers?

Yes. A 16-digit PAN like 4532015112830366 will tokenize to another 16-digit number. Note that the tokenized number will not pass a Luhn check — that is expected and intentional.

How do I know if detokenization succeeded?

There is no cryptographic indicator. If you need to verify correctness, store a hash of the original plaintext and compare it after detokenization.

Is the service stateless?

The tokenization logic itself is stateless — no lookup table is stored and the original value is mathematically derived from the token + key + tweak + encoding. In multi-tenant mode, the service connects to PostgreSQL to resolve API keys and application configurations, but each request remains independent with no session state.

Single-Tenant Mode

The default operating mode. A single AES key is loaded at startup and shared across all requests. Suitable for a single application or when you manage isolation at the infrastructure level.

Required environment variables

export TOKENIZATION_KEY="$(python3 -c 'import os,base64; print(base64.b64encode(os.urandom(32)).decode())')"
export API_KEY="$(openssl rand -hex 24)"

docker run -d -p 8000:8000 \
  -e TOKENIZATION_KEY="$TOKENIZATION_KEY" \
  -e API_KEY="$API_KEY" \
  tokenizer:0.3

Multi-Tenant Mode & Management Console

When MULTI_TENANT=true, the tokenizer resolves keys from PostgreSQL on every request. Each application has its own AES key, configured via the AgileTrust Console — a Next.js admin UI with SSO authentication.

Architecture

• Console (Next.js, port 3000): create tenants, applications, and API keys. Sign in with Google, Microsoft, or Okta.
• Tokenizer (FastAPI, port 8000): resolves the API key from the DB, decrypts the application's provider_config, caches the key for KEY_CACHE_TTL seconds.
• PostgreSQL: shared database. The console has read-write access; the tokenizer uses a read-only connection.

Multi-tenant environment variables (tokenizer)

Quick start (Docker Compose)

export POSTGRES_PASSWORD="$(openssl rand -hex 16)"
export NEXTAUTH_SECRET="$(openssl rand -base64 32)"
export MASTER_ENCRYPTION_KEY="$(python3 -c 'import os,base64; print(base64.b64encode(os.urandom(32)).decode())')"
export GOOGLE_CLIENT_ID="your-client-id"
export GOOGLE_CLIENT_SECRET="your-client-secret"

# Run Prisma migration (first time only)
cd console && npm install && npx prisma migrate deploy && cd ..

# Start all 3 services
docker compose -f docker-compose.full.yml up -d

First login flow

1. Open http://localhost:3000 and sign in with your OAuth provider.
2. The first user for your email domain becomes owner of a new tenant. Subsequent users from the same domain are added as viewers.
3. Go to Applications → New Application. Choose a key provider (System Vault is the default — no external vault needed).
4. Open the application and click New Key. Copy the API key — it is shown only once.
5. Use the API key in the X-API-Key header when calling the tokenizer.

API key rotation vs. encryption key rotation

• API key rotation: generates a new tok_… key for the same application. The encryption key and all tokens remain valid.
• Encryption key rotation (System Vault only): generates a new AES key. All existing tokens become non-recoverable — FPE with the wrong key returns a plausible-looking but incorrect value without any error signal.

Key Providers

In single-tenant mode, select the provider with the KEY_PROVIDER env var on the tokenizer. In multi-tenant mode, each application selects its own provider in the console — no env vars needed on the tokenizer.

env — Environment variable (default)

The AES key is supplied directly as a base64-encoded env var. Suitable for single-tenant deployments or local development.

system_vault — Database-stored key (multi-tenant default)

The console generates a random AES-256 key at application creation time, encrypts it with MASTER_ENCRYPTION_KEY using AES-256-GCM, and stores the ciphertext in the applications.provider_config column. No external vault or cloud credentials are required.

This is the recommended starting point for most deployments. The key never leaves the database unencrypted.

aws_secretsmanager — AWS Secrets Manager

The tokenizer calls GetSecretValue at startup (single-tenant) or on cache miss (multi-tenant) to retrieve the base64-encoded AES key stored as a plaintext secret.

KEY=$(python3 -c 'import os,base64; print(base64.b64encode(os.urandom(32)).decode())')
aws secretsmanager create-secret \
  --name agiletrust/tokenization-key \
  --secret-string "$KEY" \
  --region us-east-1

aws_kms — AWS KMS

The AES key is encrypted with an AWS KMS Customer Managed Key (CMK) and stored as a base64-encoded ciphertext blob. The tokenizer calls kms:Decrypt to recover the plaintext key at runtime.

KEY=$(python3 -c 'import os,base64; print(base64.b64encode(os.urandom(32)).decode())')
CIPHERTEXT=$(aws kms encrypt \
  --key-id alias/my-cmk \
  --plaintext "$KEY" \
  --query CiphertextBlob \
  --output text \
  --region us-east-1)
# Store CIPHERTEXT as AWS_KMS_KEY_REF

azure_keyvault — Azure Key Vault

The AES key is stored as a secret in Azure Key Vault. Authentication uses azure-identity's DefaultAzureCredential, which supports managed identity, service principal, and developer credentials automatically.

gcp_secretmanager — GCP Secret Manager

The AES key is stored as a secret version in GCP Secret Manager. Authentication uses Application Default Credentials (ADC) — the service account running the container is granted access.

oci_vault — OCI Vault

The AES key is stored as a secret in Oracle Cloud Infrastructure Vault. Authentication uses the OCI SDK config file or instance principal when running on OCI Compute.

Console Environment Variables

The management console is a Next.js application configured entirely through environment variables. Set these in a .env file at the root of console/, or pass them directly to Docker.

* At least one OAuth provider must be configured.

SSO / Identity Providers

The console authenticates users via OAuth 2.0 / OIDC through NextAuth.js v5. Configure at least one of the four supported providers below. All enabled providers appear on the sign-in page simultaneously.

Google

1. Open Google Cloud Console → APIs & Services → Credentials.
2. Click Create Credentials → OAuth 2.0 Client ID.
3. Set application type to Web application.
4. Under Authorized redirect URIs, add:
   {NEXTAUTH_URL}/api/auth/callback/google
5. Copy the Client ID and Client Secret into your env vars.

Microsoft Entra ID (Azure AD)

1. Open Azure portal → Microsoft Entra ID → App registrations.
2. Click New registration. Set a name; under Supported account types choose your desired scope.
3. Under Redirect URI, select Web and enter:
   {NEXTAUTH_URL}/api/auth/callback/azure-ad
4. After creation, note the Application (client) ID and Directory (tenant) ID.
5. Go to Certificates & secrets → New client secret. Copy the secret value (not the ID).

Okta

1. Open Okta Admin Console → Applications → Create App Integration.
2. Select OIDC – OpenID Connect as the sign-in method, then Web Application as the application type.
3. Under Sign-in redirect URIs, add:
   {NEXTAUTH_URL}/api/auth/callback/okta
4. Copy the Client ID and Client Secret from the application settings.
5. The Issuer URL is shown in Security → API → Authorization Servers. Default: https://your-org.okta.com/oauth2/default.

Keycloak

1. Open your Keycloak Admin Console → select your realm → Clients → Create client.
2. Set Client type to OpenID Connect and enter a Client ID.
3. Enable Client authentication (confidential access type).
4. Under Valid redirect URIs, add:
   {NEXTAUTH_URL}/api/auth/callback/keycloak
5. Save. Go to the Credentials tab and copy the Client Secret.
6. The issuer URL is your realm URL: https://<host>/realms/<realm-name>.

Roles & Permissions

The console enforces a three-level role hierarchy. A user with a higher-ranked role inherits all permissions of lower-ranked roles.

• The first user to sign in for a given email domain becomes owner.
• Subsequent users from the same domain join as viewer.
• An owner can promote a viewer to admin, promote an admin to owner, or demote any user.
• An owner cannot remove or demote themselves if they are the last owner of the tenant.

Managing Applications

An application represents a logical service that will call the tokenizer API. Each application has its own AES encryption key and generates its own API keys. Requires admin or owner role.

Create an application

1. Navigate to Applications in the sidebar.
2. Click New Application.
3. Enter a Name (required) and optional Description.
4. Select a Key Provider. system_vault is the default — the console auto-generates and stores the AES key; no external vault credentials are needed.
5. For cloud providers, fill in the required config fields (e.g., secret_id and region for AWS Secrets Manager).
6. Click Create. The application is created and its key is provisioned immediately.

Key provider config fields (per provider)

Edit or delete an application

Open the application's detail page and use the Edit or Delete buttons. Deleting an application immediately revokes all its API keys — any ongoing requests using those keys will receive 403 Forbidden.

Managing API Keys

API keys authenticate calls to the tokenizer. Each key is scoped to a single application and resolves that application's AES key. Requires admin or owner role.

Create a key

1. Open an application's detail page.
2. Click New Key.
3. Optionally enter a Label (e.g., production, staging) and an Expiry date.
4. Click Generate. The full key is displayed exactly once — copy it immediately.

Key format: tok_{tenant_slug}_{32 random chars}
Example: tok_acmecorp_K7mP2xQnRvW9jLdF8bTsZeYhAcGiNuXo

Revoke a key

Click Revoke next to the key. The key stops working immediately — the tokenizer rejects it with 403 Forbidden on the next request.

Rotate a key

Click Rotate. This atomically generates a new key and revokes the old one. The new plaintext is shown once. Use rotation to replace a key that may have been exposed without disrupting service between generation and deployment.

Managing Users

Users join automatically — anyone who signs in with an email address belonging to the tenant's domain is added as a viewer. No invitation step is required.

View users

Navigate to Users in the sidebar to see all users, their roles, last login time, and OAuth provider.

Change a user's role

Click the role badge next to a user to open the role selector. Requires owner. Available transitions:

• viewer → admin or owner
• admin → viewer or owner
• owner → admin or viewer (only if at least one other owner exists)

Remove a user

Click Remove next to the user. Requires owner. The user's session is invalidated and they cannot sign in again unless they re-authenticate (they would then rejoin as viewer).

Audit Log

Every mutating action performed through the console is recorded in the audit log. Navigate to Audit Log in the sidebar to view and paginate entries.

Each entry records: timestamp, actor (user name + email), IP address, action type, target resource, and an optional JSON details blob.