NoSQL Datastore Schema Management

Celerity makes NoSQL datastore schemas a first-class concern, bridging the gap between development and data teams. With SQL databases, data teams can introspect information_schema to discover what exists. With NoSQL databases (DynamoDB, Cosmos DB, Firestore), there is no database-level schema to introspect. The "schema" exists only in application code, scattered across handler functions and tribal knowledge, and data teams building pipelines on top of a NoSQL table are flying blind.

Celerity solves this by providing a declarative schema definition that serves as a single source of truth for the intended structure of your data, with tooling for validation, type generation, contracts, data-shape drift detection and exports.

For SQL database schema management, see the SQL Database Schema Management guide.

How It Works

Unlike SQL schema management, there are no migrations and no DDL. NoSQL databases are schemaless by design, so field-level changes do not require database operations. Instead, Celerity's schema management for NoSQL focuses on visibility, conformance, contracts and data team tooling. Validation is part of that, but the layer that carries the data-team guarantee is the one that observes every writer rather than only the ones going through a Celerity SDK: celerity schema drift, run on a schedule.

Schema YAML (desired state) ──► Schema Manager ──► Diff + Contracts ──► State Update
                                      ▲
                              Current state from
                              previous deployment

The schema manager computes the difference between the current schema (from the previous deployment) and the desired schema (from the YAML file). It does not connect to the database or execute any operations against it. All infrastructure changes (indexes, TTL, keys) are handled by the Bluelink deploy engine separately.

How NoSQL Differs from SQL

Bluelink / Celerity Split

The split between Bluelink and Celerity is different from SQL:

• Bluelink provider handles: Table creation, keys, secondary indexes, TTL config and capacity settings, the infrastructure concerns defined in the blueprint spec
• Celerity handles: Schema definition (the field-level YAML), SDK validation, data team contracts, type codegen, schema state tracking, exports

Schema Definition Format

Project File Structure

✅ Available in v0

A typical Celerity project with NoSQL schema management follows this structure:

my-app/
├── .celerity/                       # Generated only (merged blueprint, compose, logs)
├── app.bp                           # Main blueprint — references schema files (.bp, .yaml, or .jsonc)
├── app.deploy.jsonc                  # Deploy target config
├── config/
│   ├── local/                       # Plaintext app config for local development
│   └── test/                        # Plaintext app config for testing
├── secrets/
│   ├── local/                       # Secrets for local development
│   └── test/                        # Secrets for testing
├── seed/
│   ├── local/                       # Seed data for local development
│   └── test/                        # Seed data for testing
├── schema-contracts.yaml            # Data team dependency contracts (optional)
├── schemas/
│   └── user-store.yaml              # Schema for userStore resource
├── scripts/
│   └── user-store/                  # Escape hatch data scripts (per datastore)
│       ├── V001__backfill_tier_field.js
│       └── V002__migrate_legacy_roles.py
├── src/
│   └── ...
└── generated/                       # Optional: codegen output
    └── ...

The blueprint references schema files via the schemaPath field on celerity/datastore resources:

Blueprint Language

version "2025-11-02"
transform "celerity-2026-02-27-draft"

resource userStore: celerity/datastore {
    metadata {
        displayName = "User Store"
        labels = {
            application = "users"
        }
    }

    spec {
        name = "users"
        keys = {
            partitionKey = "id",
            sortKey = "createdAt"
        }
        schemaPath = "./schemas/user-store.yaml"
        scriptsPath = "./scripts/user-store"
        indexes = [
            { name = "emailIndex", fields = ["email"] },
            { name = "tierCreatedIndex", fields = ["tier", "createdAt"] }
        ]
        timeToLive = {
            fieldName = "expiresAt",
            enabled = true
        }
    }
}

YAML

version: 2025-11-02
transform: celerity-2026-02-27-draft

resources:
  userStore:
    type: "celerity/datastore"
    metadata:
      displayName: "User Store"
      labels:
        application: "users"
    spec:
      name: "users"
      keys:
        partitionKey: "id"
        sortKey: "createdAt"
      schemaPath: "./schemas/user-store.yaml"
      scriptsPath: "./scripts/user-store"
      indexes:
        - name: "emailIndex"
          fields: ["email"]
        - name: "tierCreatedIndex"
          fields: ["tier", "createdAt"]
      timeToLive:
        fieldName: "expiresAt"
        enabled: true

Note that keys, indexes and timeToLive remain in the blueprint spec. They are infrastructure concerns that the Bluelink provider needs to create the table and its secondary indexes on the deploy target (e.g. a DynamoDB table and GSIs on AWS). The schemaPath points to the field-level schema that describes the data, which is a Celerity concern.

What Goes Where

Schema YAML Format

✅ Available in v0

Schema files define the intended structure of items in a datastore using Celerity-native field types. One schema file per celerity/datastore resource.

# schemas/user-store.yaml
description: "User accounts and profiles. Partition key: id, sort key: createdAt."
owner: "platform-team"
tags: ["pii", "core-entity"]

required: ["id", "createdAt", "email", "name"]

fields:
  id:
    type: "string"
    description: "UUID. Partition key."
    classification: "internal-id"

  createdAt:
    type: "number"
    description: "Unix timestamp (ms). Sort key. Set at account creation, never updated."

  email:
    type: "string"
    description: "Primary email address. Unique per user (enforced in application)."
    classification: "pii"

  name:
    type: "string"
    description: "Display name."
    classification: "pii"

  tier:
    type: "string"
    description: "Subscription tier: free, pro, enterprise. Added 2025-03. Items before this date may not have this field."
    default: "free"
    tags: ["business-metric"]

  isActive:
    type: "boolean"
    description: "Whether the account is active. Inactive accounts are soft-deleted."
    nullable: true

  roles:
    type: "array"
    description: "Authorization roles assigned to the user."
    items:
      type: "string"
      description: "Role identifier (e.g., admin, editor, viewer)"

  profile:
    type: "object"
    description: "User profile details. Optional — may not exist for accounts created via API."
    nullable: true
    fields:
      bio:
        type: "string"
        nullable: true
        description: "Free-text biography"
        classification: "pii"
      avatarUrl:
        type: "string"
        nullable: true
        description: "URL to avatar image"
      preferences:
        type: "object"
        description: "User-configurable preferences"
        fields:
          theme:
            type: "string"
            description: "UI theme: light, dark, system"
            default: "system"
          notifications:
            type: "boolean"
            description: "Whether email notifications are enabled"
            default: true

  lastLogin:
    type: "number"
    nullable: true
    description: "Unix timestamp (ms) of last login. Null if user has never logged in."
    tags: ["engagement-metric"]

  expiresAt:
    type: "number"
    nullable: true
    description: "Unix timestamp (s) for TTL. Set for temporary/trial accounts."

Design choices:

• Field types are Celerity-native: string, number, boolean, object and array map naturally to DynamoDB, Cosmos DB and Firestore types. Unlike SQL schemas which use engine-native types (varchar, jsonb, etc.), NoSQL field types are portable across providers.
• required list at the top level: Required means "every item MUST have this field" from the application's perspective.
• nullable: A field can exist but be null. Different from "not required" (field may not exist at all).
• default: The value the application should write if none is provided. Not enforced by the database; used for codegen and SDK validation.
• Nested objects and arrays: Recursive structure supporting deeply nested documents.
• Data evolution notes in descriptions: e.g., "Added 2025-03. Items before this date may not have this field." This is a NoSQL reality: old items don't automatically get new fields.

Field Types

Rich Metadata

The following fields have no effect on validation. They exist for documentation, data governance and tooling:

These fields make the schema YAML self-documenting for data team consumption. They are included in schema exports and used by schema contracts for dependency tracking.

Multiple Entities (Single-Table Design)

✅ Available in v0

Storing multiple entity types in a single table is a common NoSQL pattern, particularly with DynamoDB. The flat schema format above describes a single item shape; for single-table designs, a schema file can instead declare a discriminator field and an entities map:

# schemas/app-store.yaml
description: "Application data. Single-table design with users and sessions."
owner: "platform-team"

discriminator: "entityType"          # Field whose value selects the entity

fields:                              # Shared fields, present on every entity (keys live here)
  pk:
    type: "string"
    description: "Partition key"
  sk:
    type: "string"
    description: "Sort key"
  entityType:
    type: "string"
    description: "Entity discriminator: user | session"

entities:
  user:
    description: "User account"
    required: ["pk", "sk", "entityType", "email"]
    fields:
      email:
        type: "string"
        classification: "pii"
      name:
        type: "string"
        nullable: true
        classification: "pii"

  session:
    description: "Login session"
    required: ["pk", "sk", "entityType", "expiresAt"]
    fields:
      expiresAt:
        type: "number"
        description: "Unix timestamp (s) for TTL"
      device:
        type: "string"
        nullable: true

• The flat format (top-level fields/required only) remains valid and is the common case; it is an implicit single entity
• Shared top-level fields apply to every entity; each entity adds its own fields and required list
• Type generation emits one item type per entity plus a discriminated union (AppStoreItem = UserItem | SessionItem)
• Contracts can scope a datastore dependency to specific entities
• Drift detection buckets sampled items by discriminator value and reports per entity, flagging items with unknown discriminator values

Entities are supported in external schema files (schemaPath) only; the inline schema field on the resource keeps the flat format.

Schema Evolution

NoSQL doesn't have migrations in the SQL sense. Field-level changes don't require database operations. Instead, schema management for NoSQL is about tracking evolution, validating conformance, and maintaining contracts.

What Changes and How

Escape Hatch Data Scripts

✅ Available in v0

Versioned scripts for data operations that cannot be expressed in the schema YAML. These live in the directory specified by scriptsPath on the celerity/datastore resource.

File Naming Convention

V<number>__<description>.<ext>

Examples:

• V001__backfill_tier_field.js
• V002__migrate_legacy_roles.py
• V003__cleanup_deprecated_fields.ts

Scripts can be written in any language, as long as they are executable and accept connection config via environment variables.

Execution Model

1. Scripts are tracked by name + content_hash in schema state
2. celerity schema diff shows pending scripts
3. Scripts are executed via celerity schema scripts run, which lists pending scripts, runs them with connection config injected as environment variables, and records each completion in schema state
4. Scripts executed out-of-band (e.g. through a batch system) can be recorded without running via celerity schema scripts run --mark-only

For local development: scripts run automatically during celerity dev run since local data is ephemeral.

Use Cases

• Backfilling new fields on existing items (UPDATE equivalent for NoSQL)
• Migrating data from one field structure to another
• Cleaning up deprecated fields
• Transforming field values (e.g., string-to-number conversion on existing items)

Evolving a Schema Safely

The core NoSQL evolution workflow is tightening a schema over data that predates it. The safe sequence for adding a required field:

Step 1: Add the field as optional. Update the schema YAML with the new field (not in required, or nullable: true). Deploy. New writes can include the field; old items are unaffected, and nothing enforces its presence yet.

Step 2: Backfill existing items. Write a data script (e.g. V004__backfill_tier.js) that scans and updates items missing the field, then run it:

celerity schema scripts run userStore --env production

Step 3: Verify with a drift scan. Confirm the backfill reached everything before tightening:

celerity schema drift userStore --env production --sample 10000

The per-field conformance report shows whether any sampled items still lack the field.

Step 4: Make the field required. Update required in the schema YAML and deploy. Nothing rejects a write for non-conformance, so stragglers do not cause production failures. They are reported by drift scans. Take this step once the drift scan reports full conformance, so the reports are actionable rather than noisy, and remember that regenerating types makes the field non-optional in first-party code from that point on.

Schema Validation Strategy

✅ Available in v0

Since the database doesn't enforce schema, Celerity provides a layered validation approach that catches issues at multiple stages.

Application-Level Validation (Opt-In)

Generated code carries the schema's checks, so an application can validate an item wherever it makes sense to, whether that be after decoding a queue message, before a write, at the edge of a module. Nothing validates implicitly on the way to the data store.

What the generated checks cover:

• Type mismatches: a number where the schema expects a string
• Missing required fields: a field absent from the required list
• Unexpected nulls: null in a field not marked nullable
• Unknown fields: a field the schema does not declare
• Nested structure: nested objects and arrays, recursively

What validation reports

Each language exposes validation the way that language already does it:

The violation shape is shared, not the call. There is no common validator interface, because nothing generic ever holds a validator: the SDK does not validate writes, and conformance checkers carries the schema rather than calling generated code. What has to agree across languages is the data that comes out:

type SchemaViolation = {
  /** The field, outermost first. Empty for a problem with the item as a whole. */
  path: string[];
  /** A stable, lowercase identifier: invalid_type, missing_required, unexpected_null, unknown_field. */
  code: string;
  /** What is wrong, in terms of the field. */
  message: string;
};

The shape matters in two places. Violations are comparable across a polyglot application, so a report from a Go handler and one from a Python handler read the same. It is also what the conformance checkers compare against: the checker and the generated validators are separate implementations of the same rules, and a shared violation shape is what lets celerity schema validate prove they agree.

Two properties hold everywhere, a validator returns violations rather than raising, so the caller decides what is fatal, and it reports every violation rather than the first, so one call produces one complete report. Pydantic's construction-time raise is the documented exception.

How the Schema Reaches Running Code

Generated code is the schema at runtime. There is no schema asset in the build artifact and nothing to load or parse at startup: celerity schema codegen compiles the schema's checks into the generated validator, so what ships is ordinary code in the application's own language.

Two things bind a schema to a deployment:

• The transformer embeds a schema hash into the datastore resource spec, so a schema change is a spec change: the deploy engine sees the diff and deployment state records exactly which schema version shipped.
• celerity schema validate --check-codegen in CI fails when generated code no longer matches the schema YAML, which is what makes the first point trustworthy. The hash describes the schema, and CI guarantees the shipped code agrees with it.

Code therefore always validates against the schema it shipped with: no runtime schema fetch, no asset to bundle, no YAML or JSON parsing at startup, and no version skew. celerity schema apply never changes the behaviour of running handlers; schema updates reach them on their next deployment. A malformed schema fails codegen, and so the build, rather than surfacing as a startup error in production.

Build-Time Validation (Generated Types)

Generated types from celerity schema codegen shift schema enforcement to compile/type-check time:

• TypeScript: Generated interfaces make the compiler catch type mismatches, missing required fields and incorrect nested structures, regardless of whether the developer uses the Celerity SDK or the native provider SDK. Because a type is erased at runtime, codegen also emits a validate... function for runtime checks for greater safety.
• Python: Generated Pydantic models provide runtime validation, IDE autocomplete and type checking via mypy/pyright.
• Go: Generated structs make the compiler catch type mismatches and unknown fields, with required and optional fields distinguished by value versus pointer. A generated Validate() method covers what the compiler cannot, such as items decoded from outside the program.

See Type Generation for details and examples.

CI Validation

celerity schema validate --check-codegen in CI ensures generated types are always in sync with the schema YAML. If someone changes the schema but doesn't regenerate types, validation fails.

Validation Layers

The rightmost column is the one to read first: every layer but the last two only ever sees first-party code. Drift detection is the one available today that observes writes Celerity did not make, and it is what a data team would depend on.

Drift Detection

✅ Available in v0

Since the database doesn't enforce schema, the actual shape of stored data can silently diverge from the schema YAML: items written before a field existed, other services writing to the same table, or writes bypassing validation entirely. For NoSQL, this data-shape drift is the central risk, and it's what keeps the "single source of truth" claim honest.

celerity schema drift <datastore> --env <env> performs a point-in-time sampled scan of the live datastore (default 1,000 items, configurable with --sample) and reports drift in both directions:

1. Conformance drift (schema → data): sampled items that don't conform to the schema, with per-field conformance percentages: missing required fields, type mismatches, unexpected nulls
2. Unknown-field drift (data → schema): fields present in sampled items but absent from the schema YAML, the undocumented shape changes introduced by other writers

When entities are defined, items are bucketed by discriminator value and reported per entity; items with an unknown discriminator value are flagged.

$ celerity schema drift userStore --env production --sample 10000

userStore (datastore: users) — sampled 10,000 of ~1.2M items:

  Conformance drift:
    tier         87.3%  ⚠ 12.7% of items missing (required field; likely pre-2025-03 items)
    roles        78.4%  ⚠ 21.6% missing (field added later)
    tier types   99.8%  ⚠ 0.2% number instead of string (12 items; legacy tier IDs)

  Unknown fields (in data, not in schema):
    ~ legacySource — present on 4.1% of sampled items

  Contracts affected:
    ⛔ user-analytics-pipeline — userStore shape drifted (blocking)

  Resolution:
    - Backfill missing fields: create a data script, then celerity schema scripts run
    - Adopt unknown fields: add them to schemas/user-store.yaml
    - Legacy type mismatches: transform via a data script, then re-scan

Drift output is contract-aware: drifted datastores are matched against schema-contracts.yaml so the report shows which downstream consumers are affected. The command exits non-zero when drift is found, so a scheduled CI job gives continuous shape tracking.

Because this is a sampled read-only scan, it doesn't need runtime plumbing. It observes what's actually in the datastore regardless of which SDK or service wrote it. Run on a schedule, this is the v0 mechanism for tracking data shape over time, and the non-zero exit code is what turns it into a notification: a failing scheduled job reaches people through whatever CI already uses. Continuous conformance checking would add lower latency and full coverage of changes, without replacing the scan, since only a scan sees items that nothing has touched since a schema changed.

Scheduled Drift Monitoring

Continuous Conformance Checking

Instead of scanning the table, a conformance checker consumes the datastore's change stream, validates each changed item against the schema, and reports violations attributed to the owners of any schema contracts covering the datastore.

Item written (by anything) ──► change stream ──► conformance checker ──► violations, routed

What it adds, and what it costs

Both approaches are writer-agnostic and both are contract-aware, with the following differences:

What separates them is which problems each one is positioned to see. A systematic problem, such as a writer that started adding an undeclared field or writing the wrong type, appears on most items, and a 1,000-item sample finds it with near-certainty. A rare non-conforming shape, one code path in a hundred or a batch import that went wrong for a subset, is what a sample is unlikely to land on, and checking every change as it is written is what catches it. Latency separates them the same way, a scan reports at whatever interval it runs on, a stream checker will within seconds to minutes of the write that caused the violation.

The difference also runs the other way. A stream checker sees only what changes, so items written before a schema tightened and never touched since are invisible to it, and those are exactly what a scan is for. The two are complements: a datastore watched continuously still wants an occasional scan.

Why it belongs with the paid service

Its output is only actionable once violations reach the team that declared the dependency, and that routing is the same paid capability. A free version emitting metrics into the application owner's monitoring would put the signal in front of the wrong team: the data team that depends on the datastore is usually in a different org boundary, which is the gap contracts exist to bridge. The scheduled scan sidesteps that by failing a CI job, which already reaches anyone with access to the CI project.

Where the checker would run

A consumer wired to the datastore's change stream, running a first-party Celerity component rather than generated code. It is the one place the normalised schema ships as an asset.

That is a deliberate split from the generated validators:

• Handlers do not carry the schema, because a cold start in a FaaS deploy target should not need to parse one.
• The checker carries the schema, because reading it once at the start of a long-lived consumer costs nothing, and because it means the checker is written once rather than once per language.

The cost of that split is two implementations of the same rules, the generated validators and the checker, with celerity schema validate failing when they disagree.

Deploy Pipeline Integration

✅ Available in v0

Schema management is integrated into the celerity deploy pipeline:

celerity deploy
  │
  ├─ Phase 1: Infrastructure
  │   Transformer converts celerity/datastore → provider-specific table (e.g. AWS DynamoDB)
  │   Provider creates/updates table, secondary indexes, TTL
  │   Stabilise (table + all indexes active)
  │
  ├─ Phase 2: Schema State Update
  │   Read desired schema from YAML
  │   Read current schema from previous deployment state
  │   Compute diff, update schema state
  │   Log any pending escape hatch scripts that should be run
  │   (No database connection — field changes don't need one)
  │
  └─ Phase 3: Application
      Deploy handler resources with datastore connection configuration
      Build artifacts include the normalised schema (hash tracked in the
      resource spec); handlers start with SDK validation in the
      configured mode (warn by default)

Local Development

✅ Available in v0

Running Locally

When celerity dev run starts an application with datastore resources:

1. A local emulator is started based on the deploy target (DynamoDB Local for AWS targets)
2. Tables are created with keys and indexes from the blueprint spec
3. Escape hatch data scripts are executed automatically (local data is ephemeral)
4. Connection environment variables are injected, pointing to the local emulator
5. The runtime starts with SDK schema validation in the configured mode (warn by default)

Testing

When celerity dev test runs an application with datastore resources:

1. An isolated emulator instance is created per test suite
2. Tables are created with keys and indexes
3. Test fixtures are loaded
4. Tests run against the fully-configured datastore
5. The instance is torn down after tests complete

CLI Commands

✅ Available in v0

The celerity schema command group works for both SQL databases and NoSQL datastores. For full command reference including all flags and configuration options, see the CLI Reference: schema.

Example: celerity schema diff

The diff output for NoSQL datastores shows field-level changes and data conformance warnings. Unlike the SQL diff, no migration file is generated, since field changes do not require database operations.

$ celerity schema diff

userStore (datastore: users):
  Schema changes:
    [+] Field 'tier' added (string, default: "free")
        ⚠ Existing items will NOT have this field. Consider a backfill script.
    [~] Field 'roles' changed: was nullable, now required
        ⚠ Existing items with null 'roles' will not conform. Consider a backfill script.
    [-] Field 'legacyId' removed from schema
        ⚠ Existing items may still contain this field.

  Infrastructure changes (handled by Bluelink):
    [+] Index 'tierCreatedIndex' added (fields: [tier, createdAt])
        This will be created as a secondary index by the deploy target's provider.

  Data scripts (pending):
    V003__cleanup_deprecated_fields.js (not yet run)

  Contracts:
    ⚠ user-reporting — userStore schema changed (notify)

  Apply with: celerity schema apply

Type Generation

✅ Available in v0 (TypeScript, Python, Go)

Generate types from the schema YAML: TypeScript interfaces, Python Pydantic models or Go structs representing datastore items, plus field name and index constants. No ORM coupling. Combine with whatever library you prefer.

Each language gets the design that language would have chosen, not a shoehorning of another's: optionality is ? in TypeScript, Optional[...] in Python and a pointer in Go; defaults are a Pydantic field default, an exported constant and a constructor respectively. What is identical across all three is the item shape on the wire, so an item written by one language's generated types reads correctly through another's.

C# follows its language runtime in v1. Languages beyond it are considered on a case by case basis.

For SDK usage examples with generated types, see the Node.js SDK - Datastore, Python SDK - Datastore and Go SDK - Datastore documentation. For the operations the generated types are used with, see Datastore SDK Operations.

TypeScript

celerity schema codegen --lang typescript --out ./src/generated/

Generated output:

// generated/user-store.ts — auto-generated, do not edit

/** User store item */
export type UserStoreItem = {
  /** UUID. Partition key. */
  id: string;
  /** Unix timestamp (ms). Sort key. */
  createdAt: number;
  /** Primary email address. */
  email: string;
  /** Display name. */
  name: string;
  /** Subscription tier: free, pro, enterprise. */
  tier?: string;
  /** Whether the account is active. */
  isActive?: boolean | null;
  /** Authorization roles. */
  roles?: string[];
  /** User profile details. */
  profile?: UserStoreProfile | null;
  /** Unix timestamp (ms) of last login. */
  lastLogin?: number | null;
  /** Unix timestamp (s) for TTL. */
  expiresAt?: number | null;
}

export type UserStoreProfile = {
  bio?: string | null;
  avatarUrl?: string | null;
  preferences?: UserStoreProfilePreferences;
}

export type UserStoreProfilePreferences = {
  theme?: string;
  notifications?: boolean;
}

/** Field name constants */
export const UserStoreFields = {
  id: "id",
  createdAt: "createdAt",
  email: "email",
  name: "name",
  tier: "tier",
  isActive: "isActive",
  roles: "roles",
  profile: "profile",
  lastLogin: "lastLogin",
  expiresAt: "expiresAt",
} as const;

/** Index names for query operations */
export const UserStoreIndexes = {
  emailIndex: "emailIndex",
  tierCreatedIndex: "tierCreatedIndex",
} as const;

/** Key schema */
export const UserStoreKeys = {
  partitionKey: "id",
  sortKey: "createdAt",
} as const;

/**
 * Validates a value against the user store schema, returning one violation per
 * problem found. An empty array means the value conforms.
 */
export function validateUserStoreItem(value: unknown): SchemaViolation[] { ... }

A TypeScript type disappears at runtime, so the generated interface alone cannot check anything once the code is running. That is why codegen also emits validateUserStoreItem. It is a plain function with no dependencies: the schema's checks are compiled into it, so it needs neither a validation library nor the schema itself. Python gets the equivalent from Pydantic's own model validation and Go from its generated Validate() method.

It takes unknown rather than UserStoreItem, because the values worth validating are the ones the compiler could not catch, such as an item decoded from a queue message or read back from the store.

Codegen also emits a typed accessor, which is how the SDK finds the validator for a given datastore:

export function userStore(client: DatastoreClient): TypedDatastore<UserStoreItem> { ... }

client.datastore("user-store") still works and is untyped. userStore(client) binds the datastore name to its item type, so reads and writes are typed and the name appears once, in generated code, rather than as a string in every handler that touches it. Python and Go generate the same accessor. It is about typing, not validation. Validation is a separate call an application makes where it wants it.

Note the generated types reflect NoSQL realities:

• Non-required fields are optional (?), since items may not have them
• Nullable fields are | null, since items may have the field set to null
• Required fields are always present
• Index names and key schema are exported as constants for use with query operations

Python

celerity schema codegen --lang python --out ./src/generated/

Generated output:

# generated/user_store.py — auto-generated, do not edit
from pydantic import BaseModel, Field
from typing import Optional

class UserStoreProfilePreferences(BaseModel):
    theme: Optional[str] = "system"
    notifications: Optional[bool] = True

class UserStoreProfile(BaseModel):
    bio: Optional[str] = None
    avatar_url: Optional[str] = None
    preferences: Optional[UserStoreProfilePreferences] = None

class UserStoreItem(BaseModel):
    # Required fields
    id: str
    created_at: float
    email: str
    name: str
    # Optional fields
    tier: Optional[str] = "free"
    is_active: Optional[bool] = None
    roles: Optional[list[str]] = None
    profile: Optional[UserStoreProfile] = None
    last_login: Optional[float] = None
    expires_at: Optional[float] = None

Go

✅ Available in v0

celerity schema codegen --lang go --out ./internal/generated/

Generated output:

// Code generated by celerity schema codegen. DO NOT EDIT.

package generated

import "github.com/newstack-cloud/celerity-go-sdk/resources"

// UserStoreItem is an item in the user store.
type UserStoreItem struct {
	// ID is a UUID. Partition key.
	ID string `json:"id"`
	// CreatedAt is a Unix timestamp (ms). Sort key.
	CreatedAt float64 `json:"createdAt"`
	// Email is the primary email address.
	Email string `json:"email"`
	// Name is the display name.
	Name string `json:"name"`
	// Tier is the subscription tier: free, pro, enterprise.
	Tier *string `json:"tier,omitempty"`
	// IsActive reports whether the account is active.
	IsActive *bool `json:"isActive,omitempty"`
	// Roles are the authorization roles assigned to the user.
	Roles []string `json:"roles,omitempty"`
	// Profile holds the user's profile details.
	Profile *UserStoreProfile `json:"profile,omitempty"`
	// LastLogin is a Unix timestamp (ms). Nil when the user has never logged in.
	LastLogin *float64 `json:"lastLogin,omitempty"`
	// ExpiresAt is a Unix timestamp (s) for TTL.
	ExpiresAt *float64 `json:"expiresAt,omitempty"`

	// Set by a read, for a write that must require the item has not changed.
	// Unexported and untagged: it is the SDK's, and never part of the item.
	rev datastore.Revision
}

type UserStoreProfile struct {
	Bio         *string                      `json:"bio,omitempty"`
	AvatarURL   *string                      `json:"avatarUrl,omitempty"`
	Preferences *UserStoreProfilePreferences `json:"preferences,omitempty"`
}

type UserStoreProfilePreferences struct {
	Theme         *string `json:"theme,omitempty"`
	Notifications *bool   `json:"notifications,omitempty"`
}

// Field names, for conditions and projections.
const (
	UserStoreFieldID        = "id"
	UserStoreFieldCreatedAt = "createdAt"
	UserStoreFieldEmail     = "email"
	UserStoreFieldName      = "name"
	UserStoreFieldTier      = "tier"
	UserStoreFieldIsActive  = "isActive"
	UserStoreFieldRoles     = "roles"
	UserStoreFieldProfile   = "profile"
	UserStoreFieldLastLogin = "lastLogin"
	UserStoreFieldExpiresAt = "expiresAt"
)

// Index names, for Query.Index.
const (
	UserStoreIndexEmail       = "emailIndex"
	UserStoreIndexTierCreated = "tierCreatedIndex"
)

// Key schema.
const (
	UserStorePartitionKey = "id"
	UserStoreSortKey      = "createdAt"
)

// NewUserStoreItem returns an item with the schema's defaults applied.
func NewUserStoreItem() UserStoreItem { ... }

// Key returns the item's key, so a handler never assembles one by hand.
func (i UserStoreItem) Key() resources.Key { ... }

// Validate reports every way the item does not conform to the schema, for
// handler code that wants to check an item itself. The returned error reports
// its violations through resources.SchemaViolations.
func (i UserStoreItem) Validate() error { ... }

// ValidateUserStoreItem checks a value that is not known to be an item, such
// as a decoded map, and reports every violation it finds.
func ValidateUserStoreItem(value any) []resources.SchemaViolation { ... }

// SetRevision and Revision carry the revision of the item a read produced, so
// that items found by a query can be written back without reading each again.
func (i *UserStoreItem) SetRevision(r datastore.Revision) { ... }
func (i UserStoreItem) Revision() datastore.Revision      { ... }

// UserStore binds the datastore name, the item type and the validator.
func UserStore(host resources.Host) resources.TypedDatastore[UserStoreItem] { ... }

Go's generation differs from TypeScript's and Python's in six places, each because of something about Go rather than about the schema:

Optionality is a pointer, not a ?. Go has no optional-field syntax, so a field that is not in required generates as a pointer with omitempty: nil means the field is absent from the item. A required field generates as a value type and is always written. Pointers rather than bare values matter here, because omitempty on a string would silently drop a deliberately-written empty string, whereas *string distinguishes "not set" from "set to empty".

Defaults are a constructor, not a field initialiser. Python gets defaults free from Pydantic and TypeScript exports them as constants, because a TypeScript type cannot carry values. Go has the same limitation but it has constructors, so NewUserStoreItem() returns an item with the item type's own defaults applied, and each nested object type gets its own constructor. Optional nested objects are left nil rather than materialised, as Defaults explains.

Validation is an explicit method. Pydantic validates on model construction, which has no Go equivalent and would be surprising if it did, so Validate() error is explicit, for handler code that wants to check an item before doing anything with it. ValidateUserStoreItem handles values that are not the item type, such as a decoded map, and returns violations rather than an error so a caller can decide what is fatal. Both call the same generated checks.

A generated Key() method. Go's datastore operations take a resources.Key rather than a map, so the partition and sort key are ordinary struct fields on the item. Generating Key() means the key is derived from the item, and a handler cannot pass a key whose fields disagree with the item it is writing.

The revision is a field on the item. Every read returns a revision, and one read returns many items. A TypeScript or Python SDK can hand back a result object per item; a Go destination is a slice of the application's own type, with nowhere to put a second value per element. So the generated type implements the SDK's revision interface and carries the value in an unexported field, which is also why that field has no tag: it is the SDK's, not part of the item, and a read strips it from the stored attributes either way. A hand-written item type implements the same two methods where it needs them.

Numbers are float64. The schema's number covers integers and floats, so float64 is the faithful mapping and matches Python's float and TypeScript's number. Integer-typed fields would need an integer type in the schema format first; without one, generating int64 would be a guess that silently truncates.

C#

🚀 Planned for v1 - C# type generation (record types) arrives with the C# runtime in v1.

Java

💬 Under consideration - Type generation follows the language runtime, and Java is considered rather than committed. It would generate record classes. See Celerity versions.

Defaults

Schema default values are applied at construction time by generated code and are never silently injected by the SDK on write.

A default belongs to the object type that declares it. A default on a nested field is part of that nested object's shape, not the item's, so every generated object type carries its own defaults and they apply when that object is constructed. This is what keeps defaults from fighting optionality: profile in the schema above is optional and has no default of its own, so nothing about preferences.theme having a default may cause an empty profile to appear on an item that never had one.

Each language expresses that rule in its own way:

Python. Pydantic field defaults on each model, including nested ones:

class UserStoreProfilePreferences(BaseModel):
    theme: Optional[str] = "system"
    notifications: Optional[bool] = True

TypeScript. Types cannot carry values, so defaults are exported as one constant per generated object type:

/** Schema defaults */
export const UserStoreDefaults = {
  tier: "free",
} as const;

export const UserStoreProfilePreferencesDefaults = {
  theme: "system",
  notifications: true,
} as const;

A single nested constant would have been the obvious alternative and is wrong: { tier: "free", profile: { preferences: { ... } } } spread over an item materialises a profile the schema says is optional.

Go. Types cannot carry values either, but Go has constructors, so NewUserStoreItem() and NewUserStoreProfilePreferences() return values with their own type's defaults applied. An item's constructor leaves optional nested objects nil; constructing one applies its defaults. The constants are generated too, as with UserStoreDefaults, for code that needs to read a default without building anything.

A field that is absent stays absent across languages: optional fields without an explicit default generate as None, ? or a nil pointer rather than materialising empty values, so an item round-tripped through one language's generated types has the same shape in the others.

Entities

When a schema defines entities, codegen emits one item type per entity plus a discriminated union, so AppStoreItem = UserItem | SessionItem in TypeScript and a Pydantic discriminated union (on the discriminator field) in Python, along with per-entity field constants.

Go does not have a union type. Generation emits a struct per entity plus a sealed interface the entity structs satisfy, so a type switch over it is exhaustive at review time and a new entity added to the schema shows up as an unhandled case rather than a runtime surprise:

switch item := item.(type) {
case generated.UserItem:
	// ...
case generated.SessionItem:
	// ...
}

A generated function decodes an item into the right entity struct by reading the discriminator field, which is the one place the discriminator value appears in code.

Schema Contracts

Schema contracts allow data teams to declare which datastores they depend on, so they are automatically informed when schema changes affect them. Any structural change to a watched datastore's schema triggers the contract's policy.

The schema manager already knows exactly what changed (fields added, removed, type changes, etc.), so contracts don't need to duplicate that information. They simply declare: "I care about this datastore. Tell me when its schema changes."

Contracts work the same way for SQL databases. See SQL Database Schema Contracts.

Contracts File Format

✅ Available in v0

Data teams maintain a contracts file in the repository alongside the blueprint. The same file can include contracts for both SQL databases and NoSQL datastores:

# schema-contracts.yaml
contracts:
  - name: "user-analytics-pipeline"
    owner: "data-team"
    dependencies:
      - datastore: "userStore"
        policy: "blocking"       # non-zero exit code if this datastore's schema changes

  - name: "engagement-reporting"
    owner: "data-team"
    dependencies:
      - datastore: "userStore"
        policy: "notify"         # warning output, zero exit code

For single-table datastores, scoping a dependency to specific entities keeps contracts quiet when unrelated entities change:

contracts:
  - name: "session-analytics"
    owner: "data-team"
    dependencies:
      - datastore: "appStore"
        entities: ["session"]    # Only fires when the session entity's schema changes
        policy: "blocking"

When a schema change touches a datastore listed in a contract, the diff output includes the full details of what changed: fields added, removed, type changes, etc. The contract itself just identifies which datastores matter.

Validation and CI Integration

✅ Available in v0

Contract checking is built into celerity schema validate. When a schema-contracts.yaml file exists and deployed state is available, validate evaluates contracts as part of its checks:

• blocking contracts affected: validate exits with a non-zero exit code, failing your CI pipeline
• notify contracts affected: validate prints warnings but does not affect the exit code

This means a single celerity schema validate step in CI covers everything: schema correctness, field type validation, and contract impact:

# Example CI step (GitHub Actions)
- name: Validate schema
  run: celerity schema validate

When a blocking contract fires, the CI output shows exactly what changed, giving the data team the information they need to review the PR. Use your platform's existing notification mechanisms (CODEOWNERS, required reviewers, Slack integrations on CI failure) to alert the right people.

Contract impact is also shown in celerity schema diff output, giving visibility into downstream effects during local development.

Deploy-Time Enforcement and Webhook Notifications

Programmatic Schema API

Data Catalog Integrations

For Data Teams

The schema YAML files serve as always-accurate documentation because they are what Celerity uses for validation, type generation and contract evaluation. For NoSQL datastores, this is even more critical than for SQL databases because there is no database-level schema to introspect. The schema YAML is the only reliable source of truth for what fields exist, their types, and who owns them.

Data teams benefit from:

• description, owner, tags, classification fields on the schema and every field make datastores self-documenting
• Schema exports (celerity schema export --format markdown) generate human-readable documentation
• Git history of schema files (git log schemas/user-store.yaml) provides full change history with PR review
• Contract definitions in schema-contracts.yaml let data teams declare and protect their dependencies
• Drift scans (celerity schema drift) report per-field conformance of live data against the schema, with affected contracts called out
• Machine-readable exports (celerity schema export --format json-schema) integrate with pipeline tools