"Schema Migration Without Ceremony"

Schema evolution is one of the most reliably painful problems in long-running systems. A sensor node deployed six months ago is still sending v1 payloads. Your edge gateway was upgraded last week and expects v2. The hub on Kubernetes is already on v3. All three need to communicate without breaking.

The standard answers — version your API endpoints, maintain backward-compatible JSON "forever," use a schema registry with runtime compatibility checks — all have the same failure mode: the schema drift is tracked somewhere outside the type system, and the compiler can't tell you when things go wrong.

AimDB's migration system takes a different approach. Every version transition is a typed, bidirectional Rust function. The chain of transitions is validated at compile time. And dispatching from any historical version to the current one happens automatically, with no runtime configuration.

The Building Block: MigrationStep

A MigrationStep is the smallest unit of schema migration — a typed conversion between two adjacent versions:

pub trait MigrationStep {
    type Older;
    type Newer;
    const FROM_VERSION: u32;
    const TO_VERSION: u32;

    fn up(older: Self::Older) -> Result<Self::Newer, MigrationError>;
    fn down(newer: Self::Newer) -> Result<Self::Older, MigrationError>;
}

Each step is bidirectional. up converts from an older schema to a newer one. down converts from a newer schema back to an older one. Both directions are required — this is what makes it possible to downgrade a running node for rollback, or to communicate with a peer that's still on an older version.

The Older and Newer associated types are concrete Rust structs. There's no serde_json::Value in sight, no string manipulation, no raw byte juggling. The conversion logic is just Rust:

pub struct TemperatureV1ToV2;

impl MigrationStep for TemperatureV1ToV2 {
    type Older = TemperatureV1;
    type Newer = Temperature;
    const FROM_VERSION: u32 = 1;
    const TO_VERSION: u32 = 2;

    fn up(v1: TemperatureV1) -> Result<Temperature, MigrationError> {
        // v1 stored temperature in any unit; v2 is always Celsius
        let celsius = match v1.unit.as_str() {
            "F" => (v1.temp - 32.0) * 5.0 / 9.0,
            "K" => v1.temp - 273.15,
            _   => v1.temp,
        };
        Ok(Temperature { schema_version: 2, celsius, timestamp: v1.timestamp })
    }

    fn down(v2: Temperature) -> Result<TemperatureV1, MigrationError> {
        Ok(TemperatureV1 {
            schema_version: 1,
            temp: v2.celsius,
            timestamp: v2.timestamp,
            unit: String::from("C"),
        })
    }
}

The struct definitions for TemperatureV1 and Temperature live side by side in aimdb-data-contracts. Old versions are kept in the crate permanently — not as technical debt, but as the canonical record of what the wire format used to look like and exactly how to convert from it. The test suite covers every version transition.

Wiring It Together: migration_chain!

A single step is useful. A validated chain of steps is what handles a system that's been running for two years with five schema versions.

The migration_chain! macro takes the step definitions and generates a MigrationChain implementation for the current type, with the chain validated entirely at compile time:

migration_chain! {
    type Current = Temperature;
    version_field = "schema_version";
    steps {
        TemperatureV1ToV2: TemperatureV1 => Temperature,
    }
}

For a longer-lived type with more versions:

migration_chain! {
    type Current = Sensor;
    version_field = "schema_version";
    steps {
        SensorV1ToV2: SensorV1 => SensorV2,
        SensorV2ToV3: SensorV2 => Sensor,
    }
}

The macro generates const assertions that enforce:

• Sequential versions: each step increments the version by exactly 1
• Chain continuity: the Newer type of one step must match the Older type of the next
• Anchored bounds: the first step must start at version 1; the last step must end at the current schema VERSION

These checks happen at compile time. A broken chain — a step that skips a version, or a type mismatch at a join point — is a compile error, not a runtime failure discovered in production.

The MigrationChain Trait

The macro generates an impl of MigrationChain for the current type:

pub trait MigrationChain: SchemaType + DeserializeOwned + Serialize {
    const MIN_VERSION: u32;

    fn migrate_from_bytes(data: &[u8]) -> Result<Self, MigrationError>;
    fn migrate_to_version(&self, target: u32) -> Result<Vec<u8>, MigrationError>;
}

migrate_from_bytes reads the schema_version field from the incoming JSON payload, then walks the chain upward to produce the current schema version. Current-version payloads are deserialized directly. Historical payloads go through each upgrade step in sequence:

// A v1 payload arrives from an old sensor node
let json = r#"{"schema_version":1,"temp":68.0,"timestamp":100,"unit":"F"}"#;

// One call — version detection and upgrade happen automatically
let reading = Temperature::migrate_from_bytes(json.as_bytes()).unwrap();
assert!((reading.celsius - 20.0).abs() < 0.01); // 68°F → 20°C

migrate_to_version does the reverse — it walks the chain downward to produce an older serialized representation:

let current = Temperature::new(20.0, 100);

// Downgrade to v1 format for a legacy consumer
let v1_bytes = current.migrate_to_version(1).unwrap();
let v1: serde_json::Value = serde_json::from_slice(&v1_bytes).unwrap();
assert_eq!(v1["temp"], 20.0);
assert_eq!(v1["unit"], "C");

Error cases are typed, not stringly. MigrationError has variants for version-too-new (the payload claims a version higher than the current schema), version-too-old (you tried to downgrade below the minimum supported version), deserialization failure, and missing version field. Callers can match on the specific error variant and handle each case explicitly.

Integration with Linkable

When a contract implements both MigrationChain and Linkable — the trait that wires it to an MQTT or WebSocket connector — auto-migration happens transparently at the connection layer:

#[cfg(all(feature = "linkable", feature = "migratable"))]
impl Linkable for Temperature {
    fn from_bytes(data: &[u8]) -> Result<Self, String> {
        Self::migrate_from_bytes(data)
            .map_err(|e| format!("Migration error: {}", e))
    }

    fn to_bytes(&self) -> Result<Vec<u8>, String> {
        serde_json::to_vec(self).map_err(|e| e.to_string())
    }
}

A connector configured with a Temperature record will silently upgrade v1 payloads that arrive from old nodes. The buffer only ever holds current-version values. Producers and consumers don't need to know that older versions exist — that complexity lives in the contracts crate, where it's tested in CI on every commit.

The Design Trade-Off

Some migration systems operate on the serialized form — a chain of JSON transformations that reshape a document from one version to the next. This is flexible (you can migrate without deserializing into a concrete type) but fragile (any structural assumption is tested at runtime, not compile time).

AimDB's approach requires a concrete Rust struct per schema version. The trade-off: you pay the small cost of keeping old structs around, and you get a migration path that the compiler fully understands. Every up and down function gets type-checked. The chain continuity is a const assertion. The roundtrip correctness is a unit test you can run in CI.

For AimDB's use case — data contracts that flow across MCU flash, edge gateways, and cloud services, potentially across version skew of months or years — compile-time correctness is worth the cost of explicit version structs. Old schema definitions are documentation as much as they are code. They tell you exactly what the wire format looked like at version 1, and exactly how it was transformed when version 2 was introduced.

Summary

Schema migration isn't glamorous. But getting it wrong silently — a v1 payload arriving at a v3 consumer, misinterpreted rather than rejected — is exactly the kind of bug that costs days to diagnose in a distributed system. AimDB's goal is to make the right thing as low-friction as the wrong thing used to be.

The migration system is part of aimdb-data-contracts and gated behind the migratable feature flag. Browse the source on GitHub or open an issue if you have questions.

Have questions? Open an issue on GitHub or join the discussion.