Skip to content

schema-evolution.md

Latest commit

 

History

History
393 lines (297 loc) · 9.93 KB

schema-evolution.md

File metadata and controls

393 lines (297 loc) · 9.93 KB
Raw
title Schema Evolution
sidebar_position 6
id schema-evolution
license Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Schema evolution allows your data structures to change over time while maintaining compatibility with previously serialized data. Fory Go supports this through compatible mode, which is the default for both xlang and native mode.

Compatible Mode Defaults

For cross-language payloads, create a Fory instance with the default xlang settings:

f := fory.New(fory.WithXlang(true))

For Go-only native-mode payloads that need schema evolution, use native mode and keep the compatible default:

f := fory.New(fory.WithXlang(false))

How It Works

With Compatible Mode

  • Type metadata is written to serialized data
  • Supports adding, removing, and reordering fields
  • Enables forward and backward compatibility

Same-Schema Optimization

  • Compact serialization without evolution metadata
  • Struct hash is checked during deserialization
  • Any schema change causes ErrKindHashMismatch

Supported Schema Changes

Adding Fields

New fields can be added; they receive zero values when deserializing old data:

// Version 1
type UserV1 struct {
    ID   int64
    Name string
}

// Version 2 (added Email)
type UserV2 struct {
    ID    int64
    Name  string
    Email string  // New field
}

f := fory.New(fory.WithXlang(true))
f.RegisterStruct(UserV1{}, 1)

// Serialize with V1
userV1 := &UserV1{ID: 1, Name: "Alice"}
data, _ := f.Serialize(userV1)

// Deserialize with V2
f2 := fory.New(fory.WithXlang(true))
f2.RegisterStruct(UserV2{}, 1)

var userV2 UserV2
f2.Deserialize(data, &userV2)
// userV2.Email = "" (zero value)

Removing Fields

Removed fields are skipped during deserialization:

// Version 1
type ConfigV1 struct {
    Host     string
    Port     int32
    Timeout  int64
    Debug    bool  // Will be removed
}

// Version 2 (removed Debug)
type ConfigV2 struct {
    Host    string
    Port    int32
    Timeout int64
    // Debug field removed
}

f := fory.New(fory.WithXlang(true))
f.RegisterStruct(ConfigV1{}, 1)

// Serialize with V1
config := &ConfigV1{Host: "localhost", Port: 8080, Timeout: 30, Debug: true}
data, _ := f.Serialize(config)

// Deserialize with V2
f2 := fory.New(fory.WithXlang(true))
f2.RegisterStruct(ConfigV2{}, 1)

var configV2 ConfigV2
f2.Deserialize(data, &configV2)
// Debug field data is skipped

Reordering Fields

Field order can change between versions:

// Version 1
type PersonV1 struct {
    FirstName string
    LastName  string
    Age       int32
}

// Version 2 (reordered)
type PersonV2 struct {
    Age       int32   // Moved up
    LastName  string
    FirstName string  // Moved down
}

Compatible mode handles this automatically by matching fields by name.

Compatible Scalar Field Changes

Compatible mode can also read selected scalar type changes for matched top-level struct fields when the serialized value converts without changing its logical value:

  • bool fields can be read from strings that are exactly "0", "1", "true", or "false". Bool values read as strings become "true" or "false", and numeric 0 and 1 can be read as bools.
  • Integer, unsigned integer, floating point, and decimal fields can be read across numeric scalar types only when the value is represented exactly by the target field type.
  • Numeric fields can be read from strings only when the string is a finite ASCII decimal literal with no whitespace, leading +, Unicode digits, separators, radix prefixes, or special values such as NaN and Infinity.
  • Numeric fields read as strings use canonical output: integers have normal decimal text, floating point values use exact plain decimal text with a decimal point, and decimals omit insignificant trailing fractional zeros.

Scalar conversion composes with pointer and optional.Optional[T] fields when the matched top-level scalar field is not reference-tracked. If a remote nullable or optional field is absent, the local field follows the normal missing/null compatible-mode behavior. Reference-tracked scalar type changes are incompatible. If a present value cannot be converted losslessly, deserialization fails with a data error instead of treating the field as missing.

Incompatible Changes

Some changes are NOT supported, even in compatible mode:

Type Changes

// NOT SUPPORTED
type V1 struct {
    Value []int32  // list of int32
}

type V2 struct {
    Value []string  // Element type changed - INCOMPATIBLE
}

Renaming Fields

// NOT SUPPORTED (treated as remove + add)
type V1 struct {
    UserName string
}

type V2 struct {
    Username string  // Different name - NOT a rename
}

This is treated as removing UserName and adding Username, resulting in data loss.

Best Practices

1. Use Compatible Mode for Persistent Data

// Default xlang payloads already use compatible mode.
f := fory.New(fory.WithXlang(true))

For Go-only native-mode data stored in databases, files, or caches, use compatible mode:

f := fory.New(fory.WithXlang(false))

2. Provide Default Values

type ConfigV2 struct {
    Host    string
    Port    int32
    Timeout int64
    Retries int32  // New field
}

func NewConfigV2() *ConfigV2 {
    return &ConfigV2{
        Retries: 3,  // Default value
    }
}

// After deserialize, apply defaults
if config.Retries == 0 {
    config.Retries = 3
}

Xlang Schema Evolution

Schema evolution works across languages:

Go (Producer)

type MessageV1 struct {
    ID      int64
    Content string
}

f := fory.New(fory.WithXlang(true))
f.RegisterStruct(MessageV1{}, 1)
data, _ := f.Serialize(&MessageV1{ID: 1, Content: "Hello"})

Java (Consumer with newer schema)

public class Message {
    long id;
    String content;
    String author;  // New field in Java
}

Fory fory = Fory.builder().withXlang(true).build();
fory.register(Message.class, 1);
Message msg = fory.deserialize(data, Message.class);
// msg.author will be null

Performance Considerations

Compatible mode mainly affects serialized size:

Aspect WithCompatible(false) Compatible mode
Serialized Size Smaller Larger (includes metadata, especially without field IDs)
Speed Fast Similar (metadata is just memcpy)
Schema Flexibility Same schema required Add, remove, and reorder fields

Note: Using field IDs (fory:"id=N") reduces metadata size in compatible mode.

Recommendation: Use compatible mode for:

  • Persistent storage
  • Cross-service communication
  • Long-lived caches

Use WithCompatible(false) only when every reader and writer always uses the same Go struct schema and you want faster serialization and smaller size. For xlang payloads, use WithCompatible(false) only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. Same-schema uses include:

  • In-memory operations
  • Same-schema communication
  • Faster serialization and smaller size

Per-Struct Opt-Out

For one struct, you can opt out of evolution metadata by implementing ForyEvolving and returning false:

type SameSchemaMessage struct {
    ID int64
}

func (SameSchemaMessage) ForyEvolving() bool {
    return false
}

Error Handling

Hash Mismatch (Native Same-Schema Mode)

f := fory.New(fory.WithXlang(false), fory.WithCompatible(false))

// Schema changed without compatible mode
err := f.Deserialize(oldData, &newStruct)
// Error: ErrKindHashMismatch

Unknown Fields

In compatible mode, unknown fields are skipped silently. To detect them:

// Currently, Fory skips unknown fields automatically
// No explicit API for detecting unknown fields

Complete Example

package main

import (
    "fmt"
    "github.com/apache/fory/go/fory"
)

// V1: Initial schema
type ProductV1 struct {
    ID    int64
    Name  string
    Price float64
}

// V2: Added fields
type ProductV2 struct {
    ID          int64
    Name        string
    Price       float64
    Description string  // New
    InStock     bool    // New
}

func main() {
    // Serialize with V1
    f1 := fory.New(fory.WithXlang(true))
    f1.RegisterStruct(ProductV1{}, 1)

    product := &ProductV1{ID: 1, Name: "Widget", Price: 9.99}
    data, _ := f1.Serialize(product)
    fmt.Printf("V1 serialized: %d bytes\n", len(data))

    // Deserialize with V2
    f2 := fory.New(fory.WithXlang(true))
    f2.RegisterStruct(ProductV2{}, 1)

    var productV2 ProductV2
    if err := f2.Deserialize(data, &productV2); err != nil {
        panic(err)
    }

    fmt.Printf("ID: %d\n", productV2.ID)
    fmt.Printf("Name: %s\n", productV2.Name)
    fmt.Printf("Price: %.2f\n", productV2.Price)
    fmt.Printf("Description: %q (zero value)\n", productV2.Description)
    fmt.Printf("InStock: %v (zero value)\n", productV2.InStock)
}

Related Topics