More detail on schema evolution please

[dsyer]

I know there's a link to another blog, but a lazy reader would like to know the punchline! Given the two (incompatible) example objects how do the consumers and producers work together? Which combinations work, and which do not?

[dsyer]

Here's a test I used to try and work out the compatibility matrix (and failed). I copied the generated classes and renamed them "Sensor1" and "Sensor2" so I could work with them both in the same class loader.

public class CodecTests {

    private AvroCodec codec;
    private Schema schema1;
    private Schema schema2;
    Sensor2 sensor2 = new Sensor2();
    Sensor1 sensor1 = new Sensor1();

    public AvroCodec avroCodec(AvroCodecProperties properties,
            SchemaRegistryClient schemaRegistryClient) throws Exception {
        AvroCodec codec = new AvroCodec();
        codec.setProperties(properties);
        codec.setSchemaRegistryClient(schemaRegistryClient);
        codec.setResolver(
                new PathMatchingResourcePatternResolver(new DefaultResourceLoader()));
        codec.init();
        return codec;
    }

    @Before
    public void init() throws Exception {
        codec = avroCodec(new AvroCodecProperties(), new LocalSchemaRegistryClient());
        sensor2.setId("id");
        sensor2.setInternalTemperature(100f);
        sensor2.setExternalTemperature(80f);
        sensor2.setAcceleration(0.1f);
        sensor2.setVelocity(1.0f);
        sensor2.setMagneticField(Arrays.asList(0f, 0f, 0f));
        sensor2.setAccelerometer(Arrays.asList(0f, 0f, 0f));
        sensor1.setId("id");
        sensor1.setTemperature(100f);
        sensor1.setAcceleration(0.1f);
        sensor1.setVelocity(1.0f);
        sensor1.setMagneticField(Arrays.asList(0f, 0f, 0f));
        sensor1.setAccelerometer(Arrays.asList(0f, 0f, 0f));
        schema1 = new Schema.Parser()
                .parse(new ClassPathResource("avro/sensor.v1.avsc").getInputStream());
        schema2 = new Schema.Parser()
                .parse(new ClassPathResource("avro/sensor.v2.avsc").getInputStream());
    }

    @Test
    public void serializeNative() throws Exception {
        byte[] src = codec.encode(sensor2);
        Sensor2 result = codec.decode(src, Sensor2.class);
        assertEquals(80f, result.getExternalTemperature(), 0.001);
    }

    @Test
    public void serializeToOld() throws Exception {
        byte[] src = codec.encode(sensor1);
        codec.setReaderSchema(schema2);
        Sensor2 result = codec.decode(src, Sensor2.class);
        assertEquals(80f, result.getExternalTemperature(), 0.001);
    }

    @Test
    public void serializeToNew() throws Exception {
        byte[] src = codec.encode(sensor2);
        codec.setReaderSchema(schema1);
        Sensor1 result = codec.decode(src, Sensor1.class);
        assertEquals(80f, result.getTemperature(), 0.001);
    }

}

Only the "native" test works (i.e. the codec can serialize and deserialize an object of type Sensor2). When you try and convert from old to new (Sensor1 to Sensor2) the temperature is ignored, and when you try and go the other way there is an exception in the codec because the "orientation" field is missing.

[viniciusccarvalho]

@dsyer Let me try that, I had both streams deployed and it works. Field renaming only works in one direction unfortunately. For example on v2 we renamed it to internalTemperature, and if v1 tries to read its ignored. There's some guidelines on avro on what you can expect when doing things such as (field renaming, adding fields, removing fields). I started the schema evolution discussion here: https://github.com/viniciusccarvalho/schema-evolution-samples#schema-evolution-recap

[viniciusccarvalho]

There should not be any difference between using the generated classes and "GenericRecord", the IMO the advantage of using generic record containers is that we don't need to deal with class renaming on the same class loader.

[viniciusccarvalho]

Here's a sample of writing V1 and reading on V2, like I said field renaming breaks forward compatibility, but not backward compatibility.
`
public class AvroVersioningTests {

Schema v1;
Schema v2;
GenericRecord sensor1;
GenericRecord sensor2;
SchemaRegistryClient client;
AvroCodec codec;

@Before
public void setup() throws Exception{
    v1 = new Schema.Parser().parse(AvroVersioningTests.class.getClassLoader().getResourceAsStream("schemas/sensor.v1.avsc"));
    v2 = new Schema.Parser().parse(AvroVersioningTests.class.getClassLoader().getResourceAsStream("schemas/sensor.v2.avsc"));
    sensor1 = new GenericData.Record(v1);
    sensor1.put("id","v1");
    sensor1.put("velocity",0.1f);
    sensor1.put("acceleration",0.1f);
    sensor1.put("accelerometer", Arrays.asList(0.1f, 0.1f, 0.1f));
    sensor1.put("magneticField", Arrays.asList(0.1f, 0.1f, 0.1f));
    sensor1.put("orientation", Arrays.asList(0.1f, 0.1f, 0.1f));
    sensor1.put("temperature",0.1f);

    sensor2 = new GenericData.Record(v2);
    sensor2.put("id","v2");
    sensor2.put("velocity",0.2f);
    sensor2.put("acceleration",0.2f);
    sensor2.put("accelerometer", Arrays.asList(0.2f, 0.2f, 0.2f));
    sensor2.put("magneticField", Arrays.asList(0.2f, 0.2f, 0.2f));
    sensor2.put("internalTemperature",0.2f);
    sensor2.put("externalTemperature",0.2f);

    client = mock(SchemaRegistryClient.class);
    when(client.register(eq(v1))).thenReturn(1);
    when(client.register(eq(v2))).thenReturn(2);
    when(client.fetch(eq(1))).thenReturn(v1);
    when(client.fetch(eq(2))).thenReturn(v2);

    codec = new AvroCodec();
    codec.setSchemaRegistryClient(client);
    codec.setProperties(new AvroCodecProperties());
}


@Test
public void writeV1ReadV2() throws Exception{
    byte[] v1Bytes = codec.encode(sensor1);
    codec.setReaderSchema(v2);
    GenericRecord result = codec.decode(v1Bytes,GenericRecord.class);
    Assert.assertEquals("v1",result.get("id").toString());
    Assert.assertEquals(0.1f, result.get("internalTemperature"));
    }

}
`

[dsyer]

Your test fails for me in the same way mine did (the last assertion is false because the internal temperature is 0).

[viniciusccarvalho]

I just ran this: https://github.com/viniciusccarvalho/schema-evolution-samples/blob/master/producer_v1/src/test/java/org/springframework/cloud/stream/schema/AvroVersioningTests.java and it passed, Tests seems to be just fine at Travis as well: https://travis-ci.org/viniciusccarvalho/schema-evolution-samples/builds

So I really don't know what could be going wrong