Skip to content

KryoSerializer Generator

Jump to bottom
javapark1 edited this page Feb 25, 2024 · 8 revisions

◀️ Eclipse 🔗 Platform: PadoGrid 1.x ▶️

Working with Kryo can be error prone when you have many domain classes. For data grid products, for example, you are left with the task of manually and individually registering each class by assigning a unique ID. Even with a few domain classes, this can quickly become a time consuming and error pone task.

To alleviate this problem, PadoGrid includes the t_generate_kryo_serializer command that generates a KryoSerializer class for the specified Java package where your domain classes are located. It automatically and optimally registers all the classes found in the specified package. This is done by assigning the specified type ID to the serializer and reserving a single byte to uniquely identify the domain classes found in the specified package. The class ID begins from 0x00 and reserves 0xFF for undefined classes allowing up to 255 classes per package. If you have more than 255 domain classes in a package, then you must split them into smaller packages with not more than 255 classes.

If KryoSerialzer already exists in the package and new classes are added in the package since KryoSerialzer is last created, then t_generate_kryo_serializer appends the new classes to the existing KryoSerializer class by incrementing the last class ID. This ensures the preservance of the existing class IDs.

The following example generates KryoSerializer in the src/main/java directory for the example.domain package that contains domain classes found in the lib/example-domain.jar file. It also picks up dependencies from the lib directory and assigns the unique ID 1200 to the generated class.

t_generate_kryo_serializer -id 1200 \
   -package example.domain \
   -dir src/main/java \
   -jar lib/example-domain.jar \
   -classpath lib

The generated class for each data grid product is shown below.

Geode/GemFire:

package example.domain;

import java.io.ByteArrayOutputStream;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.apache.geode.DataSerializer;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;

/**
 * This class is generated by the PadoGrid Code Generator. You may modify this
 * class as needed. It will not be overwritten when you generate it again later.
 *
 * @since Mon Apr 25 08:51:05 EDT 2022
 * @author dpark
 * @see <a href="https://github.com/padogrid/padogrid">https://github.com/padogrid/padogrid</a>
 */
public class KryoSerializer extends DataSerializer {

	private static final ThreadLocal<Kryo> kryoThreadLocal = new ThreadLocal<Kryo>() {

		@Override
		protected Kryo initialValue() {
			Kryo kryo = new Kryo();
			return kryo;
		}
	};

	private static Class<?>[] classes = new Class<?>[] {
		Customer.class, Order.class, Category.class, Employee.class
	};

	private static Map<Class<?>, Integer> classMap = new HashMap<Class<?>, Integer>(classes.length, 1f);

	static {
		int i = 0;
		for (Class<?> clazz : classes) {
			classMap.put(clazz, i++);
		}
	}

	public static List<Class<?>> getClassList()
	{
		return Collections.unmodifiableList(Arrays.asList(classes));
	}

	public static Integer getClassId(Class<?> clazz) {
		return classMap.get(clazz);
	}

	public static int getLastClassId()
	{
		return classes.length - 1;
	}

	@Override
	public Class<?>[] getSupportedClasses() {
		return classes;
	}

	@Override
	public int getId() {
		return 1200;
	}

	private int getWriteTypeId(Object obj)
	{
		if (obj == null) {
			return 0xFF;
		}
		Class<?> clazz = obj.getClass();
		return classMap.get(clazz);
	}

	private Object readKryo(Input input, Kryo kryo, int typeId)
	{
		if (typeId < 0 || typeId >= classes.length) {
			return null;
		}
		kryo.register(classes[typeId]);
		return kryo.readObject(input, classes[typeId]);
	}

	@Override
	public boolean toData(Object obj, DataOutput out) throws IOException {
		 Kryo kryo = kryoThreadLocal.get();
		 out.writeByte(getWriteTypeId(obj));
		 kryo.register(obj.getClass());
		 ByteArrayOutputStream baos = new ByteArrayOutputStream();
         Output output = new Output(baos);
         kryo.writeObject(output, obj);
         output.flush();
         out.writeInt(baos.size());
         out.write(baos.toByteArray());
         return true;
	}

	@Override
	public Object fromData(DataInput in) throws IOException, ClassNotFoundException {
		byte typeId = in.readByte();
		int size = in.readInt();
		byte[] buffer = new byte[size];
		in.readFully(buffer);
        Input input = new Input(buffer);
        Kryo kryo = kryoThreadLocal.get();
        return readKryo(input, kryo, typeId);
	}
}

Hazelcast:

package example.domain;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.nio.serialization.StreamSerializer;

/**
 * This class is generated by the PadoGrid Code Generator. You may modify this
 * class as needed. It will not be overwritten when you generate it again later.
 *
 * @since Mon Apr 25 12:16:32 EDT 2022
 * @author dpark
 * @see <a href="https://github.com/padogrid/padogrid">https://github.com/padogrid/padogrid</a>
 */
public class KryoSerializer implements StreamSerializer<Object> {

	private static final ThreadLocal<Kryo> kryoThreadLocal = new ThreadLocal<Kryo>() {

		@Override
		protected Kryo initialValue() {
			Kryo kryo = new Kryo();
			return kryo;
		}
	};

	private static Class<?>[] classes = new Class<?>[] {
		Customer.class, Order.class, Category.class, Employee.class
	};

	private static Map<Class<?>, Integer> classMap = new HashMap<Class<?>, Integer>(classes.length, 1f);

	static {
		int i = 0;
		for (Class<?> clazz : classes) {
			classMap.put(clazz, i++);
		}
	}

	public static List<Class<?>> getClassList()
	{
		return Collections.unmodifiableList(Arrays.asList(classes));
	}

	public static Integer getClassId(Class<?> clazz) {
		return classMap.get(clazz);
	}

	public static int getLastClassId()
	{
		return classes.length - 1;
	}

	@Override
	public int getTypeId() {
		return 1200;
	}

	@Override
	public void destroy() {
		kryoThreadLocal.remove();
	}

	private int getWriteTypeId(Object obj)
	{
		if (obj == null) {
			return 0xFF;
		}
		Class<?> clazz = obj.getClass();
		return classMap.get(clazz);
	}

	private Object readKryo(Input input, Kryo kryo, int typeId)
	{
		if (typeId < 0 || typeId >= classes.length) {
			return null;
		}
		kryo.register(classes[typeId]);
		return kryo.readObject(input, classes[typeId]);
	}

	@Override
	public void write(ObjectDataOutput odout, Object obj) throws IOException {
		 Kryo kryo = kryoThreadLocal.get();
		 odout.writeByte(getWriteTypeId(obj));
		 kryo.register(obj.getClass());
         Output output = new Output((OutputStream) odout);
         kryo.writeObject(output, obj);
         output.flush();
	}

	@Override
	public Object read(ObjectDataInput odin) throws IOException {
		byte typeId = odin.readByte();
		InputStream in = (InputStream) odin;
        Input input = new Input(in);
        Kryo kryo = kryoThreadLocal.get();
        return readKryo(input, kryo, typeId);
	}
}

Once the KryoSerializer class is generated, instead of registering each domain class, i.e., Customer.class, Order.class, Category.class, and Employee.class, you would simply register the KryoSerializer class in the data grid as follows.

Geode/GemFire cache.xml:

<cache>
...
    <serialization-registration>
        <serializer>
            <class-name>example.KryoSerializer</class-name>
        </serializer>
    </serialization-registration>
...
</cache>

Hazelcast hazelcast.xml:

<hazelcast>
...
       <serialization>
             <serializers>
                  <global-serializer override-java-serialization="true">
                  example.domain.KryoSerializer
                  </global-serializer>
             </serializers>
        </serialization>
...
</hazelcast>

References

The following bundles provide step-by-step instructions for generating Avro and Kryo classes using PadoGrid code generators.

  1. Geode/GemFire Kryo/Avro Code Generator, https://github.com/padogrid/bundle-geode-1-app-kryo_codegen.
  2. Hazelcast Kryo/Avro Code Generator, https://github.com/padogrid/bundle-hazelcast-4n5-app-kryo_codegen.

◀️ Eclipse 🔗 Platform: PadoGrid 1.x ▶️

PadoGrid Manual

Overview

Operations

Tools

Platforms

Clouds

Pado

Geode/GemFire

Hazelcast/Jet

ComputeDB/SnappyData

Coherence

Hadoop

Kafka/Confluent

Mosquitto

Redis

Spark

Clone this wiki locally