1. Overview
In programming, a buffer is a region of memory used to temporarily store data while it’s being moved from one place to another. Sometimes, we use it to serialize or convert an object to and from a byte form.
In this tutorial, we’ll see how to convert a Kotlin data class to a ByteBuffer using different forms of serialization.
2. What Is a ByteBuffer?
A ByteBuffer is a class used in Java and Kotlin to handle and manipulate binary data (bytes). It’s part of the java.nio package, which is efficient for I/O operations. Its primary use is for reading from and writing to channels (like files, sockets, etc.) where data transfers in byte form.
ByteBuffer operates in two main modes: writing mode (putting data into the buffer) and reading mode (getting data from the buffer). We typically switch between these modes using the flip() method.
A ByteBuffer has a capacity, the maximum number of bytes it can hold. It also has a position representing the current index where the next read or write operation occurs.
The typical use cases are:
- Data Serialization: When converting an object into a binary format (e.g., saving a class instance as bytes), we often use a ByteBuffer to store the raw bytes before transmitting or writing them to a file.
- Network Communication: When sending or receiving data over a network (sockets), the system typically represents the data as a series of bytes. A ByteBuffer can temporarily store these bytes.
- File I/O: When reading from or writing to a file in binary mode, ByteBuffer is an intermediary between the file and the application.
3. Manual Serialization
Let’s start with an example of a manual serialization of a data class. For instance, we can have a User class:
data class User(val id: Int, val name: String, val age: Int)
3.1. From Data Class to ByteBuffer
We can define an extension function for the User to serialize to ByteBuffer:
fun User.manualToByteBuffer(): ByteBuffer {
val nameBytes = this.name.toByteArray(Charsets.UTF_8)
val buffer = ByteBuffer.allocate(Int.SIZE_BYTES + nameBytes.size + Int.SIZE_BYTES)
buffer.putInt(this.id)
buffer.put(nameBytes)
buffer.putInt(this.age)
buffer.flip() // Prepare the buffer for reading
return buffer
}
As we can see, it involves manually coding the process of converting an object into a byte representation. Therefore, we need to define the capacity we allocate:
val buffer = ByteBuffer.allocate(Int.SIZE_BYTES + nameBytes.size + Int.SIZE_BYTES)
We can then put the bytes into the buffer and flip it to make it available to read from the beginning:
buffer.putInt(this.id)
// add more bytes to the buffer
buffer.flip()
This approach allows us to define exactly how data is converted to bytes but requires more code and effort to handle data layout and conversion. Furthermore, it can be inflexible if the data size varies or is unpredictable.
3.2. From ByteBuffer to Data Class
Likewise, we can write the opposite serialization, from ByteBuffer to User, still using an extension function:
fun ByteBuffer.manualToUser(): User {
val id = this.int
val nameBytes = ByteArray(this.remaining() - Int.SIZE_BYTES)
this.get(nameBytes)
val name = String(nameBytes, Charsets.UTF_8)
val age = this.int
return User(id, name, age)
}
Still, we can see that we have to manually calculate the byte position before returning the User object. For example, for the name attribute, given we extracted the id:
val nameBytes = ByteArray(this.remaining() - Int.SIZE_BYTES)
3.3. Test
Let’s create a test case for the data class to bytes manual serialization and vice versa:
@Test
fun `Manual serialize and deserialize with fixed size`() {
val user = User(id = 1, name = "Alice", age = 30)
val serializedUser = user.manualToByteBuffer()
val deserializedUser = serializedUser.manualToUser()
assertEquals(user, deserializedUser)
}
Going through the serialization and deserialization process, we expect the result to equal the input.
4. Dynamic Serialization With Internal Libraries
In practice, allocating a fixed byte size has limitations and a lack of flexibility, so we often use SDK or external library serialization.
Kotlin data classes can implement the Serializable interface. Therefore, we need to define our data class to implement Serializable:
class UserInputOutputStream(val id: Int, val name: String, val age: Int) : Serializable
4.1. ByteArrayOutputStream
We can look at ObjectOutputStream to serialize the entire object into bytes and then convert it into a ByteBuffer.
Let’s implement a function to transform our data class into an output stream:
fun UserInputOutputStream.outputStreamByteBuffer(): ByteBuffer {
val byteArrayOutputStream = ByteArrayOutputStream()
val objectOutputStream = ObjectOutputStream(byteArrayOutputStream)
objectOutputStream.writeObject(this)
objectOutputStream.flush()
val byteArray = byteArrayOutputStream.toByteArray()
return ByteBuffer.wrap(byteArray)
}
We create a stream from the data class without manually defining the buffer’s size or the field slots:
val objectOutputStream = ObjectOutputStream(byteArrayOutputStream)
objectOutputStream.writeObject(this)
Finally, we convert the stream into a byte array:
val byteArray = byteArrayOutputStream.toByteArray()
4.2. ByteArrayInputStream
Likewise, we can look at ObjectInputStream for deserialization. By converting from ByteBuffer, we read the data class object from the input stream:
fun ByteBuffer.inputStreamToUser(): UserInputOutputStream {
val byteArray = ByteArray(this.remaining())
this.get(byteArray)
val byteArrayInputStream = ByteArrayInputStream(byteArray)
val objectInputStream = ObjectInputStream(byteArrayInputStream)
return objectInputStream.readObject() as UserInputOutputStream
}
In this case, we’ll need to use the readObject() method to reconstruct the data class object.
4.3. Test
Let’s wrap in a test case for the data class to and from bytes using byte array input/output stream:
@Test
fun `Serialize and deserialize using byte array output and input stream`() {
val user = UserInputOutputStream(id = 1, name = "Alice", age = 30)
val serializedUser = user.outputStreamByteBuffer()
val deserializedUser = serializedUser.inputStreamToUser()
assertEquals(user, deserializedUser)
}
Again, we expect the result from the input/output stream conversion to equal the input.
5. Kotlin Serialization Project
We can also look at kotlin-serialization for further examples of internal libraries. Although it’s not part of the core Kotlin standard library, it’s an official Kotlin library developed and maintained by JetBrains, the creators of Kotlin.
This library allows us to use a variety of serialization formats, such as JSON, Protobuf, or Avro, making it widely regarded as the go-to solution for serialization in Kotlin.
In this case, we need to define our data class class to be @Serializable:
@Serializable
data class UserSerialization(val id: Int, val name: String, val age: Int)
5.1. From Data Class to JSON
Let’s make a function to convert to a JSON object:
fun UserSerialization.userToJson(): ByteBuffer {
val jsonString = Json.encodeToString(this)
val byteArray = jsonString.toByteArray()
return ByteBuffer.wrap(byteArray)
}
First, we need to encode the string into the JSON format. Then, we’ll wrap the byte arrays into the ByteBuffer.
5.2. From JSON to Data Class
Let’s make a function to convert from a JSON object:
fun ByteBuffer.jsonToUser(): UserSerialization {
val byteArray = ByteArray(this.remaining())
this.get(byteArray)
val jsonString = String(byteArray, StandardCharsets.UTF_8)
return Json.decodeFromString(jsonString)
}
We’ll decode the data class from the JSON string during deserialization.
5.3. Test
Let’s wrap in a test case for the data class to and from bytes using JSON serialization:
@Test
fun `Serialize and deserialize using kotlin serialization library`() {
val user = UserSerialization(id = 1, name = "Alice", age = 30)
val serializedUser = user.userToJson()
val deserializedUser = serializedUser.jsonToUser()
assertEquals(user, deserializedUser)
}
As always, we expect the result from JSON serialization to equal the input.
6. Serialization With External Libraries
There are many options if we want to use external libraries. For example, we can look at Kryo. Another good option that is widely used in applications is Jackson.
6.1. From Data Class to Kryo
For serialization, we can use the writeObject() of Kryo in combination with the byte array output stream:
fun User.toByteBufferWithKryo(kryo: Kryo): ByteBuffer {
val byteArrayOutputStream = ByteArrayOutputStream()
val output = Output(byteArrayOutputStream)
kryo.writeObject(output, this)
output.close()
val byteArray = byteArrayOutputStream.toByteArray()
return ByteBuffer.wrap(byteArray)
}
Notably, we could also use a manual allocation:
val byteBuffer = ByteBuffer.allocate(1024) // Fixed size buffer
val output = ByteBufferOutput(byteBuffer)
6.2. From Kryo to Data Class
For deserialization, we can use the readObject() of Kryo in combination with the byte array input stream:
fun ByteBuffer.toUserWithKryo(kryo: Kryo): User {
val byteArray = ByteArray(this.remaining())
this.get(byteArray)
val byteArrayInputStream = ByteArrayInputStream(byteArray)
val input = Input(byteArrayInputStream)
return kryo.readObject(input, User::class.java).also { input.close() }
}
6.3. Test
We need to register the Kryo serializer in our test suite:
private val kryo = Kryo().apply {
register(User::class.java)
}
Then, we can create a test case for the data class to and from bytes using Kryo serialization:
@Test
fun `Serialize and deserialize using Kryo library`() {
val user = User(id = 1, name = "Alice", age = 30)
val serializedUser = user.toByteBufferWithKryo(kryo)
val deserializedUser = serializedUser.toUserWithKryo(kryo)
assertEquals(user, deserializedUser)
}
As always, we expect the result from the Kryo conversion to equal the input.
7. Conclusion
In this article, we saw how to convert a data class to and from a ByteBuffer. First, we saw how to manually create a byte buffer or use the built-in input/output stream option with dynamic allocation. If we want to write more efficient code, we can look at the Kotlin serialization library or use external libraries such as Kyro or Jackson.