R++
This blog has moved
Sunday, March 22, 2020
Moving on...
Thanks for visiting this blog.
After having this blog for more than 13 years, I have decided to retire it. My new blog can be found at rohitsm.com hosted on Github Pages (rohitsm.github.io is the URL to be precise). All the content hosted here will continue to remain accessible at the URL oldblog.rohitsm.com.
Thursday, July 4, 2019
Serializing Spark dataframes to Avro using KafkaAvroSerializer
I recently worked on a project that used Spark Structured Streaming using Apache Spark, Confluent Schema Registry and Apache Kafka. Due to some versioning constraints between the various components, I had to write a custom implementation of the KafkaAvroSerializer class for serializing Spark Dataframes into Avro format. The serialized data was then published to Kafka. This post is based on the examples specified in the Confluent documentation here.
In newer versions of Confluent Schema Registry, lot of the implementations detailed below have been simplified and much easier to use. The standard recommended usage of the Confluent KafkaAvroSerializer is fairly simple in that it requires you to set it as one of the Kafka properties that is used when initializing a KafkaProducer:
props.put(...)
...
...
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class
val producer = new KafkaProducer(props);
Thursday, January 17, 2019
My 2018 Reading Challenge
Bad Blood by John Carreyrou
Sunday, August 14, 2016
Book Review: The Intel Trinity: How Robert Noyce, Gordon Moore, and Andy Grove Built the World's Most Important Company
The Intel Trinity: How Robert Noyce, Gordon Moore, and Andy Grove Built the World's Most Important Company by Michael S. Malone
My rating: 5 of 5 stars
I began reading this book on a whim having already been familiar with the backgrounds of the three founders - Robert Noyce, Gordon Moore and Andy Grove. Michael Malone chronicles the birth of Intel to its present state methodically and brilliantly. During the course of the narrative, the book beautifully digresses to give the reader a detailed look into the early lives of the three founders starting with Noyce, Moore and finally Grove.
The book acknowledges that it borrows from the individual biographies of its founders but doesn't come across as redundant at any point. Deeply engaging in its narrative and often poignant in its story line, I found this book beautifully compiled and enjoyed every bit of it to the point that I was a little sad when it ended.
If you are a fan of biographies, you would find this to be a great book. I'd rate is So-good-I-can't-stop-telling-you-about-it good!
Sunday, November 2, 2014
#Humblebrag - This blog got featured in a documentary on Aaron Swartz
How do I know its mine? The novice level of the crudely written code is uniquely me. I was supposed to improve upon it later but I never really got around to actually doing it. Meanwhile some people who forked the code from my Github repo did a far better job anyway. This project was more of a tribute than anything else and so it was a humbling experience to have randomly spotted it while watching the documentary again yesterday.
Tuesday, July 1, 2014
30 Day Challenge - July 2014 and a brief summary
Here are the list of 30 day challenges that I've attempted in the past. I've had fun doing some of them and have blogged about my experiences.
Thursday, January 2, 2014
Report on 30 Day Challenge - Read 15 books in 30 days
This post has been long overdue but considering that I did not attempt to complete the remaining four books until now, I guess this post is not so ill-timed after all. I should mention here that my previous (best) failed attempt at this same challenge was 12 books. I am hoping to beat this sometime in future but that would depend on whether, and when, I would get my hands on a stack of books this engaging.