My thoughts on data engineering and related topics
Now, we move on to implement the Insert operation. As you’ll see, it’s pretty similar to the Update operation laid out in the last post. Insert target record with values purely obtained from SparkSQL row…
Read More With both the Update and Insert facilities now in place, we are ready to implement the Update and Insert operations. We tackle the Update operation first in this post. Update target record with values purely obtained…
Read More Class JdbcRecordInsertFromSparkSqlRow We are now ready to move on to implement the Insert facility. It’s similar to the Update one. In fact, slightly simpler. Here it is: Instantiation To perform an insert operation for a…
Read More Class JdbcRecordUpdateFromSparkSqlRow Now it’s time to work on the Update facility. The implementation follows: Instantiation To perform an Update operation for a Spark partition, the client code needs to instantiate an instance of this class.…
Read More With the class JdbcRecordWriteFromSparkSqlRow in place, we are pretty close to getting done with implementing the Update and Insert facilities. The majority of the remaining work, as you’ll see soon, is about constructing their SQL…
Read More The previous post introduced the commonalities in implementing the update and insert operations. Now, we translate that insight into some classes presented in this post. First, let’s look at the class sitting at the top…
Read More Where Are We? So where are we now? Let’s take a step back. We started with the set of requirements for the write operation from a Spark dataframe to a database table. Mainly, we’re required…
Read More Now, after all the runtime and operational issues have been addressed, let’s move on to looking at code quality. We need to make it more robust, flexible, and maintainable. In other words, the code need to…
Read More The Problem The previous post suggests a fix to execSQL() to handle null values. We are not done yet. Let’s take a look at the main function updateTable() in approach IV: In the call updateDF.foreachPartition()…
Read More The Problem As alluded in Approach IV post, that implementation would not work in real production data. What’s going on? Now’s the time to deal with it. Take a look at this key line of…
Read More