My thoughts on data engineering and related topics
Where Are We? So where are we now? Let’s take a step back. We started with the set of requirements for the write operation from a Spark dataframe to a database table. Mainly, we’re required…
Read More Now, after all the runtime and operational issues have been addressed, let’s move on to looking at code quality. We need to make it more robust, flexible, and maintainable. In other words, the code need to…
Read More The Problem The previous post suggests a fix to execSQL() to handle null values. We are not done yet. Let’s take a look at the main function updateTable() in approach IV: In the call updateDF.foreachPartition()…
Read More The Problem As alluded in Approach IV post, that implementation would not work in real production data. What’s going on? Now’s the time to deal with it. Take a look at this key line of…
Read More This is a bonus post. It chronicles an activity that doesn’t generate a concrete outcome, yet reflects the considerations emerging at this point in an engineering project. Motivation Now approach IV is chosen. Approach III…
Read More Approach III, although the best so far, still doesn’t quite make it. Hence, we now move on to this next one. Summary Each row in the updateDF dataframe gets extracted individually, and explicitly updates the…
Read More The first two approaches don’t cut it. We now move on to the third one, in search of a solution that can reasonably perform the Update operation. Summary We dump the updateDF dataframe to a…
Read More We now explore the second approach using the Update operation as the first test. Following is its examination. Summary In a nutshell, this approach writes update data to a temp table, then uses database trigger…
Read More We now start exploring different approaches with the Update operation as the first evaluation criterion. Following is the first potential solution. Summary This first approach is the most naive, which relies most heavily on available…
Read More So far, we have formulated the problem we are facing: how to have Spark write data from a dataframe to a relational table, such that this operation satisfies a set of requirements? Look at it…
Read More