My thoughts on data engineering and related topics
With the Failure Handling facility in place, we are now ready to fully complete the case study. As you’ll see, it’s pretty straightforward and elegant. The Update Operation Failure Handling when Updating the target record with…
Read More Yay, we are done! Finally, it’s over!!! Oh, wait! Wait for a second! Don’t celebrate too prematurely! We’re not done yet! Just close! Look, what about the Failure Reporting Requirement? We haven’t done anything about…
Read More Now, we move on to implement the Insert operation. As you’ll see, it’s pretty similar to the Update operation laid out in the last post. Insert target record with values purely obtained from SparkSQL row…
Read More With both the Update and Insert facilities now in place, we are ready to implement the Update and Insert operations. We tackle the Update operation first in this post. Update target record with values purely obtained…
Read More Class JdbcRecordInsertFromSparkSqlRow We are now ready to move on to implement the Insert facility. It’s similar to the Update one. In fact, slightly simpler. Here it is: Instantiation To perform an insert operation for a…
Read More Class JdbcRecordUpdateFromSparkSqlRow Now it’s time to work on the Update facility. The implementation follows: Instantiation To perform an Update operation for a Spark partition, the client code needs to instantiate an instance of this class.…
Read More With the class JdbcRecordWriteFromSparkSqlRow in place, we are pretty close to getting done with implementing the Update and Insert facilities. The majority of the remaining work, as you’ll see soon, is about constructing their SQL…
Read More The previous post introduced the commonalities in implementing the update and insert operations. Now, we translate that insight into some classes presented in this post. First, let’s look at the class sitting at the top…
Read More Where Are We? So where are we now? Let’s take a step back. We started with the set of requirements for the write operation from a Spark dataframe to a database table. Mainly, we’re required…
Read More Now, after all the runtime and operational issues have been addressed, let’s move on to looking at code quality. We need to make it more robust, flexible, and maintainable. In other words, the code need to…
Read More