My thoughts on data engineering and related topics
“If you’re not embarrassed by the first version of your product, you’ve released too late” – Reid Hoffman Following this philosophy for a startup’s minimum viable product by Reid Hoffman, the renowned entrepreneur and…
Read More I might or might not push the code to GitHub. Let me see. If I do, until then, here’s the listing of the essence of the code referenced in this series. They are generally heavily…
Read More Our long and fun journey now comes to the end. About Spark Apache Spark, as an analytic processing framework, is designed to process data in bulk. Particularly, when it comes to writing to an RDBMS, the…
Read More With the Failure Handling facility in place, we are now ready to fully complete the case study. As you’ll see, it’s pretty straightforward and elegant. The Update Operation Failure Handling when Updating the target record with…
Read More Yay, we are done! Finally, it’s over!!! Oh, wait! Wait for a second! Don’t celebrate too prematurely! We’re not done yet! Just close! Look, what about the Failure Reporting Requirement? We haven’t done anything about…
Read More Now, we move on to implement the Insert operation. As you’ll see, it’s pretty similar to the Update operation laid out in the last post. Insert target record with values purely obtained from SparkSQL row…
Read More With both the Update and Insert facilities now in place, we are ready to implement the Update and Insert operations. We tackle the Update operation first in this post. Update target record with values purely obtained…
Read More Class JdbcRecordInsertFromSparkSqlRow We are now ready to move on to implement the Insert facility. It’s similar to the Update one. In fact, slightly simpler. Here it is: Instantiation To perform an insert operation for a…
Read More Class JdbcRecordUpdateFromSparkSqlRow Now it’s time to work on the Update facility. The implementation follows: Instantiation To perform an Update operation for a Spark partition, the client code needs to instantiate an instance of this class.…
Read More With the class JdbcRecordWriteFromSparkSqlRow in place, we are pretty close to getting done with implementing the Update and Insert facilities. The majority of the remaining work, as you’ll see soon, is about constructing their SQL…
Read More