My thoughts on data engineering and related topics
The Problem As alluded in Approach IV post, that implementation would not work in real production data. What’s going on? Now’s the time to deal with it. Take a look at this key line of…
Read More This is a bonus post. It chronicles an activity that doesn’t generate a concrete outcome, yet reflects the considerations emerging at this point in an engineering project. Motivation Now approach IV is chosen. Approach III…
Read More Approach III, although the best so far, still doesn’t quite make it. Hence, we now move on to this next one. Summary Each row in the updateDF dataframe gets extracted individually, and explicitly updates the…
Read More The first two approaches don’t cut it. We now move on to the third one, in search of a solution that can reasonably perform the Update operation. Summary We dump the updateDF dataframe to a…
Read More We now explore the second approach using the Update operation as the first test. Following is its examination. Summary In a nutshell, this approach writes update data to a temp table, then uses database trigger…
Read More We now start exploring different approaches with the Update operation as the first evaluation criterion. Following is the first potential solution. Summary This first approach is the most naive, which relies most heavily on available…
Read More So far, we have formulated the problem we are facing: how to have Spark write data from a dataframe to a relational table, such that this operation satisfies a set of requirements? Look at it…
Read More Given the background of the case study explained in details in the previous post, now we focus on the requirements of the LOAD step of the SparkSQL jobs in the pipeline 1. The 5 Requirements…
Read More This is a long post. Reading through and getting a good understanding of it will help you follow closely the progressive development in future posts. However, the essence, the takeaway, as alluded in the Introduction…
Read More At a high level, in a typical enterprise data infrastructure, there usually exists at least two relatively separate systems: a transactional system (OLTP), and an analytical system (OLAP, Data Warehouse). An ETL data pipeline would…
Read More