My thoughts on data engineering and related topics
So far, we have formulated the problem we are facing: how to have Spark write data from a dataframe to a relational table, such that this operation satisfies a set of requirements? Look at it…
Read More Given the background of the case study explained in details in the previous post, now we focus on the requirements of the LOAD step of the SparkSQL jobs in the pipeline 1. The 5 Requirements…
Read More This is a long post. Reading through and getting a good understanding of it will help you follow closely the progressive development in future posts. However, the essence, the takeaway, as alluded in the Introduction…
Read More At a high level, in a typical enterprise data infrastructure, there usually exists at least two relatively separate systems: a transactional system (OLTP), and an analytical system (OLAP, Data Warehouse). An ETL data pipeline would…
Read More I launch my blog with this fairly long post series (expected 20+ posts). I hope to write and release it gradually over the next few weeks. It might be considered of a case study discussion,…
Read More