Skip to content

Data in the New Age

My thoughts on data engineering and related topics

  • Home
  • About
  • Blog
  • Contact

The End is Near – Preface (From the co-writers, GPT-4 and Dzung Doan, to the Readers)

  • Written byadmin
  • Posted on May 10, 2023May 10, 2023

“If you’re not embarrassed by the first version of your product, you’ve released too late” – Reid Hoffman   Following this philosophy for a startup’s minimum viable product by Reid Hoffman, the renowned entrepreneur and…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 23: Appendix

  • Written byadmin
  • Posted on November 2, 2018November 3, 2018

I might or might not push the code to GitHub. Let me see. If I do, until then, here’s the listing of the essence of the code referenced in this series. They are generally heavily…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 22: Conclusion

  • Written byadmin
  • Posted on November 2, 2018November 5, 2018

Our long and fun journey now comes to the end. About Spark Apache Spark, as an analytic processing framework, is designed to process data in bulk. Particularly, when it comes to writing to an RDBMS, the…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 21: The Grand Finale – Write Operations with Failure Handling

  • Written byadmin
  • Posted on October 25, 2018November 2, 2018

With the Failure Handling facility in place, we are now ready to fully complete the case study. As you’ll see, it’s pretty straightforward and elegant. The Update Operation Failure Handling when Updating the target record with…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 20: The Failure Handling Facility

  • Written byadmin
  • Posted on October 23, 2018October 26, 2018

Yay, we are done! Finally, it’s over!!! Oh, wait! Wait for a second! Don’t celebrate too prematurely! We’re not done yet! Just close! Look, what about the Failure Reporting Requirement? We haven’t done anything about…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 19: The Insert Operation

  • Written byadmin
  • Posted on October 20, 2018October 25, 2018

Now, we move on to implement the Insert operation. As you’ll see, it’s pretty similar to the Update operation laid out in the last post. Insert target record with values purely obtained from SparkSQL row…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 18: The Update Operation

  • Written byadmin
  • Posted on October 18, 2018October 20, 2018

With both the Update and Insert facilities now in place, we are ready to implement the Update and Insert operations. We tackle the Update operation first in this post. Update target record with values purely obtained…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 17: The Insert Facility

  • Written byadmin
  • Posted on October 16, 2018October 20, 2018

Class JdbcRecordInsertFromSparkSqlRow We are now ready to move on to implement the Insert facility. It’s similar to the Update one. In fact, slightly simpler. Here it is: Instantiation To perform an insert operation for a…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 16: The Update Facility

  • Written byadmin
  • Posted on October 13, 2018October 20, 2018

Class JdbcRecordUpdateFromSparkSqlRow Now it’s time to work on the Update facility. The implementation follows: Instantiation To perform an Update operation for a Spark partition, the client code needs to instantiate an instance of this class.…

Read More

Apache Spark’s fine-grained processing of RDBMS records part 15: Literal values in Sql string template

  • Written byadmin
  • Posted on October 11, 2018October 16, 2018

With the class JdbcRecordWriteFromSparkSqlRow in place, we are pretty close to getting done with implementing the Update and Insert facilities. The majority of the remaining work, as you’ll see soon, is about constructing their SQL…

Read More

Recent Posts

  • Agent <-> MCP Server interaction
  • WC AI Assistant (beta): Agent, MCP, Vibe Coding, AI Engineering
  • WC AI Assistant (beta): From the Product / User Experience perspective
  • “Toward World Cup 2026” Book Series – First Book: Lamine Yamal
  • World Cup AI Assistant

Recent Comments

  • World Cup AI Assistant - Data in the New Age on WC AI Assistant (beta): Agent, MCP, Vibe Coding, AI Engineering
  • David Eristavi on The End is Near – Eps 2: The Return of von Manstein

Archives

  • October 2025
  • September 2025
  • August 2025
  • July 2025
  • April 2024
  • March 2024
  • June 2023
  • May 2023
  • November 2018
  • October 2018
  • September 2018

Categories

  • AI (Engineering,…)
  • Data Engineering
  • My Books
  • My Novels

Meta

  • Register
  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Posts pagination

«Previous Posts 1 2 3 4 5 6 Next Posts»
  • LinkedIn
Copyright © 2026 . All rights reserved.
Blogg theme designed by Blogging Theme Styles