Match and update incoming transactions in a Postgres database (JDBC, Postgres, AWS, Kafka)

  • 17 March 2022
  • 0 replies
  • 39 views

 

Streaming data pipeline that potentially has updates to transactions. Look up for matching records in the database (postgres), filter transaction using a condition to determine if transactions are updates vs new.  For updates, mask sensitive data and writes them to postgres db table.  New transaction are written to an AWS bucket and unprocessed transactions (pipeline errors) are written to a kafka destination for further analysis.


0 replies

Be the first to reply!

Reply