Skip to main content

Hi team,

I am doing data drift lab exercise in streamset academy. In that i am  following the step as they are mentioned in the video , in that video it is displaying as 54 records consuming from that kafka but while i am doing the same as they mentioned but i am able to consume only 4 records(twitter data) 

And also not able see the alert messages. could anyone help me through this please. 

 

@vasanthi, please export your pipeline and share.

Hi @saleempothiwala  this is my pipelines

@vasanthi 

 

I had a look at both the pipelines. Some things i noted:

  1. First pipeline is reading from a table twitter_profile. You need to run this pipeline and load some data to Kafka
  2. Make changes to number of fields or change one of the field names in the table twitter_profile.
  3. Add new data to the twitter_profile table
  4. Run Pipeline 1 again to add new data to kafka topic
  5. Make sure you enable both the rules in Data Drift. I can only see 1 rule for name change being enabled.
  6. Run your second pipeline. It should then alert you that number of fields have changed and also that one of the column names have changed.

The alerts can be triggered easily by using a Dev Data Generator as the source in Pipeline 1.

I hope it helps.

Thank you @saleempothiwala it’s work fine.

 

Reply


Terms & Conditions