Skip to main content

I have a requirement to write jobs logs to snowflake database table with jobs metrics data like job start time, end time, offset committed, no.of rows processed and errors if any etc.. Can you please help how to achieve this?

@pradeep2420 where is all that information stored? Any source should be able to write to Snowflake without issues

 

Hi Alex,

Thanks for the quick response, I think log metadata information is stored in data collector server which we hosted in AWS EC2 instance. How do I access this metadata through pipeline?

Hi Pradeep,

I don’t think this is something we are doing, and I don’t think it is a straight way to do this, of course you can use Directory Origin to read those files… but I don’t think it is a good idea since it might break something due to files locked and at the end those files might have way more information that what you need.

And since they are “internal” files, it might not be safe to play with them.

Reply


Terms & Conditions