Question

Global variable to reduce memory usage and API calls

  • 30 July 2022
  • 1 reply
  • 94 views
sperandio
Rank

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all  previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).

With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. 

Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record? 

1 reply

Dimas Cabré i Chacón
Rank
Userlevel 2
Badge

With your approach, one reasonable solution would be using option Multiple Values Behavior with setting Split into multiple records and then connect to an Stream Selector that would send to Trash unmatched records (that is records whose product id is not the product id of the expanded attributes. As far as your pipeline is reasonably fast and your batches not very big, you would be reclaiming wasted memory very quickly, thus dodging memory issues.

Reply


Terms & Conditions