I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).
With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that.
Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record?