Skip to main content

I am creating a simple pipeline to read all from mongo collection and store into different collection,

But my mongo DB origin is able to read only one record (even through i have 33 records in source) - what is the issue with my configuration ?

 

 

@raj 

Please configure the event as given below and check if it helps.

Add pipeline finisher in origin by adding pre-condition as given below.

 

${record:eventType() == 'no-more-data'}

 

 

Event Generation

The MongoDB origin can generate events when it completes processing all available data and the configured batch wait time has elapsed.

MongoDB origin events can be used in any logical way. For example:

  • With the Pipeline Finisher executor to stop the pipeline and transition the pipeline to a Finished state when the origin completes processing available data.

    When you restart a pipeline stopped by the Pipeline Finisher executor, the origin continues processing from the last-saved offset unless you reset the origin.

    For an example, see Stopping a Pipeline After Processing All Available Data.

  • With a destination to store event information.

    For an example, see Preserving an Audit Trail of Events.

For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.

Event Records

Event records generated by the MongoDB origin have the following event-related record header attributes. Record header attributes are stored as String values:

Record Header Attribute Description
sdc.event.type Event type. Uses the following event type:
  • no-more-data - Generated after the origin completes processing all available objects and the number of seconds configured for Max Batch Wait Time has elapsed.

    The origin must create at least one record for the pipeline before generating the event record.
sdc.event.version Integer that indicates the version of the event record type.
sdc.event.creation_timestamp Epoch timestamp when the stage created the event.

The MongoDB origin can generate the following event record:

no-more-data

The MongoDB origin generates a no-more-data event record when the origin completes processing all available records and the number of seconds configured for Max Batch Wait Time elapses without any new objects appearing to be processed.

If no data is available for processing when the pipeline starts, the origin generates the record only after creating at least one record for the pipeline.

No-more-data event records generated by the origin have the sdc.event.type set to no-more-data and include the following fields:

Event Record Field Description
record-count Number of records successfully generated since the pipeline started or since the last no-more-data event was created.
error-count Number of error records generated since the pipeline started or since the last no-more-data event was created.

Thanks for the response.

Issue with initial offset property - I am able to read all records from source collection now.

I am stuck in next step, creating CRUD operation for mongo db destination.

Pls share some example related to Mongo DB CRUD operation. 

Reply


Terms & Conditions