I am creating a simple pipeline to read all from mongo collection and store into different collection,
But my mongo DB origin is able to read only one record (even through i have 33 records in source) - what is the issue with my configuration ?
Userlevel 5
+1
- Headliner
- 477 replies
-
15 March 2023
Please configure the event as given below and check if it helps.
Add pipeline finisher in origin by adding pre-condition as given below.
${record:eventType() == 'no-more-data'}
Event Generation
The MongoDB origin can generate events when it completes processing all available data and the configured batch wait time has elapsed.
MongoDB origin events can be used in any logical way. For example:
- With the Pipeline Finisher executor to stop the pipeline and transition the pipeline to a Finished state when the origin completes processing available data.
When you restart a pipeline stopped by the Pipeline Finisher executor, the origin continues processing from the last-saved offset unless you reset the origin.
For an example, see Stopping a Pipeline After Processing All Available Data.
- With a destination to store event information.
For an example, see Preserving an Audit Trail of Events.
For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Event Records
Event records generated by the MongoDB origin have the following event-related record header attributes. Record header attributes are stored as String values:
| Record Header Attribute | Description |
|---|---|
| sdc.event.type | Event type. Uses the following event type:
|
| sdc.event.version | Integer that indicates the version of the event record type. |
| sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
The MongoDB origin can generate the following event record:
no-more-data
The MongoDB origin generates a no-more-data event record when the origin completes processing all available records and the number of seconds configured for Max Batch Wait Time elapses without any new objects appearing to be processed.
If no data is available for processing when the pipeline starts, the origin generates the record only after creating at least one record for the pipeline.
No-more-data event records generated by the origin have the sdc.event.type set to no-more-data and include the following fields:
| Event Record Field | Description |
|---|---|
| record-count | Number of records successfully generated since the pipeline started or since the last no-more-data event was created. |
| error-count | Number of error records generated since the pipeline started or since the last no-more-data event was created. |
- Fan
- 1 reply
-
15 March 2023
Thanks for the response.
Issue with initial offset property - I am able to read all records from source collection now.
I am stuck in next step, creating CRUD operation for mongo db destination.
Pls share some example related to Mongo DB CRUD operation.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.