MongoDB Output

Description

The MongoDB Output transform can output data to a MongoDB database collection.

For additional information about MongoDB, see the MongoDB documentation.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

Options

General

Transform name : Specify the unique name of the MongoDB Output transform in the pipeline.

Output options tab

The Output options tab provides controls for inserting data into a MongoDB collection. If the specified collection does not exist, it is created before a document is inserted.

Enter the following information in the fields on this tab:

Option	Description
MongoDB Connection	the MongoDB connection to use for this MongoDB Output transform.
Collection	Specify the target collection for the output. When a valid hostname and port has been set, you can click Get Collections to retrieve the names of existing collections within a selected database. If the specified collection does not exist, it will be created before data is inserted.
Batch insert size	Specify the batch size for bulk insert operations. The default value is 100 rows.
Truncate collection	Select to delete existing data in the target collection before inserting new data.
Update	Sets the update write method for the specified database and collection. The Upsert and Modifier update options are not available unless the Update field is selected.
Upsert	Select to change the write method from insert to upsert. The upsert method replaces a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. A new record is created if match conditions fail for an update.
Multi-update	Select to update all matching documents for each update or upsert operation.
Modifier update	Select to enable modifiers ($ operators) to be used to modify individual fields within matching documents. All matching documents are updated when the Multi-update option is selected. To update more than one matching document, select Modifier update and Upsert. Selecting Modifier update, Upsert, and Multi-update applies updates to all matching documents, instead of just the first.
Number of retries for write operations	Specify the number of times that a write operation is attempted.
Delay, in seconds, between retry attempts	Specify the number of seconds to wait before the next retry.

Option

Description

MongoDB Connection

the MongoDB connection to use for this MongoDB Output transform.

Collection

Specify the target collection for the output. When a valid hostname and port has been set, you can click Get Collections to retrieve the names of existing collections within a selected database. If the specified collection does not exist, it will be created before data is inserted.

Batch insert size

Specify the batch size for bulk insert operations. The default value is 100 rows.

Truncate collection

Select to delete existing data in the target collection before inserting new data.

Update

Sets the update write method for the specified database and collection.

The Upsert and Modifier update options are not available unless the Update field is selected.

Upsert

Select to change the write method from insert to upsert. The upsert method replaces a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. A new record is created if match conditions fail for an update.

Multi-update

Select to update all matching documents for each update or upsert operation.

Modifier update

Select to enable modifiers ($ operators) to be used to modify individual fields within matching documents. All matching documents are updated when the Multi-update option is selected.

To update more than one matching document, select Modifier update and Upsert. Selecting Modifier update, Upsert, and Multi-update applies updates to all matching documents, instead of just the first.

Number of retries for write operations

Specify the number of times that a write operation is attempted.

Delay, in seconds, between retry attempts

Specify the number of seconds to wait before the next retry.

Mongo document fields tab

Use the Mongo document fields tab to define how field values coming into the transform are written to a Mongo document. The Modifier policy column controls when the execution of a modifier operation affects a particular field. You can use modifier policies when the data for one Mongo document is split over several incoming Hop rows or when it is not possible to execute different modifier operations that affect the same field simultaneously.

There are 2 helper buttons you can use: * Get fields :Populates the Name column of the table with the names of the incoming fields. * Preview document structure : Opens a dialog showing the structure that will be written to MongoDB in JSON format.

Enter the following information in the fields on this tab:

Column Field Description

Column	Field Description
Name	Names of the incoming fields.
Mongo document path	The hierarchical path to fields in a document in dot notation format.
Use field name	Whether to use the incoming field name as the final entry in the path. The values are Y (use incoming field names) and N (do not use incoming field names). When set to Y, a preceding period (.) is assumed.
NULL values	Specifies whether to insert null values in the database. The values are: - Insert NULL - Ignore
JSON	Indicates the incoming value is a JSON document.
Match field for update	Indicates whether to match a field when performing an upsert operation. The first document in the collection that matches all fields tagged as Y in this column is replaced with the new document constructed with incoming values for all the defined field paths. If a matching document does not exist, then a new document is inserted into the collection.
Modifier operation	Specify in-place modifications of existing document fields. The modifiers are: - N/A - `$set` : Sets the value of a field. - `$inc` : Sets the value of a field if the field does not exist. If the field exists, increases (or decreases, with a negative value) the value of a field. - `$push` : Sets the value of a field if the field does not exist. If the field exists, appends the value of a field. - `$` : (the positional operator for matching inside of arrays).
Modifier policy	Controls when execution of a modifier operation affects a field. The values are: - `Insert&Update` : The operation is executed whether or not a match exists in the collection (default). The Insert&Update modifier policy (upsert) allows you to specify fields to match when performing an upsert operation. Upsert only replaces the first matching document. Modifier upserts can be used to replace certain field values in multiple documents. - `Insert` : The operation is executed on an insert only (when the match conditions fail) - `Update` : The operation is executed when match conditions succeed.

Name

Names of the incoming fields.

Mongo document path

The hierarchical path to fields in a document in dot notation format.

Use field name

Whether to use the incoming field name as the final entry in the path. The values are Y (use incoming field names) and N (do not use incoming field names). When set to Y, a preceding period (.) is assumed.

NULL values

Specifies whether to insert null values in the database. The values are:

- Insert NULL

- Ignore

JSON

Indicates the incoming value is a JSON document.

Match field for update

Indicates whether to match a field when performing an upsert operation. The first document in the collection that matches all fields tagged as Y in this column is replaced with the new document constructed with incoming values for all the defined field paths. If a matching document does not exist, then a new document is inserted into the collection.

Modifier operation

Specify in-place modifications of existing document fields.

The modifiers are:

- N/A

- $set : Sets the value of a field.

- $inc : Sets the value of a field if the field does not exist. If the field exists, increases (or decreases, with a negative value) the value of a field.

- $push : Sets the value of a field if the field does not exist. If the field exists, appends the value of a field.

- $ : (the positional operator for matching inside of arrays).

Modifier policy

Controls when execution of a modifier operation affects a field. The values are:

- Insert&Update : The operation is executed whether or not a match exists in the collection (default). The Insert&Update modifier policy (upsert) allows you to specify fields to match when performing an upsert operation. Upsert only replaces the first matching document. Modifier upserts can be used to replace certain field values in multiple documents.

- Insert : The operation is executed on an insert only (when the match conditions fail)

- Update : The operation is executed when match conditions succeed.

Example

Here is an example of how you can define a document structure with an arbitrary hierarchy. Use the following input data and document field definitions to create the example document structure in MongoDB:

Input data

first, last, address, age
Bob, Jones ,"13 Bob Street", 34
Fred, Flintstone, "10 Rock Street",50
Zaphod, Beeblebrox, "Beetlejuice 1", 356
Noddy,Puppet,"Noddy Land",5

Document field definitions

Name	Mongo document path	Use field name	JSON	Match field for update	Modifier operation	Modifier policy
first	top1	Y	N	N	N/A	Insert&Update
last	array[O]	Y	N	N	N/A	Insert&Update
address	array[O]	Y	N	N	N/A	Insert&Update
age	array[O]	Y	N	N	N/A	Insert&Update

Name

Mongo document path

Use field name

NULL values

JSON

Match field for update

Modifier operation

Modifier policy

first

top1

N/A

Insert&Update

last

array[O]

N/A

Insert&Update

address

array[O]

N/A

Insert&Update

age

array[O]

N/A

Insert&Update

Document structure

{
  "top1" : {
    "first" : "<string val>"
   },
  "array" : [ { "last" : "<string val>" , "address" : "<string val>"}],
  "age" : "<integer val>"
}

Create/drop indexes tab

Use the Create/drop indexes tab to create and drop indexes on one or more fields. Unless unique indexes are being used, MongoDB allows duplicate records to be inserted. Indexing is performed after all rows have been processed by the transform.

You can use the Show indexes button to display a list of existing indexes.

Enter the following information in the fields in this tab:

Field	Description
Index fields	Specify a single index (using one field) or a compound index (using multiple fields). Compound indexes are specified by a comma-separated list of paths. Use dot notation to specify the path to a field to use in the index. An optional direction indicator can be specified: 1 for ascending or -1 for descending.
Index opp	Specify whether to create or drop an index.
Unique	Specify whether to index only fields with unique values.
Sparse	Specify whether to index only documents that have the indexed field.

Field

Description

Index fields

Specify a single index (using one field) or a compound index (using multiple fields). Compound indexes are specified by a comma-separated list of paths. Use dot notation to specify the path to a field to use in the index. An optional direction indicator can be specified: 1 for ascending or -1 for descending.

Index opp

Specify whether to create or drop an index.

Unique

Specify whether to index only fields with unique values.

Sparse

Specify whether to index only documents that have the indexed field.

Create/drop indexes example

The following options defines the creation of a compound index of the "first" and "age" fields in ascending order:

Index fields	Index opp	Unique	Sparse
top1.first,age	Create	N	N

Index fields

Index opp

Unique

Sparse

top1.first,age

Create