Changes to your DataFlow
All of the nodes in a NiFi cluster run the same data flow. Any changes that you make to your data flow on one node are replicated on the other nodes.
TIP: Although a Connector appears in the data flow on all nodes, it only runs on a single node. The node is chosen automatically by NiFi. If that node becomes unavailable, another node takes over. Subsequent processors, such as KeyView Extraction and Filtering, run simultaneously on all of the nodes in your cluster.
The FlowFiles that a connector produces need to be distributed across all of the nodes in the cluster. This does not happen automatically. OpenText recommends that you configure the output connection from the connector and set the Load Balance Strategy to Round Robin.
-
Right-click the output queue (success relationship) for the connector and click Configure.
The Edit Connection dialog box opens.
- Click the Settings tab.
- In the Load Balance Strategy list, click Round Robin.
To run a NiFi cluster, you must use an external database for storing state information. Many NiFi Ingest processors need to store state information. For example, Knowledge Discovery Connectors store information about what they have retrieved from a data repository. This information needs to be in an external database so that it is accessible to all of the nodes in the cluster. Configure the connection to your database server by creating a database service (see Create a Database Service). When you configure the connectors in your data flow, set the property State Database Service to the name of the database service that you created.
The files that your connectors download from your data repositories must also be accessible to all of the nodes in a cluster. When you configure your connectors, set the property adv:IngestSharedPath
to a location, such as a shared folder, that is accessible from all of the nodes in the cluster. Alternatively, set the property adv:FlowFileEmbedFiles
to TRUE
, so that the binary file content is included in the FlowFiles created by the connector. For more information about these properties, see Advanced Connector Properties.