Apache NiFi is used to automate and control data flows between systems. It provides us with a web-based interface that can collect, process, and analyze data.
NiFi is known for its ability to build automated data flow pipelines between systems. It particularly supports many different types of sources and destinations such as: file systems, relational databases, non-relational databases, etc. Additionally, NiFi supports data operations like filtering, editing, adding and removing content.
Main Components
FlowFile
Represents a unit of data being processed in the flow, for example a text record, an image file, etc. It consists of 2 parts:
- Content: the actual data it represents
- Attribute: properties of the flow file (key-value)
FlowFile Processor
These are the components that perform the work in NiFi. They contain pre-built code that executes tasks in various scenarios with input and output. Processors generate FlowFiles. Processors operate in parallel with each other.
Connection
Connection plays the role of connecting between processors. Additionally, it serves as a queue containing FlowFiles that haven’t been processed yet:
- Determines how long FlowFiles exist in the queue
- Distributes FlowFiles to nodes in the cluster (load balancing)
- Determines the frequency at which FlowFiles are released to the system
System Architecture
- Web server: provides interface for users to perform operations
- Flow Controller: provides resources for system operation processes
- Extensions: Includes components that build data flows in NiFi: processors responsible for processing and routing; Various logs, Controller services containing functions used by other extensions
- FlowFile repository: Only stores FlowFile metadata since FlowFiles already store data
- Content repository: Stores actual data being processed in the flow. NiFi saves all data versions before and after processing
- Provenance repository: Stores complete history of FlowFiles
Notable Features of NiFi
Data Source Management Capabilities
- Ensures security: Each data unit in the flow is stored as an object called FlowFile. It records information about where the data block is being processed, where it’s moving to, etc. Provenance Repo is used to store FlowFiles helping us trace them
- Data Buffering: Solves the problem of data write speed being faster than data read speed between two processing blocks. This data will be stored in RAM, and if it exceeds a threshold, it will be saved to hard disk
- Priority setting: In data processing, there is data that must be processed with priority
- Trading off speed and fault tolerance: NiFi supports configuration to balance these two factors
Easy to Use
- NiFi supports UI for building data flows
- Reusability when we can save data flows as templates
- Visual history monitoring
Horizontal Scaling
Suppose NiFi running on just 1 server can’t handle the processing, what do we do -> run the same dataflow on multiple servers. Administrators don’t need to modify all dataflows on different servers, they just need to make changes once and it will automatically replicate to other servers.
NiFi cluster follows the Zero-master principle, meaning all servers have the same job but process different data. One Node will be selected as cluster manager (Cluster Coordinator) through Zookeeper. All nodes in the cluster will send status to this node, and this node is responsible for disconnecting nodes that don’t send status within a certain time period. Additionally, this Node will be responsible for new nodes joining the cluster. New nodes must connect to this node first for data to be updated.