How to Enhance Data Pipeline with Two-Way Communication in NiFi Based Data Integration Platform
Data Integration Platforms, particularly those based on NiFi, offer two-way communication between a variety of data sources. This feature allows for a robust and dynamic data pipeline that can cater to a wide range of data integration needs.
Data Source Categories
NiFi based Data Integration Platforms can connect to a multitude of data sources. Here are a few categories of data sources that can be connected:
- RDBMS: MS SQL Server, MySQL, PostgreSQL, Google Cloud SQL, Redshift, etc.
- File Formats: CSV, XML, JSON, Parquet, Excel, etc.
- Big Data: HDFS, Hive, Spark SQL, Flume, etc.
- Rest API
- Social Feeds: Facebook, Twitter, Instagram, LinkedIn, etc.
- No SQL: Cassandra, Mongo DB, Elasticsearch, etc.
- Cloud Services: AWS, Google Cloud, and Azure.
- Online File Systems: FTP, SFTP
- Messaging systems and protocols: Email, Kafka, AMQP, JMS, MQTT, IMAP, POP3, etc.
Moreover, the platform supports over 250 individual connectors. In addition to this we can also add our own custom connector
Data Processors
Data Integration Platforms have processors that can read, write, and modify data in many data sources. Here are a few examples:
-
Data reading processors: GetFile, GetKafka, GetHDFS, ConsumeKafka, GetHTTP, GetSharepoint, GetFTP, GetMongo, GetRethinkDB, FetchGCSObject.
-
Data writing processors: PutFile, PutKafka, PutHDFS, PutFTP, PutKudu, PutLambda, PutMicrosoftTeams, PutS3Object, PutSQL, PutSlack.
-
Data modifying processors: TransformXml, UpdateRecord, UpdateSalesforceRecord, ReplaceText, ConvertRecord, DeleteS3Object, ExcelConverter, UpdateAttribute, ResizeImage.
Additional Features
Data Integration Platforms expose the REST API for all of its in-built features. This allows for control over any of the platform’s operations in an external (custom) application with the necessary authentication.
If your use case differs from the features mentioned above, please provide more details about your expectations for two-way communication in Data Integration Platforms.
References
