Furthermore (obviously) this user should have access to the Kafka topic.įigure 3: Details of the Kerberos Credential Service Note that nifi should have read and execute permissions over the keytab file, and that the Keytab location should correspond to an existing path in the NiFi node you are using. Kerberos Principal: Kerberos Keytab: /opt/cloudera/security/keytabs/valeriodimatteo.keytab.
Of course, you should adapt these values to the identity you decide to use. In our case, we run it with the credentials listed below (they do not necessarily have to be the same as the ones running the producer).
The Kerberos Credentials include the Keytab and the Principal details. NiFi essentially needs to somehow know the same when starting this processor, hence the need to set these parameters.įigure 2: Details of the ConsumeKafka processor with Kerberos settings Notice how we are specifying the same information that we had to pass to the kafka-console-producer in the previous section, in the client.properties and nf files. Kerberos Credential Service: A new one which we create for this purpose, specifying the Kerberos keytab location in the NiFi node(s) and the principal name as which we want to run the producer.Kerberos Service Name: kafka, unless you named the Kafka service account differently.In order to run our flow in a kerberized environment, we need to change a few properties related to the Security Protocol and the Kerberos credentials: You also need to set a group ID, even though it is not relevant for our specific case. It asks for the broker and the topic name. This processor is required to listen to the Kafka topic in which our producer is sending log messages. In order to fulfil Kerberos requirements, we are going to modify the first and last processors: ConsumeKafka and PutHiveStr eaming. Write the record to a pre-existing Hive tableįigure 1: Complete view of the NiFi pipeline.Infer the CSV schema and convert it to an Avro one.Listen to the Kafka topic and consume its records.We will start off with the same NiFi pipeline we built in our previous article. More information about these settings can be found here. # tail -f simulated.log 2> /dev/null | kafka-console-producer -broker-list :9092 -topic logsTopic nfig client.properties # export KAFKA_OPTS=".config=/home/your_user/kafka_example/nf" To create a keytab, we run the ktutil utility.
#Get file path filter to filter csv nifi password
In our previous article, we were able to run Kafka producers without worrying about credentials. These are not optimal settings for production environments, and are only valid for development/testing Our cluster has 1 Zookeper node and 1 Kafka broker node.To run this example, you need a Kerberized CDH cluster with NiFi.
#Get file path filter to filter csv nifi how to
Let’s have a look at how to set these properties correctly. Similarly, Kafka producers require Kerberos information to be passed via certain configuration files. NiFi’s processors, of course, allow for Kerberos credentials to be specified but the details of the process are not always clear to everybody. Kerberized clusters require keytabs to be created and distributed in order for service accounts and user accounts to run jobs. If you are running a cluster in an enterprise, or if you are managing it for a client of yours, chances are that it is secured and integrated with Kerberos. This time, we will go one step further and show how to adapt that pipeline to a Kerberized environment. In our previous article we showed how to set up a streaming pipeline to write records to Hive in real-time, using Kafka and NiFi.
0 kommentar(er)
