Use Lightingest.exe to push data in to Azure Data explorer table and perform the authentication using Application Id and secret.

Problem Statement:

I was looking for the exact command through which I can push data in to Azure Data explorer table using Lightingest.exe, however this took sometime and it seems that finding the exact documentation could be little tricky.

What is LightIngest.exe?

LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. The utility can pull source data from a local folder or from an Azure blob storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration.

Goal:

Here the end goal is to read the content of my  "test.json.gz"  file, which is kept in "C:\test" folder of my local machine and then write it to the kusto "TestTable" table, I am jotting down all the details below.

Kusto cluster name:        https://xyz.kusto.windows.net

Kusto Database name:    KustoDB

Kusto table name:           TestTable

Application Id:                bbbbb282-27ra-82ff-cgf0-5sx6444xd76l

Application secret:          xxxxxxxxxxxxxxxxxxxxxxxx

Tenant Id:                        854f013f-7f24-23e8-agf45-cds56732452

Solution:

You can run the below command, so you can use Lightingest.exe to push the code and also here the authentication will happen using the Application Id and not using your local credentials hence you can use the same code in Azure function or any other cloud offering.

<<Folder which has Lightingest.exe>>\LightIngest.exe "Data Source=https:///xyz.kusto.windows.net;Initial Catalog=KustoDB;Application Client Id=bbbbb282-27ra-82ff-cgf0-5sx6444xd76l;Application Key=xxxxxxxxxxxxxxxxxxxxxxxx;AAD Federated Security=True;dSTS Federated Security=False;Authority Id=854f013f-7f24-23e8-agf45-cds56732452" -table:TestTable -source:"C:\test" -pattern:"*.json.gz" -format:json

PS: LightIngest - download it as part of the Microsoft.Azure.Kusto.Tools NuGet package