Use Lightingest.exe to push data in to Azure Data explorer table and perform the authentication using Application Id and secret.

Problem Statement:

I was looking for the exact command through which I can push data in to Azure Data explorer table using Lightingest.exe, however this took sometime and it seems that finding the exact documentation could be little tricky.

What is LightIngest.exe?

LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. The utility can pull source data from a local folder or from an Azure blob storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration.

Goal:

Here the end goal is to read the content of my  "test.json.gz"  file, which is kept in "C:\test" folder of my local machine and then write it to the kusto "TestTable" table, I am jotting down all the details below.

Kusto cluster name:        https://xyz.kusto.windows.net

Kusto Database name:    KustoDB

Kusto table name:           TestTable

Application Id:                bbbbb282-27ra-82ff-cgf0-5sx6444xd76l

Application secret:          xxxxxxxxxxxxxxxxxxxxxxxx

Tenant Id:                        854f013f-7f24-23e8-agf45-cds56732452

Solution:

You can run the below command, so you can use Lightingest.exe to push the code and also here the authentication will happen using the Application Id and not using your local credentials hence you can use the same code in Azure function or any other cloud offering.

<<Folder which has Lightingest.exe>>\LightIngest.exe "Data Source=https:///xyz.kusto.windows.net;Initial Catalog=KustoDB;Application Client Id=bbbbb282-27ra-82ff-cgf0-5sx6444xd76l;Application Key=xxxxxxxxxxxxxxxxxxxxxxxx;AAD Federated Security=True;dSTS Federated Security=False;Authority Id=854f013f-7f24-23e8-agf45-cds56732452" -table:TestTable -source:"C:\test" -pattern:"*.json.gz" -format:json

PS: LightIngest - download it as part of the Microsoft.Azure.Kusto.Tools NuGet package

Querying a Kusto table using PowerShell Core 7.0 fails and throws error

Issue: 

kusto table query using PowerShell Core 7.0 fails when hosted in Azure function, the Code works fine when executed in local machine with PowerShell version 5.* however the same code doesn't work in my local machine when I change the PowerShell version from 5.* to 7.0 and Azure function (which is also running PS version 7.0). PowerShell .net client library is downloaded from this location https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/powershell/powershell

I am trying AAD authentication using secret, below is the sample code and the error I am getting when executing it in Azure Function with PowerShell core version 7.0

Sample Code

$packagesRoot = "$PSScriptRoot\tools\netcoreapp2.1"[System.Reflection.Assembly]::LoadFrom("$packagesRoot\Kusto.Data.dll")
$clusterUrl = "https://cluster.kusto.windows.net;Fed=True"
$databaseName = "DatabaseName"
$kcsb1 = New-Object Kusto.Data.KustoConnectionStringBuilder ($clusterUrl, $databaseName)

#Using Azure AD application Authentication
$applicationId = "ClientId_here"
$applicationKey = "ClientSecret_here"
$authority = "TenantId_here"
$kcsb = $kcsb1.WithAadApplicationKeyAuthentication($applicationId, $applicationKey, $authority)
$queryProvider = [Kusto.Data.Net.Client.KustoClientFactory]::CreateCslQueryProvider($kcsb)
$query = 'KustoQuery_here'
$crp = New-Object Kusto.Data.Common.ClientRequestProperties
$crp.ClientRequestId = "MyPowershellScript.ExecuteQuery." + [Guid]::NewGuid().ToString()
$crp.SetOption([Kusto.Data.Common.ClientRequestProperties]::OptionServerTimeout, [TimeSpan]::FromSeconds(30))

#Executing the query
$reader = $queryProvider.ExecuteQuery($query, $crp)

Error received when executing the code in PowerShell Core 7.0:

PS C:\Program Files\PowerShell\7>  C:\Users\Administrator\Desktop\Kusto7.0\TestKusto.ps1
Please make sure you have the .Net Kusto libraries downloaded at C:\Users\Administrator\Desktop\Kusto7.0
Location is C:\Users\Administrator\Desktop\Kusto7.0\tools\netcoreapp2.1

GAC    Version        Location
---    -------        --------
False  v4.0.30319     C:\Users\Administrator\Desktop\Kusto7.0\tools\netcoreapp2.1\Kusto.Data.dll
MethodInvocationException: C:\Users\Administrator\Desktop\Kusto7.0\TestKusto.ps1:53
Line |
  53 |  $queryProvider = [Kusto.Data.Net.Client.KustoClientFactory]::CreateCs …
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "CreateCslQueryProvider" with "1" argument(s): "The type initializer for
     | 'Kusto.Data.Common.KustoTrustedEndpoints' threw an exception."

Solution:

Found that the issue is happening only on PowerShell Core 7.0, I changed the PowerShell core version of Azure function to 7.2 and the code started working. I did install PowerShell version 7.0 and &.7.2 in my local machine and executed the same code to rule out any issues with Azure function and it showed the exact same behavior, the code executed with PowerShell Core version 7.2 in my local machine and started throwing errors on PowerShell Core 7.0, which concludes that .net kusto library has some issues with PowerShell Core version 7.0.

Windows Defender Credential Guard.

What is Credential Guard

Credential Guard is a virtualization-based isolation technology which prevents attackers from stealing credentials that could be used for pass the hash attacks, Attackers often attempt to extract any stored credentials. A prime target is the LSASS.exe process, which stores NTLM and Kerberos credentials. Credential Guard prevents attackers from dumping credentials stored in LSASS by running LSASS in a virtualized based container that even a user with highest privileges cannot access,Thus preventing the stealing of the credentials.

Credential Guard Requirements (Software and Hardware)

To provide basic protections against OS level attempts to read Credential Manager domain credentials, NTLM and Kerberos derived credentials, Windows Defender Credential Guard uses:

• Windows 10 ,Windows 2016 or above.

• Support for Virtualization-based security (required)

• Secure boot (required)

• TPM 1.2 or 2.0 (preferred - provides binding to hardware), either discrete or firmware

• UEFI lock (preferred - prevents attacker from disabling with a simple registry key change)

The Virtualization-based security requires:

• 64-bit CPU

• CPU virtualization extensions plus extended page tables

• Windows hypervisor (does not require Hyper-V Windows Feature to be installed)

Requirements for running Windows Defender Credential Guard in Hyper-V virtual machines

Credential Guard can protect secrets in a Hyper-V virtual machine, just as it would on a physical machine. When Credential Guard is deployed on a VM, secrets are protected from attacks inside the VM, Below are the requirement to run Credential Guard in a Hyper-V VM.

• The Hyper-V host must have an IOMMU, and run at least Windows Server 2016 or Windows 10 version 1607.

• The Hyper-V virtual machine must be Generation 2, have an enabled virtual TPM, and be running at least Windows Server 2016 or Windows 10.

Application compatibility with Credential guard

When Windows Defender Credential Guard is enabled, specific authentication capabilities are blocked, so applications that require such capabilities will break. Applications should be tested prior to deployment to ensure compatibility.

Note: Enabling Windows Defender Credential Guard on domain controllers is not supported. The domain controller hosts authentication services which integrate

with processes isolated when Windows Defender Credential Guard is enabled, causing crashes. Windows Defender Credential Guard does not provide protections

for the Active Directory database or the Security Accounts Manager (SAM).

Enable Windows Defender Credential Guard by using Group Policy

You can use Group Policy to enable Windows Defender Credential Guard.

• Open Group Policy Management Console, Edit a Group Policy object and go to Computer Configuration -> Administrative Templates -> System -> Device Guard.

• Double-click Turn On Virtualization Based Security, and then click the Enabled option.

• In the Select Platform Security Level box, choose Secure Boot or Secure Boot and DMA Protection.

• In the Credential Guard Configuration box, click Enabled with UEFI lock, and then click OK. If you want to be able to turn off Windows Defender Credential Guard remotely, choose Enabled without lock.

To enforce processing of the group policy, you can run the command gpupdate /force

Enable Credential Guard using Device Guard and Credential Guard hardware readiness tool

Device Guard and Credential Guard hardware readiness tool is a complete package which allows you to enable/disable/check the status of Credential Guard

and Device Guard. This tool can be downloaded from this link, This is a Powershell script which has the checks written in it and it does the job for you without

the need for you to manually make any changes.

Note: If Execution-Policy is not already set to allow running script, then you should manually set it as below and then use the readiness script:

Set-ExecutionPolicy Unrestricted

How to read the output of the Script:

• Red Errors: Basic hardware/firmware features are missing that will prevent enabling and using DG/CG.

• Yellow Warnings: This device is capable of running DG/CG, but some additional security qualifications are absent. To learn more, please go through: https://aka.ms/dgwhcr

• Green Messages: This device is fully compliant with DG/CG requirements.

To Verify if this device is Credential Guard Capable/Whether Credential Guard can be enabled or not, Please run the below command.

DG_Readiness.ps1 -Capable -CG

To enable only Credential Guard only, Please run the below command.

DG_Readiness.ps1 -Enable -CG

To Verify if Credential Guard is enabled, Please run the below command.

DG_Readiness.ps1 -Ready -CG

Note: It is recommended to reboot the machine after the script is run so that Credential guard related settings can be applied.

Verify that Credential Guard is running using msinfo32

You can view System Information to check that Windows Defender Credential Guard is running on a system.

• Click Start, type msinfo32 and then click System Information.
• Click System Summary.
• Confirm that Credential Guard is shown next to Virtualization-based security Services Configured.

Verify Credential guard is running using Powershell

You can verify whether Credential Guard is enabled or not by running the below Powershell command.

Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard

Value of SecurityServicesConfigured field indicates whether the Credential Guard or HVCI service has been configured.

• 0 value for SecurityServicesConfigured indicate no services configured.
• 1 value for SecurityServicesConfigured indicate Credential Guard is configured.
• 2 value for SecurityServicesConfigured indicate HVCI is configured.

Note: Presence of both the value 1 and 2 for the field SecurityServicesConfigured indicate that both Credential Guard and HVCI is enabled.

How to create Recovery database in Exchange 2016 (with Snapshot).

Hello Friends, Today I will show you how you can create a Recovery database in Exchange 2016. Recovery database can be very useful if you want to restore few emails of a user mailbox or want to restore the whole mailbox. Recovery database is a feature in Exchange 2016 which helps us to restore a mailbox from a back up copy of mailbox database. Recovery database allows us to merge the emails of a mailbox from a back up copy of a mailbox database with a mailbox which is currently residing on a mounted copy of mailbox database.

I have an user MSFT , The user had an  important email on his mailbox however the user accidentally deleted all the email on his mailbox now the user wants those emails back.

I have restored a copy of the mailbox database from my back up utility, The name of the mailbox database is mailbox database is "Mailbox Database 1978255080.edb".
I will use the restored mailbox database to restore the deleted emails for the MSFT mailbox.

Run the below command from Exchange Powershell to create a Recovery database.

New-MailboxDatabase -Recovery -Name <RDBName> -Server <ServerName> -EdbFilePath <RDBPathandFileName>

Once the Recovery database is created then restart the Microsoft Exchange Information store service. Run the below command from Exchange Powershell.

Restart-Service MSExchangeIS

We will have to mount the recovery database now however before mounting the Recovery database we will have to bring the Recovery  database in to clean shutdown so that the Recovery database can be mounted.

Run the below command to repair the Recovery database.

eseutil /p "C:\Recovery\Mailbox Database 1978255080.edb"

Note:- My database couldn't be repaired using soft recovery so I have used hard repair switch to repair the mailbox database. You can also use the soft recovery switch to repair the mailbox database.

Recovery database is in clean shutdown, We can check by running the below command.

eseutil /mh "C:\Recovery\Mailbox Database 1978255080.edb"

Run the below command to mount the Recovery database.

Mount-database <Recovery database>

Run the Below command to find out the mailboxes that resides in the Recovery database.

Get-MailboxStatistics -Database <Recovery database>

The mailbox that needs to be restored should show up. We can see that MSFT mailbox is in the Recovery database.

Run the below command to restore the mailbox with name "MSFT" from Recovery database to the user mailbox that has the email address msft@shubh.com.

New-MaiboxRestoreRequest -SourceDatabase Rec_db -SourceStoreMailbox "MSFT" -TargetMailbox msft@shubh.com

Once the command is run, The Restore process will be initiated. We can see here the restore process is in Queued state. We can check the status of the restore process by running the below command.

Get-MailboxRestoreRequest

Now the MSFT mailbox is restored. The accidentally deleted important email is back in the mailbox.

Run the below command to Remove the restore request, This can be achieved using Remove-MailboxRestoreRequest command.

Get-MailboxRestoreRequest -Status Completed | Remove-MailboxRestoreRequest

Note:- A mailbox restore can be done using different attributes, Below is another example to restore a mailbox using different attributes.

You can find out the mailbox GUID using the below command.

Get-MailboxStatistics -Database rec_db |fl DisplayName,*GUID*

and then use the below command to restore the mailbox that has the MailboxGUID 44b7f929-786f-4fd1-a001-db903279b9f3 on mailbox database Rec_db to the target mailbox with the alias MSFT.

New-MailboxRestoreRequest -SourceDatabase Rec_db -SourceStoreMailbox 44b7f929-786f-4fd1-a001-db903279b9f3 -TargetMailbox msft

How to Install Exchange 2016

Hello Friends Today i will show how you can install Exchange 2016 server.I have installed 2012 R2 server and a Domin controller. I have downloaded the ISO of Exchange 2016. I will run the below script from powershell as this will enable the  features which is a pre requisite for Exchange 2016 Mailbox Role.


Install-WindowsFeature NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-ADDS

　
If you also plan to install management tools then run the below command from Windows PowerShell.


Enable-WindowsOptionalFeature -Online -FeatureName IIS-ManagementScriptingTools,IIS-ManagementScriptingTools,IIS-IIS6ManagementCompatibility,IIS-LegacySnapIn,IIS-ManagementConsole,IIS-Metabase,IIS-WebServerManagementTools,IIS-WebServerRole

　

It is recommended to reboot the server so that the changes can come in to effect.
Run the Exchange 2016 setup file as an Administrator, This will extract the Exchange set up.
Once the file is extracted, Navigate to the Extracted folder on Command prompt and run the below command to prepare the schema.


setup.exe /prepareschema /iacceptexchangeserverlicenseterms

Now we need to prepare the AD, Since this is the first Exchange server in the environment so we will have to give the Organization name. Chose the organization name as per your convenience, I am choosing the Organization name as "First Organization".

Run the below command from the command prompt.


setup.exe /prepareAd /Organizationname:"First Organization" /iacceptexchangeserverlicenseterms

Now we will start the install of the Mailbox Role and Management Tools.
To install Management tools you will require .NET Framework 4.5.2


Run the below command to Start the install.


setup.exe /m:install /r:Mailbox, mt /iacceptexchangeserverlicenseterms


The set up might fail if some of the patches are not installed or Feature is not enabled, I was asked to install Unified Communications Managed API 4.0 Runtime and enable the Media Foundation feature.I installed the patch and enabled the Feature. Initiated the install again.

The installation completed successfully.


　


　


　


　


　


　

How to Restore a User/Group using Adrestore (Adrestore.exe) tool.

Hello Friends , Today i will show you how you can use the Adrestore(Adrestore.exe) tool to restore a deleted user/Group etc. When you delete a user, The user is not completely deleted from Active directory however the user is put in to deleted object container and the deleted user remains in the deleted object container till it's Tomstone period. Recovering a user using Adrestore doesn't bring back all the attributes of the user however it bring back only a subset of the attributes.

The main points about Adrestore tool is as below.

1)The Adrestore tool can be used to recover a user/group.

2)Adrestore doesn't bring back all the attributes of the user.

3)After the recover, The user will be disabled so the user has to be enabled.

4)You will have to manually set a password to the user as the password attribute can't be recovered.

Every user has a "IsDeleted" attribute, When a user is deleted then the "IsDeleted" attribute is set to"True" Which implies that the user is deleted.

I have created a user "netadmin", I have added the below Group membership to the user as the user is an Admin user.

1)Domain Admins.

2)Enterprise Admins.

3)Schema Admins.

4)Domain Users.

Now i accidentally deleted the user netadmin, I will recover the user using Adrestore.

Adrestore is a Microsoft tool which can be downloaded from the below link.

https://live.sysinternals.com/adrestore.exe

Download the Adrestore tool and save it on a folder.

Open command prompt and go the location where Adrestore resides.

We will be using -r switch to restore the deleted user.

Type Adrestore.exe -r and hit enter.

Once you hit enter , The Adrestore will show all the deleted users/Group one by one and will ask your permission if you want to restore the user/group.

If you know the name of the deleted user and you don't want the Adrestore tool to check all the deleted user then you can also use the command  "Adrestore.exe -r Username" to restore the user.

Once the user is restored then it will show up in Active Directory user and Computers console however the user will be disabled.

We will have to enable the user and then reset the password to use the user account again.
Now the user is recovered however most of it's Attributes are stripped off.

Before the netadmin user was member of Domain admins, Schema Admin, Enterprise Admins and Domain users group however after the restore the user is only a member of Domain users so all the other attributes of the user has to be added manually.