Kusto Queries on AKS Clusters

Kusto query language can be used to get insights into Azure Kubernetes Service (AKS) clusters. Container insights collects data from AKS clusters and forwards it to Log Analytics workspace, if enabled for a cluster. This data is available for querying in the Azure Monitor. Here is an example of how you can query the pods not in running state in specific namespaces. 

KubePodInventory 
| where Namespace in ("dv","test","prod") 
| where ContainerStatus != "Running" 
| where ContainerStatusReason !in ("", "Completed") 
| distinct Namespace, Name 
Run 
Time range : Last 24 hours 
E] Save v 
14 Share v 
1 
2 
3 
4 
5 
KubePodInventory 
I where Namespaceän ('dv', 'test', 'prod') 
I where ContainerStatusu != 
"Running" 
I where ContainerStatusReasone ! in ('I", "Completed") 
I distinct Namespace, Name 
Results Chart 
@ Display time (UTC+OO:OO) v 
Columns v 
Completed. Showing results from the last 24 hours. 
> 
Namespace 
dv 
prod 
Y Name Y 
app2 
app2

The following query includes the name of the AKS cluster and renders the output as a stacked bar chart.

KubePodInventory 
| where Namespace in ("dv","test","prod") 
| where ContainerStatus != "Running" 
| where ContainerStatusReason !in ("", "Completed") 
| distinct ClusterName, Namespace, Name 
| summarize dcount(Name) by ClusterName, Namespace 
| render columnchart kind=stacked100 
Run 
Time range : Last 24 hours 
E] Save v 
14 Share v 
+ New alert rule 
Export v 
Pin to dashboard 
1 
2 
3 
4 
5 
6 
7 
8 
KubePodInventory 
I where Namespace in ('dv', 'test', 'prod') 
I where ContainerStatus ! = 
"Running" 
I where ContainerStatusReason ! in ('I", "Completed") 
I distinct ClusterName, Namespace, Name 
summarize dcount(Name) by ClusterName, Namespace 
render columnchart kind=stacked100 
Results 
Chart 
@ Display time (UTC+OO:OO) v 
Completed. Showing results from the last 24 hours. 
00:00.7 
2 records 
z 
o 
100 
50 
dv 
O 
aksdemol 
ClusterName 
prod 
Activate Windows 
Go to Settings to activate Windows.

You can include multiple AKS clusters in the scope in which this query is executed by clicking on [Select scope] hyperlink.


Create an Azure Dashboard panel with this output by clicking on [Pin to dashboard] button.

AIG Demo v/ 
Private dashboard 
New dashboard v 
Auto refresh : Off 
Analytics 
rq-aksdashboard 
CD Refresh Full screen 
UTC Time : Past 24 hours 
aksdemol 
test 
62 Edit Share 
Add filter 
ClusterName 
Download 
aksdem02 
Clone 
e Assign 
z 
o 
100 
75 
50 
25 
dv 
prod

You can also execute this Kusto query directly using powershell.

$workspaceName = "DefaultWorkspace-6637b095-xxxx-xxxx-xxxx-xxxxxxxxxxx-EUS" 
$workspaceRG = "defaultresourcegroup-eus" 
$WorkspaceID = (Get-AzOperationalInsightsWorkspace -Name $workspaceName -ResourceGroupName $workspaceRG).CustomerID 

$query = 'KubePodInventory | where Namespace in ("dv","test","prod") | where ContainerStatus != "Running" | where ContainerStatusReason !in ("", "Completed") | distinct ClusterName, Namespace, Name | summarize dcount(Name) by ClusterName, Namespace' 

$result = Invoke-AzOperationalInsightsQuery -WorkspaceId $WorkspaceID -Query $query -Timespan (New-TimeSpan -days 1) 
$result.results 

This allows you to include the results of your custom Kusto queries in any reports you might run using Azure Automation Runbooks.

PowerShell v Q) 
PS /home/ash> 
PS /home/ash> 
PS /home/ash> 
Q 
o 
PS /home/ash> $workspaceName 
"DefaultWorkspace-6637b095 
PS / home/ash> $workspaceRG 
"defaultresourcegroup-eus" 
PS / home/ash> $WorkspaceID 
(Get-AzOperationa11nsightsWorkspace 
-Name $workspaceName -ResourceGroupName $workspaceRG) . CustomerID 
'KubePodInventory I where Namespace in ("dv" , "test" , "prod") I where ContainerStatus ! = 
"Running" I where Container 
PS / home/ash> $query 
I distinct ClusterName, Namespace, Name I summarize dcount(Name) by ClusterName, Namespace' 
PS / home/ash> $result 
Invoke-AzOperationa11nsightsQuery -Workspaceld $WorkspaceID -Query $query -Timespan (New-TimeSpan 
-days 3) 
PS / home/ash> $result.results 
ClusterName Namespace dcount_Name 
aksdemol 
aksdemol 
aksdem02 
dv 
prod 
test 
1 
1 
1 
PS / home/ash> 
PS / home/ash> 
PS /home/ash>

AKS – Adding SSH Keys to VMSS Nodes

You can connect to Azure Kubernetes Service (AKS) nodes using ssh. It is documented here: Connect with SSH to Azure Kubernetes Service (AKS) cluster nodes for maintenance or troubleshooting.

I needed to access nodes on the System node pool for collecting some logs recently, but the process documented above was not working for me. It turns out that there were two different issues, both related to adding your SSH keys to the nodes in a virtual machine scale set (VMSS).

This is the az cli command that adds the ssh keys to the VMSS:

az vmss extension set  \
    --resource-group $CLUSTER_RESOURCE_GROUP \
    --vmss-name $SCALE_SET_NAME \
    --name VMAccessForLinux \
    --publisher Microsoft.OSTCExtensions \
    --version 1.4 \
    --protected-settings "{\"username\":\"azureuser\", \"ssh_key\":\"$(cat ~/.ssh/id_rsa.pub)\"}"

I was running this command from powershell on a windows host. So, the first modification I needed was to escape the double quotes by doubling them. You can also use back ticks instead of doubling the double quotes.

az vmss extension set  \
    --resource-group $CLUSTER_RESOURCE_GROUP \
    --vmss-name $SCALE_SET_NAME \
    --name VMAccessForLinux \
    --publisher Microsoft.OSTCExtensions \
    --version 1.4 \
    --protected-settings "{\""username\"":\""azureuser\"", \""ssh_key\"":\""$(cat ~/.ssh/id_rsa.pub)\""}"

The second issue was due to the fact that ssh-keygen adds a comment to the id_rsa.pub file. By default, it adds username@hostname as the comment at the end of the file . Normally, this comment wouldn’t cause any problems, but in this case it was causing an error – “Invalid escape sequence \uXXXX”.

The problem was that my domain user name starts with character ‘u’ and “domain\userid1” was being interpreted as containing a unicode character “\useri” which is obviously not a valid unicode character. The workaround here is to either delete the comment from the id_rsa.pub file (it works just fine without it) or to override the default comment when generating the ssh key by specifying your own comment via -C command line option.

ssh-keygen -C mycomment

Finally, there is another way to execute az vmss extension set command to get around both of these issues by creating a json file and passing that as the value of –protected-settings argument. Here is how –

  • Create a json file, protected_settings.json, with the following content:
{
	"username" : "azureuser",
	"ssh_key" : "REPLACE_THIS_WITH_CONTENT_OF_ID_RSA_PUB_FILE"
}
  • Pass this file as the command line argument:
az vmss extension set  \
    --resource-group $CLUSTER_RESOURCE_GROUP \
    --vmss-name $SCALE_SET_NAME \
    --name VMAccessForLinux \
    --publisher Microsoft.OSTCExtensions \
    --version 1.4 \
    --protected-settings protected_settings.json

Works like a charm!

AKS Supported Kubernetes Versions

Azure Kubernetes Service (AKS) supports specific versions of Kubernetes.
It is necessary to regularly monitor the release of new versions and upgrade your AKS clusters to supported versions in order to remain in compliance with AKS Kubernetes Version Support Policy.

AKS announces the planned date of a new minor version release and corresponding old version deprecation via AKS Release notes at least 30 days prior to removal. An email notification is sent to the subscription administrators with the planned version removal dates. You get 30 days from version removal to upgrade to a supported minor version release. Patch versions can be released anytime and you get 30 days from the removal date to upgrade to a supported patch version.

You should test new target Kubernetes versions and upgrade your AKS clusters in a timely manner. For that, it is necessary to proactively monitor the AKS release notes. This can easily become a chore in a large Enterprise environment. Here is a Powershell script to make it easier to stay on top of Kubernetes version releases in AKS and publish/share it with others: Get-AksSupportedVersions.

$aksVersionsJson = az aks get-versions --location eastus

$aksVersions = $aksVersionsJson | ConvertFrom-Json
$aksVersions.orchestrators.upgrades.orchestratorVersion 

$data = $aksVersions.orchestrators | Select-Object `
    -Property @{Name="Version";Expression={$_.orchestratorVersion}} `
            , @{Name="Default";Expression={$_.default}} `
            , @{Name="Preview";Expression={$_.isPreview}} `
            , @{Name="Upgrades";Expression={$_.upgrades.orchestratorVersion -join ", "}}

$versionTable = $data | ConvertTo-Html -Fragment 
$versionTableString = $versionTable | Out-String
$html = New-Object -ComObject "HTMLFile"
$html.IHTMLDocument2_write($versionTableString)
$tables = $html.body.getElementsByTagName("table")

$rptString = ""
ForEach($table in $tables){
    ForEach($row in $table.rows){
            $cellCount = 0
            ForEach($cell in $row.cells){
                $cellCount++
                if(($cellCount -eq 2) -and ($cell.innertext -eq 'True'))
                {
                    $row.className = "OkStatus"
                }
                if(($cellCount -eq 3) -and ($cell.innertext -eq 'True'))
                {
                    $row.className = "WarningStatus"
                }
        }
    }
    $rptString += $table.outerHTML
}

$reportDate = $(get-date -DisplayHint DateTime) | Out-String
$fileTS = $(get-date -Format "yyyyMMdd") 
$fileName = "aks-versions-$fileTS.html"

$rptTitle = "<h1>AKS Kubernetes Versions</h1><p>$reportDate</p>"
$report = ConvertTo-Html -Title "AKS Versions" -Body "$rptTitle $rptString" -Head $header
$report | Out-File "$fileName"

It creates an HTML report of currently supported Kubernetes versions in AKS along with their respective upgrade paths. Here is an example of the report generated by the script referenced above :

AKS Kubernetes Versions

Sunday, November 8, 2020 8:47:42 PM

Version Default Preview Upgrades
1.16.13 1.16.15, 1.17.9, 1.17.11
1.16.15 1.17.9, 1.17.11
1.17.9 1.17.11, 1.18.6, 1.18.8
1.17.11 True 1.18.6, 1.18.8
1.18.6 1.18.8, 1.19.0
1.18.8 1.19.0
1.19.0 True

Reference: How To Create An HTML Report With PowerShell

Ethereum Blockchain-As-A-Service in Azure Cloud

Ethereum Blockchain (EBaas) as a Service provided by Microsoft Azure and ConsenSys allows for enterprise customers and partners to play, learn, and fail fast at a low cost in a ready-made dev/test/production environment. It will allow them to create private, public and consortium based Blockchain environments very quickly in Azure. In this session, you will learn how to get started with prototyping building blocks of a decentralized application using EBaas in Windows Azure.

Venue: Triangle Azure User Group

Slides : download.

KeyNode with Node.js and Microsoft Azure

KeyNode is a application to issue and verify software license keys. Technology stack for KeyNode is Node.js, MongoDB and Microsoft Azure.

I had built this functionality with C9.io (a cloud-based IDE with a built-in source code repository and debugger), mongohq (MongoDB as a service – now part of compose.io) and appfog (Cloud PAAS built on top of CloudFoundry). It used SMTP/gmail to email license files. That was the version I created a couple of years ago to issue tamper-proof signed xml license files for CodeDemo (a code snippet tool for developers, presenters and instructors).

For KeyNode (open source) I switched to a different toolset : Visual Studio Code and Windows Azure, simplified the code to remove signed xml file and open-sourced it on GitHub. Signed xml allowed offline verification in CodeDemo (a Wpf/Desktop app). Removing signed xml requires verification to happen online. I am working on adding the web endpoint for verification of license keys. This version uses SendGrid to email license keys. KeyNode is deployed as a Windows Azure Web App. The Azure Web App is on Continuous Deployment feed from the source code repository on GitHub.

I created and tested this Node.js application locally without IIS and deployed it as an Azure Web App without making any changes to the code at all. Node.js applications are hosted in Azure under IIS with iisnode. Iisnode is a native IIS module that allows hosting of node.js applications in IIS on Windows. Read more about iisnode here. Iisnode architecture also makes it significantly easier to take advantages of scalability afforded by Azure.

KeyNode is a work in progress. My plan is to use this as the basis for further explorations in the following areas :

  • DevOps, Docker and Microservices (at miniature scale of course!)
  • Create a Web UI with Express (a Node.js web application framework)
  • Integrate with Azure Storage/Queues
  • and more…

I invite you to check out the live site on Azure and fork it for your own experiments : KeyNode on GitHub.

Resources :

Photo Credit : Piano Keyboard (www.kpmalinowski.pl)

PackRaft with Xamarin and Microsoft Azure

PackRaft is a cross-platform mobile application built with Xamarin. It integrates with Microsoft Azure services, such as Azure Table Storage and Azure App Service. It is created with Xamarin.Forms for iOS and Android (Windows version will be coming soon).

packraft-android-riders   packraft-iphone-assignments

The purpose of creating this sample application was to expand expertise in cross-platform mobile development with Windows Azure integration, specifically with Xamarin.Forms and Azure App Service. With Xamarin.Forms, it is possible to achieve close to 100% code reuse across these platforms for certain type of apps. Azure App Service provide a set of rich cloud backend services for your mobile applications.

PackRaft-Platform

PackRaft allows the user to setup a fleet of vehicles of various seating capacities. Groups of riders are assigned to these rides according to the count of riders in each group. You can specify that certain groups be kept together on a single vehicle – and the app tries to honor those requests if possible. I did this via a rudimentary implementation of a bin packing algorithm. This implementation is deployed as an API App in Microsoft Azure.

PackRaft uses Azure Mobile Services SDK to persist Fleet data to Azure Table Storage and to authenticate users with Facebook and Microsoft accounts. Twitter and Google account authentication is also supported by Azure Mobile Services SDK out of the box and those will also be enabled in PackRaft soon.

PackRaft is available on Android, iPhone and iPad.
Get it on Google Play

Download iOS app on App Store