⚠️Disclaimer: This is-by-design feature, and depending how you look at it, it is either desired, or undesired mechanism. The term of bypass does not indicate malicious use here, as some might pick it up from the semantic context.
The main point is to understand what Private API server means in AKS at the context of using the run command API available in Azure Resource Manager API.
- For those who want to ensure that private network connectivity is the only way to reach the API server, run command API presents something that might be wise to mitigate, or at least audit
- For those that look for Azure RBAC protected way to access private API server, without* network limitations it might ease some burden, and provides option where the API server is still protected from direct internet access. Using KubeCTL for example still requires line of sight to the API endpoint.
*Unless conditional access is used to limit Azure Management Access.
Background
Run command is based on Azure Resource Manager, so invoking commands does not require the use of KubeCTL in the client side, or knowledge of the API server address (or line-of-sight) to API server for that matter.
Running the command via Azure Resource Manager API
Code snippet (without dependencies)
- ClusterToken audience is = 6dae42f8-4368-4678-94ff-3960e28e3630
var opt = {
method: "post",
url: "https://management.azure.com/subscriptions/3539c2a2-cd25-48c6-b295-14e59334ef1c/resourcegroups/rg-aks-refinstall/providers/microsoft.containerservice/managedclusters/akssvc55/runCommand?api-version=2021-10-01",
headers: {
Authorization: `Bearer ${token}`,
"content-type": "application/json",
},
data: {
"command": "kubectl get pods -n kube-system",
"context": "",
clusterToken:"This token for the audience 6dae42f8-4368-4678-94ff-3960e28e3630"
}
}
var {
data,
headers
} = await axios(opt).catch((error) => {
console.log(error.response.data.error)
})
console.log(data, headers)
opt.method = "get"
opt.url = headers.location
delete opt.data
var {
data,
headers
} = await axios(opt).catch((error) => {
console.log(error)
})
console.log(data)
Response example
{
provisioningState: "Succeeded",
exitCode: 0,
startedAt: "2022-02-16T15:51:41Z",
finishedAt: "2022-02-16T15:51:42Z",
logs: "NAME READY STATUS RESTARTS AGE azure-ip-masq-agent-224vj 1/1 Running 0 119m azure-ip-masq-agent-9bbz9 1/1 Running 0 119m azure-ip-masq-agent-qrrnj 1/1 Running 0 119m azure-npm-4znhv 1/1 Running 0 119m azure-npm-qdf7b 1/1 Running 0 119m azure-npm-sfkv2 1/1 Running 0 119m azure-policy-647d486958-f9d6t 1/1 Running 0 93m azure-policy-webhook-84884d989b-vpg2j 1/1 Running 0 93m coredns-845757d86-gz4wq 1/1 Running 0 119m coredns-845757d86-h4gqf 1/1 Running 0 123m coredns-autoscaler-5f85dc856b-wgjxj 1/1 Running 0 123m csi-azuredisk-node-fpttj 3/3 Running 0 119m csi-azuredisk-node-g4j82 3/3 Running 0 119m csi-azuredisk-node-pbqn4 3/3 Running 0 119m csi-azurefile-node-jdwnr 3/3 Running 0 119m csi-azurefile-node-kb5lh 3/3 Running 0 119m csi-azurefile-node-t8fc5 3/3 Running 0 119m ingress-appgw-deployment-645cbf99f6-qj4gq 1/1 Running 0 123m kube-proxy-4n4qr 1/1 Running 0 119m kube-proxy-mpzvr 1/1 Running 0 119m kube-proxy-qjqhj 1/1 Running 0 119m metrics-server-6bc97b47f7-rrx6t 1/1 Running 0 123m tunnelfront-c8957f9df-jm6np 1/1 Running 0 121m ",
}
Further references
Running the command with AZ CLI
https://docs.microsoft.com/en-us/azure/aks/command-invoke#use-command-invoke-to-run-a-single-command
Mitigations
There are various approaches to mitigate this behavior (if it is not desired)
- Use kubernetes permissions that do not include the following annotation
{"aks.azure.com/runCommand":"true"} - Deploy deny role in Azure, or use custom roles that do not include the following permissions
Microsoft.ContainerService/managedClusters/runcommand/actionMicrosoft.ContainerService/managedclusters/commandResults/read
Auditing
Auditing gets command and creation details
AzureDiagnostics
| where Category == "kube-audit"
| extend l= parse_json(log_s)
| where l.responseObject contains "aks.azure.com/runCommand"
| mv-expand privateApiBypass= l.responseObject.spec.containers
| extend cmd = split(tostring(privateApiBypass.command),'"/bin/sh","-c","')[1]
| distinct tostring(cmd), tostring(l.responseObject.metadata.labels.createdBy),tostring(l.responseObject.status.phase)
Ending swords
This is clearly a double edged sword. Bare in mind, that in this case the API server is not reached directly, and not either by KubeCTL directly. I can see arguments for both cases, but my recommendation, is that customers that require that their API server is not reachable without VPN, or other private access do review this information.
SecureCloudBlog 
0 comments on “AKS – understanding bypass of network limitations for private API server via Azure Resource Manager”