Uncategorized

AKS – understanding bypass of network limitations for private API server via Azure Resource Manager

⚠️Disclaimer: This is-by-design feature, and depending how you look at it, it is either desired, or undesired mechanism. The term of bypass does not indicate malicious use here, as some might pick it up from the semantic context.

The main point is to understand what Private API server means in AKS at the context of using the run command API available in Azure Resource Manager API.

  • For those who want to ensure that private network connectivity is the only way to reach the API server, run command API presents something that might be wise to mitigate, or at least audit
  • For those that look for Azure RBAC protected way to access private API server, without* network limitations it might ease some burden, and provides option where the API server is still protected from direct internet access. Using KubeCTL for example still requires line of sight to the API endpoint.

*Unless conditional access is used to limit Azure Management Access.

https://docs.microsoft.com/en-us/azure/aks/private-clusters#options-for-connecting-to-the-private-cluster

Background

Run command is based on Azure Resource Manager, so invoking commands does not require the use of KubeCTL in the client side, or knowledge of the API server address (or line-of-sight) to API server for that matter.

Permissions needed

Running the command via Azure Resource Manager API

Code snippet (without dependencies)

  • ClusterToken audience is = 6dae42f8-4368-4678-94ff-3960e28e3630
  var opt = {
    method: "post",
    url: "https://management.azure.com/subscriptions/3539c2a2-cd25-48c6-b295-14e59334ef1c/resourcegroups/rg-aks-refinstall/providers/microsoft.containerservice/managedclusters/akssvc55/runCommand?api-version=2021-10-01",
    headers: {
      Authorization: `Bearer ${token}`,
      "content-type": "application/json",
    },
    data: {
      "command": "kubectl get pods -n kube-system",
      "context": "",
      clusterToken:"This token for the audience 6dae42f8-4368-4678-94ff-3960e28e3630"
    }
  }

  var {
    data,
    headers
  } = await axios(opt).catch((error) => {
    console.log(error.response.data.error)
  })

  console.log(data, headers)

  opt.method = "get"
  opt.url = headers.location

  delete opt.data
  var {
    data,
    headers
  } = await axios(opt).catch((error) => {
    console.log(error)
  })
  console.log(data)

Response example

{
  provisioningState: "Succeeded",
  exitCode: 0,
  startedAt: "2022-02-16T15:51:41Z",
  finishedAt: "2022-02-16T15:51:42Z",
  logs: "NAME                                        READY   STATUS    RESTARTS   AGE azure-ip-masq-agent-224vj                   1/1     Running   0          119m azure-ip-masq-agent-9bbz9                   1/1     Running   0          119m azure-ip-masq-agent-qrrnj                   1/1     Running   0          119m azure-npm-4znhv                             1/1     Running   0          119m azure-npm-qdf7b                             1/1     Running   0          119m azure-npm-sfkv2                             1/1     Running   0          119m azure-policy-647d486958-f9d6t               1/1     Running   0          93m azure-policy-webhook-84884d989b-vpg2j       1/1     Running   0          93m coredns-845757d86-gz4wq                     1/1     Running   0          119m coredns-845757d86-h4gqf                     1/1     Running   0          123m coredns-autoscaler-5f85dc856b-wgjxj         1/1     Running   0          123m csi-azuredisk-node-fpttj                    3/3     Running   0          119m csi-azuredisk-node-g4j82                    3/3     Running   0          119m csi-azuredisk-node-pbqn4                    3/3     Running   0          119m csi-azurefile-node-jdwnr                    3/3     Running   0          119m csi-azurefile-node-kb5lh                    3/3     Running   0          119m csi-azurefile-node-t8fc5                    3/3     Running   0          119m ingress-appgw-deployment-645cbf99f6-qj4gq   1/1     Running   0          123m kube-proxy-4n4qr                            1/1     Running   0          119m kube-proxy-mpzvr                            1/1     Running   0          119m kube-proxy-qjqhj                            1/1     Running   0          119m metrics-server-6bc97b47f7-rrx6t             1/1     Running   0          123m tunnelfront-c8957f9df-jm6np                 1/1     Running   0          121m ",
}

Further references

https://docs.microsoft.com/en-us/rest/api/aks/managed-clusters/run-command
https://docs.microsoft.com/en-us/azure/aks/command-invoke

Running the command with AZ CLI

https://docs.microsoft.com/en-us/azure/aks/command-invoke#use-command-invoke-to-run-a-single-command

Mitigations

There are various approaches to mitigate this behavior (if it is not desired)

  1. Use kubernetes permissions that do not include the following annotation {"aks.azure.com/runCommand":"true"}
  2. Deploy deny role in Azure, or use custom roles that do not include the following permissions Microsoft.ContainerService/managedClusters/runcommand/action  Microsoft.ContainerService/managedclusters/commandResults/read

Auditing

Auditing gets command and creation details

AzureDiagnostics
| where Category == "kube-audit"
| extend l= parse_json(log_s)
| where l.responseObject contains "aks.azure.com/runCommand"
| mv-expand privateApiBypass= l.responseObject.spec.containers
| extend cmd = split(tostring(privateApiBypass.command),'"/bin/sh","-c","')[1]
| distinct  tostring(cmd), tostring(l.responseObject.metadata.labels.createdBy),tostring(l.responseObject.status.phase)

Ending swords

This is clearly a double edged sword. Bare in mind, that in this case the API server is not reached directly, and not either by KubeCTL directly. I can see arguments for both cases, but my recommendation, is that customers that require that their API server is not reachable without VPN, or other private access do review this information.

0 comments on “AKS – understanding bypass of network limitations for private API server via Azure Resource Manager

Vastaa

Täytä tietosi alle tai klikkaa kuvaketta kirjautuaksesi sisään:

WordPress.com-logo

Olet kommentoimassa WordPress.com -tilin nimissä. Log Out /  Muuta )

Twitter-kuva

Olet kommentoimassa Twitter -tilin nimissä. Log Out /  Muuta )

Facebook-kuva

Olet kommentoimassa Facebook -tilin nimissä. Log Out /  Muuta )

Muodostetaan yhteyttä palveluun %s

%d bloggaajaa tykkää tästä: