End-user authentication with Istio and KeyCloak on the AWS EKS environment

When evaluating Istio to use in our AWS EKS clusters environment, I found it is a little bit confusing with end-user authentication which cost me a couple days to set up a running scenario. Moreover, most of the blog posts and online documents only mention end-user authentication with Auth0 (a proprietary authentication solution) or very limited to other software such as KeyCloak. This article describes how I did the configuration to make it work with KeyCloak as well as briefly explaining the authentication flow of Istio.

As you may know, Istio introduces two types of authentication which are Transport Authentication and Origin Authentication [0]. Transport Authentication is used for the service to service authentication while Origin Authentication is used for end-user authentication. But, when it comes to real configuration, it looks like I have to apply both types if I want to set up the scenario as follows:

Figure 1: Expected scenario
What I expected to have are:
  • End-user requests will be authenticated with Origin Authentication type only (with JWT token)
  • Service to service will be authenticated with Transport Authentication type only (mTLS)
But, it turned out with Istio policies applied, all the requests to a particular service (httpbin) have to be authenticated using both authentication types or only one type can be applied as presented in Figure 2. One thing should be noted that the mTLS authentication of the end-user requests is provided by the istio-ingressgateway.
Figure 2: Both requests from the end-user and other services (e.g., sleep) to httpbin have to use the access token and mTLS

Here is how the scenario was set up:

1. Create a new EKS cluster [1]

2. Deploy Istio on the EKS cluster you just created on a separate namespace (istio-system) [2]

3. Deploy KeyCloak on a separate namespace (keycloak):
  • Download this k8s deployment YAML file as keycloak.yaml:

  • Run this helm command to deploy KeyCloak on your EKS cluster:
helm upgrade --install keycloak stable/keycloak -f keycloak.yml

Now you have a KeyCloak instance running and you can login to the public address of KeyCloak with the admin username (keycloak) and password (mykeycloakadminpasswd). You can find the public address of the KeyCloak's web interface by running:

kubectl get svc | grep keycloak-http

4. Set up the client on KeyCloak

  • Create a new realm name istio and set it up like this

  • Create the client as following, take note the client ID and secret

5. Create the foo namespace

kubectl create ns foo

6. Deploy httpbin and sleep service in the foo namespace

kubectl apply -f <(istioctl kube-inject -f httpbin.yaml) -n foo

kubectl apply -f <(istioctl kube-inject -f sleep.yaml) -n foo

7. Create a Gateway type service and VirtualService

kubectl apply -f httpbin-gateway.yaml

After that, you can access the httpbin service via the istio-ingressgateway. Get the public address of the istio-ingressgateway by running this command:

kubectl get svc | grep istio-ingressgateway

8. Globally enabling Istio mutual TLS

kubectl apply -f - <<EOF
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
  name: "default"
  - mtls: {}

9. Config the client-side (the requesting services) to use mTLS

kubectl apply -f - <<EOF
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
  name: "default"
  namespace: "istio-system"
  host: "*.local"
      mode: ISTIO_MUTUAL

10. Apply the policy to add end-user authentication

kubectl apply -f - <<EOF
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
  name: "jwt-keycloak"
  - name: httpbin
  - mtls: {}
  - jwt:
      issuer: "http://<keycloak public address>/auth/realms/istio"
      jwksUri: "http://keycloak public address/auth/realms/istio/protocol/openid-connect/certs"
      - istio
  principalBinding: USE_ORIGIN

11. Testing

Because I don't implement any application to handle the user authentication flow so to test the scenario, I just use get the access token by calling the KeyCloak token endpoint using HTTP client (e.g., postman for Mac)

  • Get the access token

  • Copy the returned access token
  • Store the token in a shell's environment variable
TOKEN='<the access token>'
  • Test requesting the httpbin service from the public
curl <ingress public address>/headers -s -o /dev/null -w "%{http_code}\n" --header "Authorization: Bearer $TOKEN"
  • Test requesting the httpbin service from the sleep service
kubectl exec $(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name}) -c sleep -n foo -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "%{http_code}\n"

If the return value is 200, it means that you can access the httpbin service in the foo namespace of your EKS cluster, otherwise, it will return a 401 value which means you are not authorized to acccess.