Secured private Docker registry in Kubernetes

If you run a Docker-based Kubernetes cluster yourself, sooner or later you will find out that you need a Docker registry to store the docker images. You might start out with a public registry out there, but often you might want to keep your images away from the public. Now if your cluster is on the cloud, you can just use the Container Registry provided by AWS EC2 or Google Cloud Platform. If your cluster is on-prem however, then you might want to keep the registry close to your cluster, hence deploying your own registry might be a good idea.

For starters, you can always use the registry addon shipped with Kubernetes. The default setup will give you an unsecured registry, so you will need to setup a DeamonSet to route a local port to the registry, so that to the workers, your registry runs on localhost:PORT, which will not trigger the secured logic of the docker daemon. Check the link for more information.

This setup is rather bad though. If a user, from his machine, wants to push his image to the registry, then he has to use kubectl to setup a proxy to the registry service, so that the service is available on his machine at localhost:PORT. This is rather inconvenient and tedious. We need a registry available at a separated host name, so that it can receive images from any machines in the network, and serve images to any workers in the Kubernetes cluster.

We start with the guide provided by Kubernetes, with a few deviations:

  • Since this is an on-prem cluster, you can not expose the registry service as LoadBalancer service. The best you could do is to expose it as a NodePort service, and then setup an nginx server to map a predefined URL to the URL of the service. Everyone will talk to the registry using this predefined URL.
  • Now since we have nginx in the middle, and we need to make nginx to answer to HTTPS request, it makes little sense to make the registry secured through TLS, because if we do that, in the proxy, nginx has to decode the request, then re-encode it before routing the request to the registry. Instead, we will only make the nginx endpoint secured, which routes the request to the unsecured docker registry service.
  • The rest of the magic is just to configure nginx to handle HTTPS requests, then route it un-encrypted to the service.

First you will need a domain certificate and key. You can obtain it from some trusted CA on the web (which takes some time), or you can generate self-signed certificate as follows:

# generates CA file "ca.crt" and CA key "privkey.pem"
openssl req -out ca.crt -new -x509 -days 900

# Generate server certificate/key pair
echo "00" > file.srl
openssl genrsa -out server.key 1024
openssl req -key server.key -new -out server.req
openssl x509 -days 900 -req -in server.req -CA ca.crt -CAkey privkey.pem -CAserial file.srl -out server.pem 

# Generate client certificate/key pair

# Either choose to encrypt the key or not
openssl genrsa -des3 -out client.key 1024    # Encrypt the client key with a passphrase
openssl genrsa -out client.key 1024          # Don't encrypt the client key
openssl req -key client.key -new -out client.req
openssl x509 -days 900 -req -in client.req -CA ca.crt -CAkey privkey.pem -CAserial file.srl -out client.pem

The registry and registry UI could be deployed in the same ReplicationController:

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-tls
  namespace: kube-system
  labels:
    k8s-app: kube-registry-tls
    version: v0
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry-tls
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry-tls
        version: v0
    spec:
      containers:
      - name: registry-tls
        image: registry:latest
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 2
            memory: 2Gi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry-tls
          protocol: TCP
      - name: registry-tls-ui
        image: konradkleine/docker-registry-frontend:v2
        env:
        - name: ENV_DOCKER_REGISTRY_HOST
          value: "localhost"
        - name: ENV_DOCKER_REGISTRY_PORT
          value: "5000"
        ports:
        - containerPort: 80
          name: registry-tls-ui
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: registry-image-storage

You will need to a PersistentVolumeClaim called registry-image-storage, but if you are in a hurry then feel free to use an emptyDir. This is where the registry stores its images.

The service definition is straightforward:

apiVersion: v1
kind: Service
metadata:
  name: kube-registry-tls
  namespace: kube-system
  labels:
    k8s-app: kube-registry-tls
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry-tls
  type: NodePort
  ports:
  - name: registry-tls
    port: 5000
    protocol: TCP
    nodePort: 30199
  - name: registry-tls-ui
    port: 80
    protocol: TCP
    nodePort: 30299

Those are NodePort services with the registry listens on port 30199 and the UI on port 30299. Depending on how you set up your cluster, you can specify any other values.

Then the nginx configuration file will look similar to this. Put it somewhere in /etc/nginx/sites-enabled:


upstream kube_worker_registry {
 server     Node_IP:30199;
 server     Node_IP:30199;
}

server {
 listen 11000 ssl;
 server_name name_of_the_nginx_server;
 access_log /var/log/nginx/name_of_the_nginx_server.access.log;

 ssl on;
 ssl_certificate /path/to/server.pem;
 ssl_certificate_key /path/to/server.key;
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_set_header X-Original-URI $request_uri;
 proxy_set_header Docker-Distribution-Api-Version registry/2.0;

 # Allow large uploads
 client_max_body_size 0;
 chunked_transfer_encoding on;

 location / {
 proxy_pass http://kube_worker_registry;
 proxy_read_timeout 900;
 }
}

You will need to put the server certificates somewhere nginx knows. It will then listen for HTTPS requests on port 11000 and route the request (un-encrypted) to the registry service in Kubernetes. Of course you can do similarly for the registry UI.

We are not done yet. In order for the docker daemon to talk to the secured nginx endpoint, we need to copy the CA certificate (ca.crt file) generated earlier to every worker in the Kubernetes cluster, and to all the machines that need to submit images to the registry.

Assume you put the certificate ca.crt somewhere, then you can install the certificate on the machines as follow:

  • On macOS:
    curl http://URL/ca.crt > ca.crt
    sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./ca.crt
    rm ca.crt
    
  • On Linux (Ubuntu):
    sudo mkdir -p /etc/docker/certs.d/(nginx_server_address)\:11000
    curl http://URL/ca.crt > ca.crt
    sudo mv ./ca.crt /etc/docker/certs.d/(nginx_server_address)\:11000/
    sudo service docker restart
    

Now check if you can push an image to the registry:

docker build -t (nginx_server_address):11000/my-image-name -f Dockerfile .
docker push (nginx_server_address):11000/my-image-name
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s