Deploying a root Certificate Authority integrated with an HSM at iFood

On this post we will describe why and how we implemented an internal Certificate Authority for issuing X.509 and SSH certificates, using an HSM to securely store the CA private key.

Symplifying SSH authorization

One reason that led us to implement our own Certificate Authority was to simplify SSH authorization to our machines in the cloud. Classic SSH authorization consists on adding a user’s public key to a trusted keys file on all machines where that user is authorized to SSH into.

This scenario brings a few issues, such as the overload of work for updating all machines with newly authorized keys and the burden of revoking keys from all authorized keys files. By using SSH certificates both of these issues are mitigated.

Setting up SSH certificates on a machine consists on adding a few lines of configuration and saving the root CA’s certificate on disk. After that, all valid certificates issued by that CA will be authorized to SSH into that machine. That happens because the SSH agent will verify the connecting client’s certificate, and verifying a certificate means it will check if it is valid (it is not expired, its signature was generated by the private key corresponding to the public key attached to it, and so on) and will check the chain of CAs that issued that certificate. If that certificate has the same root CA as the one configured as trusted, the connection will be authorized.

As per the authorization revocating, with regular SSH keys we would have to remove a revoked key from all machines. With certificates, we can use the so called passive revocation, where all issued certificates have a short lifespan, ensuring that no revoked access will persist for too long.

Supporting mTLS

A second motivation for implementing our Certificate Authority is our service mesh project. This project aims to, among many other changes to our environment, add mTLS authentication between all applications and services.

For that task, several leaf Certificate Authorities will be deployed, where those will be issued by the Root CA.

Smallstep

We used Smallstep, a certificate management toolkit, as the base software of our Root CA. With it you can deploy Certificate Authorities (CAs) for SSH and X.509 certificates. One of its great features includes setting up external authorization provisioners, which allows you to integrate it to your company’s SSO/IDP, and integration with HSMs using PKCS#11.

Infrastructure and architecture

The figure below shows an overview of the deployment of our root CA.

The application is running on multiple pods on our Kubernetes Cluster. Integration with our IDP was done via smallstep native configuration.

We also configured an HSM for storing the CA’s private key. The idea is that the private key never leaves the HSM.

Ensuring high availability with Smallstep

Smallstep relies on local configuration files, for storing static configuration, and a database for storing more volatile information, such as the issued certificates.

As we wanted to have multiple instances of the CA, we adapted the step configuration files to be generated upon the pod startup. All values and secrets are retrieved from our secrets Vault and inserted into a configuration template by a startup script, which then starts step-ca. This way, all pods will be started with the same configuration parameters, allowing multiple instances of the application to coexist.

Command line interface for SSH certificates

A Command Line Interface, which is basically a wrapper on the step-cli, was created so users could issue their SSH certificates. To make usage even easier, it also checks for the step-cli binary on the user’s computer and installs it if it is not there yet.

As it is configured on the step ca instances, iFood’s single sign on is used as an authentication mechanism, making it even easier for the user to authenticate.

Usage

After downloading the ifood-ca CLI on the client side, the user has to, on the first run, bootstrap the CA with the command:

$ ifood-ca ssh init-ca

After initializing, the user can issue a certificate with:

$ ifood-ca ssh issue john.doe@ifood.com.br
new issued certificate for:  john.doe@ifood.com.br
iFood CA not configured!
Bootstraping the iFood SSH CA..
The iFood SSH CA client was successfully initialized!

listing certificates on ssh agent:
256 SHA256:3tAzj3DmEz5DjwpBaVS1ZhRVyTAG2TBZJ0RZXicqhfc john.doe@ifood.com.br (ECDSA-CERT)

The above command will generate a Certificate Sign Request (CSR), contact the CA and send the CSR, getting the signed certificate and adding it to the ssh-agent. The user can simply SSH into a machine after that.

Issuing X.509 certificates

X.509 certificates will rarely be issued by the root CA, as it will only issue certificates for other CAs, thus we used the standard step CLI for that task.

Conclusion

In this post we explained how we deployed a root CA integrated with an HSM at iFood for issuing SSH certificates and standard certificates for mTLS on our mesh environment. Smallstep proved to be a very powerful tool on the task with its diverse range of integrations. Integrating it with an HSM added robustes to the application, as the CA private key is securely stored on that device.