Setting up TLS Certificates for Custom Domains on the Fly

When we were brainstorming on various deployment techniques to set up ToolJet on AWS EC2 instances, one common requirement that came up was to support TLS certificates for custom domains. We wanted a solution that was easy and transparent so that it feels seamless to our users.
After careful thinking, we decided not to package our existing development docker-compose setup and instead decided to run ToolJet natively on the EC2 instance.
With docker out of the picture, we needed a fast and performant webserver/proxy which can proxy requests to both the ToolJet client(frontend) and the ToolJet server(backend) while at the same time handle and terminate TLS connections.

OpenResty caught our eye since it is built on top of the battle-tested Nginx web-server coupled with an embedded LuaJIT engine that can run Lua scripts. OpenResty thus has a wide variety of plugins that provide additional functionalities.
One such plugin called lua-resty-auto-ssl, allows OpenResty to automatically and transparently issue and renew TLS certificates from Let's Encrypt as and when requests are received.
This is exactly what we wanted, and is the topic of this blog post. So without further delay, let's look at how we can set up OpenResty to issue TLS certificates on the fly.

• Provision a new instance running a Linux distribution of your choice from your favourite cloud provider. (For this tutorial, we've used Ubuntu 18.04).
• ssh into the instance and install some pre-requisites by running:
  sudo apt-get -y install wget gnupg ca-certificates apt-utils curl
  (If you are using a different Linux distribution, your package manager might be different from what we've used here.)

• Import OpenResty's GPG key by running:
  wget -O - https://openresty.org/package/pubkey.gpg | sudo apt-key add -
• Add the OpenResty's official APT repository to your apt source list:
  echo "deb http://openresty.org/package/ubuntu $(lsb_release -sc) main" \ | sudo tee /etc/apt/sources.list.d/openresty.list
• Update the apt index:
  sudo apt-get update
• Finally, install OpenResty by running:
  sudo apt-get -y install openresty

(If you're using a different Linux distribution, checkout https://openresty.org/en/installation.html for installation instructions.)

• First, we need to install the Lua package manager luarocks.
  You can do that by running: sudo apt-get install -y luarocks
• Next, install the lua-resty-auto-ssl plugin by running:
  sudo luarocks install lua-resty-auto-ssl

• By default, OpenResty writes the access logs and error logs to /var/log/openresty. Create this directory by running:
  sudo mkdir /var/log/openresty.
• Create a new directory for lua-resty-auto-ssl to read and write custom domain TLS certificates from. It's recommended to create this directory under /etc with appropriate permissions.
  Run sudo mkdir /etc/resty-auto-ssl to create the directory.
• Since we've used sudo to create the directory. The newly created
  /etc/resty-auto-ssl directory would be owned by the root user and the root user group. So, if we need OpenResty to write new certificates to this directory, we would need to run OpenResty as the root user.Running public-facing applications on a server as the root user is a big security risk and is something we always should strive to avoid. Most Linux distros come with a user and group called www-data (You should tell OpenResty to run its worker processes as this user in your nginx.conf file. We'll talk more about this later.).
  For now, change the ownership of the directory to the user and group www-data by running:

  sudo chown www-data:www-data /etc/resty-auto-ssl


• Now, create a fallback certificate that OpenResty can use in cases where certificate requests for a domain fail for some reason.
  (This is a self-signed certificate and won't be trusted by the client's browser/OS).
  Run the following command to generate a self-signed certificate in the /etc/ssl directory :

sudo openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 \   -subj '/CN=sni-support-required-for-valid-ssl' \    -keyout /etc/ssl/resty-auto-ssl-fallback.key \   -out /etc/ssl/resty-auto-ssl-fallback.crt 

We now modify the configuration file that OpenResty uses so that it can start using the lua-resty-auto-ssl plugin.

• Go to the configuration directory by running: cd /etc/openresty
• Remove the existing Nginx configuration file by running
  rm nginx.conf
• Create a new configuration file with sudo vi nginx.conf and paste the following contents:

user www-data;events {  worker_connections 1024;}http {  lua_shared_dict auto_ssl 1m;  lua_shared_dict auto_ssl_settings 64k;  resolver 8.8.8.8 ipv6=off;  init_by_lua_block {    auto_ssl = (require "resty.auto-ssl").new()    auto_ssl:set("allow_domain", function(domain)      return true    end)    auto_ssl:init()  }  init_worker_by_lua_block {    auto_ssl:init_worker()  }  server {    listen 443 ssl;    ssl_certificate_by_lua_block {      auto_ssl:ssl_certificate()    }    ssl_certificate /etc/ssl/resty-auto-ssl-fallback.crt;    ssl_certificate_key /etc/ssl/resty-auto-ssl-fallback.key;  }  server {    listen 80;    location /.well-known/acme-challenge/ {      content_by_lua_block {        auto_ssl:challenge_server()      }    }  }  server {    listen 127.0.0.1:8999;    client_body_buffer_size 128k;    client_max_body_size 128k;    location / {      content_by_lua_block {        auto_ssl:hook_server()      }    }  }}

By specifying user www-data; in the config file, we've told OpenResty to start as the user www-data.

Tip:
Since LetsEncrypt's production environment has very strict rate limits for failed validations, I would recommend setting the CA to LetsEncrypt's staging ACME URL. You can do that by using the following init_by_lua_block .

init_by_lua_block  {        auto_ssl = (require "resty.auto-ssl").new()    auto_ssl:set("allow_domain", function(domain)    return true    end)    auto_ssl:set("ca", "https://acme-staging-v02.api.letsencrypt.org/directory")auto_ssl:init()  }

Once everything is working fine, you can remove the line auto_ssl:set("ca", "https://acme-staging-v02.api.letsencrypt.org/directory") and the plugin would start using LetsEncrypt's production URL.

(See more configuration options at: https://github.com/auto-ssl/lua-resty-auto-ssl)

Now that we've configured the configuration file, we need to restart the OpenResty process so that the new changes are reflected. Since the OpenResty installation automatically creates a systemctl service for it, we can use that to restart the service.

Run sudo systemctl restart openresty to restart Openresty.

And voila, TLS certificates will now automatically get generated for any domain/subdomain that is pointing to your server's IP address.

Note:
An astute reader would have noticed a contradiction with what I've said earlier. We've used the www-data user in the config file to precisely get away from running OpenResty as the root user, but here we are starting OpenResty as a root user with sudo systemctl restart openresty.

:/

What's the deal here?

The answer lies in the way *nix systems handle port permissions. In Linux, port numbers below 1024 are reserved for the superuser (i.e. root), so a normal user cannot bind to ports in this range. Since our web server listens on port 80(HTTP) and 443(HTTPS), OpenResty/Nginx needs to run as root.
But, this is precisely what we're trying to avoid right?
Nginx solves this problem by having two sorts of processes, the master process and the worker process. The idea is that the master process runs as root and binds to privileged ports. The worker processes on the other hand run as the user specified in the config file( www-data here). As you would have already guessed, these worker processes do the actual request processing.

We would love to discuss your thoughts about this article, feel free to use the comments section below. Also, we would love you to check out ToolJet on GitHub: https://github.com/ToolJet/ToolJet/.