Xiang

Powered by Jekyll, with Hyde theme by @mdo

Multitenant Phoenix application deployment with SSL using HAProxy

01 Aug 2018

Table of Contents

Recently I needed to deploy two Phoenix applications on a CentOS server that I was provided with. A normal way to do so would be to use subdomains, i.e. using appA.domain.com and appB.domain.com. However, in this case I was not able to configure the DNS records for the server. I had to thus resort to appending path segments to the domain, i.e. domain.com/appA and domain.com/appB, and rely on a reverse proxy to route the request to the corresponding app.

HAProxy seemed an attractive option for the reverse proxy, as its configuration syntax is clean and straightforward. However, there is relatively few documentation for it compared with Nginx, and it took me a while to figure everything out. I’d like to share my experience in this article.

The commands are tested on a fresh CentOS 7.5 VM.

Installing HAProxy

This section documents the steps to install the newest HAProxy and get it running. You may also just choose to install it using the package manager of your system and skip ahead to the next section after verifying the installation. Part of this section is based on this online guide.

The HAProxy version in the CentOS repo is not up to date. To install the newest version of HAProxy, first download the source code from http://www.haproxy.org/#down.

wget http://www.haproxy.org/download/1.8/src/haproxy-1.8.13.tar.gz
tar -xzvf haproxy-1.8.13.tar.gz
cd haproxy-1.8.13

You may first need to install the dependencies with

sudo yum install gcc pcre-static pcre-devel openssl openssl-devel

before the compilation.

You can then compile HAProxy with

sudo make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1

A few notes about the flags:

After the compilation finishes, run

sudo make install

to install it.

Then, some initialization steps need to be performed:

sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy
sudo mkdir -p /etc/haproxy
sudo mkdir -p /var/lib/haproxy
# Useful if you want to enable the default stats page for HAProxy
sudo touch /var/lib/haproxy/stats
sudo cp ~/haproxy-1.8.13/examples/haproxy.init /etc/init.d/haproxy
sudo chmod 755 /etc/init.d/haproxy
sudo systemctl daemon-reload
sudo chkconfig haproxy on
sudo useradd -r haproxy

Before we are able to use HAProxy, we may also need to open up the ports 80 and 443 to outside traffic, if they aren’t open already. You can either do it by directly modifying iptables, or use the firewall-cmd frontend that comes with CentOS 7. The following is the direct iptables command:

sudo iptables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo systemctl reload iptables

Now we can add a basic configuration file with sudo vi /etc/haproxy/haproxy.cfg:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    option forwardfor
    option http-server-close

frontend http_front
    bind *:80
    stats uri /haproxy?stats
    default_backend http_back

backend http_back
    balance roundrobin
    server web1 127.0.0.1:80 check

We can then start up HAProxy with

sudo systemctl start haproxy

You should be able to see the HAProxy stats page at http://example.com/haproxy?stats. This means you have successfully installed HAProxy.

Phoenix-specific HAProxy configurations

Now that HAProxy runs normally, let’s edit the configuration file /etc/haproxy/haproxy.cfg so that it is able to serve two different Phoenix apps, distinguishing them using path segments.

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    option forwardfor
    option http-server-close

frontend http_front
    # We don't really need the stats page now.
    # stats uri /haproxy?stats
    bind *:80
    acl appA_url url_beg /appA
    acl appB_url url_beg /appB
    use_backend appA if appA_url
    use_backend appB if appB_url

backend appA
    http-request set-path %[path,regsub(/appA/?,/)]
    balance roundrobin
    server web1 127.0.0.1:4001 check

backend appB
    http-request set-path %[path,regsub(/appB/?,/)]
    balance roundrobin
    server web1 127.0.0.1:5001 check

  scope "/", App do
    pipe_through :browser
    get "/", PageController, :index
  end

which expects to serve the homepage at http://example.com/, the router will still receive the incoming HTTP requests as if they were for http://example.com/ instead of for http://example.com/appA/ , and thus serve the pages correctly.

(We will also need to change a part of Phoenix configuration for this to work. This will be covered in the next section.)

Don’t forget to restart HAProxy with sudo systemctl restart haproxy after saving the changes.

Checking for errors in the HAProxy configuration file

N.B.: Sometimes the haproxy service might fail to start due to errors in `haproxy.cfg`. The following command checks for errors in the config file:

sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg

Configuring Phoenix Apps

Now we need to also configure the relevant Phoenix apps so that they work in conjunction with HAProxy. For this we need to modify App.Endpoint in config/prod.exs:

config :app, App.Endpoint,
  url: [host: "example.com", path: "/appA"],
  http: [port: 4001],
  cache_static_manifest: "priv/static/cache_manifest.json"

Needless to say, the same edits are to be performed on appB, with the path as "/appB" and the port as 5001.

To ensure that the applications automatically start on boot, create a systemd service for each app in the folder /lib/systemd/system:

app-a.service:

[Unit]
Description=AppA

[Service]
Type=simple
User=username
Group=groupname
Restart=on-failure
Environment=MIX_ENV=prod "PORT=4001"
Environment=LANG=en_US.UTF-8

WorkingDirectory=/path/to/appA
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

app-b.service:

[Unit]
Description=AppB

[Service]
Type=simple
User=username
Group=groupname
Restart=on-failure
Environment=MIX_ENV=prod "PORT=5001"
Environment=LANG=en_US.UTF-8

WorkingDirectory=/path/to/appB
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

and enable them with sudo systemctl enable app-a.service sudo systemctl enable app-b.service. Start them with systemctl start.

If everything is set up correctly up to this point, you should now be able to access the applications at http://example.com/appA and http://example.com/appB.

(The systemd configuration is taken from an ElixirForum thread by yurko, where further details and discussions can be found.)

Working with Distillery

Instead of the plain

WorkingDirectory=/path/to/appB
ExecStart=/usr/local/bin/mix phoenix.server

we might want to use distillery to package our Phoenix apps. This also allows us to stop and restart the apps more easily. The corresponding systemd config would be:

[Service]
Type=simple
User=user
Group=group
WorkingDirectory=/path/to/appB
ExecStart=/path/to/appB/_build/prod/rel/appB/bin/appB foreground
ExecStop=/path/to/appB/_build/prod/rel/appB/bin/appB stop
Environment=LANG=en_US.UTF-8
Environment=MIX_ENV=prod
Environment=RELEASE_MUTABLE_DIR=/path/to/appB/_build/prod/rel/appB/var/tmp
LimitNOFILE=65536
UMask=0027
SyslogIdentifier=appB
Restart=always
RestartSec=5

Adding TLS support via HAProxy

Normally we’d want to serve our apps via HTTPS, which HAProxy also supports.

First, we need to concatenate all the related keys/certs (.pem files) into one. The order required by HAProxy is:

For example, if your cert files are stored under /etc/pki/tls/certs/, you would run

DOMAIN='example.com' sudo -E bash -c 'cat /etc/pki/tls/private/key_example.pem  /etc/pki/tls/certs/cert_example.pem /etc/pki/tls/certs/chain.pem > /etc/haproxy/certs/$DOMAIN.pem'

If you use letsencrypt, it’s likely that you already have a fullchain.pem that you can use directly.

DOMAIN='example.com' sudo -E bash -c 'cat /etc/letsencrypt/live/privkey.pem /etc/letsencrypt/live/fullchain.pem > /etc/haproxy/certs/$DOMAIN.pem'

There is also a letsencrypt plugin for HAProxy, though I haven’t used it myself.

Then, we need to modify /etc/haproxy/haproxy.cfg again:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 2048
    # TLS
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    # Disable SSL v3, which is insecure
    ssl-default-bind-options no-sslv3

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    option forwardfor
    option http-server-close

frontend http_front
    bind *:80
    http-request add-header X-Forwarded-Proto http
    acl appA_url url_beg /appA
    acl appB_url url_beg /appB
    use_backend appA if appA_url
    use_backend appB if appB_url

frontend https_front
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
    http-request add-header X-Forwarded-Proto https
    acl appA_url url_beg /appA
    acl appB_url url_beg /appB
    use_backend appA if appA_url
    use_backend appB if appB_url

backend appA
    http-request redirect scheme https if ! { ssl_fc }
    http-request set-path %[path,regsub(/appA/?,/)]
    balance roundrobin
    server web1 127.0.0.1:4001 check

backend appB
    http-request redirect scheme https if ! { ssl_fc }
    http-request set-path %[path,regsub(/appB/?,/)]
    balance roundrobin
    server web1 127.0.0.1:5001 check

Restart haproxy with sudo systemctl restart haproxy. Now, the apps should be up and running under https://example.com/appA and https://example.com/appB.

(Here is an alternative configuration for the frontend and backend portions):

frontend http_front
    bind *:80
    http-request add-header X-Forwarded-Proto http
    http-request redirect scheme https

frontend https_front
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
    http-request add-header X-Forwarded-Proto https
    acl appA_url url_beg /appA
    acl appB_url url_beg /appB
    use_backend appA if appA_url
    use_backend appB if appB_url

backend appA
    http-request set-path %[path,regsub(/appA/?,/)]
    balance roundrobin
    server web1 127.0.0.1:4001 check

backend appB
    http-request set-path %[path,regsub(/appB/?,/)]
    balance roundrobin
    server web1 127.0.0.1:5001 check