ClusterControl is a great tool to deploy and manage databases clusters - if you are into MySQL, you can easily deploy clusters based on both traditional MySQL master-slave replication, Galera Cluster or MySQL NDB Cluster. To achieve high availability, deploying a cluster is not enough though. Nodes may (and will most probably) go down, and your system has to be able to adapt to those changes.
This adaptation can happen at different levels. You can implement some kind of logic within the application - it would check the state of cluster nodes and direct traffic to the ones which are reachable at the given moment. You can also build a proxy layer which will implement high availability in your system. In this blog post, we’d like to share some tips on how you can achieve that using ClusterControl.
Deploying HAProxy using the ClusterControl
HAProxy is the standard - one of the most popular proxies used in connection with MySQL (but not only, of course). ClusterControl supports deployment and monitoring of HAProxy nodes. It also helps to implement high availability of the proxy itself using keepalived.
Deployment is pretty simple - you need to pick or fill in the IP address of a host where HAProxy will be installed, pick port, load balancing policy, decide if ClusterControl should use existing repository or the most recent source code to deploy HAProxy. You can also pick which backend nodes you’d like to have included in the proxy configuration, and whether they should be active or backup.
By default, the HAProxy instance deployed by ClusterControl won’t work correctly with a master-slave replication setup - it’s designed to implement round-robin type of load-balancing (e.g., for Galera Cluster where all nodes are writeable). There’s a way to go around this issue, though - in the following repository you can find a check script which is intended to work with MySQL Replication. You will need to replace the check deployed by ClusterControl with this particular file.
Keepalived is used to add high availability to the proxy layer. When you have at least two HAProxy nodes in your system, you can install Keepalived from the ClusterControl UI.
You’ll have to pick two HAProxy nodes and they will be configured as an active - standby pair. A Virtual IP would be assigned to the active server and, should it fail, it will be reassigned to the standby proxy. This way you can just connect to the VIP and all your queries will be routed to the currently active and working HAProxy node.
You can find more details in how the internals are configured by reading through our HAProxy tutorial.
Deploying MaxScale using ClusterControl
While HAProxy is a rock-solid proxy and very popular choice, it lacks database awareness, e.g., read-write split. The only way to do it in HAProxy is to create two backends and listen on two ports - one for reads and one for writes. This is, usually, fine but it requires you to implement changes in your application - the application has to understand what is a read and what is a write, and then direct those queries to the correct port. It’d be much easier to just connect to a single port and let the proxy decide what to do next - this is something HAProxy cannot do as what it does is just routing packets - no packet inspection is done and, especially, it has no understanding of the MySQL protocol.
MaxScale solves this problem - it talks MySQL protocol and it can (among other things) perform a read-write split. Installation of MaxScale from ClusterControl is simple - you want to go to Manage -> Load Balancer section and fill the “Install MaxScale” tab with the required data.
In short, we need to pick where MaxScale will be installed, what admin user and password it should have, which user it should use to connect to the database. Next, we can pick number of threads MaxScale should use, ports and which nodes should be added to the loadbalancer.
By default MaxScale is configured with two ways of accessing the database. You can use Round Robin listener on port 4006 - it will split connections between the available nodes in a round-robin fashion. If you want to use MaxScale’s ability to perform a read/write split, you need to connect to port 4008. Once connected,MaxScale will begin to parse your MySQL traffic and route it according to what queries you execute. In short, SELECT queries will be routed to slaves (or, in case of Galera Cluster, all nodes except of one picked as a master), remaining traffic will hit the master. Explicitly opened transactions will also open on the master only. In MySQL replication, the master is self-explanatory - it’s a master node which will be used by MaxScale. In Galera Cluster things are slightly different as it’s a multi-master environment. What MaxScale does is to check wsrep_local_index value on all Galera nodes - the one with the lowest index will be treated as a master. In case master goes down, another lowest-valued node is picked.
MaxScale, as every proxy, can become a single point of failure and it has to be made redundant to achieve high availability. There are a couple of methods to do that. One of them is to collocate MaxScale on the web nodes. The idea here is that, most of the time, the MaxScale process will work just fine and the reason for its unavailability is that the whole node went down. In such case, if MaxScale is collocated with the web node, not much harm has been done because that particular web node will not be available either.
Another method, not supported directly from the ClusterControl UI, is to use Keepalived in a similar way like we did in the case of HAProxy.
You can do most of the hard work from the UI - you need to deploy MaxScale on two hosts. Then, you want to deploy HAProxy on the same two nodes - so that ClusterControl would allow you to install Keepalived. That’s what you need to do next - install Keepalived, making sure the IP addresses of HAProxy you picked are the same as the IPs of your MaxScale nodes. Once Keepalived has been set up by ClusterControl, you can just remove the HAProxy nodes.
Next step requires CLI access - you need to log into both MaxScale (and Keepalived) nodes and edit the Keepalived configuration. On Centos it is located in /etc/keepalived/keepalived.conf. What you need to do is, basically, pass it through sed ‘s/haproxy/maxscale/g’ - replace all mentions of haproxy with maxscale and then apply changes through restart of Keepalived on both nodes. Last step would be to edit MaxScale’s configuration and make sure it listens on either Virtual IP or on all interfaces - restart MaxScale to apply your changes and you are all set: your MaxScale health will be monitored by Keepalived and, in case of failure of the active node, Virtual IP will be moved to the standby node.
Given how MaxScale decides which one of the nodes in a Galera Cluster is a “master”, as long as you do not use any stickiness (by default, the master node is a node with the lowest wsrep_local_index which means that old master will become a master again, if it ever comes back online - you can change that so MaxScale would stick to the new master), all MaxScale nodes will see the cluster in the same way. You just have to make sure all nodes of the cluster have been added to MaxScale.