State Snapshot Transfer (SST) is one of the two ways used by Galera to perform initial syncing when a node is joining a cluster, until the node is declared as synced and part of the “primary component”. Depending on the dataset size and workload, SST could be lightning fast, or an expensive operation which will bring your database service down on its knees.

SST can be performed using 3 different methods:

• mysqldump
• rsync (or rsync_wan)
• xtrabackup (or xtrabackup-v2, mariabackup)

Most of the time, xtrabackup-v2 and mariabackup are the preferred options. We rarely see people running on rsync or mysqldump in production clusters.

The Problem

When SST is initiated, there are several processes triggered on the joiner node, which are executed by the "mysql" user:

$ ps -fu mysql
UID         PID   PPID  C STIME TTY          TIME CMD
mysql    117814 129515  0 13:06 ?        00:00:00 /bin/bash -ue /usr//bin/wsrep_sst_xtrabackup-v2 --role donor --address 192.168.55.173:4444/xtrabackup_sst//1 --socket /var/lib/mysql/mysql.sock --datadir
mysql    120036 117814 15 13:06 ?        00:00:06 innobackupex --no-version-check --tmpdir=/tmp/tmp.pMmzIlZJwa --user=backupuser --password=x xxxxxxxxxxxxxx --socket=/var/lib/mysql/mysql.sock --galera-inf
mysql    120037 117814 19 13:06 ?        00:00:07 socat -u stdio TCP:192.168.55.173:4444
mysql    129515      1  1 Oct27 ?        01:11:46 /usr/sbin/mysqld --wsrep_start_position=7ce0e31f-aa46-11e7-abda-56d6a5318485:4949331

While on the donor node:

mysql     43733      1 14 Oct16 ?        03:28:47 /usr/sbin/mysqld --wsrep-new-cluster --wsrep_start_position=7ce0e31f-aa46-11e7-abda-56d6a5318485:272891
mysql     87092  43733  0 14:53 ?        00:00:00 /bin/bash -ue /usr//bin/wsrep_sst_xtrabackup-v2 --role donor --address 192.168.55.172:4444/xtrabackup_sst//1 --socket /var/lib/mysql/mysql.sock --datadir /var/lib/mysql/  --gtid 7ce0e31f-aa46-11e7-abda-56d6a5318485:2883115 --gtid-domain-id 0
mysql     88826  87092 30 14:53 ?        00:00:05 innobackupex --no-version-check --tmpdir=/tmp/tmp.LDdWzbHkkW --user=backupuser --password=x xxxxxxxxxxxxxx --socket=/var/lib/mysql/mysql.sock --galera-info --stream=xbstream /tmp/tmp.oXDumYf392
mysql     88827  87092 30 14:53 ?        00:00:05 socat -u stdio TCP:192.168.55.172:4444

SST against a large dataset (hundreds of GBytes) is no fun. Depending on the hardware, network and workload, it may take hours to complete. Server resources may be saturated during the operation. Despite throttling is supported in SST (only for xtrabackup and mariabackup) using --rlimit and --use-memory options, we are still exposed to a degraded cluster when you are running out of majority active nodes. For example, if you are unlucky enough to find yourself with only one out of three nodes running. Therefore, you are advised to perform SST during quiet hours. You can, however, avoid SST by taking some manual steps, as described in this blog post.

Stopping an SST

Stopping an SST needs to be done on both the donor and the joiner nodes. The joiner triggers SST after determining how big the gap is when comparing the local Galera seqno with cluster's seqno. It executes the wsrep_sst_{wsrep_sst_method} command. This will be picked by the chosen donor, which will start streaming out data to the joiner. A donor node has no capabilities of refusing to serve snapshot transfer, once selected by Galera group communication, or by the value defined in wsrep_sst_donor variable. Once the syncing has started and you want to revert the decision, there is no single command to stop the operation.

The basic principle when stopping an SST is to:

• Make the joiner look dead from a Galera group communication point-of-view (shutdown, fence, block, reset, unplug cable, blacklist, etc)
• Kill the SST processes on the donor

One would think that killing the innobackupex process (kill -9 {innobackupex PID}) on the donor would be enough, but that is not the case. If you kill the SST processes on donor (or joiner) without fencing off the joiner, Galera still can see the joiner as active and will mark the SST process as incomplete, thus respawning a new set of processes to continue or start over again. You will be back to square one. This is the expected behaviour of /usr/bin/wsrep_sst_{method} script to safeguard SST operation which is vulnerable to timeouts (e.g., if it is long-running and resource intensive).

Let's look at an example. We have a crashed joiner node that we would like to rejoin the cluster. We would start by running the following command on the joiner:

$ systemctl start mysql # or service mysql start

A minute later, we found out that the operation is too heavy at that particular moment, and decided to postpone it later during low traffic hours. The most straightforward way to stop an xtrabackup-based SST method is by simply shutting down the joiner node, and kill the SST-related processes on the donor node. Alternatively, you can also block the incoming ports on the joiner by running the following iptables command on the joiner:

$ iptables -A INPUT -p tcp --dport 4444 -j DROP
$ iptables -A INPUT -p tcp --dport 4567:4568 -j DROP

Then on the donor, retrieve the PID of SST processes (list out the processes owned by "mysql" user):

$ ps -u mysql
   PID TTY          TIME CMD
117814 ?        00:00:00 wsrep_sst_xtrab
120036 ?        00:00:06 innobackupex
120037 ?        00:00:07 socat
129515 ?        01:11:47 mysqld

Finally, kill them all except the mysqld process (you must be extremely careful to NOT kill the mysqld process on the donor!):

$ kill -9 117814 120036 120037

Then, on the donor MySQL error log, you should notice the following line appearing after ~100 seconds:

2017-10-30 13:24:08 139722424837888 [Warning] WSREP: Could not find peer: 42b85e82-bd32-11e7-87ae-eff2b8dd2ea0
2017-10-30 13:24:08 139722424837888 [Warning] WSREP: 1.0 (192.168.55.172): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)

At this point, the donor should return to the "synced" state as reported by wsrep_local_state_comment and the SST process is completely stopped. The donor is back to its operational state and is able to serve clients in full capacity.

For the cleanup process on the joiner, you can simply flush the iptables chain:

$ iptables -F

Or simply remove the rules with -D flag:

$ iptables -D INPUT -p tcp --dport 4444 -j DROP
$ iptables -D INPUT -p tcp --dport 4567:4568 -j DROP

The similar approach can be used with other SST methods like rsync, mariabackup and mysqldump.

Throttling an SST (xtrabackup method only)

Depending on how busy the donor is, it's a good approach to throttle the SST process so it won't impact the donor significantly. We've seen a number of cases where, during catastrophic failures, users were desperate to bring back a failed cluster as a single bootstrapped node, and let the rest of the members catch up later. This attempt reduces the downtime from the application side, however, it creates additional burden on this “one-node cluster”, while the remaining members are still down or recovering.

Xtrabackup can be throttled with --throttle=<rate of IO/sec> to simply limit the number of IO operation if you are afraid that it will saturate your disks, but this option is only applicable when running xtrabackup as a backup process, not as an SST operator. Similar options are available with rlimit (rate limit) and can be combined with --use-memory to limit the RAM usage. By setting up values under [sst] directive inside the MySQL configuration file, we can ensure that the SST operation won't put too much load on the donor, even though it can take longer to complete. On the donor node, set the following:

[sst]
rlimit=128k
inno-apply-opts="--use-memory=200M"

More details on the Percona Xtrabackup SST documentation page.

However, there is a catch. The process could be so slow that it will never catch up with the transaction logs that InnoDB is writing, so SST might never complete. Generally, this situation is very uncommon, unless if you really have a very write-intensive workload or you allocate very limited resources to SST.

Conclusions

SST is critical but heavy, and could potentially be a long-running operation depending on the dataset size and network throughput between the nodes. Regardless of the consequences, there are still possibilities to stop the operation so we can have a better recovery plan at a better time.

Having a load balancer or reverse proxy in front of your MySQL or MariaDB server does add a little bit of complexity to your database setup, which might lead to some, things behaving differently. Theoretically, a load balancer which sits in front of MySQL servers (for example an HAProxy in front of a Galera Cluster) should just act like a connection manager and distribute the connections to the backend servers according to some balancing algorithm. MySQL, on the other hand, has its own way of managing client connections. Ideally, we would need to configure these two components together so as to avoid unexpected behaviours, and narrow down the troubleshooting surface when debugging issues.

If you have such setup, it is important to understand these components as they can impact the overall performance of your database service. In this blog post, we will dive into MySQL's max_connections and HAProxy maxconn options respectively. Note that timeout is another important parameter that we should know, but we are going to cover that in a separate post.

MySQL's Max Connections

The number of connections permitted to a MySQL server is controlled by the max_connections system variable. The default value is 151 (MySQL 5.7).

To determine a good number for max_connections, the basic formulas are:

Where,

And,

By using the above formulas, we can calculate a suitable max_connections value for this particular MySQL server. To start the process, stop all connections from clients and restart the MySQL server. Ensure you only have the minimum number of processes running at that particular moment. You can use 'mysqladmin' or 'SHOW PROCESSLIST' for this purpose:

$ mysqladmin -uroot -p processlist
+--------+------+-----------+------+---------+------+-------+------------------+----------+
| Id     | User | Host      | db   | Command | Time | State | Info             | Progress |
+--------+------+-----------+------+---------+------+-------+------------------+----------+
| 232172 | root | localhost | NULL | Query   |    0 | NULL  | show processlist |    0.000 |
+--------+------+-----------+------+---------+------+-------+------------------+----------+
1 row in set (0.00 sec)

From the above output, we can tell that only one user is connected to the MySQL server which is root. Then, retrieve the available RAM (in MB) of the host (look under 'available' column):

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3778        1427         508         148        1842        1928
Swap:          2047           4        2043

Just for the info, the 'available' column gives an estimate of how much memory is available for starting new applications, without swapping (only available in kernel 3.14+).

Then, specify the available memory, 1928 MB in the following statement:

mysql> SELECT ROUND((1928 - (ROUND((@@innodb_buffer_pool_size + @@innodb_additional_mem_pool_size + @@innodb_log_buffer_size + @@query_cache_size + @@tmp_table_size + @@key_buffer_size) / 1024 / 1024))) / (ROUND(@@read_buffer_size + @@read_rnd_buffer_size + @@sort_buffer_size + @@thread_stack + @@join_buffer_size + @@binlog_cache_size) / 1024 / 1024)) AS 'Possible Max Connections';
+--------------------------+
| Possible Max Connections |
+--------------------------+
|                      265 |
+--------------------------+

From this example, we can have up to 265 MySQL connections simultaneously according to the available RAM the host has. It doesn't make sense to configure a higher value than that. Then, append the following line inside MySQL configuration file, under the [mysqld] directive:

max_connections = 265

Restart the MySQL service to apply the change. When the total simultaneous connections reaches 265, you would get a "Too many connections" error when trying to connect to the mysqld server. This means that all available connections are in use by other clients. MySQL actually permits max_connections+1 clients to connect. The extra connection is reserved for use by accounts that have the SUPER privilege. So if you face this error, you should try to access the server as a root user (or any other SUPER user) and look at the processlist to start the troubleshooting.

HAProxy's Max Connections

HAProxy has 3 types of max connections (maxconn) - global, defaults/listen and default-server. Assume an HAProxy instance configured with two listeners, one for multi-writer listening on port 3307 (connections are distributed to all backend MySQL servers) and another one is single-writer on port 3308 (connections are forwarded to a single MySQL server):

global
    ...
    maxconn 2000 #[a]
    ...
defaults
    ...
    maxconn 3 #[b]
    ...
listen mysql_3307
    ...
    maxconn 8 #[c]
    balance leastconn
    default-server port 9200 maxqueue 10 weight 10 maxconn 4 #[d]
    server db1 192.168.55.171 check
    server db2 192.168.55.172 check
    server db3 192.168.55.173 check
listen mysql_3308
    ...
    default-server port 9200 maxqueue 10 weight 10 maxconn 5 #[e]
    server db1 192.168.55.171 check
    server db2 192.168.55.172 check backup #[f]

Let’s look at the meaning of some of the configuration lines:

global.maxconn [a]

The total number of concurrent connections that are allowed to connect to this HAProxy instance. Usually, this value is the highest value of all. In this case, HAProxy will accept a maximum of 2000 connections at a time and distribute them to all listeners defined in the HAProxy process, or worker (you can run multiple HAProxy processes using nbproc option).

HAProxy will stop accepting connections when this limit is reached. The "ulimit-n" parameter is automatically adjusted to this value. Since sockets are considered equivalent to files from the system perspective, the default file descriptors limit is rather small. You will probably want to raise the default limit by tuning the kernel for file descriptors.

defaults.maxconn [b]

Defaults maximum connections value for all listeners. It doesn't make sense if this value is higher than global.maxconn.

If "maxconn" line is missing under the "listen" stanza (listen.maxconn), the listener will obey this value. In this case, mysql_3308 listener will get maximum of 3 connections at a time. To be safe, set this value equal to global.maxconn, divided by the number of listeners. However, if you would like to prioritize other listeners to have more connections, use listen.maxconn instead.

listen.maxconn [c]

The maximum connections allowed for the corresponding listener. The listener takes precedence over defaults.maxconn if specified. It doesn't make sense if this value is higher than global.maxconn.

For a fair distribution of connections to backend servers like in the case of a multi-writer listener (mysql_3307), set this value as listen.default-server.maxconn multiply by the number of backend servers. In this example, a better value should be 12 instead of 8 [c]. If we chose to stick with this configuration, db1 and db2 are expected to receive a maximum of 3 connections each, while db3 will receive a maximum of 2 connections (due to leastconn balancing), which amounts to 8 connections in total. It won't hit the limit as specified in [d].

For single-writer listener (mysql_3308) where connections should be allocated to one and only one backend server at a time, set this value to be the same or higher than listen.default-server.maxconn.

listen.default-server.maxconn [d][e]

This is the maximum number of connections that every backend server can receive at a time. It doesn't make sense if this value is higher than listen.maxconn or defaults.maxconn. This value should be lower or equal to MySQL's max_connections variable. Otherwise, you risk exhausting the connections to the backend MySQL server, especially when MySQL's timeout variables are configured lower than HAProxy's timeouts.

In this example, we've set each MySQL server to only get a maximum of 4 connections at a time for multi-writer Galera nodes [d]. While the single-writer Galera node will get a maximum of 3 connections at a time, due to the limit that applies from [b]. Since we specified "backup" [f] to the other node, the active node will at once get all 3 connections allocated to this listener.

The above explanation can be illustrated in the following diagram:

To sum up the connections distribution, db1 is expected to get a maximum number of 6 connections (3 from 3307 + 3 from 3308). The db2 will get 3 connections (unless if db1 goes down, where it will get additional 3) and db3 will stick to 2 connections regardless of topology changes in the cluster.

Connection Monitoring with ClusterControl

With ClusterControl, you can monitor MySQL and HAProxy connection usage from the UI. The following screenshot provides a summary of the MySQL connection advisor (ClusterControl -> Performance -> Advisors) where it monitors the current and ever used MySQL connections for every server in the cluster:

For HAProxy, ClusterControl integrates with HAProxy stats page to collect metrics. These are presented under the Nodes tab:

From the above screenshot, we can tell that each backend server on multi-writer listener gets a maximum of 8 connections. 4 concurrent sessions are running. These are highlighted in the top red square, while the single-writer listener is serving 2 connections and forwarding them to a single node respectively.

Conclusion

Configuring the maximum connections for HAProxy and MySQL server is important to ensure good load distribution to our database servers, and protect the MySQL servers from overloading or exhausting its connections.

Most organizations do not realize they have a problem with database backups until they need to restore the data and find it’s not there or not in the form that they were expecting.

The designated administrator managing the database environments must be prepared for situations where any failure may cause an impact to the availability, integrity, or usability of a database or application. Reacting to these failures is a key component of the administrator’s responsibilities and their ability to react correctly depends on whether they have a well-planned strategy for database backups and recovery.

Pixar’s “Toy Story 2” famously almost never happened due a command line mis-run causing the movie to be deleted and an in-effective backup strategy in place. That movie went on to take in nearly $500 million dollars worldwide in box office… money that, without the fact that one team member made their own personal backup, may have never been made.

ClusterControl provides you with sophisticated backup and failover features using a point-and-click interface to easily restore your data if something goes wrong and can be your DBA-sidekick when it comes to building an effective backup strategy. There are many aspects to consider though when building such a strategy.

Here at Severalnines we have database experts who have written much about the topic and in this blog we will collect the top resources to help you build your own database backup strategy for MySQL and MariaDB databases more specifically.

If you are running a MySQL or MariaDB environment our best resource for you is the free whitepaper “The DevOps Guide to Database Backups for MySQL and MariaDB.” The guide covers the two most popular backup utilities available for MySQL and MariaDB, namely mysqldump and Percona XtraBackup. It further covers topics such as how database features like binary logging and replication can be leveraged in backup strategies and provides best practices that can be applied to high availability topologies in order to make database backups reliable, secure and consistent.

In addition to the whitepaper there are two webinars focused on backups that you can watch on-demand. “MySQL Tutorial - Backup Tips for MySQL, MariaDB & Galera Cluster” and “Become a MySQL DBA - Deciding on a Relevant Backup Solution.” Each of these webinars offer tips and best practices on building a backup plan and summarize much of the content that is available throughout our website.

Here are our most popular and relevant blogs on the topic...

Overview of Backup and Restores

In the blog “Become a MySQL DBA - Backup and Restore” we provide a high-level overview of backups and restores when managing a MySQL environment. Included in the blog is an overview of different backup methodologies, overview of logical and physical backups, and some best practices and guidelines you can follow.

The Impact of MySQL Storage Engines on Backups

In the blog “The Choice of MySQL Storage Engine and its Impact on Backup Procedures” we discuss how the selection of different types of storage engines (like MyISAM, InnoDB, etc) can have an impact on your backup strategy.

Building a Backup Strategy and Plan

In our blog “mysqldump or Percona XtraBackup? Backup Strategies for MySQL Galera Cluster” we discuss the different options available to you when making your backup and restore plan with special focus on doing it in a way that does not affect performance.

Making Sure You Perform a Good Backup

In our blog “How to Perform Efficient Backups for MySQL and MariaDB” we discuss a number of ways to backup MySQL and MariaDB, each of which comes with pros and cons.

Using ClusterControl for Backups

In the blog “ClusterControl Tips & Tricks - Best Practices for Database Backups” we should how to effectively manage your backup plan using ClusterControl. With ClusterControl you can schedule logical or physical backups with failover handling and easily restore backups to bootstrap nodes or systems.

Additional Blogs

There are several more blogs that have been written over the years that can also aid you in ensuring your backups are performed successfully and efficiently. Here’s a list of them...

Full Restore of a MySQL or MariaDB Galera Cluster from Backup

Performing regular backups of your database cluster is imperative for high availability and disaster recovery. This blog post provides a series of best practices on how to fully restore a MySQL or MariaDB Galera Cluster from backup.

Read the Blog

What’s New in ClusterControl 1.4 - Backup Management

This blog post covers the new backup features available in ClusterControl version 1.4.

Read the Blog

ClusterControl Tips & Tricks: Customizing your Database Backups

ClusterControl follows some best practices to perform backups using mysqldump or Percona xtrabackup. Although these work for the majority of database workloads, you might still want to customize your backups. This blog shows you how.

Read the Blog

Architecting for Failure - Disaster Recovery of MySQL/MariaDB Galera Cluster

Whether you use unbreakable private data centers or public cloud platforms, Disaster Recovery (DR) is indeed a key issue. This is not about copying your data to a backup site and being able to restore it, this is about business continuity and how fast you can recover services when disaster strikes.

Read the Blog

Using BitTorrent Sync to Transfer Database Backups Offsite

BitTorrent Sync is a simple replication application providing encrypted bidirectional file transfers that can run behind NAT and is specifically designed to handle large files. By leveraging the simplicity of Bittorrent Sync, we can transfer backup files away from our cluster, enhancing the backups availability and reducing the cost of broken backup, where you can regularly verify your backups off-site.

Read the Blog

How to Clone Your Database

If you are managing a production database, chances are high that you’ve had to clone your database to a different server than the production server. The basic method of creating a clone is to restore a database from a recent backup onto a different database server. Other methods include replicating from a source database while it is up, in which case it is important the original database be unaffected by any cloning procedure.

Read the Blog

Not Using MySQL? Here are some resources we have to help with other database technologies…

Become a MongoDB DBA: MongoDB Backups

This is our fifth post in the “Become a MongoDB DBA” blog series - how do you make a good backup strategy for MongoDB, what tools are available and what you should watch out for.

Read the Blog

Become a MongoDB DBA: Recovering your Data

This is our sixth post in the “Become a MongoDB DBA” blog series - how do you recover MongoDB using a backup.

Read the Blog

Become a PostgreSQL DBA - Logical & Physical PostgreSQL Backups

Taking backups is one of the most important tasks of a DBA - it is crucial to the availability and integrity of the data. Part of our Become a PostgreSQL DBA series, this blog post covers some of the backup methods you can use with PostgreSQL.

Read the Blog

Today we are excited to announce the 1.5 release of ClusterControl - the all-inclusive database management system that lets you easily deploy, monitor, manage and scale highly available open source databases - and load balancers - in any environment: on-premise or in the cloud.

ClusterControl 1.5 provides an array of exciting new backup functionalities to ensure that your data is secure and available whenever disaster strikes. The release also provides expanded PostgreSQL, MariaDB and MySQL NDB Cluster support.

Release Highlights

Cloud Services (AWS S3 and Google Cloud Storage)

• Manual upload or schedule backups to be uploaded after completion to the cloud.
• Download and restore backups from a cloud storage.

Backup (MySQL)

• Backup individual databases separately (mysqldump only).
• Upload, download and restore backups stored in the cloud.
• Trigger a verification and restore of a backup after N hours of completion.
• Rebuild a replication slave by staging it from an existing backup.
• Add a new replication slave by staging it from an existing backup.

PostgreSQL

• New backup method pg_basebackup which makes a binary copy of the database files.
• Synchronous replication failover (support for synchronous_standby_names).
• Support for HAProxy with Keepalived.
• Support for PostgreSql 10.

ProxySQL

• Mass import existing database users into ProxySQL.
• Add and modify scheduler scripts.

Misc

• MariaDB v10.2 support (Galera and MySQL Replication).
• MySQL Cluster(NDB) v7.5 support.
• Added support to show and filter DB status variables for MongoDB nodes.
• HTML formatted alarm and digest emails.
• Multiple NICs supports when deploying Load balancers (HAProxy).
• Continuous improvements to UX/UI and performance.
• New cmon-cloud process and clud client to handle cloud services.
• New Ops Report: Database Growth

View Release Details and Resources

• 1.5 ChangeLog
• ClusterControl Administration Guide
• Upgrade instructions

New and Enhanced Backup Functions

ClusterControl 1.5 marks the first steps towards providing full seamless integration between ClusterControl and the Cloud. It provides our users with the options of migrating their data to the cloud while still benefiting from the features and functionality that ClusterControl has to offer.

• AWS & Google Cloud Services Integration: ClusterControl now offers full integration for database backups for Amazon Web Services (AWS) and Google Cloud Services. Backups can now be executed, scheduled, downloaded and restored directly from your cloud provider of choice. This ability provides increased redundancy, better disaster recovery options, and benefits in both performance and cost savings.
• New Backup Wizard: ClusterControl 1.5 comes with a complete redesign of our backup wizard. The wizard lets you perform and schedule backups using a variety of options and technologies to ensure your data is protected, accurate and ready to be restored on a moment's notice.
• Single Database Backup & Restores for MySQL: With ClusterControl 1.5 you now have the ability to backup and restore individual MySQL databases rather than being forced to do it all at one time. This new feature can reduce the burden on those users who manage hundreds of databases within their environment.
• Automatic Backup Verification: ClusterControl 1.5 brings you peace of mind by automatically verifying the integrity of your backup data. It will perform a restore of your backup data, and alert in case of issues.
• Creating & Restoring Slaves from Backup: In the past a user would need to leverage their Master to create or restore new slaves… this creates an undue burden on the system as it performs a full backup on the master. In ClusterControl 1.5, you can now create new nodes or rebuild existing slaves from existing backup files, removing the burden on the master and ensuring you continue to operate at peak performance.

Improved Support for PostgreSQL

ClusterControl continues to expand its support for PostgreSQL. In our new 1.5 release, we can now support PostgreSQL 10 which offers a number of new features such as support for local replication, declarative table partitioning, improved query parallelism, quorum commit for synchronous replication, and more!

• PostgreSQL Proxy and Virtual IP: ClusterControl now supports failover of database connections with HAProxy, Keepalived and Virtual IP - a battle-tested combination.
• PostgreSQL Synchronous Replication Failover: ClusterControl 1.5 now fully supports Synchronous Replication Failover for PostgreSQL. This function minimizes the risk of data loss between Master and Slaves should something break with the Master before it has a chance to fully synchronize. This allows you to increase reliability across your servers.
• New Backup Method: PostgreSQL has an additional backup method pg_basebackup that can be used for online binary backups. Backups taken with pg_basebackup can be used later for point-in-time recovery and as the starting point for a log shipping or streaming replication standby servers.

Improved Support for ProxySQL

ProxySQL enables MySQL, MariaDB and Percona XtraDB database systems to easily manage intense, high-traffic database applications without losing availability.

• Script Scheduling: ClusterControl now provides support for the ProxySQL scheduler, a cron-like module which allows the load balancer to start external scripts on a regular interval. This provides users with full control of what scripts run and when, allowing you to tweak your instances performance at a very granular level.
• User Management: Rather than create or removing users one at a time from ProxySQL, ClusterControl now allows you to import a large batch of users.

Support for New Versions of MariaDB & MySQL NDB Cluster

• MariaDB 10.2: ClusterControl now supports MariaDB Server 10.2. This new version provides even greater integration with Galera Cluster, MariaDB’s HA solution of choice, and also features enhancements to SQL like windows functions, common table expressions, and JSON functions.
• MySQL NDB Cluster 7.5: ClusterControl now supports MySQL Cluster (NDB) 7.5. The latest MySQL Cluster release using the NDB file system offers a variety of improvements and optimizations. These improvements include enhancing the ndbinfo database, records-per-key optimization, and many more!

Additional New Functionalities

• New Notification Emails: In the 1.5 release of ClusterControl, we have revamped the design of the email notifications and alerts which are sent from the system to make them easier to read and understand.
• New Operational Report: ClusterControl 1.5 introduces a new operational report called “Database Growth.” The report shows a snapshot of your database as it exists today as well as data for the last day, last week, and last month - useful for capacity planning. The report also allows you to click into each period to deep dive into the tables to track growth at an even more granular level.

At the core of ClusterControl is its automation, as is ensuring that your data is is securely backed up and ready for restoration whenever something goes wrong. Having an effective backup strategy and disaster recovery plan is key to the success of any application or environment.

In our latest release, ClusterControl 1.5, we have introduced a number of enhancements for backing up MySQL and MariaDB-based systems.

One of the key improvements is the ability to backup from ClusterControl to the cloud provider of your choice. Cloud providers like Google Cloud Services and Amazon S3 each offer virtually unlimited storage, reducing local space needs. This allows you to retain your backup files longer, for as long as you would like and not have concerns around local disk space.

Let’s explore all the exciting new backup features for ClusterControl 1.5...

Backup/Restore Wizard Redesign

First of all, you will notice backup and restore wizards have been revamped to better improve the user experience. It will now load as a side menu on the right of the screen:

The backup list is also getting a minor tweak where backup details are displayed when you click on the particular backup:

You will be able to view backup location and which databases are inside the backup. There are also options to restore the backup or upload it into the cloud.

PITR Compatible Backup

ClusterControl performs the standard mysqldump backup with separate schema and data dumps. This makes it easy to restore partial backups. However, it breaks the consistency of the backup (schema and data are dumped in two separate sessions), thus it cannot be used to provision a slave or point-in-time recovery.

A mysqldump PITR-compatible backup contains one single dump file, with GTID info, binlog file and position. Thus, only the database node that produces binary log will have the "PITR compatible" option available, as highlighted in the screenshot below:

When PITR compatible option is toggled, the database and table fields are greyed out since ClusterControl will always perform the backup against all databases, events, triggers and routines of the target MySQL server.

The following lines will appear in the first ~50 lines of the completed dump file:

$ head -50 mysqldump_2017-11-07_072250_complete.sql
...
-- GTID state at the beginning of the backup
--
SET @@GLOBAL.GTID_PURGED='20dc5247-4a98-ee18-73af-5c79373388ee:1-1681';

--
-- Position to start replication or point-in-time recovery from
--
CHANGE MASTER TO MASTER_LOG_FILE='binlog.000001', MASTER_LOG_POS=2457790;
...

The information can be used to build slaves from backup, or perform point-in-time recovery together with binary logs, where you can start the recovery from the MASTER_LOG_FILE and MASTER_LOG_POS reported in the dump file using "mysqlbinlog" utility. Note that binary logs are not backed up by ClusterControl.

Build Slaves from Backup

Another feature is the ability to build a slave directly from a PITR-compatible backup, instead of doing it from a chosen master. This is a huge advantage as it offloads the master server. This option can be used with MySQL Replication or Galera Cluster. An existing backup can be used to rebuild an existing replication slave or add a new replication slave during the staging phase, as shown in the following screenshot:

Once the staging completes, the slave will connect to the chosen master and start catching up. Previously, ClusterControl performed a streaming backup directly from the chosen master using Percona Xtrabackup. This could impact performance of the master when scaling out a large dataset, despite the operation being non blocking on the master. With the new option, if the backup is stored on ClusterControl, only these hosts (ClusterControl + the slave) will be busy when staging the data on the slave.

Backup to Cloud

Backups can now be automatically uploaded in the cloud. This requires a ClusterControl module to be installed, called clustercontrol-cloud (Cloud integration module) and clustercontrol-clud (Cloud download/upload CLI) which are available in v1.5 and later. The upgrade instructions have been included with these packages and they come without any extra configuration. At the moment, the supported cloud platforms are Amazon Web Services and Google Cloud Platform. Cloud credentials are configured under ClusterControl -> Settings -> Integrations -> Cloud Providers.

When creating or scheduling a backup, you should see the following additional options when "Upload Backup to the cloud" is toggled:

The feature allows a one time upload or to schedule backups to be uploaded after completion (Amazon S3 or Google Cloud Storage). You can then download and restore the backups as required.

Custom Compression for mysqldump

This feature was in fact first introduced with ClusterControl v1.4.2 after its release. We added a backup compression level based on gzip. Previously, ClusterControl used the default backup compression (level 6) if the backup destination was on the controller node. The lowest compression (level 1 - fastest, less compression) was used if the backup destination was on the database host itself, to ensure minimal impact to the database during the compressing operation.

In this version, we have polished the compression aspect and you can now customize the compression level, regardless of the backup destination. When upgrading your ClusterControl instance, all the scheduled backups will be automatically converted to use level 6, unless you explicitly edit them in v1.5.

Backup compression is vital when your dataset is large, combined with a long backup retention policy, while storage space is limited. Mysqldump, which is text-based, can benefit from compression with savings of up to 60% of disk space of the original file size. On some occasions, the highest compression ratio is the best option to go, although it comes at the price of longer decompression when restoring.

Bonus Feature: Automatic Backup Verification

As old sysadmins say - A backup is not a backup if it's not restorable. Backup verification is something that is usually neglected by many. Some sysadmins have developed in-house routines for this, usually more manual than automated. Automating it is hard, mainly due to the complexity of the operation as a whole - starting from host provisioning, MySQL installation and preparation, backup files transfer, decompression, restore operation, verification procedures and finally cleaning up the system after the process. All these hassles make people neglect such an important aspect of a reliable backup. In general a backup restore test should be done at least once a month, or in case of significant changes in data size or database structure. Find a schedule that works for you and formalize it with a scheduled event.

ClusterControl can automate the backup verification by performing the restoration on a fresh host, without compromising any of the verification procedures mentioned above. This can be done after some delay, or right after the backup has completed. It will report the backup status based on the exit code of the restore operation, perform automatic shutdown if the backup is verified, or simply let the restored host run so you perform additional manual verifications on the data.

When creating or scheduling a backup, you will have additional options if "Verify Backup" is toggled:

If "Install Database Software" is enabled, ClusterControl will remove any existing MySQL installation on the target host and reinstall the database software with the same version as the existing MySQL server. Otherwise, if you have a specific setup for the restored host, you can skip this option. The rest of the options are self-explanatory.

Bonus Feature: Don’t Forget PostgreSQL

In addition to all this great functionality for MySQL and MariaDB ClusterControl 1.5 also now provides PostgreSQL with an additional backup method (pg_basebackup) that can be used for online binary backups. Backups taken with pg_basebackup can be used later for point-in-time recovery and as the starting point for a log shipping or streaming replication standby servers.

That’s it for now. Do give ClusterControl v1.5 a try, play around with the new features and let us know what you think.

Announced as part of the ClusterControl 1.5 release, we now provide full support for MariaDB version 10.2. This new version provides even greater integration with Galera Cluster, MariaDB’s HA solution of choice, and also features enhancements to SQL like window functions, common table expressions, and JSON functions.

MariaDB is the fastest growing open source database, reaching more than 60 million developers worldwide through its inclusion in every major Linux distribution, as well as a growing presence in the world’s leading cloud providers. Its widespread use across Linux distributions and cloud platforms, as well as its ease of use, have quickly made MariaDB the open source database standard for the modern enterprise.

MariaDB Server was listed in the recent OpenStack survey as the number one and two database technologies in use today.

What’s New in Version 10.2?

MariaDB Server 10.1 brought the default built-in integration of Galera Cluster to allows its users to achieve the ultimate in high availability. Severalnines was an early adopter of this clustering technology and was excited to see MariaDB embrace it for HA.

Here are some of the enhancements included in the new 10.2 version as announced by MariaDB…

• SQL enhancements like window functions, common table expressions and JSON functions allow new use cases for MariaDB Server
• Standard MariaDB Server replication has further optimizations
• Many area limitations have been removed, which allows easier use and there is no need for limitation handling on the application level
• MyRocks, a new storage engine developed by Facebook, has been introduced, which will further enrich the use cases for MariaDB Server (NOTE: This new Storage Engine is also now available for MariaDB deployments in ClusterControl, however ClusterControl does not yet support MyRocks specific monitoring.)

Window Functions

Window functions are popular in Business Intelligence (BI) where more complex report generation is needed based on a subset of the data, like country or sales team metrics. Another common use case is where time-series based data should be aggregated based on a time window instead of just a current record, like all rows inside a certain time span.

As analytics is becoming more and more important to end users, window functions deliver a new way of writing performance optimized analytical SQL queries, which are easy to read and maintain, and eliminates the need to write expensive subqueries and self-joins.

Common Table Expressions

Hierarchical and recursive queries are usually implemented using common table expressions (CTEs). They are similar to derived tables in a FROM clause, but by having an identification keyword WITH, the optimizer can produce more efficient query plans. Acting as an automatically created temporary and named result set, which is only valid for the time of the query, it can be used for recursive and hierarchical execution, and also allows for reuse of the temporary dataset. Having a dedicated method also helps to create more expressive and cleaner SQL code.

JSON Functions

JSON (JavaScript Object Notation), a text-based and platform independent data exchange format, is used not only to exchange data, but also as a format to store unstructured data. MariaDB Server 10.2 offers more than 24 JSON functions to allow querying, modification, validation and indexing of JSON formated data, which is stored in a text-based field of a database. As a result, the powerful relational model of MariaDB can be enriched by working with unstructured data, where required.

Through the use of virtual columns, the JSON function, JSON_VALUE and the newest indexing feature of MariaDB Server 10.2 on virtual columns, JSON values will be automatically extracted from the JSON string, stored in a virtual column and indexed providing the fastest access to the JSON string.

Using the JSON function JSON_VALID, the new CHECK CONSTRAINTS in MariaDB Server 10.2 guarantee that only JSON strings of the correct JSON format can be added into a field.

Binary Log Based Rollback

The enhanced mysqlbinlog utility delivered with MariaDB Server 10.2 includes a new point-in-time rollback function, which allows a database or table to revert to an earlier state, and delivers binary log based rollback of already committed data. The tool mysqlbinlog is not directly modifying any data, it is generating an “export file” including the reverted statements of the transactions, logged in a binary log file. The created file can be used with the command line client or other SQL tool to execute the included SQL statements. This way all committed transactions up to a given timestamp will be rolled back.

In the case of addressing logical mistakes like adding, changing or deleting data, so far the only possible way has been to use mysqlbinlog to review transactions and fix the problems manually. However, this often leads to data inconsistency because corrections typically only address the wrong statement, thereby ignoring other data dependencies.

Typically caused by DBA or user error, restoring a huge database can result in a significant outage of service. Rolling back the last transactions using point-in-time roll back takes only the time of the extract, a short review and the execution of the reverted transactions – saving valuable time, resources and service.

Why MariaDB?

With several MySQL options to choose from, why select MariaDB as the technology to power your application? Here are some of the benefits to selecting MariaDB...

• MariaDB is built on a modern architecture that is extensible at every layer: client, cluster, kernel and storage. This extensibility provides two major advantages. It allows for continual community innovation via plugins and it makes it easy for customers to configure MariaDB to support a wide variety of use cases from OLTP to OLAP.
• MariaDB develops features and enhancements that are part of its own roadmap, independent from Oracle / MySQL. This allows MariaDB to accept and attract broader community innovation, as well as to add internally developed new features that make it easier to migrate from proprietary systems to open source MariaDB.
• MariaDB is engineered to secure the database at every layer, making it a trusted general-purpose database used in industries such as government and banking that require the highest level security features.
• MariaDB offers support for a variety of storage engines, including NoSQL support, giving its users several choices to determine the one which will work best with their environment.
• MariaDB has deployed many performance enhancing improvements including query optimizations which, in several benchmark tests, let's MariaDB perform 3-5% better than a similarly configured MySQL environment.

ClusterControl for MariaDB

ClusterControl provides support for each of the top MariaDB technologies...

• MariaDB Server: MariaDB Server is a general purpose database engineered with an extensible architecture to support a broad set of use cases via pluggable storage engines – such as InnoDB, MyRocks and Spider.
  • Built-in asynchronous master/slave replication
  • Dynamic columns that allows different rows to store different data in the same column
  • Built-in encryption
  • Query optimization
  • Improved schema compatibility
• MariaDB Cluster: MariaDB Cluster is made for today’s cloud based environments. It is fully read-write scalable, comes with synchronous replication, allows multi-master topologies, and guarantees no lag or lost transactions.
  • Synchronous replication with no slave lag or lost transactions
  • Active-active multi-master topology
  • Read and write to any cluster node
  • Automatic membership control, with failed nodes dropped from the cluster
  • Automatic node joining
  • True row-level parallel replication
  • Direct client connections, native MariaDB look and feel
  • Both read and write scalability
• MariaDB MaxScale: MariaDB MaxScale is a database proxy that extends the high availability, scalability, and security of MariaDB Server while at the same time simplifying application development by decoupling it from underlying database infrastructure.
  • Includes Database Firewall and DoS protection
  • Read-Write Splitting
  • Data Masking
  • Schema-based Sharding
  • Query Caching

The ClusterControl Community Edition is a free-to-use, all-in-one database management system that allows you to easily deploy and monitor the top open source database technologies like MySQL, MariaDB, Percona, MongoDB, PostgreSQL, Galera Cluster and more. It also allows you to import and monitor your existing database stack.

Free Database Deployment

The ClusterControl Community Edition ensures your team can easily and securely deploy production-ready open source database stacks that are built using battle-tested, proven methodologies. You don’t have to be a database expert to utilize the ClusterControl Community Edition - deploying the most popular open sources databases is easy with our point-and-click interface. Even if you are a master of deploying databases, ClusterControl’s point-and-click deployments will save you time and ensure your databases are deployed correctly, removing the chance for human error. There is also a CLI for those who prefer the command line, or need to integrate with automation scripts.

The ClusterControl Community Edition is not restricted to a single database technology and supports the major flavors and versions. With it you’re able to apply point-and-click deployments of MySQL standalone, MySQL replication, MySQL Cluster, Galera Cluster, MariaDB, MariaDB Cluster, Percona XtraDB and Percona Server for MongoDB, MongoDB itself and PostgreSQL!

Free Database Monitoring

The ClusterControl Community Edition makes monitoring easy by providing you the ability to look at all your database instances across multiple data centers or drill into individual nodes and queries to pinpoint issues. Offering a high-level, multi-dc view as well as a deep-dive view, ClusterControl lets you keep track of your databases so you can keep them running at peak performance.

In addition to monitoring the overall stack and node performance you can also monitor the specific queries to identify potential errors that could affect performance and uptime.

Why pay for a monitoring tool when the ClusterControl Community Edition gives you a great one for free!

Free Database Developer Studio

The Developer Studio provides you a set of monitoring and performance advisors to use and lets you create custom advisors to add security and stability to your database infrastructures. It lets you extend the functionality of ClusterControl, which helps you detect and solve unique problems in your environments.

We even encourage our users to share the advisors they have created on GitHub by adding a fork to our current advisor bundle. If we like them and think that they might be good for other users we’ll include them in future ClusterControl releases.

Why Should I Use the ClusterControl Community Edition?

These are just a few of the reasons why you should use ClusterControl as your system to deploy and monitor your open source database environments…

• You can deploy knowing you are using proven methodologies and industry best practices.
• If you are just getting started with open source database technology ClusterControl makes it easy for the beginner to deploy and monitor your stacks removing human error and saving you time.
• If you are not familiar with orchestration programs like Puppet and Chef? Don’t worry! The ClusterControl Community Edition uses a point-and-click GUI to make it easy to get your environment production-ready.
• The ClusterControl Community Edition gives you deployment and monitoring in one battle-tested all-in-one system. Why use one tool for scripting only to use a different tool for monitoring?
• If you are not sure what database technology is right for your application? The ClusterControl Community Edition supports nearly two dozen database versions that you can try.
• Have a load balancer running on an existing stack? With the ClusterControl Community Edition you can import and deploy your existing and already configured load balancer to run alongside your database instances.

If you are ready to give it a try click here to download and install the latest version of ClusterControl. Each install comes with the option to activate a 30-day enterprise trial as well.

The ClusterControl Enterprise Edition provides you will a full suite of management and scaling features in addition to the deployment and monitoring functions offered as part of the free Community Edition. You also have the ability to deploy, configure and manage the top open source load balancing and caching technologies to drive peak performance for your mission-critical applications.

Whether you have been benefiting from the free resources included in the Community Edition or have evaluated the product through the Enterprise Trial, we’ll walk you through how our licensing works and explain how to get you up-and-running with all the automation and scaling that ClusterControl Enterprise has to offer.

How to Upgrade from Community to Enterprise

While using the ClusterControl Community Edition you may have clicked on a feature and got a pop-up indicating that it was not included in the version you are using. When this happens you have two options. You can activate (or extend) your Enterprise Trial OR you can contact sales to purchase an enterprise license.

Enterprise Trial

The ClusterControl Enterprise trial provides you with free access to our full suite of features for 30 days. The purpose of this trial is to allow you to “kick the tires” using your environments and applications to make sure that ClusterControl meets your needs.

With the trial you have access to all our Community features plus: Custom Dashboards, Load Balancers, Configuration Management, Backup and Restore, Automatic Node and Cluster Recovery, Role Based Access Control, Key Management, LDAP, SSL Encryption Scaling, and more!

The trial also grants you Enterprise Level access to our support teams 24/7. We want to make sure that you have the best experience during your trial and also introduce you to our amazing support that you can count on when you become a customer of Severalnines.

At the end of your trial, you will have the option to meet with our sales team to continue with ClusterControl Enterprise on a paid license. Or you may also continue with our ClusterControl Community Edition, which you can use for free - forever.

Extending Your Trial

Sometimes thirty days isn’t enough time to evaluate a product as extensive as ClusterControl. In these situations we can sometimes grant an extension to allow you some more time to evaluate the product. This extension can be requested from the product itself and you will be contacted by an account manager to arrange for the extension.

Purchasing a Commercial License

ClusterControl offers three separate plans and different support options. Our account managers are available to assist, and recommend the best plan. We also offer volume discounts for larger orders. In short, we will work very hard to make sure our price meets your needs and budget. Once we’ve all signed on the dotted line, you will then be provided with Commercial License keys that you can put into your already deployed environment (or into a new one) which will then immediately grant you full access to the entire suite of ClusterControl features that you have contracted.

Benefits of Upgrading

While the free ClusterControl community version provides rich features that allow you to easily and securely deploy and monitor your open source databases, the Enterprise Edition provides much much more!

These are just some of the features awaiting you in the Enterprise Edition...

• Advanced Backup & Restoration: With ClusterControl you can schedule logical or physical backups with failover handling and easily restore backups to bootstrap nodes or systems.
• Automated Failover: ClusterControl includes advanced support for failure detection and handling; it also allows you to deploy different proxies to integrate them with your HA stack.
• Topology Changes: Making topology changes with ClusterControl is easy; it does all the background work to elect a new master, deploy fail-over slave servers, rebuild slaves in case of data corruption, and maintain load balancer configurations to reflect all the changes.
• Load Balancing: Load balancers are an essential component in database high availability; especially when making topology changes transparent to applications and implementing read-write split functionality and ClusterControl provides support for ProxySQL, HAProxy, and Maxscale.
• Advanced Security: ClusterControl removes human error and provides access to a suite of security features automatically protecting your databases from hacks and other threats. Operational Reports come in handy, whether you need to show you are meeting your SLAs or wish to keep track of the historical data of your cluster.
• Scaling: Easily add and remove nodes, resize instances, and clone your production clusters with ClusterControl.

In short, ClusterControl is an all-inclusive database management system that removes the need for your team to have to cobble together multiple tools, saving you time and money.

If you ever have any issues during this process you can always consult the documentation or contact us. If you need support you can contact us here.

Helping users securely automate and manage their open source databases has been at the core of our efforts from the inception of Severalnines.

And ever since the first release of our flagship product, ClusterControl, it’s always been about making it as easy and secure as possible for users to deploy complex, open source database cluster technologies in any environment.

Since our first steps with deployment, automation and management we’ve perfected the art of securely deploying highly available open source database infrastructures by developing ClusterControl from a deployment and monitoring tool to a full-blown automation and management system adopted by thousands of users worldwide.

As a result, ClusterControl can be used today to deploy, monitor, and manage over a dozen versions of the most popular open source database technologies - on premise or in the cloud.

Whether you’re looking to deploy MySQL standalone, MySQL replication, MySQL Cluster, Galera Cluster, MariaDB, MariaDB Cluster, Percona XtraDB and Percona Server for MongoDB, MongoDB itself and PostgreSQL - ClusterControl has you covered.

In addition to the database stores, users can also deploy and manage load balancing technologies such as HAProxy, ProxySQL, MaxScale and Keepalived.

Using ClusterControl, database clusters can be either deployed new or existing ones imported.

A deployment wizard makes it easy and secure to deploy production-ready database clusters with a point and click interface that walks the users through the deployment process step by step.

Select Deploy or Import Cluster

Walk through of the Deploy Wizard

View your cluster list

Deploying with the ClusterControl CLI

Users can also chose to work with our CLI, which allows for easy integration with infrastructure orchestration tools such as Ansible etc.

s9s cluster
  --create
  --cluster-type=galera
  --nodes='10.10.10.26;10.10.10.27;10.10.10.28'
  --vendor=percona
  --cluster-name=PXC_CENTOS7
  --provider-version=5.7
  --os-user=vagrant   --wait

The ClusterControl deployment supports multiple NICS and templated configurations.

In short, ClusterControl provides:

• Topology-aware deployment jobs for MySQL, MariaDB,Percona, MongoDB and PostgreSQL
• Self-service and on-demand
• From standalone nodes to load-balanced clusters
• Your choice of barebone servers, private/public cloud and containers

To see for yourself, download ClusterControl today and give us your feedback.

Galera Cluster’s automatic node provisioning simplifies the complexity of scaling out a database cluster with guaranteed data consistency. SST and IST improve the usability of initial data synchronization without the need to manually backup the database and copy it to the new node. Combine this with Galera's ability to tolerate different network setups (e.g, WAN replication), we can now migrate the database between different isolated networks with zero service disruption.

In this blog post, we are going to look into how to migrate our MySQL Galera Cluster without downtime. We will move the database from Amazon Web Service (AWS) EC2 to Google Cloud Platform (GCP) Compute Engine, with the help of a relay node. Note that we had a similar blog post in the past, but this one uses a different approach.

The following diagram simplifies our migration plan:

Old Site Preparation

Since both sites cannot communicate with each other due to security group or VPC isolation, we need to have a relay node to bridge these two sites together. This node can be located on either site, but must able to connect to one or more nodes on the other side on port 3306 (MySQL), 4444 (SST), 4567 (gcomm) and 4568 (IST). Here is what we already have, and how we will scale the old site:

You can also use an existing Galera node (e.g, the third node) as the relay node, as long as it has connectivity to the other side. The downside is that the cluster capacity will be reduced to two, because one node will be used for SST and relaying the Galera replication stream between sites. Depending on the dataset size and connection between sites, this can introduce database reliability issues on the current cluster.

So, we are going to use a fourth node, to reduce the risk on the current production cluster when syncing to the other side. First, create a new instance in the AWS Dashboard with a public IP address (so it can talk to the outside world) and allow the required Galera communication ports (TCP 3306, 4444, 4567-4568).

Deploy the fourth node (relay node) on the old site. If you are using ClusterControl, you can simply use "Add Node" feature to scale the cluster out (don't forget to setup passwordless SSH from ClusterControl node to this fourth host beforehand):

Ensure the relay node is in sync with the current cluster and is able to communicate to the other side.

From the new site, we are going to connect to the relay node since this is the only node that has connectivity to the outside world.

New Site Deployment

On the new site, we will deploy a similar setup with one ClusterControl node and three-node Galera Cluster. Both sites must use the same MySQL version. Here is our architecture on the new site:

With ClusterControl, the new cluster deployment is just a couple of clicks away and a free feature in the community edition. Go to ClusterControl -> Deploy Database Cluster -> MySQL Galera and follow the deployment wizard:

Click Deploy and monitor the progress under Activity -> Jobs -> Create Cluster. Once done, you should have the following on the dashboard:

At this point, you are having two separate Galera Clusters - 4 nodes at the old site and 3 nodes at the new site.

Connecting Both Sites

On the new site (GCP), pick one node to communicate with the relay node on the old site. We are going to pick galera-gcp1 as the connector to the relay node (galera-aws4). The following diagram illustrates our bridging plan:

The important things to configure are the following parameters:

• wsrep_sst_donor: The wsrep_node_name of the donor node. On galera-gcp1, set the donor to galera-aws4.
• wsrep_sst_auth: SST user credentials in username:password format must follow the old site (AWS).
• wsrep_sst_receive_address: The IP address that will receive SST on the joiner node. On galera-gcp1, set this to the public IP address of this node.
• wsrep_cluster_address: Galera connection string. On galera-gcp1, add the public IP address of galera-aws4.
• wsrep_provider_options:
  • gmcast.segment: Default is 0. Set a different integer on all nodes in GCP.

1. On the relay node (galera-aws4), retrieve the wsrep_node_name:

   $ mysql -uroot -p -e 'SELECT @@wsrep_node_name'
   Enter password:
   +-------------------+
   | @@wsrep_node_name |
   +-------------------+
   | 10.0.0.13         |
   +-------------------+

2. On galera-gcp1's my.cnf, set wsrep_sst_donor value to the relay node's wsrep_node_name and wsrep_sst_receive_address to the public IP address of galera-gcp1:

   wsrep_sst_donor=10.0.0.13
   wsrep_sst_receive_address=35.197.136.232

3. On all nodes on GCP, ensure the wsrep_sst_auth value is identical following the old site (AWS) and change the Galera segment to 1 (so Galera knows both sites are in different networks):

   wsrep_sst_auth=backupuser:mysecretP4ssW0rd
   wsrep_provider_options="base_port=4567; gcache.size=512M; gmcast.segment=1"

4. On galera-gcp1, set the wsrep_cluster_address to include the relay node's public IP address:

   wsrep_cluster_address=gcomm://10.148.0.2,10.148.0.3,10.148.0.4,13.229.247.149

   **Only modify wsrep_cluster_address on galera-gcp1. Don't modify this parameter on galera-gcp2 and galera-gcp3.

5. Stop all nodes on GCP. If you are using ClusterControl, go to Cluster Actions dropdown -> Stop Cluster. You are also required to turn off automatic recovery at both cluster and node levels, so ClusterControl won't try to recover the failed nodes.

6. Now the syncing part. Start galera-gcp1. You can see from the MySQL error log on the donor node that SST is initiated between the the relay node (10.0.0.13) using a public address on galera-gcp1 (35.197.136.232):

   2017-12-19T13:58:04.765238Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '35.197.136.232:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/m
   ysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '''' --gtid 'df23adb8-b567-11e7-8c50-a386c8cc7711:151181')
   2017-12-19T13:58:04.765468Z 5 [Note] WSREP: DONOR thread signaled with 0
           2017-12-19T13:58:15.158757Z WSREP_SST: [INFO] Streaming the backup to joiner at 35.197.136.232 4444
   2017-12-19T13:58:52.512143Z 0 [Note] WSREP: 1.0 (10.0.0.13): State transfer to 0.0 (10.148.0.2) complete.

   Take note that, at this point of time, galera-gcp1 will be flooded with following lines:

   2017-12-19T13:32:47.111002Z 0 [Note] WSREP: (ed66842b, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.0.0.118:4567 timed out, no messages seen in PT3S
   2017-12-19T13:32:48.111123Z 0 [Note] WSREP: (ed66842b, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.0.0.90:4567 timed out, no messages seen in PT3S
   2017-12-19T13:32:50.611462Z 0 [Note] WSREP: (ed66842b, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.0.0.25:4567 timed out, no messages seen in PT3S

   You can safely ignore this warning since galera-gcp1 keeps trying to see the remaining nodes beyond the relay node on AWS.

7. Once SST on galera-gcp1 completes, ClusterControl on GCE won't be able to connect the database nodes, due to missing GRANTs (existing GRANTs have been overridden after syncing from AWS). So here is what we need to do after SST completes on galera-gcp1:

   mysql> GRANT ALL PRIVILEGES ON *.* TO cmon@'10.148.0.5' IDENTIFIED BY 'cmon' WITH GRANT OPTION;

   Once this is done, ClusterControl will correctly report the state of galera-gcp1 as highlighted below:

8. The last part is to start the remaining galera-gcp2 and galera-gcp3, one node at a time. Go to ClusterControl -> Nodes -> pick the node -> Start Node. Once all nodes are synced, you should get 7 as the cluster size:

The cluster is now operating on both sites and scaling out is complete.

Decommissioning

Once the migration completes and all nodes are in synced, you can start to switch your application to the new cluster on GCP:

At this point MySQL data is replicated to all nodes until decommissioning. The replication performance will be as good as the farthest node in the cluster permits. The relay node is critical, as it broadcasts writesets to the other side. From the application standpoint, it's recommended to write to only one site at a time, which means you will have to start redirecting reads/writes from AWS and serve them from GCP cluster instead.

To decommission the old database nodes and move to the cluster on GCP, we have to perform a graceful shutdown (one node at a time) on AWS. It is important to shut down the nodes gracefully, since the AWS site holds the majority number of nodes (4/7) for this cluster. Shutting them down all at once will cause the cluster on GCP to go into non-primary state, forcing the cluster to refuse operation. Make sure the last node to shutdown on the AWS side is the relay node.

Don't forget to update the following parameters on galera-gcp1 accordingly:

• wsrep_cluster_address - Remove the relay node public IP address.
• wsrep_sst_donor - Comment this line. Let Galera auto pick the donor.
• wsrep_sst_receive_address - Comment this line. Let Galera auto pick the receiving interface.

Your Galera Cluster is now running on a completely new platform, hosts and network without a second of downtime to your database service during migration. How cool is that?

As we wrap up our last blog of 2017 we wanted to reflect on what content we have been creating that’s been resonating and generating the most interest with our readers. We will continue to deliver the best technical content we can for MySQL, Galera Cluster, PostgreSQL, MariaDB, and MongoDB in 2018.

Here is some of our most popular content from 2017…

Top Database Blogs for 2017

• MySQL on Docker: There are four blogs on this topic which consistently show up in top tens reports but “MySQL Docker Containers: Understanding the Basics” takes the crown. This is the first in our MySQL on Docker blog series and it discusses what a database container is and some tips for getting started with this booming new technology.
• Installing Kubernetes Cluster with 3 Minions on CentOS 7 to Manage Pods and Services: This “how-to” blog provides step-by-step instructions on how to deploy on CentOS 7 and also provides a basic understanding of the components. It goes through the process of setting up a Kubernetes Master and Minions (nodes) and concludes with creating pods and services.
• How to Bootstrap MySQL or MariaDB Galera Cluster: This blog details how to start MySQL Galera Cluster or MariaDB Cluster, which is different than the way you do it for MySQL server or MySQL Cluster. Called a “cluster bootstrap” this process is an initial setup to introduce a database node as a primary component.
• 9 Tips for Going in Production with Galera Cluster for MySQL: A blog that was later turned into a webinar, this “how to” post covers nine tips to make sure you're ready for production before you go live.
• How to Set Up Asynchronous Replication from Galera Cluster to Standalone MySQL server with GTID: This blog shows how to utilize GTID to enable hybrid replication (combining Galera and Asynchronous MySQL Replication in the same setup).
• Scaling WordPress and MySQL on Multiple Servers for Performance: This blog details how to combine technologies like Percona XtraDB Cluster, GlusterFS, Apache with load balancing to ensure HA WordPress setup.
• Avoiding Deadlocks in Galera - Setup HAProxy for Single-Node Writes and Multi-node Reads: While HAProxy is not fully deployed and configurable inside of ClusterControl this blog details how to avoid common challenges when using the technology with Galera Cluster.
• Understanding What Has Changed Between MySQL 5.6 and 5.7 Before Upgrading: The newest version of MySQL was released back in 2015 but it is still being upgraded across many systems and this blog details all the changes that occured. It is also a except from our very popular Migrating to MySQL 5.7 Whitepaper.
• How to Avoid SST When Adding a New Node to Galera Cluster for MySQL or MariaDB: This blog details how to avoid State Snapshot Transfer (SST) for MySQL Galera Setups when adding additional nodes to your setup.
• MySQL Load Balancing with ProxySQL Overview: ProxySQL was one of our major focuses for ClusterControl this year so it’s exciting to see an overview of this exciting new technology making our top list. While this blog explains manual deployment it is now completely integrated in the ClusterControl GUI.

Top Blogs by Technology

While MySQL and MySQL Galera Cluster dominate our most popular content we blog about a lot of different technologies and methodologies on the Severalnines blog. Here are some of the most popular blogs in 2017 for non-MySQL topics.

• Top MongoDB Blog - High Availability Log Processing with Graylog, MongoDB and Elasticsearch: Graylog is an open-source log management tool and this blog details how you can use it in conjunction with MongoDB, Elasticsearch and ClusterControl.
• Top PostgreSQL Blog - How to Setup Streaming Replication for High Availability: This “how to” blog lets you know step-by-step how to deploy streaming replication for PostgreSQL including using ClusterControl to add nodes.
• Top DevOps Blog - DevOps Considerations for Production-ready Database Deployments: If you are a DevOps professional and have suddenly found yourself responsible for the database environments in your setup this blog is key to detailing what you need to know to effectively achieve high availability and ensure your environment operates correctly.
• Top ClusterControl Blog - How to Deploy and Manage Multi-Master Replication Setups with ClusterControl 1.4: This blog details the new deployment wizard that was released in ClusterControl 1.4. With this new wizard you can easily deploy and configure a variety of open source database technologies including Master-Master replication topologies.
• Top NinesControl Blog - Deploying and Monitoring MySQL and MongoDB Clusters in the Cloud with NinesControl: In this blog we detail the deployment and monitoring features found in NinesControl, a system to easily deploy open sources databases to the cloud of your choice.

If there are some blog topics you would like us to cover in 2018 please list them in the comments below.

Join Laurent Blume, Unix Systems Engineer & PCI Specialist and Vinay Joosery, CEO at Severalnines, as they discuss all there is to know about how to achieve PCI compliance for MySQL & MariaDB with ClusterControl in this new webinar on January 30th.

The Payment Card Industry Data Security Standard (PCI-DSS) is a set of technical and operational requirements defined by the PCI Security Standards Council (PCI SSC) to protect cardholder data. These standards apply to all entities that store, process or transmit cardholder data – with requirements for software developers and manufacturers of applications and devices used in those transactions.

PCI data that resides in a MySQL or MariaDB database must of course also adhere to these requirements, and database administrators must follow best practices to ensure the data is secured and compliant. The PCI standards are stringent and can easily require a spiraling amount of time spent on meeting their requirements. Database administrators can end up overwhelmed when using software that was not designed for compliance, often because it long predates PCI itself, as is the case for most database systems in use today.

That is why, as often as possible, reliable tools must be chosen to help with that compliance, easing out the crucial parts. Each time the compliance for one requirement can be shown to be implemented, working, and logged accordingly, time will be saved. If well-designed, it will only require regular software upgrades, a yearly review and a moderate amount of tweaking to follow the standard's evolution over time.

This new webinar focuses on PCI-DSS requirements for a MySQL or MariaDB database back-end managed by ClusterControl in order to help meet these requirements. It will provide a MySQL and MariaDB user focussed overview of what the PCI standards mean, how they impact database management and provide valuable tips and tricks on how to achieve PCI compliance for MySQL & MariaDB with ClusterControl.

Sign up here!

Date, Time & Registration

Europe/MEA/APAC

Tuesday, January 30th at 09:00 GMT / 10:00 CET (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, January 30th at 09:00 PT (US) / 12:00 ET (US)

Register Now

Agenda

• Introduction to the PCI-DSS standards
• The impact of PCI on database management
• Step by step review of the PCI requirements
• How to meet the requirements for MySQL & MariaDB with ClusterControl
• Conclusion
• Q&A

Speakers

Laurent Blume, Unix Systems Engineer, PCI Specialist

Laurent’s career in IT started in 2000, his work since evolved from POS terminals for a jewelry store chain to infrastructure servers in a government aerospace R&D organization, even touching supercomputers. One constant throughout was the increasing need for security.

For the past 6 years, he has been in charge of first implementing, then keeping up with the PCI-DSS compliance of critical transnational payment authorization systems. Its implementation for databases has been an essential part of the task. For the last few years, it has expanded to the design and productization of MariaDB cluster backends for mobile contactless payments.

Vinay Joosery, CEO & Co-Founder, Severalnines

Vinay is a passionate advocate and builder of concepts and business around distributed database systems.

Prior to co-founding Severalnines, Vinay held the post of Vice-President EMEA at Pentaho Corporation - the Open Source BI leader. He has also held senior management roles at MySQL / Sun Microsystems / Oracle, where he headed the Global MySQL Telecoms Unit, and built the business around MySQL's High Availability and Clustering product lines. Prior to that, Vinay served as Director of Sales & Marketing at Ericsson Alzato, an Ericsson-owned venture focused on large scale real-time databases.

Security is one of the most important aspects of running a database. Whether you are a developer or a DBA, if you are managing the database, it is your responsibility to safeguard your data and protect it from any kind of unauthorized access. The unfortunate fact is that many organizations do not protect their data, as we’ve seen from the new wave of MongoDB ransomware attacks in September 2017. We had earlier published a blog on how to secure MongoDB databases.

In this blog post, we’ll have a look into how to secure your databases using ClusterControl. All of the features described here are available in version 1.5.1 of ClusterControl (released on December 23, 2017). Please note that some features are only available for certain database types.

Backup Encryption

ClusterControl 1.5.1 introduced a new feature called backup encryption. All encrypted backups are marked with a lock icon next to it:

You can use this feature on all backup methods (mysqldump, xtrabackup, mongodump, pg_dump) supported by ClusterControl. To enable encryption, simply toggle on the "Enable Encryption" switch when scheduling or creating the backup. ClusterControl automatically generates a key to encrypt the backup. It uses AES-256 (CBC) encryption algorithm and performs the encryption on-the-fly on the target server. The following command shows an example of how ClusterControl performs a mysqldump backup:

$ mysqldump --defaults-file=/etc/my.cnf --flush-privileges --hex-blob --opt --no-create-info --no-data --triggers --routines --events --single-transaction --skip-comments --skip-lock-tables --skip-add-locks --databases db1 | gzip -6 -c | openssl enc -aes-256-cbc -pass file:/var/tmp/cmon-094508-e0bc6ad658e88d93.tmp | socat - TCP4:192.168.55.170:9999'

You would see the following error if you tried to decompress an encrypted backup without decrypting it first with the proper key:

$ gunzip mysqldump_2018-01-03_175727_data.sql.gz
gzip: mysqldump_2018-01-03_175727_data.sql.gz: not in gzip format

The key is stored inside the ClusterControl database, and can be retrieved from the cmon_backup.metadata file for a particular backup set. It will be used by ClusterControl when performing restoration. Encrypting backups is highly recommended, especially when you want to secure your backups offsite like archiving them in the cloud.

MySQL/PostgreSQL Client-Server Encryption

Apart from following the recommended security steps during deployment, you can increase the reliability of your database service by using client-server SSL encryption. Using ClusterControl, you can perform this operation with simple point and click:

You can then retrieve the generated keys and certificates directly from the ClusterControl host under /var/lib/cmon/ca path to establish secure connections with the database clients. All the keys and certificates can be managed directly under Key Management, as described further down.

Database Replication Encryption

Replication traffic within a Galera Cluster can be enabled with just one click. ClusterControl uses a 2048-bit default key and certificate generated on the ClusterControl node, which is transferred to all the Galera nodes:

A cluster restart is necessary. ClusterControl will perform a rolling restart operation, taking one node at a time. You will see a green lock icon next to the database server (Galera indicates Galera Replication encryption, while SSL indicates client-server encryption) in the Hosts grid of the Overview page once encryption is enabled:

All the keys and certificates can be managed directly under Key Management, as described further down.

Key Management

All the generated keys and certificates can be managed directly from the ClusterControl UI. Key Management allows you to manage SSL certificates and keys that can be provisioned on your clusters:

If the certificate has expired, you can simply use the UI to generate a new certificate with proper key and Certificate Authority (CA), or import an existing key and certificate into ClusterControl host.

Security Advisors

Advisors are mini-programs that run in ClusterControl. They perform specific tasks and provide advice on how to address issues in areas such as performance, security, log management, configuration, storage space and others. Each advisor can be scheduled like a cron job, and run as a standalone executable within the ClusterControl UI. It can also be run via the ClusterControl 's9s' command line client.

ClusterControl enables two security advisors for MySQL-based systems:

• Access from any host ('%') - Identifies all users that use a wildcard host from the mysql system table, and lets you have more control over which hosts are able to connect to the servers.
• Check number of accounts without a password - Identifies all users who do not have a password in the mysql system table.

For MongoDB, we have the following advisors:

• MongoDB authentication enabled - Check whether the MongoDB instance is running with authentication mode enabled.
• Authorization check - Check whether MongoDB users are authorized with too permissive role for access control.

For more details on how does ClusterControl performs the security checks, you can look at the advisor JavaScript-like source code under Manage -> Developer Studio. You can see the execution results from the Advisors page:

Multiple Network Interfaces

Having multiple NICs on the database hosts allows you to separate database traffic from management traffic. One network is used by the database nodes in order to communicate to each other, and this network is not exposed to any public network. The other network is used by ClusterControl, for management purposes. ClusterControl is able to deploy such a multi-network setup. Consider the following architecture diagram:

To import the above database cluster into ClusterControl, one would specify the primary IP address of the database hosts. Then, it is possible to choose the management network as well as the data network:

ClusterControl can also work in an environment without Internet access, with the databases being totally isolated from the public network. The majority of the features will work just fine. If the ClusterControl host is configured with Internet, it is also capable of cloning the database vendor's repository for the internet-less database servers. Just go to Settings (top menu) -> Repositories -> Create New Repository and set the options to fit the target database server environment:

The mirroring may take about 10 to 20 minutes depending on the internet connection, you will see the new item in the list later on. You can then pick this repository instead when scaling or deploying a new cluster, without the need for the database hosts to have any Internet connection (note that the operating system’s offline repository should be in place as well).

MySQL Users Management

The MySQL privilege system ensures that all users can perform only the operations they are allowed to. Granting is critical as you don't want to give all users complete access to your database, but you need users to have the necessary permissions to run queries and perform daily tasks.

ClusterControl provides an interactive user interface to manage the database schemas and privileges. It unifies the accounts on all MySQL servers in the cluster and simplifies the granting process. You can easily visualize the database users, so you avoid making mistakes.

As you can see in the above screenshot, ClusterControl greyed out unnecessary privileges if you only want to grant a user to a database (shopdb). "Require SSL?" is only enabled if the client/server SSL encryption is enabled while the administration privilege checkboxes are totally disabled if a specific database is defined. You can also inspect the generated GRANT statement at the bottom of the wizard, to see the statement that ClusterControl will execute to create this user. This helper looks pretty simple, but creating users and granting privileges can be error-prone.

ClusterControl also provides a list of inactive users for all database nodes in the cluster, showing off the accounts that have not been used since the last server restart:

This alerts the administrator for unnecessary accounts that exist, and that could potentially harm the server. The next step is to verify if the accounts are no longer active, and you can simply use the "Drop Selected User" option in order to remove them. Make sure you have enough database activity to ensure the list generated by ClusterControl is accurate. The longer the server uptime, the better.

Always Keep Up-to-date

For production use, it’s highly recommended for you to install the database-related packages from the vendor’s repository. Don’t rely on the default operating system repository, where the packages are usually outdated. If you are running in a cluster environment like Galera Cluster, or even MySQL Replication, you always have the choice to patch the system with minimal downtime.

ClusterControl supports automatic minor version rolling upgrade for MySQL/MariaDB with a single click. Just go to Manage -> Upgrades -> Upgrade and choose the appropriate major version for your running cluster. ClusterControl will then perform the upgrade, on one node at a time. The node will be stopped, then software will be updated, and then the node will be started again. If a node fails to upgrade, the upgrade process is aborted and the admin is notified. Upgrades should only be performed when there is as little traffic as possible on the cluster.

Major versions upgrades (e.g, from MySQL 5.6 to MySQL 5.7) are intentionally not automated. Major upgrades usually require uninstallation of the existing packages, which is a risky task to automate. Careful planning and testing is necessary for such kind of upgrades.

Database security is an important aspect of running your database in production. From all the incidents we frequently read about in the news (and there are probably many others that are not publicized), it is clear that there are groups busy out there with bad intentions. So, make sure your databases are well protected.

Security of data is a top priority these days. Sometimes it’s enforced by external regulations like PCI-DSS or HIPAA, sometimes it’s because you care about your customers’ data and your reputation. There are numerous aspects of security that you need to keep in mind - network access, operating system security, grants, encryption and so on. In this blog post, we’ll give you 10 tips on what to look at when securing your MySQL or MariaDB setup.

1. Remove users without password

MySQL used to come with a set of pre-created users, some of which can connect to the database without a password or, even worse, anonymous users. This has changed in MySQL 5.7 which, by default, comes only with a root account that uses the password you choose at installation time. Still, there are MySQL installations which were upgraded from previous versions and these installations keep the legacy users. Also, MariaDB 10.2 on Centos 7 comes with anonymous users:

MariaDB [(none)]> select user, host, password from mysql.user where user like '';
+------+-----------------------+----------+
| user | host                  | password |
+------+-----------------------+----------+
|      | localhost             |          |
|      | localhost.localdomain |          |
+------+-----------------------+----------+
2 rows in set (0.00 sec)

As you can see, those are limited only to access from localhost but regardless, you do not want to have users like that. While their privileges are limited, they still can run some commands which may show more information about the database - for example, the version may help identify further vectors of attack.

[root@localhost ~]# mysql -uanonymous_user
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 19
Server version: 10.2.11-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SHOW GRANTS\G
*************************** 1. row ***************************
Grants for @localhost: GRANT USAGE ON *.* TO ''@'localhost'
1 row in set (0.00 sec)
MariaDB [(none)]> \s
--------------
mysql  Ver 15.1 Distrib 10.2.11-MariaDB, for Linux (x86_64) using readline 5.1
Connection id:        19
Current database:
Current user:        anonymous_user@localhost
SSL:            Not in use
Current pager:        stdout
Using outfile:        ''
Using delimiter:    ;
Server:            MariaDB
Server version:        10.2.11-MariaDB MariaDB Server
Protocol version:    10
Connection:        Localhost via UNIX socket
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
UNIX socket:        /var/lib/mysql/mysql.sock
Uptime:            12 min 14 sec
Threads: 7  Questions: 36  Slow queries: 0  Opens: 17  Flush tables: 1  Open tables: 11  Queries per second avg: 0.049
--------------

Please note that users with very simple passwords are almost as insecure as users without any password. Passwords like “password” or “qwerty” are not really helpful.

2. Tight remote access

First of all, remote access for superusers - this is taken care of by default when installing the latest MySQL (5.7) or MariaDB (10.2) - only local access is available. Still, it’s pretty common to see superusers being available for various reasons. The most common one, probably because the database is managed by humans who want to make their job easier, so they’d add remote access to their databases. This is not a good approach as remote access makes it easier to exploit potential (or verified) security vulnerabilities in MySQL - you don’t need to get a connection to the host first.

Another step - make sure that every user can connect to MySQL only from specific hosts. You can always define several entries for the same user (myuser@host1, myuser@host2), this should help to reduce a need for wildcards (myuser@’%’).

3. Remove test database

The test database, by default, is available to every user, especially to the anonymous users. Such users can create tables and write to them. This can potentially become a problem on its own - any writes would add some overhead and reduce database performance. Currently, after the default instalation, only MariaDB 10.2 on Centos 7 is affected by this - Oracle MySQL 5.7 and Percona Server 5.7 do not have the ‘test’ schema available.

[root@localhost ~]# mysql -uanonymous_user
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.2.11-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SHOW GRANTS\G
*************************** 1. row ***************************
Grants for @localhost: GRANT USAGE ON *.* TO ''@'localhost'
1 row in set (0.00 sec)
MariaDB [(none)]> USE test;
Database changed
MariaDB [test]> CREATE TABLE testtable (a INT);
Query OK, 0 rows affected (0.01 sec)
MariaDB [test]> INSERT INTO testtable VALUES (1), (2), (3);
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0
MariaDB [test]> SELECT * FROM testtable;
+------+
| a    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

Of course, it may still happen that your MySQL 5.7 has been upgraded from previous versions in which the ‘test’ schema was not removed - you should take care of this and check if you have it created.

4. Obfuscate access to MySQL

It is well known that MySQL runs on port 3306, and its superuser is called ‘root’. To make things harder, it is quite simple to change this. To some extent, this is an example of security through obscurity but it may at least stop automated attempts to get access to the ‘root’ user. To change port, you need to edit my.cnf and set ‘port’ variable to some other value. As for users - after MySQL is installed, you should create a new superuser (GRANT ALL … WITH GRANT OPTION) and then remove existing ‘root@’ accounts.

5. Network security

Ideally, MySQL would be not available through the network and all connections would be handled locally, through the Unix socket. In some setups, this is possible - in that case you can add the ‘skip-networking’ variable in my.cnf. This will prevent MySQL from using any TCP/IP communication, only Unix socket would be available on Linux (Named pipes and shared memory on Windows hosts).

Most of the time though, such tight security is not feasible. In that case you need to find another solution. First, you can use your firewall to allow traffic only from specific hosts to the MySQL server. For instance, application hosts (although they should be ok with reaching MySQL through proxies), the proxy layer, and maybe a management server. Other hosts in your network probably do not need direct access to the MySQL server. This will limit possibilities of attack on your database, in case some hosts in your network would be compromised.

If you happen to use proxies which allow regular expression matching for queries, you can use them to analyze the SQL traffic and block suspicious queries. Most likely your application hosts shouldn’t run “DELETE * FROM your_table;” on a regular basis. If it is needed to remove some data, it can be executed by hand, locally, on the MySQL instance. You can create such rules using something like ProxySQL: block, rewrite, redirect such queries. MaxScale also gives you an option to block queries based on regular expressions.

6. Audit plugins

If you are interested in collecting data on who executed what and when, there are several audit plugins available for MySQL. If you use MySQL Enterprise, you can use MySQL Enterprise Audit which is an extension to MySQL Enterprise. Percona and MariaDB also have their own version of audit plugins. Lastly, McAfee plugin for MySQL can also be used with different versions of MySQL. Generally speaking, those plugins collect more or less the same data - connect and disconnect events, queries executed, tables accessed. All of this contains information about which user participated in such event, from what host it logged from, when did it happen and so on. The output can be XML or JSON, so it’s much easier to parse it than parsing general log contents (even though the data is rather similar). Such output can also be sent to syslog and, further, some sort of log server for processing and analysis.

7. Disable LOAD DATA LOCAL INFILE

If both server and client has the ability to run LOAD DATA LOCAL INFILE, a client will be able to load data from a local file to a remote MySQL server. This, potentially, can help to read files the client has access to - for example, on an application server, one could access any file that the HTTP server has access to. To avoid it, you need to set local-infile=0 in the my.cnf

8. File privileges

You have to keep in mind that MySQL security also depends on the operating system setup. MySQL stores data in the form of files. The MySQL server writes plenty of information to logs. Sometimes this information contains data - slow query log, general log or binary log, for example. You need to make sure that this information is safe and accesible only to users who have to access it. Typically it means that only the root and the user under whose rights MySQL is running, should have access to all MySQL-related files. Most of the time it’s a dedicated user called ‘mysql’. You should check MySQL configuration files and all the logs generated by MySQL and verify that they are not readable by other users.

9. SSL and Encryption of Data in Transit

Preventing people from accessing configuration and log files is one thing. The other issue is to make sure data is securely transferred over the network. With an exception of setups where all the clients are local and use Unix socket to access MySQL, in majority of cases, data which forms a result set for a query, leaves the server and is transferred to the client over the network. Data can also be transferred between MySQL servers, for example via standard MySQLreplication or within a Galera cluster. Network traffic can be sniffed, and through those means, your data would be exposed.

To prevent this from happening, it is possible to use SSL to encrypt traffic, both server and client-side. You can create an SSL connection between a client and a MySQL server. You can also create an SSL connection between your master and your slaves, or between the nodes of a Galera cluster. This will ensure that all data that is transferred is safe and cannot be sniffed by an attacker who gained access to your network.

The MySQL documentation covers in detail how to setup SSL encryption. If you find it too cumbersome, ClusterControl can help you deploy a secure environment for MySQL replication or Galera cluster in a couple of clicks:

10. Encryption of Data at Rest

Securing data in transit using SSL encryption only partially solves the problem. You need to take care also of data at rest - all the data that is stored in the database. Data at rest encryption can also be a requirement for security regulations like HIPAA or PCI DSS. Such encryption can be implemented on multiple levels - you can encrypt the whole disk on which the files are stored. You can encrypt only the MySQL database through functionality available in the latest versions of MySQL or MariaDB. Encryption can also be implemented in the application, so that it encrypts the data before storing it in the database. Every option has its pros and cons: disk encryption can help only when disks are physically stolen, but the files would not be encrypted on a running database server. MySQL database encryption solves this issue, but it cannot prevent access to data when the root account is compromised. Application level encryption is the most flexible and secure, but then you lose the power of SQL - it’s pretty hard to use encrypted columns in WHERE or JOIN clauses.

All flavors of MySQL provide some sort of data at rest encryption. Oracle’s MySQL uses Transparent Data Encryption to encrypt InnoDB tablespaces. This is available in the commercial MySQL Enterprise offering. It provides an option to encrypt InnoDB tablespaces, other files which also store data in some form (for example, binary logs, general log, slow query log) are not encrypted. This allows the toolchain (MySQL Enterprise Backup but also xtrabackup, mysqldump, mysqlbinlog) to work correctly with such setup.

Starting from MySQL 5.7.11, the community version of MySQL also got support for InnoDB tablespace encryption. The main difference compared to the enterprise offering is the way the keys are stored - keys are not located in a secure vault, which is required for regulatory compliance. This means that starting from Percona Server 5.7.11, it is also possible to encrypt InnoDB tablespace. In the recently published Percona Server 5.7.20, support for encrypting binary logs has been added. It is also possible to integrate with Hashicorp Vault server via a keyring_vault plugin, matching (and even extending - binary log encryption) the features available in Oracle’s MySQL Enterprise edition.

MariaDB added support for data encryption in 10.1.3 - it is a separate, enhanced implementation. It gives you the possibility to not only encrypt InnoDB tablespaces, but also InnoDB log files. As a result, data is more secure but some of the tools won’t work in such configuration. Xtrabackup will not work with encrypted redo logs - MariaDB created a fork, MariaDB Backup, which adds support for MariaDB encryption. There are also issues with mysqlbinlog.

No matter which MySQL flavor you use, as long as it is a recent version, you would have options to implement data at rest encryption via the database server, making sure that your data is additionally secured.

Securing MySQL or MariaDB is not trivial, but we hope these 10 tips will help you along the way.

Requires ClusterControl 1.2.11 or later. Applies to MySQL based clusters.

During the life cycle of Database installation it is common that new user accounts are created. It is a good practice to once in a while verify that the security is up to standards. That is, there should at least not be any accounts with global access rights, or accounts without password.

Using ClusterControl, you can at any time perform a security audit.

In the User Interface go to Manage > Developer Studio. Expand the folders so that you see s9s/mysql/programs. Click on security_audit.js and then press Compile and Run.

If there are problems you will clearly see it in the messages section:

Enlarged Messages output:

Here we have accounts that can connect from any hosts and accounts which do not have a password. Those accounts should not exist in a secure database installation. That is rule number one. To correct this problem, click on mysql_secure_installation.js in the s9s/mysql/programs folder.

Click on the dropdown arrow next to Compile and Run and press Change Settings. You will see the following dialog and enter the argument “STRICT”:

Then press Execute. The mysql_secure_installation.js script will then do on each MySQL database instance part of the cluster:

1. Delete anonymous users
2. Dropping 'test' database (if exists).
3. If STRICT is given as an argument to mysql_secure_installation.js it will also do:
   • Remove accounts without passwords.

In the Message box you will see:

The MySQL database servers part of this cluster have now been secured and you have reduced the risk of compromising your data.

You can re-run security_audit.js to verify that the actions have had effect.

Happy Clustering!

PS.: To get started with ClusterControl, click here!

This video, based on last weeks blog “Ten Tips to Achieve MySQL and MariaDB Security”, walks you through ten different items to keep in mind when deploying a MySQL or MariaDB database to production.

Database security is an essential part of any system. With more and more news reports of widespread data breaches coming in from around the world, there is no better time to check your environments and make sure you have implemented these basic steps to remain secure.

ClusterControl for Database Security

ClusterControl provides advanced deployment, monitoring and management features to ensure your databases and their data are secure. It ensures that your open source database deployments always adhere to basic security model setups for each technology.

ClusterControl provides the Package Summary Operational Report that shows you how many technology and security patches are available to upgrade and can even execute the upgrades for you!

In addition ClusterControl offers…

• Secure Deployments
  Every technology has its own unique security features and ClusterControl ensures that what should be enabled is enabled during deployment. This eliminates the risk of human error which could otherwise result in leaving the database vulnerable because of a security setting oversight.
• Communication Security
  ClusterControl provides the ability to install a purchased or self-signed SSL certificate to encrypt the communications between the server and the client. Replication traffic within a Galera Cluster can also be encrypted. Keys for these certificates are entered into and managed by ClusterControl.
• Backup Security
  Backups are encrypted at rest using AES-256 CBC algorithm. An auto generated key will be stored in the cluster's configuration file under /etc/cmon.d. The backup files are transferred in encrypted format. Users can now secure their backups for offsite or cloud storage with the flip of a checkbox. This feature is available for select backup methods for MySQL, MongoDB & PostgreSQL.
• User Management
  ClusterControl’s advanced user management features allow you to restrict read or write access to your data at the database or table level. ClusterControl also provides advisors that check that all of your users have proper passwords, and even comes with checks to make sure any part of your database is not open to the public.
• Reports & Auditing
  ClusterControl provides reporting and audit tools to ensure you remain compliant, whether it is to an industry standard or to your own requirements. It also provides several Developer Studio Advisors that check your database environment to ensure that it is secure. You can even create your own security advisors to automate your own best practices. In addition, several Operational Reports found in ClusterControl can provide you with information you need to know to ensure your database environment is secure.

Download ClusterControl today to take advantage of these database security features.

As a distributed database system, Galera Cluster requires additional security measures as compared to a centralized database. Data is distributed across multiple servers or even datacenters perhaps. With significant data communication happening across nodes, there can be significant exposure if the appropriate security measures are not taken.

In this blog post, we are going to look into some tips on how to secure our Galera Cluster. Note that this blog builds upon our previous blog post - How to Secure Your Open Source Databases with ClusterControl.

Firewall & Security Group

The following ports are very important for a Galera Cluster:

• 3306 - MySQL
• 4567 - Galera communication and replication
• 4568 - Galera IST
• 4444 - Galera SST

From the external network, it is recommended to only open access to MySQL port 3306. The other three ports can be closed down from the external network, and only allows them for internal access between the Galera nodes. If you are running a reverse proxy sitting in front of the Galera nodes, for example HAProxy, you can lock down the MySQL port from public access. Also ensure the monitoring port for the HAProxy monitoring script is opened. The default port is 9200 on the Galera node.

The following diagram illustrates our example setup on a three-node Galera Cluster, with an HAProxy facing the public network with its related ports:

Based on the above diagram, the iptables commands for database nodes are:

$ iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 3306 -j ACCEPT
$ iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 4444 -j ACCEPT
$ iptables -A INPUT -p tcp -s 10.0.0.0/24 --dports 4567:4568 -j ACCEPT
$ iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 9200 -j ACCEPT

While on the load balancer:

$ iptables -A INPUT -p tcp --dport 3307 -j ACCEPT

Make sure to end your firewall rules with deny all, so only traffic as defined in the exception rules is allowed. You can be stricter and extend the commands to follow your security policy - for example, by adding network interface, destination address, source address, connection state and what not.

MySQL Client-Server Encryption

MySQL supports encryption between the client and the server. First we have to generate the certificate. Once configured, you can enforce user accounts to specify certain options to connect with encryption to a MySQL server.

The steps require you to:

1. Create a key for Certificate Authority (ca-key.pem)
2. Generate a self-signed CA certificate (ca-cert.pem)
3. Create a key for server certificate (server-key.pem)
4. Generate a certificate for server and sign it with ca-key.pem (server-cert.pem)
5. Create a key for client certificate (client-key.pem)
6. Generate a certificate for client and sign it with ca-key.pem (client-cert.pem)

Always be careful with the CA private key (ca-key.pem) - anybody with access to it can use it to generate additional client or server certificates that will be accepted as legitimate when CA verification is enabled. The bottom line is all the keys must be kept discreet.

You can then add the SSL-related variables under [mysqld] directive, for example:

ssl-ca=/etc/ssl/mysql/ca-cert.pem
ssl-cert=/etc/ssl/mysql/server-cert.pem
ssl-key=/etc/ssl/mysql/server-key.pem

Restart the MySQL server to load the changes. Then create a user with the REQUIRE SSL statement, for example:

mysql> GRANT ALL PRIVILEGES ON db1.* TO 'dbuser'@'192.168.1.100' IDENTIFIED BY 'mySecr3t' REQUIRE SSL;

The user created with REQUIRE SSL will be enforced to connect with the correct client SSL files (client-cert.pem, client-key.pem and ca-cert.pem).

With ClusterControl, client-server SSL encryption can easily be enabled from the UI, using the "Create SSL Encryption" feature.

Galera Encryption

Enabling encryption for Galera means IST will also be encrypted because the communication happens via the same socket. SST, on the other hand, has to be configured separately as shown in the next section. All nodes in the cluster must be enabled with SSL encryption and you cannot have a mix of nodes where some have enabled SSL encryption, and others not. The best time to configure this is when setting up a new cluster. However, if you need to add this on a running production system, you will unfortunately need to rebootstrap the cluster and there will be downtime.

All Galera nodes in the cluster must use the same key, certificate and CA (optional). You could also use the same key and certificate created for MySQL client-server encryption, or generate a new set for this purpose only. To activate encryption inside Galera, one has to append the option and value under wsrep_provider_options inside the MySQL configuration file on each Galera node. For example, consider the following existing line for our Galera node:

wsrep_provider_options = "gcache.size=512M; gmcast.segment=0;"

Append the related variables inside the quote, delimited by a semi-colon:

wsrep_provider_options = "gcache.size=512M; gmcast.segment=0; socket.ssl_cert=/etc/mysql/cert.pem; socket.ssl_key=/etc/mysql/key.pem;"

For more info on the Galera's SSL related parameters, see here. Perform this modification on all nodes. Then, stop the cluster (one node at a time) and bootstrap from the last node that shut down. You can verify if SSL is loaded correctly by looking into the MySQL error log:

2018-01-19T01:15:30.155211Z 0 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '192.168.10.61:,192.168.10.62:,192.168.10.63:'
2018-01-19T01:15:30.159654Z 0 [Note] WSREP: SSL handshake successful, remote endpoint ssl://192.168.10.62:53024 local endpoint ssl://192.168.10.62:4567 cipher: AES128-SHA compression:

With ClusterControl, Galera Replication encryption can be easily enabled using the "Create SSL Galera Encryption" feature.

SST Encryption

When SST happens without encryption, the data communication is exposed while the SST process is ongoing. SST is a full data synchronization process from a donor to a joiner node. If an attacker was able to "see" the full data transmission, the person would get a complete snapshot of your database.

SST with encryption is supported only for mysqldump and xtrabackup-v2 methods. For mysqldump, the user must be granted with "REQUIRE SSL" on all nodes and the configuration is similar to standard MySQL client-server SSL encryption (as described in the previous section). Once the client-server encryption is activated, create a new SST user with SSL enforced:

mysql> GRANT ALL ON *.* TO 'sst_user'@'%' IDENTIFIED BY 'mypassword' REQUIRE SSL;

For rsync, we recommend using galera-secure-rsync, a drop-in SSL-secured rsync SST script for Galera Cluster. It operates almost exactly like wsrep_sst_rsync except that it secures the actual communications with SSL using socat. Generate the required client/server key and certificate files, copy them to all nodes and specify the "secure_rsync" as the SST method inside the MySQL configuration file to activate it:

wsrep_sst_method=secure_rsync

For xtrabackup, the following configuration options must be enabled inside the MySQL configuration file under [sst] directive:

[sst]
encrypt=4
ssl-ca=/path/to/ca-cert.pem
ssl-cert=/path/to/server-cert.pem
ssl-key=/path/to/server-key.pem

Database restart is not necessary. If this node is selected by Galera as a donor, these configuration options will be picked up automatically when Galera initiates the SST.

SELinux

Security-Enhanced Linux (SELinux) is an access control mechanism implemented in the kernel. Without SELinux, only traditional access control methods such as file permissions or ACL are used to control the file access of users.

By default, with strict enforcing mode enabled, everything is denied and the administrator has to make a series of exceptions policies to the elements of the system require in order to function. Disabling SELinux entirely has become a common poor practice for many RedHat based installation nowadays.

Depending on the workloads, usage patterns and processes, the best way is to create your own SELinux policy module tailored for your environment. What you really need to do is to set SELinux to permissive mode (logging only without enforce), and trigger events that can happen on a Galera node for SELinux to log. The more extensive the better. Example events like:

• Starting node as donor or joiner
• Restart node to trigger IST
• Use different SST methods
• Backup and restore MySQL databases using mysqldump or xtrabackup
• Enable and disable binary logs

One example is if the Galera node is monitored by ClusterControl and the query monitor feature is enabled, ClusterControl will enable/disable the slow query log variable to capture the slow running queries. Thus, you would see the following denial in the audit.log:

$ grep -e denied audit/audit.log | grep -i mysql
type=AVC msg=audit(1516835039.802:37680): avc:  denied  { open } for  pid=71222 comm="mysqld" path="/var/log/mysql/mysql-slow.log" dev="dm-0" ino=35479360 scontext=system_u:system_r:mysqld_t:s0 tcontext=unconfined_u:object_r:var_log_t:s0 tclass=file

The idea is to let all possible denials get logged into the audit log, which later can be used to generate the policy module using audit2allow before loading it into SELinux. Codership has covered this in details in the documentation page, SELinux Configuration.

SST Account and Privileges

SST is an initial syncing process performed by Galera. It brings a joiner node up-to-date with the rest of the members in the cluster. The process basically exports the data from the donor node and restores it on the joiner node, before the joiner is allowed to catch up on the remaining transactions from the queue (i.e., those that happened during the syncing process). Three SST methods are supported:

• mysqldump
• rsync
• xtrabackup (or xtrabackup-v2)

For mysqldump SST usage, the following privileges are required:

• SELECT, SHOW VIEW, TRIGGER, LOCK TABLES, RELOAD, FILE

We are not going to go further with mysqldump because it is probably not often used in production as SST method. Beside, it is a blocking procedure on the donor. Rsync is usually a preferred second choice after xtrabackup due to faster syncing time, and less error-prone as compared to mysqldump. SST authentication is ignored with rsync, therefore you may skip configuring SST account privileges if rsync is the chosen SST method.

Moving along with xtrabackup, the following privileges are advised for standard backup and restore procedures based on the Xtrabackup documentation page:

• CREATE, CREATE TABLESPACE, EVENT, INSERT, LOCK TABLE, PROCESS, RELOAD, REPLICATION CLIENT, SELECT, SHOW VIEW, SUPER

However for xtrabackup's SST usage, only the following privileges matter:

• PROCESS, RELOAD, REPLICATION CLIENT

Thus, the GRANT statement for SST can be minimized as:

mysql> GRANT PROCESS,RELOAD,REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost' IDENTIFIED BY 'SuP3R@@sTr0nG%%P4ssW0rD';

Then, configure wsrep_sst_auth accordingly inside MySQL configuration file:

wsrep_sst_auth = sstuser:SuP3R@@sTr0nG%%P4ssW0rD

Only grant the SST user for localhost and use a strong password. Avoid using root user as the SST account, because it would expose the root password inside the configuration file under this variable. Plus, changing or resetting the MySQL root password would break SST in the future.

MySQL Security Hardening

Galera Cluster is a multi-master replication plugin for InnoDB storage engine, which runs on MySQL and MariaDB forks. Therefore, standard MySQL/MariaDB/InnoDB security hardening recommendations apply to Galera Cluster as well.

This topic has been covered in numerous blog posts out there. We have also covered this topic in the following blog posts:

• Ten Tips on How to Achieve MySQL and MariaDB Security
• ClusterControl Tips & Tricks: Securing your MySQL Installation
• How to Secure Your Open Source Databases with ClusterControl

The above blog posts summarize the necessity of encrypting data at rest and data in transit, having audit plugins, general security guidelines, network security best practices and so on.

Use a Load Balancer

There are a number of database load balancers (reverse proxy) that can be used together with Galera - HAProxy, ProxySQL and MariaDB MaxScale to name some of them. You can set up a load balancer to control access to your Galera nodes. It is a great way of distributing the database workload between the database instances, as well as restricting access, e.g., if you want to take a node offline for maintenance, or if you want to limit the number of connections opened on the Galera nodes. The load balancer should be able to queue connections, and therefore provide some overload protection to your database servers.

ProxySQL, a powerful database reverse-proxy which understands MySQL and MariaDB, can be extended with many useful security features like query firewall, to block offending queries from the database server. The query rules engine can also be used to rewrite bad queries into something better/safer, or redirect them to another server which can absorb the load without affecting any of the Galera nodes. MariaDB MaxScale also capable of blocking queries based on regular expressions with its Database Firewall filter.

Another advantage having a load balancer for your Galera Cluster is the ability to host a data service without exposing the database tier to the public network. The proxy server can be used as the bastion host to gain access to the database nodes in a private network. By having the database cluster isolated from the outside world, you have removed one of the important attacking vectors.

That's it. Always stay secure and protected.

It’s kind of difficult to start a blog with something original that hasn’t been written before when blogging about attending or sponsoring a particular event or conference…

What can I say?

FOSDEM 2018 is round the corner? Ready for FOSDEM 2018? Join us next week at FOSDEM 2018? All set for FOSDEM 2018 next week? The possibilities are endless…

So here we go ;-)

FOSDEM 2018 D-7!

If you’re in Europe and into open source (databases) make sure not to miss FOSDEM 2018 in Brussels, which takes place next week from the 2nd (unofficially) to the 4th of February (officially).

Thousands of open source enthusiasts will be gathering in the Belgium capital to talk latest technologies, concepts and ideas all around open source software.

If you haven’t made plans to attend yet, you can find all the details on the FOSDEM website.

Why and how are we at Severalnines involved?

Well, we help users automate and manage their open source databases and are historically closely tied to the MySQL database (on a personal and technology level) - though we also support PostgreSQL and MongoDB of course.

And the MySQL Community in Europe has always taken advantage of the FOSDEM conference to get together and exchange on the latest and greatest to do with MySQL. The group organising this particular aspect of FOSDEM is the MySQL & Friends group. With that, FOSDEM hosts a MySQL DevRoom dedicated to the open source database and related technologies.

And since the MySQL community has always been friendly towards food and drink, there is a yearly MySQL Community Dinner that takes place the evening before the official FOSDEM start.

That’s where we come in (alongside other illustrious names) since we’re co-sponsoring this year’s MySQL Community Dinner again!

This year the dinner is sponsored by Oracle MySQL, MariaDB, Facebook, Percona and our true selves, and we’d like to wish all participants a great evening and get together.

If you haven’t booked your tickets yet, there’s still time do so here.

Have fun at FOSDEM and enjoy the MySQL Community Dinner!

Security of data is critical for any organisation. It’s an important aspect that can heavily influence the design of the database environment. When deciding upon which MySQL flavour to use, you need to take into consideration the security features available from the different server vendors. In this blog post, we’ll come up with a short comparison of the latest versions of the MySQL Community Edition from Oracle, Percona Server and MariaDB:

mysqld  Ver 5.7.20-19 for Linux on x86_64 (Percona Server (GPL), Release 19, Revision 3c5d3e5d53c)
mysqld  Ver 5.7.21 for Linux on x86_64 (MySQL Community Server (GPL))
mysqld  Ver 10.2.12-MariaDB for Linux on x86_64 (MariaDB Server)

We are going to use Centos 7 as the operating system - please keep in mind that results we present here may be slightly different on other distributions like Debian or Ubuntu. We’d also like to focus on the differences and will not cover the commonalities - Percona Server and MariaDB are flavors of MySQL, so some of the security features (e.g., how access privileges of MySQL files look like) are shared among them.

Initial security

Users

Both Percona Server and MySQL Community Server comes with a randomly generated temporary password for the root user. You need to check the contents of MySQL’s error log to find it:

2018-01-19T13:47:45.532148Z 1 [Note] A temporary password is generated for root@localhost: palwJu7uSL,g

Once you log in, a password change is forced upon you:

[root@localhost ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.21

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from mysql.user;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

Password has to be strong enough, this is enforced by the validate_password plugin:

mysql> alter user root@localhost identified by 'password123.';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> alter user root@localhost identified by 'password123.A';
Query OK, 0 rows affected (0.00 sec)

MariaDB does not generate a random root password and it provides passwordless access to the root account from (and only from) localhost.

[root@localhost ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.2.12-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SELECT CURRENT_USER();
+----------------+
| CURRENT_USER() |
+----------------+
| root@localhost |
+----------------+
1 row in set (0.00 sec)

This is not a big issue during the initial deployment phase, as the DBA is supposed to configure and secure access to the database later on (by running mysql_secure_installation for example). The bigger problem here is that a good practice is not enforced by MariaDB. If you don’t have to setup a strong password for the root user, it could be that nobody changes it later and passwordless access will remain. Then this would become a serious security threat.

Another aspect we’d like to look at is anonymous, passwordless access. Anonymous users allow anyone to get in, it doesn’t have to be a predefined user. If such access is passwordless, it means that anyone can connect to MySQL. Typically such account has only USAGE privilege but even then it is possible to print a status (‘\s’) which contains information like MySQL version, character set etc. Additionally, if ‘test’ schema is available, such user has the ability to write to that schema.

Both MySQL Community Server and Percona server do not have any anonymous users defined in MySQL:

mysql> select user, host, authentication_string from mysql.user;
+---------------+-----------+-------------------------------------------+
| user          | host      | authentication_string                     |
+---------------+-----------+-------------------------------------------+
| root          | localhost | *EB965412B594F67C8EB611810EF8D406F2CF42BD |
| mysql.session | localhost | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
| mysql.sys     | localhost | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE |
+---------------+-----------+-------------------------------------------+
3 rows in set (0.00 sec)

On the other hand, MariaDB is open for anonymous, passwordless access.

MariaDB [(none)]> select user,host,password from mysql.user;
+------+-----------------------+----------+
| user | host                  | password |
+------+-----------------------+----------+
| root | localhost             |          |
| root | localhost.localdomain |          |
| root | 127.0.0.1             |          |
| root | ::1                   |          |
|      | localhost             |          |
|      | localhost.localdomain |          |
+------+-----------------------+----------+
6 rows in set (0.00 sec)

In addition to that, the ‘test’ schema is available - which allows anonymous users to issue writes to the database.

[root@localhost ~]# mysql -umyanonymoususer
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 10.2.12-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use test;
Database changed
MariaDB [test]> CREATE TABLE mytab (a int);
Query OK, 0 rows affected (0.01 sec)

MariaDB [test]> INSERT INTO mytab VALUES (1), (2);
Query OK, 2 rows affected (0.02 sec)
Records: 2  Duplicates: 0  Warnings: 0

MariaDB [test]> SELECT * FROM mytab;
+------+
| a    |
+------+
|    1 |
|    2 |
+------+
2 rows in set (0.00 sec)

This poses a serious threat, and needs to be sorted out. Else, it can be easily exploited to attempt to overload the server with writes.

Data in transit security

MySQL Community Server and both of its forks support the use of SSL to encrypt data in transit. This is extremely important for Wide Area Networks, but also shouldn’t be overlooked in a local network. SSL can be used both client and server-side. Regarding server-side configuration (to encrypt traffic from master to slaves, for example), it looks identical across the board. There is a difference though when it comes to client-side SSL encryption, introduced in MySQL 5.7. Prior to 5.7, one had to generate SSL keys and CA’s and define them in the configurations of both server and client. This is how MariaDB’s 10.2 SSL setup looks like. In both MySQL Community Server 5.7 and in Percona Server 5.7 (which is based on MySQL 5.7), there is no need to pre-generate keys. It is all done automatically, in the background. All you need to do is to enable SSL on your client by setting the correct ‘--ssl-mode’. For MySQL’s CLI client, this is not even needed as it enables SSL by default:

[root@localhost ~]# mysql -p -h127.0.0.1
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.21 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> \s
--------------
mysql  Ver 14.14 Distrib 5.7.21, for Linux (x86_64) using  EditLine wrapper

Connection id:        6
Current database:
Current user:        root@localhost
SSL:            Cipher in use is DHE-RSA-AES256-SHA
Current pager:        stdout
Using outfile:        ''
Using delimiter:    ;
Server version:        5.7.21 MySQL Community Server (GPL)
Protocol version:    10
Connection:        127.0.0.1 via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:        3306
Uptime:            2 days 21 hours 51 min 52 sec

Threads: 1  Questions: 15  Slow queries: 0  Opens: 106  Flush tables: 1  Open tables: 99  Queries per second avg: 0.000
--------------

On the other hand MariaDB would require additional configuration as SSL is disabled by default:

[root@localhost ~]# mysql -h127.0.0.1
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 18
Server version: 10.2.12-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> \s
--------------
mysql  Ver 15.1 Distrib 10.2.12-MariaDB, for Linux (x86_64) using readline 5.1

Connection id:        18
Current database:
Current user:        root@localhost
SSL:            Not in use
Current pager:        stdout
Using outfile:        ''
Using delimiter:    ;
Server:            MariaDB
Server version:        10.2.12-MariaDB MariaDB Server
Protocol version:    10
Connection:        127.0.0.1 via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:        3306
Uptime:            2 days 22 hours 26 min 58 sec

Threads: 7  Questions: 45  Slow queries: 0  Opens: 18  Flush tables: 1  Open tables: 12  Queries per second avg: 0.000
--------------

Data at rest encryption

First of all, backups - there are freely available backup tools like xtrabackup or MariaDB Backup (which is a fork of xtrabackup). These allow to create encrypted backups of all three MySQL flavors we discuss in this blog post.

All three flavours support encryption of the running database, but there are differences in what pieces of data are encrypted.

The MySQL Community Server supports encryption of InnoDB tablespaces only. Keys used for encryption are stored in files (which is not compliant with regulations - keys should be stored in a vault - something which MySQL Enterprise supports). Percona Server is based on MySQL Community Server, so it also supports encryption of InnoDB tablespaces. Recently, in Percona Server 5.7.20, support for encryption of general tablespaces (compared to only individual ones in previous versions and MySQL Community Edition) was added. Support for encryption of binary logs was also added. Percona Server comes with a keyring_vault plugin, which can be used to store keys in Hashicorp Vault server, making Percona Server 5.7.20 compliant with regulatory requirements regarding data at rest encryption.

MariaDB 10.2 has more advanced data-at-rest encryption support. In addition to tablespace and binary/relay log encryption, it has support for encrypting InnoDB redo logs. Currently, it is the more complete solution regarding data encryption.

Audit logging

All three MySQL flavors have support for audit logging. Their scope is pretty much comparable: connect and disconnect events, queries executed, tables accessed. The logs contain information about which user participated in such event, from what host the user logged from, the time it happened, and similar info. Such events can be also logged via syslog and stored on an external log server to enable log analysis and parsing.

Data masking, SQL firewall

All of the discussed MySQL flavors work with some kind of tool which would allow implementing data masking, and would be able to block SQL traffic based on some rules. Data masking is a method of obfuscating some data outside of the database, but before it reaches client. An example would be credit card data which is stored in plain text in the database, but when a developer wants to query such data, she will see ‘xxxxxxxx...’ instead of numbers. The tools we are talking here are ProxySQL and MaxScale. MaxScale is a product of MariaDB Corporation, and is subscription-based. ProxySQL is a free to use database proxy. Both proxies can be used with any of the MySQL flavours.

That’s all for today folks. For further reading, check out these 10 tips for securing your MySQL and MariaDB databases.

We get some nice feedback with regards to our product ClusterControl, especially how easy it is to install and get going. Installing new software is one thing, but using it properly is another.

It is not uncommon to be impatient to test new software and one would rather toy around with a new exciting application than to read documentation before getting started. That is a bit unfortunate as you may miss important features or misunderstand how to use them.

This blog series covers all the basic operations of ClusterControl for MySQL, MongoDB & PostgreSQL with examples on how to make the most of your setup. It provides you with a deep dive on different topics to save you time.

These are the topics covered in this series:

• Deploying the first clusters
• Adding your existing infrastructure
• Performance and health monitoring
• Making your components HA
• Workflow management
• Safeguarding your data
• Protecting your data
• In depth use case

In today’s post, we’ll cover installing ClusterControl and deploying your first clusters.

Preparations

In this series, we will make use of a set of Vagrant boxes but you can use your own infrastructure if you like. In case you do want to test it with Vagrant, we made an example setup available from the following Github repository: https://github.com/severalnines/vagrant

Clone the repo to your own machine:

$ git clone git@github.com:severalnines/vagrant.git

The topology of the vagrant nodes is as follows:

• vm1: clustercontrol
• vm2: database node1
• vm3: database node2
• vm4: database node3

You can easily add additional nodes if you like by changing the following line:

4.times do |n|

The Vagrant file is configured to automatically install ClusterControl on the first node and forward the user interface of ClusterControl to port 8080 on your host that runs Vagrant. So if your host’s ip address is 192.168.1.10, you will find the ClusterControl UI here: http://192.168.1.10:8080/clustercontrol/

Installing ClusterControl

You can skip this if you chose to use the Vagrant file, and get the automatic installation. But installation of ClusterControl is straightforward and will take less than five minutes.

With the package installation, all you have to do is to issue the following three commands on the ClusterControl node to get it installed:

$ wget http://www.severalnines.com/downloads/cmon/install-cc
$ chmod +x install-cc
$ ./install-cc   # as root or sudo user

That’s it: it can’t get easier than this. If the installation script has not encountered any issues, then ClusterControl should be installed and up and running. You can now log into ClusterControl on the following URL: http://192.168.1.210/clustercontrol

After creating an administrator account and logging in, you will be prompted to add your first cluster.

Deploy a Galera cluster

You will be prompted to create a new database server/cluster or import an existing (i.e., already deployed) server or cluster:

We are going to deploy a Galera cluster. There are two sections that need to be filled in. The first tab is related to SSH and general settings:

To allow ClusterControl to install the Galera nodes, we use the root user that was granted SSH access by the Vagrant bootstrap scripts. In case you chose to use your own infrastructure, you must enter a user here that is allowed to do passwordless SSH to the nodes that ClusterControl will control. Just keep in mind that you have to setup passwordless SSH from ClusterControl to all database nodes by yourself beforehand.

Also make sure you disable AppArmor/SELinux. See here why.

Then, proceed to the second stage and specify the database related information and the target hosts:

ClusterControl will immediately perform some sanity checks each time you press Enter when adding a node. You can see the host summary by hovering over each defined node. Once everything is green, it means that ClusterControl has connectivity to all nodes, you can click Deploy. A job will be spawned to build the new cluster. The nice thing is that you can keep track of the progress of this job by clicking on the Activity -> Jobs -> Create Cluster -> Full Job Details:

Once the job has finished, you have just created your first cluster. The cluster overview should look like this:

In the nodes tab, you can do about any operation you normally would do on a cluster. The query monitor gives you a good overview of both running and top queries. The performance tab will help you keep a close eye on the performance of your cluster and also features the advisors that help you act proactively on trends in data. The backup tab enables you to easily schedule backups and store them on local or cloud storage. The manage tab enables you to expand your cluster or make it highly available for your applications through a load balancer.

All this functionality will be covered in later blog posts in this series.

Deploy a MySQL Replication Cluster

Deploying a MySQL Replication setup is similar to Galera database deployment, except that it has an additional tab in the deployment dialog where you can define the replication topology:

You can set up standard master-slave replication, as well as master-master replication. In case of the latter, only one master will remain writable at a time. Keep in mind that master-master replication doesn't come with conflict resolution and guaranteed data consistency, as in the case of Galera. Use this setup with caution, or look into Galera cluster. Once everything is green and you have clicked Deploy, a job will be spawned to build the new cluster.

Again, the deployment progress is available under Activity -> Jobs.

To scale out the slave (read copy), simply use the “Add Node” option in the cluster list:

After adding the slave node, ClusterControl will provision the slave with a copy of the data from its master using Xtrabackup or from any existing PITR compatible backups for that cluster.

Deploy PostgreSQL Replication

ClusterControl supports the deployment of PostgreSQL version 9.x and higher. The steps are similar with MySQL Replication deployment, where at the end of the deployment step, you can define the database topology when adding the nodes:

Similar to MySQL Replication, once the deployment completes, you can scale out by adding replications slave to the cluster. The step is as simple as selecting the master and filling in the FQDN for the new slave:

ClusterControl will then perform the necessary data staging from the chosen master using pg_basebackup, configure the replication user and enable the streaming replication. The PostgreSQL cluster overview gives you some insight into your setup:

Just like with the Galera and MySQL cluster overviews, you can find all the necessary tabs and functions here: the query monitor, performance, backup tabs all enable you to do the necessary operations.

Deploy a MongoDB Replica Set

Deploying a new MongoDB Replica Set is similar to the other clusters. From the Deploy Database Cluster dialog, pick MongoDB ReplicatSet, define the preferred database options and add the database nodes:

You can either choose to install Percona Server for MongoDB from Percona or MongoDB Server from MongoDB, Inc (formerly 10gen). You also need to specify the MongoDB admin user and password since ClusterControl will deploy by default a MongoDB cluster with authentication enabled.

After installing the cluster, you can add an additional slave or arbiter node into the replica set using the "Add Node" menu under the same dropdown from the cluster overview:

After adding the slave or arbiter to the replica set, a job will be spawned. Once this job has finished it will take a short while before MongoDB adds it to the cluster and it becomes visible in the cluster overview:

Final thoughts

With these three examples we have shown you how easy it is to set up different clusters from scratch in only a couple of minutes. The beauty of using this Vagrant setup is that, as easy as spawning this environment, you can also take it down and then spawn again. Impress your fellow colleagues by showing how quickly you can setup a working environment.

Of course it would be equally interesting to add existing hosts and already-deployed clusters into ClusterControl, and that’s what we'll cover next time.