Tag Archives: Replication

What is Galera Arbitrator?

Galera Arbitrator (garbd) is a stateless daemon that can act like a node in a Galera cluster. It is normally used to avoid split-brain situation which mostly occurs because of hardware/link failure, as a result of which the cluster gets divided into two parts and each part remains operational thinking they are in majority (primary component). This may lead to inconsistent data sets. Garbd should be installed on a separate machine. However, it can share the machine running load-balancer. It is interesting to note that as garbd joins the cluster, it makes a request for SST.

Let us now try to add galera arbitrator (garbd) to an existing 2-node MariaDB Galera cluster. Check out this post for steps to setup a MariaDB Galera cluster. We start by connecting to one of the nodes to get the size and name of the cluster.

MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 2     |
+--------------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> show variables like 'wsrep_cluster_name';
+--------------------+------------------+
| Variable_name      | Value            |
+--------------------+------------------+
| wsrep_cluster_name | my_wsrep_cluster |
+--------------------+------------------+
1 row in set (0.00 sec)

Now, lets start the garbd as a daemon:

$ cd galera-23.2.7-src/garb

$ ./garbd
    --address='gcomm://127.0.0.1:4567'
    --options='gmcast.listen_addr=tcp://127.0.0.1:4569'
    --group='my_wsrep_cluster'
    --log=garbd.err
    --daemon

That’s it, garbd should now be running and connected to the cluster. Let’s verify this by checking wsrep_cluster_size again.

MariaDB [test]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.00 sec)

Great! So, the new garbd is now part of the cluster.

Setting up MariaDB Galera Cluster on Ubuntu

MariaDB Galera Cluster is a multi-master synchronous replication system. In this article, I would be setting up a 3-node cluster on a single machine running Ubuntu. However, in a production scenario it is advised to run each cluster node on a separate box in a WAN.

Requirements

  1. MariaDB Galera Cluster
    • Download it from the official site : https://downloads.mariadb.org/mariadb-galera/, OR
    • Install it using Advanced packaging tool (APT), steps can be found in erkules’ blog post. OR
    • Build it from source : lp:~maria-captains/maria/maria-5.5-galera
      Note : Build would require additional cmake options : WITH_WSREP=ON and WITH_INNODB_DISALLOW_WRITES=1
  2. Galera wsrep provider (libgalera_smm.so)
  3. Some extra Ubuntu packages (in case you choose to build Galera from source!)
    • scons (Build utility, replacement for make)
    • check (Unit test framework for C)
    • libboost-dev
    • libboost-program-options-dev
    • libboost-system-dev (for 23.2.7)
    • libssl-dev

Setup

Now that we have all requirements in place, lets bring up the cluster nodes.

  1. Node#1 : Start 1st node (at port 4001 for instance) with empty cluster address (–wsrep_cluster_address=’gcomm://’).
     $ mysqld
        --no-defaults 
        --basedir=.
        --datadir=./data
        --port=4001
        --socket=/tmp/mysql_4001.sock
        --binlog_format=ROW
        --wsrep_provider=/path-to-galera-provider/libgalera_smm.so
        --wsrep_cluster_address='gcomm://'

    There are certain points to note before we proceed :

    • wsrep_provider option points to the galera wsrep provider, i.e. libgalera_smm.so library.
    • wsrep_cluster_address contains the address of existing cluster members. But in this case, as it is the very first node, the address must be empty.

      Important! In case you are planning to put these options in a config file (my.cnf) – once the cluster is up and running, make sure to change this option to hold a valid address. Failure to do so would disable the node’s capability to auto-join the cluster in case it restarts after a shutdown or crash.

    • From the server’s error log make a note of the base_host & base_port (default 4567) of wsrep. This information would be required to start the subsequent nodes.
  2. Node#2 : Start 2nd node at a different port (4002).
     $ mysqld
        --no-defaults
        --basedir=.
        --datadir=./data
        --port=4002
        --socket=/tmp/mysql_4002.sock
        --binlog_format=ROW
        --wsrep_provider=/path-to-galera-provider/libgalera_smm.so
        --wsrep_cluster_address='gcomm://127.0.0.1:4567'
        --wsrep_provider_options='gmcast.listen_addr=tcp://127.0.0.1:4568'

    Here, we have to specify the address of 1st node via wsrep_cluser_address option. It consists of base_host & base_port of the 1st node that we noted earlier in step 1 (i.e. gcomm://127.0.0.1:4567). Also, as we are starting this 2nd node on the same machine, in order to avoid port conflict, we must provide a different port for wsrep provider to listen to via gmcast.listen_addr wsrep provider option (tcp://127.0.0.1:4568).

  3. Node#3 : As with 2nd node, 3rd (and all subsequent nodes) can be started by same set of options with appropriate selection of available ports.
     $ mysqld
        --no-defaults
        --basedir=.
        --datadir=./data
        --port=4003
        --socket=/tmp/mysql_4003.sock
        --binlog_format=ROW
        --wsrep_provider=/path-to-galera-provider/libgalera_smm.so
        --wsrep_cluster_address='gcomm://127.0.0.1:4567'
        --wsrep_provider_options='gmcast.listen_addr=tcp://127.0.0.1:4569'

The cluster should now be up and running with 3 nodes. This can easily be verified and monitored further by inspecting server’s status & system variables :

MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.06 sec)

MariaDB [(none)]> show variables like 'wsrep%';
...
MariaDB [(none)]> show status like 'wsrep%';
...

Before I close, let me list out some important options which were omitted from this article for brevity :

  • default_storage_engine=INNODB
  • innodb_autoinc_lock_mode=2
  • innodb_locks_unsafe_for_binlog=1
  • wsrep_sst_auth=”user:pass” : to be used by SST (Snapshot state transfer) script

That’s all for now!