Showing posts with label percona. Show all posts
Showing posts with label percona. Show all posts

Tuesday, October 23, 2012

Hotbackup of Galera Cluster for MySQL using Xtrabackup


Hot backups are important for high availability, they can run without blocking the application. Percona Xtrabackup is a great tool for backing up InnoDB data. 
We have created a backup tool called s9s_backup that makes Xtrabackup really easy to use, and is fully integrated with ClusterControl, which means that you can schedule backups with ease and view the backups that you have taken, and also restore the backups with no pain.
s9s_backup is available in the lastest version of ClusterControl or you can download it here.

Is mysqldump totally useless then?

No. If you would like to isolate and load only one table, mysqldump is great, or if you want to load your data into another storage engine or database (e.g. NDB). Then mysqldump comes in very hand. Within ClusterControl you can create schedules to make a mysqldumps, full and incremental backups using Xtrabackup.

How to use s9s_backup

s9s_backup is included in ClusterControl and automatically installed in either /usr/bin or /usr/local/cmon/bin
Running/usr/bin/s9s_backup prints out the set of options that s9s_backup supports:
/usr/bin/s9s_backup <--backup|--list|--install|--purge|--restore>  

Full Backup

/usr/bin/s9s_backup --backup full  /path/to/my.cnf     <backupdir>
This will take a full backup of the host where you run the tool and store the backup in <backupdir>/full/backupdir.
s9s_backup writes a log  (you can change it easily in the s9s_backup script if you want it on screen instead) to /tmp/s9s_backup.log.
Status information is written into the CMON Database, so you can see if a backup has failed or succeeded, size information, md5 etc.

Incremental Backup 

/usr/bin/s9s_backup --backup incr  /path/to/my.cnf    <backupdir>
This will take an incremental backup of the host where you run the tool and store the backup in <backupdir>/incremental.

In order to take an incremental backup you must have already made a full backup on the host. So when creating the backup schedule (in either cron or ClusterControl), make user the first backup that will run is a full backup, else the incremental backups will fail.
s9s_backup writes a log to /tmp/s9s_backup.log

List backups

/usr/bin/s9s_backup --list
This prints out a list of backup sets (a full backup plus the subsequent incremental backups) together with a backup id. This is called the backup set:
root@server01:~# ./s9s_backup --list
===========================================================================
Full backup taken '2012-October-23_07:40:37':
   server:     server01
   backup set: 2
   status:     completed
   location:   /backups//full/backup-full-...tar.gz (796873 bytes)
   lsn:        8791838
   md5:        31f84543c997a28a6bca7917776a5fac
   >> incremental backup taken '2012-October-23_07:58:48':
      server:   server01
      parent:   2
      status:   completed
      location: /backups//incremental/backup-incr-...tar.gz (2037959 bytes)
      lsn:      35177200
      md5:      5f845dd7a478fa0a99b4d97f6868c747
The list of a backup set is ordered by the LSN (Logical Sequence Number) and the backup, once restored, will be restored in LSN order.
The incremental backups have as parent the full backup, so you can verify that the incremental backup indeed belongs to the correct backup set (parent). Orphaned backups cannot be restored.&nbsp

Restore backups 

Do not try to unpack the files in the backup set files manually. Instead you should use s9s_backup’s restore option to do it in the correct way and in the correct order:
/usr/bin/s9s_backup --restore <backup id>  <my.cnf>  <restoredir>
To restore a backup, you must specify a backup set id. From the ClusterControl Web interface you can easily see which backup files are part of a backup set, or you can run the --list command.

The restoredir specifies where the restored backup files in the backup set will be applied and combined by xtrabackup. Please note that the restoredir is not, and must not be the same as the mysql datadir.  The increments are restored in LSN order to ensure you have a consistent restore point. This means that you as a user don’t have to worry about what order to apply the increments in. s9s_backup takes care of that, which is great when you are under pressure.

When restoring, s9s_backup compares the md5sum of the backup files when the backup was taken with the backup files that are currently restored. If there is a mismatch, s9s_backup will abort the restoration.

Once the restore is completed, the restoredir contains a complete restored backup that can be copied back to the mysql datadir.

Assuming you have a total cluster crash, and the cluster is unable to repair itself, then do on one node only:
  1. Make sure that the ClusterControl controller is stopped to prevent auto-recovery (you are now in charge of the recovery). On the controller do: service cmon stop
  2. Verify the other mysql servers are stopped.
  3. cd <restoredir>/<backup set id>
  4. make sure the is empty (use e.g rm -rf /var/lib/mysql) before running the innobackupex –copy-back command in the next step
  5. innobackupex  --copy-back  <restoredir>/<backup set id>
  6. chown mysql:mysql –R <mysql datadir>
  7. mysqld_safe --defaults-file=/path/to/my.cnf --wsrep-cluster-address='gcomm://' &  
  8. Start the ClusterControl controller (on the controller do: service cmon start),and it will orchestrate the synchronization of the other nodes.

Purge backups 

If you want to purge old backups, or failed backups, you can do:
/usr/bin/s9s_backup --purge <retention_period>   
Backups older than retention_period will be removed from the file system.
The retention, if not specified on the command line, will be fetched from the cmon database (in the cmon_configuration table), which you can set from the Web interface under Cluster Settings.

Installation of Xtrabackup and Cron Job

If Percona Xtrabackup is not already installed, then s9s_backup will automatically download and install Xtrabackup.
To install Xtrabackup and to also install a cron job do:
/usr/bin/s9s_backup --install cron
The 'cron' at the end means Install Cron job.
This will install Percona Xtrabackup on this server, but no cron job:
/usr/bin/s9s_backup --install 
In both cases above, the Percona Xtrabackup will be downloaded and installed.
The cron job is defined in the s9s_backup script, and you can easily modify it to suit your needs. The cron job is installed in /etc/cron.d/s9s_backup
The cron job will take a full backup every Sunday at 0300 and then incremental backups Mon-Sat at 0300.

If you want to modify the cron job table you can do it directly in the s9s_backup script or post installation by changing /etc/cron.d/s9s_backup

Finally, you can of course edit, change and make whatever modifications you like to the s9s_backup script if you want to customize it further.

Thursday, October 13, 2011

Multi-master and read slaves using Severalnines

This blog post shows how you can use the Severalnines Configurator for MySQL Replication to deploy a Multi-master replication setup, and install ClusterControl.
You can also watch videos showing what is described below or read an even more detailed tutorial.

When the deployment is finished you have a set of tools to manage and monitor replication, as well as to add new slaves, and to perform failover.
You can choose:
  • The number of slaves you wish to connect to the master->relay master.
  • whether you want to use MySQL 5.5.x or Percona 5.5.x
  • Semi-sync replication or not (semi sync really recommended)
  • cloud provider (on premise/EC2/Rackspace
Please note that your hosts must resolve to a valid ip. If 'hostname -i' resolves to nothing or to 127.0.0.1 it will not work.

We recommend also that you don't use bi-directional replication. It may sound like a good idea, but in practice it is challenging to get it working well.

When deployed you can easily perform tasks as:
  • Add new slaves
  • Failover
  • View replication health information
  • Stop/start replication links
  • Stage slaves
The Configurator for MySQL Replication is a wizard-like application and you have to enter details about the setup you want to have. When this is done you will get a tar.gz that contains the deployment and management scripts.

ClusterControl

The deployment process will also install ClusterControl which is a set of monitoring agents and functionality to manage the database installation. The agents are deployed on each server and collects host information and information from the local MySQL server a particular agent is monitoring (such as replication statistics and status information).

One server (dedicated) is denoted the ClusterControl server. It is holding a database, CMON DB, that contains data about the monitored hosts, and the reporting data from the local agents is stored in this database. On this server a Controller is running and faciliates failover etc.

Moreover, the ClusterControl server exposes a web interface that can be used to graphically see the health of the replication cluster (after the deployment is done, take a web browser and point it to http://clustercontrolserver/cmon :

From ClusterControl you can then upload a schema, and start loading in data into a database. As well as doing GRANTs etc. In the Enterprise version you an also define performance probes (to make sure you system is not degrading over time), tune queries, try out new queries before putting them in production. You also have a Query Monitor and explains at your hands.

Failover is automatic (if the master fails, the relay will become the new master). When the master comes up again, it will connect to the new master and sync up, become the relay, and the slaves will failover to the relay. We also make use of the replication features of MySQL 5.5 (such as relay recovery) and a bunch of other techniques.

You can also add slaves or remove slaves with the click of a button, as well as scheduling backups.

Installation
When you have finished the wizard you get a package that you should deploy on the ClusterControl server:
tar xvfz s9s-mysql-55.tar.gz
cd s9s-mysql-55/mysql/scripts/install
./deploy.sh

The 'deploy.sh' script will create data directories, install initd scripts, create mysql users, apply database GRANTs (those you defined in the Configurator), and install ClusterControl.

A while in 'deploy.sh' replication will be started, and it can look like this:

*******************************************************************************
* Starting replication *
*******************************************************************************
Master hosts
------------
10.30.30.31: up
10.30.30.32: up
Slave hosts
------------
10.30.30.33: up
10.30.30.34: up
starting replication between 10.30.30.31 --> 10.30.30.32 (change master=1, reset master=1)
replication [started] 10.30.30.31 --> 10.30.30.32
starting replication between 10.30.30.32 --> 10.30.30.33 (change master=1, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.33
starting replication between 10.30.30.32 --> 10.30.30.34 (change master=1, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.34
master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
10.30.30.31 10.30.30.32 ok binlog.000003:250 binlog.000003| 250| 250| 0
--- slaves follows ---
10.30.30.32 10.30.30.33 ok binlog.000001:107 binlog.000001| 107| 107| 0
10.30.30.32 10.30.30.34 ok binlog.000001:107 binlog.000001| 107| 107| 0
After this, ClusterControl will be installed. If you are using RPM based installation there can be rpm dependency problems (we install libmysqlclient.so.16), but it can conflict with what comes with your distribution. Please let us know if this happens, contact us on community@severalnines.com.

Starting/stopping Replication and basic examples


After having executed deploy.sh, you can view the status of the mysql servers:
cd s9s-mysql-55/mysql/scripts/
./status.sh -a
Master hosts
------------
10.30.30.31: up
10.30.30.32: up
Slave hosts
------------
10.30.30.33: up
10.30.30.34: up
To view the replication status:
cd s9s-mysql-55/mysql/scripts/
./repl-status.sh -a
master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
10.30.30.31 10.30.30.32 ok binlog.000003:250 binlog.000003| 250| 250| 0
--- slaves follows ---
10.30.30.32 10.30.30.33 ok binlog.000001:107 binlog.000001| 107| 107| 0
10.30.30.32 10.30.30.34 ok binlog.000001:107 binlog.000001| 107| 107| 0
To stop replication between 10.30.30.32 (relay master) to 10.0.30.33 you can do:
cd s9s-mysql-55/mysql/scripts/
./stop-repl.sh -m 10.30.30.32 -s 10.30.30.33
stopping replication between 10.30.30.32 --> 10.30.30.33
replication [stopped] 10.30.30.32 --> 10.30.30.33
To start replication again you can do:
cd s9s-mysql-55/mysql/scripts/
./start-repl.sh -m 10.30.30.32 -s 10.30.30.33
starting replication between 10.30.30.32 --> 10.30.30.33 (change master=0, reset master=0)
replication [started] 10.30.30.32 --> 10.30.30.33

Other options for 'start-repl.sh' is -r (for reset master, dangerous), -c (change master). Normally these options are not needed but you can use them in conjuction with -l (logpos) and -f (binlog file) if you want to start replication from a particular position.

And we can check the replication status:
ubuntu@ip-10-49-122-115:~/s9s-mysql-55/mysql/scripts$ ./repl-status.sh -a
serverid master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
1 10.48.207.130 10.48.139.24 ok binlog.000005:494 binlog.000005| 494| 494| 0
2 replication not activated - you must start replication on this link.
--- slaves follows ---
3 10.48.139.24 10.49.122.56 ok binlog.000005:485 binlog.000005| 485| 485| 0
4 10.48.139.24 10.49.110.183 ok binlog.000005:485 binlog.000005| 485| 485| 0

Good luck and don't hesitate to contact us at feedback@severalnines.com or community@severalnines.com if you have any problems or whatever it may be. You can also book a demo this if you want to know more.

Monday, April 11, 2011

Setting up Multi-master and read slaves using Severalnines




Read the detailed replication tutorial instead.


This tutorial shows how you can use the Severalnines Configurator for MySQL Replication to deploy a Multi-master replication setup, and install ClusterControl.

When the deployment is finished you have a set of tools to manage and monitor replication, as well as to add new slaves, and to perform failover.
You can choose:
  • if you wish also setup bi-directional replication between the masters
  • the number of slaves you wish to connect to the master
  • whether you want to use MySQL 5.5.10 or Percona latest stable
  • Handlersocket (percona) or Semi-sync replication (MySQL 5.5)
  • cloud provider (on premise/EC2/Rackspace)
When deployed you can easily perform tasks as:
  • Add new slaves
  • Failover
  • View replication health information
  • Stop/start replication links
  • Stage slaves
The Configurator for MySQL Replication is a wizard-like application and you have to enter details about the setup you want to have. When this is done you will get a tar.gz that contains the deployment and management scripts.

ClusterControl

The deployment process will also install ClusterControl which is a set of monitoring agents and functionality to manage the database installation. The agents are deployed on each server and collects host information and information from the local MySQL server a particular agent is monitoring (such as replication statistics and status information).

One server (either a standalone or one that will be part of the Replication cluster) is denoted the ClusterControl server. It is holding a database, CMON DB, that contains data about the monitored hosts, and the reporting data from the local agents is stored in this database.

Moreover, the ClusterControl server exposes a web interface that can be used to graphically see the health of the replication cluster (after the deployment is done, take a web browser and point it to http://clustercontrolserver/cmon :

We are in the process of adding a Query Analyzer and shaping up the web interface.

Installation
When you have finished the wizard you get a package that you should deploy on the ClusterControl server:
tar xvfz s9s-mysql-55.tar.gz
cd s9s-mysql-55/mysql/scripts/install
./deploy.sh

The 'deploy.sh' script will create data directories, install initd scripts, create mysql users, apply database GRANTs (those you defined in the Configurator), and install ClusterControl.

Starting Replication and basic examples


After having executed deploy.sh should do:
cd s9s-mysql-55/mysql/scripts/
./list-servers.sh -a
master : 10.48.207.130 [ up]
master : 10.48.139.24 [ up]
slave : 10.49.122.56 [ up]
slave : 10.49.110.183 [ up]
To verify the mysql servers are running and then:
cd s9s-mysql-55/mysql/scripts/
./start-repl.sh --master -i 1 -c

The command above will do a CHANGE MASTER (-c) and start replication from MySQL Server 1 to MySQL Server 2.

This step is not automated since you may want to start bi-directional replication between the masters. In that case you do:
./start-repl.sh --master -a -c

The slaves are then started with:
./start-repl.sh --slave -a -c
Which means, start all slaves (-a) and change master on those slaves.

Then you can run
./repl-status.sh  -a
serverid master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
1 10.48.207.130 10.48.139.24 ok binlog.000005:107 binlog.000005| 107| 107| 0
2 replication not activated - you must start replication on this link.
--- slaves follows ---
3 10.48.139.24 10.49.122.56 ok binlog.000005:107 binlog.000005| 107| 107| 0
4 10.48.139.24 10.49.110.183 ok binlog.000005:107 binlog.000005| 107| 107| 0

We got two masters, and the master with server-id=1 replicates to server-id=2, and the master on server-id=2 replicates to the slaves (3 and 4).

Now we are ready to provision data
./mysql.sh -h 10.48.207.130 -ppassword

mysql> create table t1 (id integer auto_increment primary key, data varchar(255))engine=innodb;
Query OK, 0 rows affected (0.02 sec)
mysql> insert into t1(data) values('hello');
And we can check the replication status:
ubuntu@ip-10-49-122-115:~/s9s-mysql-55/mysql/scripts$ ./repl-status.sh -a
serverid master_host -->slave_host status master_status slave_status [binlog|m_pos|exec_pos|lag]
1 10.48.207.130 10.48.139.24 ok binlog.000005:494 binlog.000005| 494| 494| 0
2 replication not activated - you must start replication on this link.
--- slaves follows ---
3 10.48.139.24 10.49.122.56 ok binlog.000005:485 binlog.000005| 485| 485| 0
4 10.48.139.24 10.49.110.183 ok binlog.000005:485 binlog.000005| 485| 485| 0


Good luck and don't hesitate to contact us at feedback@severalnines.com if you have any problems or whatever it may be. You can also book a demo this if you want to know more.