MySQL Cluster Configurator v3

Tweet Version 3.0 of the severalnines/Configurator has been released:

• Improved scripts (a lot of cleanup), prefixing the script output with the hostname:
  E.g,:
  Cluster: Cluster Start
Cluster: STARTING MANAGEMENT SERVERS
ps-ndb01: Starting management server (nodeid=1)
ps-ndb01: Copying ../config/config.ini to /etc/mysql
ps-ndb01: Started management server (nodeid=1, pid=28253)
...
  
• Reduced number of scripts
  start-cluster-initial.sh --> start-cluster.sh --initial
rolling-restart-initial.sh --> rollling-restart.sh --initial
start-ndbd-<host>-<id>-initial.sh --> start-ndbd-<host>-<id>.sh --initial
  
• Better error handling during rolling restarts
• Scripts are checking if Node Failure handling is ready before starting nodes
• start-ndbd-<host>-<id>.sh - new parameters [--initial] [--nowait] [--ignore]
--initial - initial node start (clear out local ndb_fs
  --nowait - script will exit as soon as data node is starting
  --ignore - ignore waiting for Node Failure handling to complete
  
• Supports MySQL Cluster 7.1.1

Here is an example of how to try out 7.1.1 using the Configurator

1. Generate a Configuration (make sure you select "MySQL Cluster 7.1.1")

• Currently 7.1.1 is only in source format so you need to have ncurses-devel/gcc/g++/make installed

2. Unpack the tarball you get and start the installation!
You are recommended to set up shared ssh keys between the "Frontend" (where you run the scripts) to the other nodes in the cluster in order to avoid typing passwords all the time.

tar xvfz mysqlcluster-71.tar.gz
cd mysqlcluster-71
cd cluster/scripts/install/
./shared-ssh-keys.sh ###WARNING - This script is experimental, let me know if it works!
./download-and-compile.sh
./install-cluster.sh
./bootstrap.sh
cd ..
./start-cluster.sh --initial

Voila!

MySQL Cluster 7.0.9b in Configurator and Sandbox

Tweet Configurator and Sandbox scripts are updated to use 7.0.9b.

You are recommended to upgrade!

Upgrade can be performend using these scripts:
Binary distribution - Upgrade script from 7.0.* -> 7.0.9b is here.
Source distribution - Upgrade script from 7.0.* -> 7.0.9b is here.

1. Copy the scripts to "install/"
2. chmod u+x upgrade-script.sh
   
3. ./upgrade-script.sh

After step 3) you need to run either:

• download-binary.sh
• download-and-compile.sh

and then

• rolling-restart.sh
  

Upgrade to 7.0.8 (with the Configurator)

Tweet MySQL Cluster 7.0.8 was released as a source distribution 30th of Sept 2009. You should upgrade if you can build from source or use the Configurator.

See below how to upgrade when using the Configurator.

7.0.8 contains a number of fixes in almost every area and you are recommended to upgrade if you experience problems with earlier version(s). Especially if you have encountered the "error 306" problem.

The Configurator v2.9 has been updated to use this version.

If you are already using the Configurator and build from source you can upgrade from MySQL Cluster 7.0.7 to 7.0.8 (and you are recommended to upgrade because of the following changes/fixes) in four steps. Here is how:

1. run the upgrade-707-to-708-src.sh script (put it in install/ and chmod u+x ./upgrade-707-to-708-src.sh):

[cluster01]# pwd
/root/mysqlcluster-70-master/cluster/scripts/install
[cluster01]# chmod u+x ./upgrade-707-to-708-src.sh
[cluster01]# ./upgrade-707-to-708-src.sh
Upgrading scripts
done - now run ./download-and-compile.sh

2. run the script download-and-compile.sh

[cluster01]# ./download-and-compile.sh
After some time it will finish compiling:

mysql-7.0.8-linux-x86_64/sql-bench/test-alter-table
mysql-7.0.8-linux-x86_64.tar.gz created
Removing temporary directory
Downloaded binary distribution to ../../repo

(the 'downloaded binary' should really read 'Copied binary').

3. When you get prompted with the following question answer 'y' (yes):

Do you want to install the binaries on the hosts now
(else run 'install-cluster.sh' later)? (y/n):y

Important! Do not run the the bootstrap.sh script!!
4. change directory to the scripts/ to directory and run the ./rolling-restart.sh script

[cluster01]# pwd
/root/mysqlcluster-70-master/cluster/scripts/
[cluster01]# ./rolling-restart.sh

This in an online procedure so it requires no downtime of the Cluster.
If you are using binary distributions then you have to wait, because they are not ready yet.
Good luck!

Ps - Configurator 2.9 (released 2nd of September 2009) contains some minor fixes:

• Adding of [mysqld] slot for administration purposes
• ./start-backup now backup the configuration files (config.ini, my.cnf and some internal files) as well.

Upgrade to 7.0.7 (with the Configurator)

Tweet MySQL Cluster 7.0.7 was released as a source distribution 1st of Sept 2009. You should upgrade if you can build from source or use the Configurator.

The Configurator v2.9 has been updated to use this version.

If you are already using the Configurator and build from source you can upgrade from MySQL Cluster 7.0.6 to 7.0.7 (and you are recommended to upgrade because of the following changes/fixes) in four steps. Here is how:

1. run the upgrade-706-to-707-src.sh script (put it in install/ and chmod u+x ./upgrade-706-to-707-src.sh):

[cluster01]# pwd
/root/mysqlcluster-70-master/cluster/scripts/install
[cluster01]# chmod u+x ./upgrade-706-to-707-src.sh
[cluster01]# ./upgrade-706-to-707-src.sh
Upgrading scripts
done - now run ./download-and-compile.sh

2. run the script download-and-compile.sh

[cluster01]# ./download-and-compile.sh
After some time it will finish compiling:

mysql-7.0.7-linux-x86_64/sql-bench/test-alter-table
mysql-7.0.7-linux-x86_64.tar.gz created
Removing temporary directory
Downloaded binary distribution to ../../repo

(the 'downloaded binary' should really read 'Copied binary').

3. When you get prompted with the following question answer 'y' (yes):

Do you want to install the binaries on the hosts now
(else run 'install-cluster.sh' later)? (y/n):y

Important! Do not run the the bootstrap.sh script!!
4. change directory to the scripts/ to directory and run the ./rolling-restart.sh script

[cluster01]# pwd
/root/mysqlcluster-70-master/cluster/scripts/
[cluster01]# ./rolling-restart.sh

This in an online procedure so it requires no downtime of the Cluster.
If you are using binary distributions then you have to wait, because they are not ready yet.
Good luck!

Ps - Configurator 2.9 (released 2nd of September 2009) contains some minor fixes:

• Adding of [mysqld] slot for administration purposes
• ./start-backup now backup the configuration files (config.ini, my.cnf and some internal files) as well.

How to upgrade from 6.3 to 7.0

Tweet In order to upgrade from 6.3 to 7.0 you must follow these rules:

1. IT IS ONLY POSSIBLE TO UPGRADE FROM 6.3 to at least 7.0.6!
2. I would recommend to upgrade from 6.3.x to 6.3.latest before doing the upgrade. If the upgrade does not work then and you have a good config.ini , you have probably hit a bug.
   
3. You cannot upgrade from 6.3 to the multi-threaded binary of 7.0.6 in one go.
   You have to upgrade from 'ndbd' (6.3) --> 'ndbd' (7.0.6)
   Then you can do 'ndbd' (7.0.6) --> 'ndbmtd' (7.0.6)

So don't try to upgrade to 7.0.5 - it will fail.
As usual when upgrading (version, configuration variables etc) you must do a rolling restart and restart:

1. ndb_mgmd (management servers)
2. ndbd (data nodes)
3. mysqld (mysql servers / direct api applications)

Distribute the 7.0.6 binaries
Copy the binaries to each host (i.e., replace existing 6.3 binaries) in the cluster.

Upgrading the Management Server(s)

In 6.3 you started the management server as (e.g):

ndb_mgmd -f /etc/mysql/config.ini

In 7.0.6 you have to start them slightly differently, but first you have to stop both (if you have two, but you really should) management servers (because of this bug):

killall ndb_mgmd

And.. I never use the management client to start/stop nodes, because I think it works sometimes which makes it difficult to file bugs on, and hard to rely on.

When you have killed both you can then do:

ndb_mgmd -f /etc/mysql/config.ini --configdir=/etc/mysql --reload

The management server in 7.0.6 writes a binary config file in --configdir, which is a configuration cache. --reload will reload the config.ini and update that cache.

Upgrading the Data Nodes

Restart the data nodes on each machine one at a time:

killall ndbd
ndbd --ndb-nodeid=X --ndb-connectstring="managementhostA;managementhostB"
ndb_waiter --ndb-connectstring="managementhostA;managementhostB"
# when ndb_waiter exits, you can restart the next data node.

Upgrading the MySQL Servers/Direct APIs

Stop and start the mysql servers:

killall mysqld  #(or mysqladmin shutdown , both sends signal 15)
mysqld_safe --defaults-file=/etc/config/my.cnf

Upgrading to multithreaded 7.0.6 data nodes

When the Cluster has been restarted with 7.0.6 you can upgrade to multithreaded data nodes.
The binary for the multithreaded data node is called 'ndbmtd'.

You need to set in config.ini (on both management servers):

[ndbd default]
..
MaxNoOfExecutionThreads=#number of cores you have (up to 8)
..

Then stop both management servers

killall ndb_mgmd

When you have killed both you can then start both by doing:

ndb_mgmd -f /etc/mysql/config.ini --configdir=/etc/mysql --reload

and then restart the data nodes one at a time:

killall ndbd
ndbmtd --ndb-nodeid=X --ndb-connectstring="managementhostA;managementhostB"
ndb_waiter --ndb-connectstring="managementhostA;managementhostB"
# when ndb_waiter exits, you can restart the next data node.

Using Severalnines Configurator

If you have a Severalnines installation of MySQL Cluster 6.3, then you can upgrade in the following way:

1. Make sure you have have the files:
mysqlcluster-xy/cluster/scripts/install/.s9s/hostnames
mysqlcluster-xy/cluster/scripts/install/.s9s/ndb_hostnames
mysqlcluster-xy/cluster/scripts/install/.s9s/mysql_hostnames
mysqlcluster-xy/cluster/scripts/install/.s9s/mgm_hostnames
Otherwise you have a too old version of the Configurator package and you need to generate a new one.
2. Generate a new Config using 7.0.6 with the same nodes, data memory, data dirs etc.
   Select "no MT" when asked for "Number of cores:"
   
   
   
3. Install the package
   Let's assume you have 6.3 in:
   

   /root/mysqlcluster-63/

   Install mysqlcluster-70.tar.gz
   

   cd /root/
   tar xvfz mysqlcluster-70.tar.gz
   cd mysqlcluster-70/cluster/scripts/install

   Copy the .s9s catalog from 6.3 so you get the hostnames
   

   cp  -r /root/mysqlcluster-63/cluster/scripts/install/.s9s  .

4. Download the binary or build the source:
   

   ./download-binary.sh
   

   or
   

   ./download-and-compile.sh
   

5. Install and perform a rolling restart
   

   ./install-cluster.sh
   cd ..
   ./rolling-restart.sh
   

6. Change from ndbd --> ndbmtd
   

   vi ../config/config.ini
   

   Locate MaxNoOfExecutionThreads and, un-comment it if needed and set it to as many cores as you have:
   

   [ndbd default]
   ...
   MaxNoOfExecutionThreads=8
   ...

   Then change in the scripts from ndbd -> ndbmtd
   

   sed -i 's#libexec/ndbd#libexec/ndbmtd#g' *.sh
   

   Important! Don' leave out 'libexec' above!
   And do a rolling restart:
   

   ./rolling-restart.sh

7. Luckily it is much easier to upgrade from e.g 7.0.6 to 7.0.x and I will show when 7.0.7 is out how to do that in two lines.
   

Online configuration update and a new script

Tweet If you are using the scripts from the Configurator (and yes, the broken link is now fixed so it is actually possible to get the config package..), and want to tweak your configuration on a started Cluster then you can do like written below.

There is also a new script in Configurator 2.2 (released 28th April 2009, PST1153AM , GMT07:53) that lets you check the values of different config parameters. Its use is described below.

Pretend we want to extend the DataMemory from 1024M to 1536M:

1. cd mysqlcluster-70/cluster/scripts
2. vi ../config/config.ini
   This is the master config.ini file and will be copied out to the management server(s). Don't change /etc/mysql/config.ini, as that will have no effect.
   
3. Set DataMemory=1536M then save and exit
   
4. sh rolling-restart.sh
5. When it has finished, verify that the Cluster has the new settings by running query-param.sh:
   sh query-param.sh DataMemory
Parameter=DataMemory
Data Node 3 : 1610612736
Data Node 4 : 1610612736
Please note that the output is in bytes.

The rolling-restart.sh script implements this procedure described in the reference manual.
Many other parameters can be changed using the above steps (e.g, MaxNoOfConcurrentOperations etc). Be careful though that you have memory available (the script does not check for that yet).

Parameters that change the file system one way or another (NoOfFragmentLogFiles, FragmentLogFileSize, Datadir, BackupDataDir, FileSystem*, InitFragmentLogfiles) must be changed with an initial rolling restart (all parameters are listed here in the ref manual):

Pretend we want to change the size of the Redolog (NoOfFragmentLogfiles).
From the beginning we have:
./query-param.sh NoOfFragmentLogFiles
Parameter=NoOfFragmentLogFiles
Data Node 3 : 6
Data Node 4 : 6

Then we modify our "template" config.ini and go ahead:


1. vi ../config/config.ini
This is the master config.ini file and will be copied out to the management server(s)
2. Set NoOfFragmentLogFiles=7 then save and exit
3. sh rolling-restart-initial.sh
4. When it has finished, verify that the Cluster has the new settings by running query-param.sh:
   ./query-param.sh NoOfFragmentLogFiles
Parameter=NoOfFragmentLogFiles
Data Node 3 : 7
Data Node 4 : 7


Have fun upgrading but remember - don't try to start with a smaller configuration on an already existing configuration (i.e., it is safe to increase parameters, but be very careful if you decrease).

Easiest way to try out new MySQL Cluster 7.0.5

Tweet Here is how to install and start MySQL Cluster 7.0.5 in eight steps:

1. Go to the configurator and setup the configuration you wish to have. Make sure you select "MySQL Cluster 7.0.x" in the first drop down.
   
2. Copy the package you either download or receive by email to one of the computers that will run one of the management servers.
3. tar xvfz mysqlcluster-70.tgz
4. cd mysqlcluster-70/scripts/install
   
5. Download MySQL Cluster 7.0 from MySQL website and put it in mysqlcluster-70/scripts/install
6. sh dist-tgz.sh
   
7. sh bootstrap.sh
   
8. cd .. && sh start-cluster-initial.sh

Okay, step 1) requires that you setup shared ssh keys and a bit of typing and clicking, but gives you a production class configuration, and scripts to manage the cluster from a single point. Including backups and restores of MySQL Cluster!

I have mostly tested the scripts on Linux - please let me know about issues on other platforms!

Configurator - new version 2.1

Tweet The Configurator has undergone some serious fixing as 2.0 had some issues.
Here is a brief list of the enhancements:

• error handing and progress reports - I have rewritten almost all scripts with better error handling and progress reports. E.g, if a node fails to start during a rolling restart, then the rolling restart script is aborted and you get a suggestion how to recover. If you run start-cluster-initial.sh on an already started cluster, it will refuse to run. There are many more changes like this to check the status of the nodes (both data nodes and sql nodes) during startup
  
• download link to package - When you have entered the email address, then you will also get a link where to download the package, in case you experience problems with receiving the email (I have raised an issue to the hosting company about this)
• restructure - I have also moved the install scripts to scripts/install and the monit scripts to scipts/extra. The install_initd_linux.sh uses chkconfig and it does not work on all linux distros. Going to fix that.
• bugs - fixed a number of small bugs and annoyances.