Monday, June 29, 2009

How to upgrade from 6.3 to 7.0

In order to upgrade from 6.3 to 7.0 you must follow these rules:
  1. IT IS ONLY POSSIBLE TO UPGRADE FROM 6.3 to at least 7.0.6!
  2. I would recommend to upgrade from 6.3.x to 6.3.latest before doing the upgrade. If the upgrade does not work then and you have a good config.ini , you have probably hit a bug.
  3. You cannot upgrade from 6.3 to the multi-threaded binary of 7.0.6 in one go.
    You have to upgrade from 'ndbd' (6.3) --> 'ndbd' (7.0.6)
    Then you can do 'ndbd' (7.0.6) --> 'ndbmtd' (7.0.6)
So don't try to upgrade to 7.0.5 - it will fail.
As usual when upgrading (version, configuration variables etc) you must do a rolling restart and restart:
  1. ndb_mgmd (management servers)
  2. ndbd (data nodes)
  3. mysqld (mysql servers / direct api applications)
Distribute the 7.0.6 binaries
Copy the binaries to each host (i.e., replace existing 6.3 binaries) in the cluster.

Upgrading the Management Server(s)

In 6.3 you started the management server as (e.g):
ndb_mgmd -f /etc/mysql/config.ini
In 7.0.6 you have to start them slightly differently, but first you have to stop both (if you have two, but you really should) management servers (because of this bug):
killall ndb_mgmd
And.. I never use the management client to start/stop nodes, because I think it works sometimes which makes it difficult to file bugs on, and hard to rely on.

When you have killed both you can then do:
ndb_mgmd -f /etc/mysql/config.ini --configdir=/etc/mysql --reload
The management server in 7.0.6 writes a binary config file in --configdir, which is a configuration cache. --reload will reload the config.ini and update that cache.

Upgrading the Data Nodes

Restart the data nodes on each machine one at a time:
killall ndbd
ndbd --ndb-nodeid=X --ndb-connectstring="managementhostA;managementhostB"
ndb_waiter --ndb-connectstring
# when ndb_waiter exits, you can restart the next data node.

Upgrading the MySQL Servers/Direct APIs

Stop and start the mysql servers:
killall mysqld  #(or mysqladmin shutdown , both sends signal 15)
mysqld_safe --defaults-file=/etc/config/my.cnf
Upgrading to multithreaded 7.0.6 data nodes

When the Cluster has been restarted with 7.0.6 you can upgrade to multithreaded data nodes.
The binary for the multithreaded data node is called 'ndbmtd'.

You need to set in config.ini (on both management servers):
[ndbd default]
MaxNoOfExecutionThreads=#number of cores you have (up to 8)
Then stop both management servers
killall ndb_mgmd
When you have killed both you can then start both by doing:
ndb_mgmd -f /etc/mysql/config.ini --configdir=/etc/mysql --reload
and then restart the data nodes one at a time:
killall ndbd
ndbmtd --ndb-nodeid=X --ndb-connectstring="managementhostA;managementhostB"
ndb_waiter --ndb-connectstring
# when ndb_waiter exits, you can restart the next data node.
Using Severalnines Configurator

If you have a Severalnines installation of MySQL Cluster 6.3, then you can upgrade in the following way:
  1. Make sure you have have the files:
    Otherwise you have a too old version of the Configurator package and you need to generate a new one.
  2. Generate a new Config using 7.0.6 with the same nodes, data memory, data dirs etc.
    Select "no MT" when asked for "Number of cores:"

  3. Install the package
    Let's assume you have 6.3 in:
    Install mysqlcluster-70.tar.gz
    cd /root/
    tar xvfz mysqlcluster-70.tar.gz
    cd mysqlcluster-70/cluster/scripts/install
    Copy the .s9s catalog from 6.3 so you get the hostnames
    cp  -r /root/mysqlcluster-63/cluster/scripts/install/.s9s  .
  4. Download the binary or build the source:
  5. Install and perform a rolling restart
    cd ..
  6. Change from ndbd --> ndbmtd
    vi ../config/config.ini
    Locate MaxNoOfExecutionThreads and, un-comment it if needed and set it to as many cores as you have:
    [ndbd default]
    Then change in the scripts from ndbd -> ndbmtd
    sed -i 's#libexec/ndbd#libexec/ndbmtd#g' *.sh
    Important! Don' leave out 'libexec' above!
    And do a rolling restart:
  7. Luckily it is much easier to upgrade from e.g 7.0.6 to 7.0.x and I will show when 7.0.7 is out how to do that in two lines.


Cyril Scetbon said...

I would be cool if ndbd could launch automatically ndbmtd if MaxNoOfExecutionThreads is set in ndb_mgmd's configuration file . What are you thinking about it ?

Cyril Scetbon said...

I tried it and everything worked. However I can no longer create/alter a table without getting this error :

mysql> create table test(id int) engine=ndb;
ERROR 1005 (HY000): Can't create table 'mydb.test' (errno: 140)
mysql> show warnings;
| Level | Code | Message |
| Error | 1296 | Got error 755 'Invalid tablespace' from NDB |
| Error | 1005 | Can't create table 'mydb.test' (errno: 140) |
2 rows in set (0,00 sec)

And I can no longer backup data with ndb_mgm :

d01:~# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> start backup;
Connected to Management Server at: XX.XX.XX.XX:1186
Waiting for completed, this may take several minutes
Node 3: Backup request from 2 failed to start. Error: 1304
Backup failed
* 3001: Could not start backup
* Sequence failure: Permanent error: Internal error
ndb_mgm> exit

Did you met the same errors ? (7.0.6)

Geert JM Vanderkelen said...


I just verified your bug report, and indeed, I get same errors. Also using 6.3.25:

I am still a bit on it, as I can't believe it..