Tuesday, October 07, 2008

Upgrade from MySQL Cluster 6.2 to 6.3

Is it possible to upgrade from MySQL Cluster 6.2 to MySQL Cluster 6.3? Of course!
Can it be done online? Sure!

Here is how!

I started off as with the cluster in the picture below:



All you need to do is as follows:
  • Copy out the new binaries (MySQL Cluster 6.3) on to all hosts
  • Install the new version (please note that you have not restarted anything yet - we are about too). The install phase is just to copy out the binaries of the new version to all hosts and make sure that you are referencing the new version of the Cluster binaries in scripts, PATHs etc.
After that you need to:
  1. restart both (all) management servers at the same time!
    (TRICK! If the management servers are using a connect-string that only references itself, then you can restart them one at a time. Otherwise they risk picking up the other ones configuration! So if you have two hosts where the management servers run, A and B, then the connect strings should be:
    • For the management server on host A: ndb_mgmd -c "A"
    • For the management server on host B: ndb_mgmd -c "B"
    • And NOT ndb_mgmd -c "A;B" , then you risk that they pick up the wrong configuration data (unless you stop them both at the same time and then restart them) and it would be really bad!
  2. ndb_mgm -e "show" --> verify that all of them have version 6.3.x. The data nodes and mysql servers will still show they are version 6.2.x
  3. restart the data nodes (one at a time) until you have restarted all of them
  4. verify with ndb_mgm -e "show" that the data nodes all have the correct version (6.3.x)
  5. restart the mysql servers (one at a time) until you have restarted all of them.
  6. verify with ndb_mgm -e "show" that the data nodes all have the correct version (6.3.x)
While I am restarting the cluster (i am using the configuration tool and monit and cmon) you see the following in cmon:



And when the rolling restart was complete I have:

Actually, for a short period of time, the web browser will show a blank page for the duration when the management servers are stopped and restarting!

If you had used the configuration tool to generate scripts and a configuration for MySQL Cluster 6.2, then you can generate a new configuration and scripts for MySQL Cluster 6.3 (the two configurations must be identical in terms of number of nodes (data memory, data nodes, management servers, mysql servers) and then use those scripts:
  1. sh ./download-and-compile.sh (download and compiles latest MySQL Cluster 6.3 - works probably only for Linux)
  2. sh ./dist.sh
or if you can find the correct version at dev.mysql.com/downloads/cluster then you can do
  1. sh ./dist-tgz.sh <name of binary dist(rpms does not work>
  2. sh ./rolling-restart.sh (sh ./rolling-restart-monit.sh if you are using monit to guard your processes).
I used the approach above together with cmon.

16 comments:

Phil Bayfield said...

Another very useful post on your blog. I've just performed an upgrade on our custer from 6.2.15 to 6.3.18 with the help of your configurator tool and your blog.

I've tried to setup cmon as well as it looks quite handy but I'm running into a few problems with it.

I've edited the startup script so all the values are correct for my system but notice the following in the messages:

Oct 13 17:11:52 web1 cmon: Starting cmon version 1.2.2 with the following parameters: --mysqldatabase=cmon --mysqluser=root --mysqlpasswd=********** --mysqlhost=localhost --mysqlport=3306 --mysqlsocket=/var/run/mysqld/mysqld.sock --ndb-connectstring=localhost --savetime-clusterlog=24 (hours) --savetime-statistics=24 (hours)
Oct 13 17:11:52 web1 cmon: If that doesn't look correct, kill cmon and restart with -? for help on the parameters, or change the params in /etc/init.d/cmon
Oct 13 17:11:52 web1 cmon: connected to mysqld
Oct 13 17:11:52 web1 cmon: Table 'backup' did not exist. Schema incomplete. Going to recreate missing tables.
Oct 13 17:11:52 web1 cmon: Recreated missing tables

No tables are created and at this point cmon seams to exit.

Johan Andersson said...

hello phil,
sorry to hear about the problems.
Can you do:
tail -f /var/log/messages
and then start again.
Can you also do
ps -ef |grep cmon

and let me know what you get for the above.

Please email me at johan at severalnines.com or go to launchpad.net/cmon and report the bug there.

Unknown said...

I try install last cmi=on release? and see this error
cmon-1.2.2.tar.gz 100% 4658KB 4.6MB/s 00:00
configure: error: Couldn't find NdbApi.h!
make: *** Не заданы цели и не найден make-файл. Останов.
make: *** Нет правила для сборки цели `install'. Останов.

Piotr Kirklewski said...

I have exactly the same problem.
Installed Cluster 6.3, and I'm trying to install cmon on first SQL node which is at the same time a management node:

[root@SQL1 cmon-1.2.2]# sh autogen.sh
[root@SQL1 cmon-1.2.2]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for g++... no
checking for c++... no
checking for gpp... no
checking for aCC... no
checking for CC... no
checking for cxx... no
checking for cc++... no
checking for cl... no
checking for FCC... no
checking for KCC... no
checking for RCC... no
checking for xlC_r... no
checking for xlC... no
checking whether we are using the GNU C++ compiler... no
checking whether g++ accepts -g... no
checking dependency style of g++... none
checking whether make sets $(MAKE)... (cached) yes
checking how to run the C preprocessor... gcc -E
checking for egrep... grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for atoi... yes
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for mysql_config executable... checking for mysql_config... mysql_config
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ANSI C... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking whether we are using the GNU C++ compiler... (cached) no
checking whether g++ accepts -g... (cached) no
checking dependency style of g++... (cached) none
checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

Johan Andersson said...

Which OS are you using?
Can you do a uname -a and email me the details.

Also email the output from mysql_config.

If you do:
make distclean
sh ./autogen.sh
./configure
make


what happens then?
Sorry for the inconvenience.

Piotr Kirklewski said...

Hi
CentOS 5

[root@SQL1 ~]# uname -a
Linux SQL1.nagileon.local 2.6.24.6 #1 SMP Thu Oct 9 07:28:09 EDT 2008 i686 i686 i386 GNU/Linux


[root@SQL1 cmon-1.2.2]# make distclean
make: *** No rule to make target `distclean'. Stop.

[root@SQL1 cmon-1.2.2]# sh ./autogen.sh

[root@SQL1 cmon-1.2.2]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c

...

executable... checking for mysql_config... no
configure: error: MySQL not found

So something has changed.
But still no joy.
Regards
Peter

Piotr Kirklewski said...

After I did :export PATH=/usr/local/mysql/bin:$PATH

and then ./configure

I'm getting the same error again:

checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

Johan Andersson said...

Hello,

I think I can see the error now:
g++ is not installed.
Can you verify this?
Just running the command:
g++
from the shell should say if it exists or nor.

If it is not there you need to install it, e.g,
sudo apt-get install g++

However, the error message is bad..
Can you let me know what happens!

johan@severalnines.com

BR
johan

Piotr Kirklewski said...

Hi Johan

You were absolutely right:)

[root@SQL1 ~]# g++
-bash: g++: command not found

yum install gcc-c++

./configure

sucess :)

Regards

Peter

Piotr Kirklewski said...

Hi Johan
I'm still having problems:

[root@SQL1 www]# /etc/init.d/cmon start
/usr/local/bin//cmon: error while loading shared libraries: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory
start on service executed successfully

I did:

LD_LIBRARY_PATH=/mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/

The output of $LD_LIBRARY_PATH is :
-bash: /mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/: is a directory

So I don't get what's wrong ?

Again - my system is CentOs 5.

Any ideas?

Regards

Peter

Piotr Kirklewski said...

I know:

ldconfig /usr/local/mysql/lib

:)

Ollie said...
This comment has been removed by the author.
Ollie said...

Firstly, looks great - I realy want to get it running!
CentoOS release 5.2 (final)
MySQL 5.1.23-ndb-6.2.15

Installed G++yum install gcc gcc-c++ autoconf automake

./configure CPPFLAGS="-I/usr/local/mysql/include/mysql"

Same error as another commenter:
checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

This is the header file dir that comes with my version of mysql cluster

tree /usr/local/mysql/include/mysql
/usr/local/mysql/include/mysql

|-- decimal.h
|-- errmsg.h
|-- keycache.h
|-- m_ctype.h
|-- m_string.h
|-- my_alloc.h
|-- my_attribute.h
|-- my_config.h
|-- my_dbug.h
|-- my_dir.h
|-- my_getopt.h
|-- my_global.h
|-- my_list.h
|-- my_net.h
|-- my_no_pthread.h
|-- my_pthread.h
|-- my_sys.h
|-- my_xml.h
|-- mysql.h
|-- mysql_com.h
|-- mysql_embed.h
|-- mysql_time.h
|-- mysql_version.h
|-- mysqld_ername.h
|-- mysqld_error.h
|-- plugin.h
|-- sql_common.h
|-- sql_state.h
|-- sslopt-case.h
|-- sslopt-longopts.h
|-- sslopt-vars.h
|-- storage
| `-- ndb
| |-- mgmapi
| | |-- mgmapi.h
| | |-- mgmapi_config_parameters.h
| | |-- mgmapi_config_parameters_debug.h
| | |-- mgmapi_debug.h
| | |-- mgmapi_error.h
| | |-- ndb_logevent.h
| | `-- ndbd_exit_codes.h
| |-- ndb_constants.h
| |-- ndb_init.h
| |-- ndb_types.h
| |-- ndb_version.h
| `-- ndbapi
| |-- Ndb.hpp
| |-- NdbApi.hpp
| |-- NdbBlob.hpp
| |-- NdbDictionary.hpp
| |-- NdbError.hpp
| |-- NdbEventOperation.hpp
| |-- NdbIndexOperation.hpp
| |-- NdbIndexScanOperation.hpp
| |-- NdbIndexStat.hpp
| |-- NdbInterpretedCode.hpp
| |-- NdbOperation.hpp
| |-- NdbPool.hpp
| |-- NdbRecAttr.hpp
| |-- NdbReceiver.hpp
| |-- NdbScanFilter.hpp
| |-- NdbScanOperation.hpp
| |-- NdbTransaction.hpp
| |-- ndb_cluster_connection.hpp
| |-- ndb_opt_defaults.h
| |-- ndbapi_limits.h
| `-- ndberror.h
`-- typelib.h

Johan Andersson said...

Ollie,

what do you get by running:
- g++ --version

- do you have mysql_config on the path. i.e, what happens if you run mysql_config from the command line in the terminal. If it says "not found" --> set the PATH (e.g export PATH=/usr/local/mysql/bin:$PATH )

- you should run configure as:
./configure
(setting the CPPFLAGS is not needed as is wrongly used).


Let me know one way or another what you get from the above.

Piotr Kirklewski said...

Hi there
I managed to install Cmon on Centos 5.0 and here is what I've done:

yum install autoconf && yum install automake

yum install gcc-c++

yum install httpd

yum install php-mysql

yum install php-gd

yum install php-cli

yum install php

cd /usr/src

wget http://www.severalnines.com/downloads/cmon/cmon-1.2.2.tar.gz

tar -xvf cmon-1.2.2.tar.gz

cd cmon-1.2.2

sh ./autogen.sh

export PATH=/usr/local/mysql/bin:$PATH

./configure --with-wwwroot=/var/www


make

make install

make install-extra

ldconfig /usr/local/mysql/lib

chkconfig --add cmon


chmod 775 /etc/init.d/cmon

chkconfig --add httpd

vim /etc/httpd/conf/httpd.conf # and change DocumentRoot to: DocumentRoot "yourdirectory"
as well as
Directory "yourdirectory"

DirectoryIndex index.php index.html index.html.var


/etc/init.d/httpd restart

ldconfig /mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/

/etc/init.d/cmon start


vim /var/www/cmon/cmon_db_conf.inc.php

change $mysql_hostname="localhost" to $mysql_hostname = ":/tmp/mysql.sock"


php -v


go to http://yourip/cmon

Unknown said...

Hello ,
I've been trying to compile CMON in a Centos 5.3 with MySQL-Cluster-gpl version 7 with no luck , i think i've all the libraries and other requirements, It fails to compile in the last step , so here's the error:

make[2]: Entering directory `/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src'
g++ -g -O2 -I/usr/include/mysql -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -DUNIV_LINUX -I/usr/include/mysql/storage/ndb -I/usr/include/mysql/storage/ndb/ndbapi -I/usr/include/mysql/storage/ndb/mgmapi -I/usr/include/mysql/ndb -I/usr/include/mysql/ndb/ndbapi -I/usr/include/mysql/ndb/mgmapi -I/usr/include/mysql -rdynamic -L/usr/lib64/mysql -lmysqlclient_r -lz -lrt -lpthread -lcrypt -lnsl -lm -lpthread -lmygcc -lndbclient -lmystrings -lmysys -o cmon connect.o logevent.o maint.o monitor.o cmon.o schema.o
connect.o: In function `connectNdbMgm(cluster_conn_t*)':
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:151: undefined reference to `ndb_mgm_is_connected'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:159: undefined reference to `ndb_mgm_destroy_handle'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:163: undefined reference to `ndb_mgm_create_handle'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:164: undefined reference to `ndb_mgm_set_connectstring'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:171: undefined reference to `ndb_mgm_connect'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:184: undefined reference to `ndb_mgm_get_connected_port'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:184: undefined reference to `ndb_mgm_get_connected_host'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:177: undefined reference to `ndb_mgm_get_latest_error_msg'
/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:177: undefined reference to `ndb_mgm_get_latest_error'
connect.o: In function `connectMySQL2(st_mysql*, char const*)':

and so on...

any ideas? there's a binary precompiled out there?

Thanks a lot!
Raul