Tuesday, October 07, 2008

Upgrade from MySQL Cluster 6.2 to 6.3

Is it possible to upgrade from MySQL Cluster 6.2 to MySQL Cluster 6.3? Of course!
Can it be done online? Sure!

Here is how!

I started off as with the cluster in the picture below:



All you need to do is as follows:
  • Copy out the new binaries (MySQL Cluster 6.3) on to all hosts
  • Install the new version (please note that you have not restarted anything yet - we are about too). The install phase is just to copy out the binaries of the new version to all hosts and make sure that you are referencing the new version of the Cluster binaries in scripts, PATHs etc.
After that you need to:
  1. restart both (all) management servers at the same time!
    (TRICK! If the management servers are using a connect-string that only references itself, then you can restart them one at a time. Otherwise they risk picking up the other ones configuration! So if you have two hosts where the management servers run, A and B, then the connect strings should be:
    • For the management server on host A: ndb_mgmd -c "A"
    • For the management server on host B: ndb_mgmd -c "B"
    • And NOT ndb_mgmd -c "A;B" , then you risk that they pick up the wrong configuration data (unless you stop them both at the same time and then restart them) and it would be really bad!
  2. ndb_mgm -e "show" --> verify that all of them have version 6.3.x. The data nodes and mysql servers will still show they are version 6.2.x
  3. restart the data nodes (one at a time) until you have restarted all of them
  4. verify with ndb_mgm -e "show" that the data nodes all have the correct version (6.3.x)
  5. restart the mysql servers (one at a time) until you have restarted all of them.
  6. verify with ndb_mgm -e "show" that the data nodes all have the correct version (6.3.x)
While I am restarting the cluster (i am using the configuration tool and monit and cmon) you see the following in cmon:



And when the rolling restart was complete I have:

Actually, for a short period of time, the web browser will show a blank page for the duration when the management servers are stopped and restarting!

If you had used the configuration tool to generate scripts and a configuration for MySQL Cluster 6.2, then you can generate a new configuration and scripts for MySQL Cluster 6.3 (the two configurations must be identical in terms of number of nodes (data memory, data nodes, management servers, mysql servers) and then use those scripts:
  1. sh ./download-and-compile.sh (download and compiles latest MySQL Cluster 6.3 - works probably only for Linux)
  2. sh ./dist.sh
or if you can find the correct version at dev.mysql.com/downloads/cluster then you can do
  1. sh ./dist-tgz.sh <name of binary dist(rpms does not work>
  2. sh ./rolling-restart.sh (sh ./rolling-restart-monit.sh if you are using monit to guard your processes).
I used the approach above together with cmon.

16 comments:

  1. Another very useful post on your blog. I've just performed an upgrade on our custer from 6.2.15 to 6.3.18 with the help of your configurator tool and your blog.

    I've tried to setup cmon as well as it looks quite handy but I'm running into a few problems with it.

    I've edited the startup script so all the values are correct for my system but notice the following in the messages:

    Oct 13 17:11:52 web1 cmon: Starting cmon version 1.2.2 with the following parameters: --mysqldatabase=cmon --mysqluser=root --mysqlpasswd=********** --mysqlhost=localhost --mysqlport=3306 --mysqlsocket=/var/run/mysqld/mysqld.sock --ndb-connectstring=localhost --savetime-clusterlog=24 (hours) --savetime-statistics=24 (hours)
    Oct 13 17:11:52 web1 cmon: If that doesn't look correct, kill cmon and restart with -? for help on the parameters, or change the params in /etc/init.d/cmon
    Oct 13 17:11:52 web1 cmon: connected to mysqld
    Oct 13 17:11:52 web1 cmon: Table 'backup' did not exist. Schema incomplete. Going to recreate missing tables.
    Oct 13 17:11:52 web1 cmon: Recreated missing tables

    No tables are created and at this point cmon seams to exit.

    ReplyDelete
  2. hello phil,
    sorry to hear about the problems.
    Can you do:
    tail -f /var/log/messages
    and then start again.
    Can you also do
    ps -ef |grep cmon

    and let me know what you get for the above.

    Please email me at johan at severalnines.com or go to launchpad.net/cmon and report the bug there.

    ReplyDelete
  3. I try install last cmi=on release? and see this error
    cmon-1.2.2.tar.gz 100% 4658KB 4.6MB/s 00:00
    configure: error: Couldn't find NdbApi.h!
    make: *** Не заданы цели и не найден make-файл. Останов.
    make: *** Нет правила для сборки цели `install'. Останов.

    ReplyDelete
  4. I have exactly the same problem.
    Installed Cluster 6.3, and I'm trying to install cmon on first SQL node which is at the same time a management node:

    [root@SQL1 cmon-1.2.2]# sh autogen.sh
    [root@SQL1 cmon-1.2.2]# ./configure
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for gawk... gawk
    checking whether make sets $(MAKE)... yes
    checking for gcc... gcc
    checking for C compiler default output file name... a.out
    checking whether the C compiler works... yes
    checking whether we are cross compiling... no
    checking for suffix of executables...
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ANSI C... none needed
    checking for style of include used by make... GNU
    checking dependency style of gcc... gcc3
    checking for g++... no
    checking for c++... no
    checking for gpp... no
    checking for aCC... no
    checking for CC... no
    checking for cxx... no
    checking for cc++... no
    checking for cl... no
    checking for FCC... no
    checking for KCC... no
    checking for RCC... no
    checking for xlC_r... no
    checking for xlC... no
    checking whether we are using the GNU C++ compiler... no
    checking whether g++ accepts -g... no
    checking dependency style of g++... none
    checking whether make sets $(MAKE)... (cached) yes
    checking how to run the C preprocessor... gcc -E
    checking for egrep... grep -E
    checking for ANSI C header files... yes
    checking for sys/types.h... yes
    checking for sys/stat.h... yes
    checking for stdlib.h... yes
    checking for string.h... yes
    checking for memory.h... yes
    checking for strings.h... yes
    checking for inttypes.h... yes
    checking for stdint.h... yes
    checking for unistd.h... yes
    checking for stdlib.h... (cached) yes
    checking for string.h... (cached) yes
    checking for unistd.h... (cached) yes
    checking for atoi... yes
    checking build system type... i686-pc-linux-gnu
    checking host system type... i686-pc-linux-gnu
    checking for mysql_config executable... checking for mysql_config... mysql_config
    checking for gcc... (cached) gcc
    checking whether we are using the GNU C compiler... (cached) yes
    checking whether gcc accepts -g... (cached) yes
    checking for gcc option to accept ANSI C... (cached) none needed
    checking dependency style of gcc... (cached) gcc3
    checking whether we are using the GNU C++ compiler... (cached) no
    checking whether g++ accepts -g... (cached) no
    checking dependency style of g++... (cached) none
    checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

    ReplyDelete
  5. Which OS are you using?
    Can you do a uname -a and email me the details.

    Also email the output from mysql_config.

    If you do:
    make distclean
    sh ./autogen.sh
    ./configure
    make


    what happens then?
    Sorry for the inconvenience.

    ReplyDelete
  6. Hi
    CentOS 5

    [root@SQL1 ~]# uname -a
    Linux SQL1.nagileon.local 2.6.24.6 #1 SMP Thu Oct 9 07:28:09 EDT 2008 i686 i686 i386 GNU/Linux


    [root@SQL1 cmon-1.2.2]# make distclean
    make: *** No rule to make target `distclean'. Stop.

    [root@SQL1 cmon-1.2.2]# sh ./autogen.sh

    [root@SQL1 cmon-1.2.2]# ./configure
    checking for a BSD-compatible install... /usr/bin/install -c

    ...

    executable... checking for mysql_config... no
    configure: error: MySQL not found

    So something has changed.
    But still no joy.
    Regards
    Peter

    ReplyDelete
  7. After I did :export PATH=/usr/local/mysql/bin:$PATH

    and then ./configure

    I'm getting the same error again:

    checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

    ReplyDelete
  8. Hello,

    I think I can see the error now:
    g++ is not installed.
    Can you verify this?
    Just running the command:
    g++
    from the shell should say if it exists or nor.

    If it is not there you need to install it, e.g,
    sudo apt-get install g++

    However, the error message is bad..
    Can you let me know what happens!

    johan@severalnines.com

    BR
    johan

    ReplyDelete
  9. Hi Johan

    You were absolutely right:)

    [root@SQL1 ~]# g++
    -bash: g++: command not found

    yum install gcc-c++

    ./configure

    sucess :)

    Regards

    Peter

    ReplyDelete
  10. Hi Johan
    I'm still having problems:

    [root@SQL1 www]# /etc/init.d/cmon start
    /usr/local/bin//cmon: error while loading shared libraries: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory
    start on service executed successfully

    I did:

    LD_LIBRARY_PATH=/mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/

    The output of $LD_LIBRARY_PATH is :
    -bash: /mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/: is a directory

    So I don't get what's wrong ?

    Again - my system is CentOs 5.

    Any ideas?

    Regards

    Peter

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Firstly, looks great - I realy want to get it running!
    CentoOS release 5.2 (final)
    MySQL 5.1.23-ndb-6.2.15

    Installed G++yum install gcc gcc-c++ autoconf automake

    ./configure CPPFLAGS="-I/usr/local/mysql/include/mysql"

    Same error as another commenter:
    checking for ndbapi headers... configure: error: Couldn't find NdbApi.h!

    This is the header file dir that comes with my version of mysql cluster

    tree /usr/local/mysql/include/mysql
    /usr/local/mysql/include/mysql

    |-- decimal.h
    |-- errmsg.h
    |-- keycache.h
    |-- m_ctype.h
    |-- m_string.h
    |-- my_alloc.h
    |-- my_attribute.h
    |-- my_config.h
    |-- my_dbug.h
    |-- my_dir.h
    |-- my_getopt.h
    |-- my_global.h
    |-- my_list.h
    |-- my_net.h
    |-- my_no_pthread.h
    |-- my_pthread.h
    |-- my_sys.h
    |-- my_xml.h
    |-- mysql.h
    |-- mysql_com.h
    |-- mysql_embed.h
    |-- mysql_time.h
    |-- mysql_version.h
    |-- mysqld_ername.h
    |-- mysqld_error.h
    |-- plugin.h
    |-- sql_common.h
    |-- sql_state.h
    |-- sslopt-case.h
    |-- sslopt-longopts.h
    |-- sslopt-vars.h
    |-- storage
    | `-- ndb
    | |-- mgmapi
    | | |-- mgmapi.h
    | | |-- mgmapi_config_parameters.h
    | | |-- mgmapi_config_parameters_debug.h
    | | |-- mgmapi_debug.h
    | | |-- mgmapi_error.h
    | | |-- ndb_logevent.h
    | | `-- ndbd_exit_codes.h
    | |-- ndb_constants.h
    | |-- ndb_init.h
    | |-- ndb_types.h
    | |-- ndb_version.h
    | `-- ndbapi
    | |-- Ndb.hpp
    | |-- NdbApi.hpp
    | |-- NdbBlob.hpp
    | |-- NdbDictionary.hpp
    | |-- NdbError.hpp
    | |-- NdbEventOperation.hpp
    | |-- NdbIndexOperation.hpp
    | |-- NdbIndexScanOperation.hpp
    | |-- NdbIndexStat.hpp
    | |-- NdbInterpretedCode.hpp
    | |-- NdbOperation.hpp
    | |-- NdbPool.hpp
    | |-- NdbRecAttr.hpp
    | |-- NdbReceiver.hpp
    | |-- NdbScanFilter.hpp
    | |-- NdbScanOperation.hpp
    | |-- NdbTransaction.hpp
    | |-- ndb_cluster_connection.hpp
    | |-- ndb_opt_defaults.h
    | |-- ndbapi_limits.h
    | `-- ndberror.h
    `-- typelib.h

    ReplyDelete
  13. Ollie,

    what do you get by running:
    - g++ --version

    - do you have mysql_config on the path. i.e, what happens if you run mysql_config from the command line in the terminal. If it says "not found" --> set the PATH (e.g export PATH=/usr/local/mysql/bin:$PATH )

    - you should run configure as:
    ./configure
    (setting the CPPFLAGS is not needed as is wrongly used).


    Let me know one way or another what you get from the above.

    ReplyDelete
  14. Hi there
    I managed to install Cmon on Centos 5.0 and here is what I've done:

    yum install autoconf && yum install automake

    yum install gcc-c++

    yum install httpd

    yum install php-mysql

    yum install php-gd

    yum install php-cli

    yum install php

    cd /usr/src

    wget http://www.severalnines.com/downloads/cmon/cmon-1.2.2.tar.gz

    tar -xvf cmon-1.2.2.tar.gz

    cd cmon-1.2.2

    sh ./autogen.sh

    export PATH=/usr/local/mysql/bin:$PATH

    ./configure --with-wwwroot=/var/www


    make

    make install

    make install-extra

    ldconfig /usr/local/mysql/lib

    chkconfig --add cmon


    chmod 775 /etc/init.d/cmon

    chkconfig --add httpd

    vim /etc/httpd/conf/httpd.conf # and change DocumentRoot to: DocumentRoot "yourdirectory"
    as well as
    Directory "yourdirectory"

    DirectoryIndex index.php index.html index.html.var


    /etc/init.d/httpd restart

    ldconfig /mysql/mysql-cluster-gpl-6.3.17-linux-i686-glibc23/lib/

    /etc/init.d/cmon start


    vim /var/www/cmon/cmon_db_conf.inc.php

    change $mysql_hostname="localhost" to $mysql_hostname = ":/tmp/mysql.sock"


    php -v


    go to http://yourip/cmon

    ReplyDelete
  15. Hello ,
    I've been trying to compile CMON in a Centos 5.3 with MySQL-Cluster-gpl version 7 with no luck , i think i've all the libraries and other requirements, It fails to compile in the last step , so here's the error:

    make[2]: Entering directory `/root/mysqlcluster-70-master/cmon/cmon-1.2.2/src'
    g++ -g -O2 -I/usr/include/mysql -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -DUNIV_LINUX -I/usr/include/mysql/storage/ndb -I/usr/include/mysql/storage/ndb/ndbapi -I/usr/include/mysql/storage/ndb/mgmapi -I/usr/include/mysql/ndb -I/usr/include/mysql/ndb/ndbapi -I/usr/include/mysql/ndb/mgmapi -I/usr/include/mysql -rdynamic -L/usr/lib64/mysql -lmysqlclient_r -lz -lrt -lpthread -lcrypt -lnsl -lm -lpthread -lmygcc -lndbclient -lmystrings -lmysys -o cmon connect.o logevent.o maint.o monitor.o cmon.o schema.o
    connect.o: In function `connectNdbMgm(cluster_conn_t*)':
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:151: undefined reference to `ndb_mgm_is_connected'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:159: undefined reference to `ndb_mgm_destroy_handle'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:163: undefined reference to `ndb_mgm_create_handle'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:164: undefined reference to `ndb_mgm_set_connectstring'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:171: undefined reference to `ndb_mgm_connect'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:184: undefined reference to `ndb_mgm_get_connected_port'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:184: undefined reference to `ndb_mgm_get_connected_host'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:177: undefined reference to `ndb_mgm_get_latest_error_msg'
    /root/mysqlcluster-70-master/cmon/cmon-1.2.2/src/connect.cpp:177: undefined reference to `ndb_mgm_get_latest_error'
    connect.o: In function `connectMySQL2(st_mysql*, char const*)':

    and so on...

    any ideas? there's a binary precompiled out there?

    Thanks a lot!
    Raul

    ReplyDelete