Tuesday, April 28, 2009

Online configuration update and a new script

If you are using the scripts from the Configurator (and yes, the broken link is now fixed so it is actually possible to get the config package..), and want to tweak your configuration on a started Cluster then you can do like written below.

There is also a new script in Configurator 2.2 (released 28th April 2009, PST1153AM , GMT07:53) that lets you check the values of different config parameters. Its use is described below.

Pretend we want to extend the DataMemory from 1024M to 1536M:
  1. cd mysqlcluster-70/cluster/scripts
  2. vi ../config/config.ini
    This is the master config.ini file and will be copied out to the management server(s). Don't change /etc/mysql/config.ini, as that will have no effect.
  3. Set DataMemory=1536M then save and exit
  4. sh rolling-restart.sh
  5. When it has finished, verify that the Cluster has the new settings by running query-param.sh:
    sh query-param.sh DataMemory
    Parameter=DataMemory
    Data Node 3 : 1610612736
    Data Node 4 : 1610612736
    Please note that the output is in bytes.
The rolling-restart.sh script implements this procedure described in the reference manual.
Many other parameters can be changed using the above steps (e.g, MaxNoOfConcurrentOperations etc). Be careful though that you have memory available (the script does not check for that yet).

Parameters that change the file system one way or another (NoOfFragmentLogFiles, FragmentLogFileSize, Datadir, BackupDataDir, FileSystem*, InitFragmentLogfiles) must be changed with an initial rolling restart (all parameters are listed here in the ref manual):

Pretend we want to change the size of the Redolog (NoOfFragmentLogfiles).
From the beginning we have:
./query-param.sh NoOfFragmentLogFiles
Parameter=NoOfFragmentLogFiles
Data Node 3 : 6
Data Node 4 : 6

Then we modify our "template" config.ini and go ahead:
  1. vi ../config/config.ini
    This is the master config.ini file and will be copied out to the management server(s)
  2. Set NoOfFragmentLogFiles=7 then save and exit
  3. sh rolling-restart-initial.sh
  4. When it has finished, verify that the Cluster has the new settings by running query-param.sh:
    ./query-param.sh NoOfFragmentLogFiles
    Parameter=NoOfFragmentLogFiles
    Data Node 3 : 7
    Data Node 4 : 7
Have fun upgrading but remember - don't try to start with a smaller configuration on an already existing configuration (i.e., it is safe to increase parameters, but be very careful if you decrease).

2 comments:

Rob said...

I tried to use the scripts on a new system and got the following error. What is the best way to debug this.

[root@hat scripts]# start-cluster-initial.sh
Connection to hat:1186 failed
Starting management server on hat
copying ../config/config.ini to hat
config.ini 100% 4001 3.9KB/s 00:00
STARTED MANAGEMENT SERVER ON hat
Sleeping for three seconds
...
Starting data node 2 on hatsql1
Starting data node 3 on hatsql2
Data nodes are starting. This can take a few minutes...
.....
.
Cluster failed to start

Johan Andersson said...

A very typical problem is that you don't have enough resources to start the cluster. How much DataMemory are you allocating? Do you have enough RAM?

Do you have enough disk space?

There are log files in the datadirs of the data nodes and the management server.

The easiest is if you can run a program called ndb_error_reporter which collects them.
Run it on 'hat':

/usr/local/mysql/mysql/bin/ndb_error_reporter /etc/mysql/config.ini

Send the bzip file it generates to johan at severalnines com
and I will have a look at it.

-j

I am working on better to present and suggest what actions to take.