MySQL Cluster: mysqld Sort aborted and error 4006

Tweet
Just a note on a problem that some people may find useful and may work for you if you have the same issue.

A client had a problem this morning with queries being aborted with the error message:

 Got temporary error 4006 'Connect failure - out of connection objects (increase MaxNoOfConcurrentTransactions)' from NDBCLUSTER

Here you would think that increasing the number of MaxNoOfConcurrentTransactions would help, but the root cause is something else (the client is not even close of hitting the default MaxNoOfConcurrentTransactions)

It turned out that during the night a couple of updates to the schema had been made (ALTER TABLEs to convert a couple of TEXT to  VARCHAR, so that a particular index could be created).

Looking in the mysql error logs we had:

120511 10:47:05 [ERROR] /usr/local/mysql/bin/mysqld: Sort aborted: Got temporary error 4006 'Connect failure - out of connection objects (increase MaxNoOfConcurrentTransactions)' from NDBCLUSTER
120511 10:47:27 [ERROR] Got error 4006 when reading table './database_name/table_name'

I suspected there was something wrong (corrupted?) with the .frm files that the mysql server is caching in the datadir, and simply did, for all mysql servers:

• stop mysqld
• remove all .frm for all ndb tables in the database
• start mysqld
• and finally a "SHOW TABLES" to get all the .frm files back up from the data nodes into the local datadir of the mysqld

Now, the problem is gone, but I can't explain how reading a table can cause error 4006...

Hope this helps someone.