Wednesday, May 7, 2014

mysqld did not start properly and how to troubleshoot

This morning, after rebooting my server, the mysqld service did not listen for incoming connections (port 3306). But I can still see the mysql process:

root@myserver:~ ps aux | grep mysql

mysql     5804  3.8 17.1 2626952 869352 ?      Ssl  07:34   4:49 /usr/sbin/mysqld
root     20822  0.0  0.0   9388   932 pts/4    R+   09:39   0:00 grep --color=auto mysql

What happened?

First, we should check the mysqld process's status:

root@myserver:~# service mysql status
mysql start/post-start, process 4394
        post-start process 4395


Then check the syslog:

root@myserver:~# tail -f /var/log/syslog
May  7 07:33:37 myserver kernel: [  146.209493] init: mysql post-start process (3786) terminated with status 1
May  7 07:33:41 myserver kernel: [  150.125783] init: mysql main process (4057) terminated with status 7
May  7 07:33:41 myserver kernel: [  150.125834] init: mysql main process ended, respawning
May  7 07:33:42 myserver kernel: [  150.379645] init: mysql post-start process (4059) terminated with status 1
May  7 07:33:45 myserver kernel: [  154.086347] init: mysql main process (4394) terminated with status 7
May  7 07:33:45 myserver kernel: [  154.086388] init: mysql main process ended, respawning
May  7 07:33:46 myserver kernel: [  154.553728] init: mysql post-start process (4395) terminated with status 1
May  7 07:33:49 myserver kernel: [  158.012980] init: mysql main process (4529) terminated with status 7
May  7 07:33:49 myserver kernel: [  158.013031] init: mysql main process ended, respawning


As you can see, the mysql process was trying to start but something hold it down. It maked the starting process of mysqld loop forever. But what was it? The log showed us nothing else.

So, try to stop the mysqld service then start it manually with '-v' parameter to get verbose outputs:

root@myserver:~# service mysql stop

root@myserver:~# mysqld -v
140507  7:34:10 InnoDB: The InnoDB memory heap is disabled
140507  7:34:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140507  7:34:10 InnoDB: Compressed tables use zlib 1.2.3.4
140507  7:34:10 InnoDB: Using Linux native AIO
140507  7:34:10 InnoDB: Initializing buffer pool, size = 1.5G
140507  7:34:11 InnoDB: Completed initialization of buffer pool
140507  7:34:11 InnoDB: highest supported file format is Barracuda.
140507  7:34:12  InnoDB: Waiting for the background threads to start
140507  7:34:13 Percona XtraDB (http://www.percona.com) 5.5.37-MariaDB-34.0 started; log sequence number 36793054164
140507  7:34:13 [Note] Plugin 'FEEDBACK' is disabled.
140507  7:34:13 [ERROR] mysqld: unknown variable 'key_buffer_zie=256M'
140507  7:34:13 [ERROR] Aborting


Here is it. There is one misconfiguration in the /etc/mysql/my.cnf. Change it and the mysqld will start successfully.