2017-06-12 143 views
0

我有一个基于MySQL Community版本的数据库出现严重问题,该数据库在数据库分区调整大小后崩溃并且不会启动备份。 db位于RHEL7.2 Linux lvm分区上,由于存储问题,该分区必须减少。MySQL 5.7 Community Edition磁盘分区调整大小后崩溃

在调整大小期间,有inode问题导致数据库文件和文件夹被移到lost + found中,我们恢复了这些文件并将它们移到了正确的位置。 重新启动mysql服务是导致这些错误:

2017-06-12T09:38:12.510405Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 
2017-06-12T09:38:12.510568Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path. 
2017-06-12T09:38:12.510628Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.13) starting as process 40601 ... 
2017-06-12T09:38:12.515354Z 0 [Note] InnoDB: PUNCH HOLE support available 
2017-06-12T09:38:12.515396Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 
2017-06-12T09:38:12.515404Z 0 [Note] InnoDB: Uses event mutexes 
2017-06-12T09:38:12.515413Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 
2017-06-12T09:38:12.515421Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3 
2017-06-12T09:38:12.515428Z 0 [Note] InnoDB: Using Linux native AIO 
2017-06-12T09:38:12.515779Z 0 [Note] InnoDB: Number of pools: 1 
2017-06-12T09:38:12.515943Z 0 [Note] InnoDB: Using CPU crc32 instructions 
2017-06-12T09:38:12.518004Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 
2017-06-12T09:38:12.527531Z 0 [Note] InnoDB: Completed initialization of buffer pool 
2017-06-12T09:38:12.529516Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 
2017-06-12T09:38:12.541493Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 
2017-06-12T09:38:12.542896Z 0 [Note] InnoDB: The log sequence number 8204 in the system tablespace does not match the log sequence number 435564720566 in the ib_logfiles! 
2017-06-12T09:38:12.542919Z 0 [Note] InnoDB: Database was not shutdown normally! 
2017-06-12T09:38:12.542926Z 0 [Note] InnoDB: Starting crash recovery. 
2017-06-12 10:38:12 0x7f4e11608740 InnoDB: Assertion failure in thread 139973275715392 in file fut0lst.ic line 85 
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA 
InnoDB: We intentionally generate a memory trap. 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com. 
InnoDB: If you get repeated assertion failures or crashes, even 
InnoDB: immediately after the mysqld startup, there may be 
InnoDB: corruption in the InnoDB tablespace. Please refer to 
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html 
InnoDB: about forcing recovery. 
09:38:12 UTC - mysqld got signal 6 ; 
This could be because you hit a bug. It is also possible that this binary 
or one of the libraries it was linked against is corrupt, improperly built, 
or misconfigured. This error can also be caused by malfunctioning hardware. 
Attempting to collect some information that could help diagnose the problem. 
As this is a crash and something is definitely wrong, the information 
collection process might fail. 

key_buffer_size=8388608 
read_buffer_size=131072 
max_used_connections=0 
max_threads=151 
thread_count=0 
connection_count=0 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68189 K bytes of memory 
Hope that's ok; if not, decrease some variables in the equation. 

Thread pointer: 0x0 
Attempting backtrace. You can use the following information to find out 
where mysqld died. If you see no messages after this, something went 
terribly wrong... 
stack_bottom = 0 thread_stack 0x40000 
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef0cfb] 
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x7af361] 
/lib64/libpthread.so.0(+0xf100)[0x7f4e111ef100] 
/lib64/libc.so.6(gsignal+0x37)[0x7f4e0fbe25f7] 
/lib64/libc.so.6(abort+0x148)[0x7f4e0fbe3ce8] 
/usr/sbin/mysqld[0x77f71c] 
/usr/sbin/mysqld(_Z22trx_undo_free_preparedP5trx_t+0x0)[0x77f4d4] 
/usr/sbin/mysqld(_Z19trx_undo_lists_initP10trx_rseg_t+0xdcc)[0x10bcacc] 
/usr/sbin/mysqld[0x10a11ec] 
/usr/sbin/mysqld[0x10a39bc] 
/usr/sbin/mysqld(_Z24trx_sys_init_at_db_startv+0x1883)[0x10aaa43] 
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x4830)[0x10721a0] 
/usr/sbin/mysqld[0xf39451] 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7fa2b1] 
/usr/sbin/mysqld[0xce3925] 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x610)[0xcea830] 
/usr/sbin/mysqld[0x7a82bd] 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x842)[0x7a97e2] 
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4e0fbceb15] 
/usr/sbin/mysqld[0x79f5e5] 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains 
information that should help you find out what is causing the crash. 

和:

2017-06-12T09:38:12.542919Z 0 [Note] InnoDB: Database was not shutdown normally! 
2017-06-12T09:38:12.542926Z 0 [Note] InnoDB: Starting crash recovery. 
2017-06-12 10:38:12 0x7f4e11608740 InnoDB: Assertion failure in thread 139973275715392 in file fut0lst.ic line 85 
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA 

我试过多种innodb_force_recovery模式和的mysqld没有启动,直到我用innodb_force_recovery = 6。当启动mysqld我不能备份数据库,因为这样的多个错误:

2017-06-12T11:10:52.217568Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`plugin` in the cache. Attempting to load the tablespace with space id 2 
2017-06-12T11:10:52.277914Z 0 [ERROR] Function 'validate_password' already exists 
2017-06-12T11:10:52.277954Z 0 [Warning] Couldn't load plugin named 'validate_password' with soname 'validate_password.so'. 
2017-06-12T11:10:52.284445Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`gtid_executed` in the cache. Attempting to load the tablespace with space id 18 
2017-06-12T11:10:52.293339Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them. 
2017-06-12T11:10:52.310829Z 0 [Warning] CA certificate ca.pem is self signed. 
2017-06-12T11:10:52.313911Z 0 [Note] Server hostname (bind-address): '*'; port: 3306 
2017-06-12T11:10:52.314120Z 0 [Note] IPv6 is available. 
2017-06-12T11:10:52.314139Z 0 [Note] - '::' resolves to '::'; 
2017-06-12T11:10:52.314179Z 0 [Note] Server socket created on IP: '::'. 
2017-06-12T11:10:52.433825Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`server_cost` in the cache. Attempting to load the tablespace with space id 19 
2017-06-12T11:10:52.449122Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`engine_cost` in the cache. Attempting to load the tablespace with space id 20 
2017-06-12T11:10:52.476635Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_leap_second` in the cache. Attempting to load the tablespace with space id 12 
2017-06-12T11:10:52.477431Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_name` in the cache. Attempting to load the tablespace with space id 8 
2017-06-12T11:10:52.490830Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone` in the cache. Attempting to load the tablespace with space id 9 
2017-06-12T11:10:52.507804Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition_type` in the cache. Attempting to load the tablespace with space id 11 
2017-06-12T11:10:52.508725Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition` in the cache. Attempting to load the tablespace with space id 10 
2017-06-12T11:10:52.513876Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`servers` in the cache. Attempting to load the tablespace with space id 3 
2017-06-12T11:10:52.563565Z 0 [Note] Event Scheduler: Loaded 0 events 
2017-06-12T11:10:52.563876Z 0 [Note] /usr/sbin/mysqld: ready for connections. 
Version: '5.7.13' socket: '/dataspace/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 

而当我试图转储数据库文件我看到这个:

mysqldump -uroot -pXXXXX --skip-lock-tables --skip-extended-insert --hex-blob office > office_dump.sql 
mysqldump: [Warning] Using a password on the command line interface can be insecure. 
mysqldump: Error 2013: Lost connection to `MySQL` server during query when dumping table `audit_log` at row: 334202 

请问我能做些什么来恢复数据并使数据库恢复在线状态?

任何援助将认真赞赏。

回答

0

我做了什么让MySQL再次运行。

  1. 备份borked $ MYSQL_HOME目录。
  2. 添加skip-grant-tables允许我以root身份登录到mysql,而不需要密码。
  3. 在/etc/my.cnf尝试了不同的innodb_force_recovery模式,直到允许mysqld启动。 (在我的情况下,它是innodb_force_recovery = 5)。
  4. 尝试和失败,转储所有数据库:

    mysqldump -uroot -pXXXXX --skip-lock-tables --skip-extended-insert --hex-blob databasename > databasename_dump.sql mysqldump: [Warning] Using a password on the command line interface can be insecure. mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table AUDIT_LOG at row: 334202

  5. 给自己买了一些Percona的支持,他们建议我遍历在$ MYSQL_HOME每个数据库目录下的* .FRM文件,为每一个创建单独的mysqldump命令。 (在此之前,并不知道frm文件代表数据库内的单个数据库表)。他们还建议在每个命令后面输入以避免Lost connection问题。 我写了这样一个脚本: for table in $(ls -1 *.frm); do table_name=$(echo $table | cut -d '.' -f 1); echo mysqldump --host=127.0.0.1 --user=root --skip-lock-tables databasename ${table_name} \> database.${table_name}.sql; echo sleep 5 >> commands.txt; done 它生成了我可以用来转储每个数据库表的所有mysqldump命令。

  6. 我通过创建commands.txt中环文件转储所有可能仍然工作表。(sh -x commands.txt

  7. 我开始了一个新的MySQL实例,创建丢失的数据库(一个或多个),和进口的所有.mysql文件转储到新的数据库中。 for each in $(ls -1 $MYSQL_HOME/recoveries/*); do mysql -uroot -pXXXX databasename < $each; sleep 5; done

  8. 确信该数据库已运行和固定与缺少表(mysqlfrm --diagnostic帮助类似的服务器)上的其他问题

  9. 当一切准备停当,服务器又回到网上,尽管有几个月的关键数据丢失。