可能是 tempdb 空间用尽或某个系统表不一致故障处理

有客户sql server数据库由于异常断电,导致数据库访问异常
QQ20250422-132721


对其做dbcc检查发现报:检查已终止。收集事实数据时检测到错误。可能是 tempdb 空间用尽或某个系统表不一致。请检查前面的错误消息。错误
QQ20250422-121636

查看sql server 日志,发现报错日志内容为:
04/22/2025 10:50:26,spid58,未知,DBCC CHECKDB (SHINVA) WITH no_infomsgs executed by sa terminated abnormally due to error state 5. Elapsed time: 0 hours 0 minutes 1 seconds.
04/22/2025 10:50:26,spid58,未知,The Database ID 7 Page (1:147440) slot 0 for LOB data type node does not exist. This is usually caused by transactions that can read uncommitted data on a data page. Run DBCC CHECKTABLE.
04/22/2025 10:50:26,spid58,未知,错误: 7105,严重性: 22,状态: 9。
对于这种情况,尝试重建LDF,和REPAIR_ALLOW_DATA_LOSS方案都失败,最终确认通过逻辑迁移的方式完成恢复,然后再次尝试dbcc一切正常,完成本次恢复任务
20250423210001

发表在 SQL Server恢复 | 标签为 , | 留下评论

11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错

今天有一个客户11.2.0.4的库由于断电报ORA-600 kcratr_nab_less_than_odr的故障

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期二 4月 22 10:56:36 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for 64-bit Windows: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 1.9174E+10 bytes
Fixed Size                  2289320 bytes
Variable Size            3355443544 bytes
Database Buffers         1.5771E+10 bytes
Redo Buffers               45932544 bytes
数据库装载完毕。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653],
[50575], [50922], [], [], [], [], [], [], []

alert日志报错信息

Tue Apr 22 05:08:12 2025
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 3880792636
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Tue Apr 22 05:08:17 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 2451 KB redo, 1442 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc  (incident=240429):
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\incident\incdir_240429\ydhl_ora_2164_i240429.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\ydhl\ydhl\trace\ydhl_ora_2164.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [4653], [50575], [50922],[],[],[],[],[],[],[]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个故障在一般在11.2.0.1版本中最为常见,处理起来比较简单,一般数据也会0丢失
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
这里处理比较简单,重建控制文件就完成了恢复工作
102328


发表在 Oracle备份恢复 | 标签为 , | 留下评论

[MY-013183] [InnoDB] Assertion failure故障处理

在一个存储故障的环境中,通过做硬件恢复,恢复出来一个mysql数据库,但是直接启动报错

[mysql@localhost bin]$ ./mysqld
2025-04-17T03:34:50.352302Z 0 [System] [MY-010116] [Server] /data/mysql/mysql/bin/mysqld (mysqld 8.0.34) starting as process 58239
2025-04-17T03:34:50.356910Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-04-17T03:34:51.031054Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_002′ Page [page id: space=4294967278, page number=160] log sequence number 1728577790947 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.031090Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.031118Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_002′ Page [page id: space=4294967278, page number=131] log sequence number 1728577833027 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.031124Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.031138Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_002′ Page [page id: space=4294967278, page number=3621] log sequence number 1728577635513 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.031142Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.031193Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_002′ Page [page id: space=4294967278, page number=167] log sequence number 1728577760219 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.042480Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_001′ Page [page id: space=4294967279, page number=184] log sequence number 1728577792529 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.042486Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.042359Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘innodb_undo_001′ Page [page id: space=4294967279, page number=1975] log sequence number 1728577800027 is in the future! Current system log sequence number 1728577469817.
2025-04-17T03:34:51.042681Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.059937Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-04-17T03:34:51.159245Z 0 [ERROR] [MY-011971] [InnoDB] Tablespace ‘xff/t_xifenfei’ Page [page id: space=153, page number=4] log sequence number 1728577926919 is in the future! Current system log sequence number 1728577498088.
2025-04-17T03:34:51.159280Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery.
2025-04-17T03:34:51.163187Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: fut0lst.ic:81:addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA thread 140491735693056
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-04-17T03:34:51Z UTC – mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=f183cd3ecfc35a4aa5da997063d5e8c97ffca986
Thread pointer: 0x7fc6bc000b60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7fc6c7ffeaf0 thread_stack 0×100000
/data/mysql/mysql/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0×41) [0x21323b1]
/data/mysql/mysql/bin/mysqld(print_fatal_signal(int)+0x2a2) [0xfef932]
/data/mysql/mysql/bin/mysqld(my_server_abort()+0×75) [0xfefb75]
/data/mysql/mysql/bin/mysqld(my_abort()+0xe) [0x212c24e]
/data/mysql/mysql/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0×309) [0x237cde9]
/data/mysql/mysql/bin/mysqld() [0x2349cf0]
/data/mysql/mysql/bin/mysqld() [0x234aa54]
/data/mysql/mysql/bin/mysqld(trx_purge(unsigned long, unsigned long, bool)+0xeb) [0x234d56b]
/data/mysql/mysql/bin/mysqld(srv_purge_coordinator_thread()+0×450) [0x23224b0]
/data/mysql/mysql/bin/mysqld(void Detached_thread::operator()<void (*)()>(void (*&&)())+0xca) [0x224bcaa]
/lib64/libstdc++.so.6(+0xc2ba3) [0x7fc731c11ba3]
/lib64/libpthread.so.0(+0x814a) [0x7fc732fe614a]
/lib64/libc.so.6(clone+0×43) [0x7fc7312eef23]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

这个报错主要含义是:

  1. 多个表空间(特别是innodb_undo_*)的日志序列号(LSN)比当前系统LSN要大,这表明可能存在数据损坏或不一致
  2. 系统最终因为断言失败而崩溃

对于这样的情况,可以通过mysql强制拉库的方式启动mysql,如果可以启动成功直接使用mysqldump导出数据,然后重建新库,如果无法启动mysql成功,那就考虑通过对单个的ibd基表进行discard+import方式进行恢复参考:MySQL 8.0版本ibd文件恢复,如果这个方法不能成功考虑直接通过工具读取ibd文件参考:frm和ibd文件数据库恢复

发表在 MySQL恢复 | 标签为 , , | 留下评论