The client called at 7am on Monday morning with a database down scenario. They arrived to messages from their users that the database was inaccessible. When they logged in, they found the database was down and that it would not start normally:
$ sqlplus / as sysdba SQL*Plus: Release 18.104.22.168.0 Production on Sun Feb 14 15:47:52 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 2147483648 bytes Fixed Size 3712904 bytes Variable Size 1476397176 bytes Database Buffers 654311424 bytes Redo Buffers 13062144 bytes ORA-00205: error in identifying control file, check alert log for more info
They looked the alert log and saw messages like this:
ALTER DATABASE MOUNT Sun Feb 14 15:47:58 2016 ORA-00210: cannot open the specified control file ORA-00202: control file: '/u01/app/oracle/recovery_area/ora12c/control02.ctl' ORA-27037: unable to obtain file status Linux-x86_64 Error: 2: No such file or directory Additional information: 3 ORA-205 signalled during: ALTER DATABASE MOUNT... Sun Feb 14 15:47:58 2016 Checker run found 1 new persistent data failures Sun Feb 14 15:48:00 2016 Using default pga_aggregate_limit of 2048 MB
Because they knew there had been storage corruption from the storage admins, they called us in a panic. Thankfully, and luckily, Oracle forces multiplexing of the control files, so this was an easy recovery. Just find a good copy of the control file and copy it over the corrupted one:
$ sqlplus / as sysdba SQL*Plus: Release 22.214.171.124.0 Production on Sun Feb 14 15:49:45 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 126.96.36.199.0 - 64bit Production SQL> show parameter control_files NAME TYPE VALUE --------------------------------- ----------- ------------------------------ control_files string /u01/app/oracle/oradata/ora12c /control01.ctl,/u01/app/oracl e/recovery_area/ora12c/control 02.ctl SQL> host $ cd /u01/app/oracle/recovery_area/ora12c $ cp /u01/app/oracle/oradata/ora12c/control01.ctl ./control02.ctl $ exit exit SQL> alter database mount; Database altered. SQL> alter database open; Database altered.
It was a very simple fix, but it really highlights knowing a lot of different recovery scenarios, including some of the easy ones.