Theme

Welcome · log in · join

Search similar:

Forums → Tech / Special Interest →

Computer Hardware Help → More adventures in data recovery

uniqs
2685

« Just picked up two 4 TB WD HDDs... Smartmontools says this • (topic move) Need Help Installing A Bluetooth Dongle In A Vista Machine »

koitsu
MVM
join:2002-07-16
Mountain View, CA

Netgear CM1000
Ubiquiti EdgeRouter X SFP
Ubiquiti Unifi UAP-AC-LITE

koitsu

MVM

2014-Jun-11 8:50 pm

More adventures in data recovery

I know I haven't been around much (new job and life have been keeping me quite busy), but I figured I'd take some time today to document a recovery I'm doing for a colleague of mine. This is also my first time dealing with a Seagate recovery (I'm used to WD disks and WD firmware behaviour).

Note: I haven't even begun the recovery process yet, but I'll try my best to do follow-ups as things progress.

The details are as follows:

- Colleague of mine has a 3TB Seagate Barracuda 7200.14 (1TB platter version), specifically model ST3000DM001-9YN166, which went bad on him -- and by bad I mean a tremendous amount of unreadable, suspect, and remapped LBAs (when I say tremendous... well, just wait for the SMART dump. It's the worst drive I've dealt with yet). He initially approached me saying "do you know how to allocate more space/room for NTFS clusters?" and I was like "wait, what? You exhausted $BadClus$ somehow? WTF?" :-)

- He managed to get "most" of the data off the drive himself (or had backups, not sure), but knew (from talking to me) to not run any programs that would either a) modify things on the drive (or as little as possible), or b) "repair" the drive (this just diminishes the chance of successful recovery if using a data recovery company -- they want the drive as untouched/unmodified as possible). However, there was a partition on the drive which he wanted files from, but couldn't read it (Windows apparently was throwing a fit over it).

- The drive was partitioned into 3 partitions using GPT. All 3 partitions consisted of NTFS-based filesystems. I point this out because the way I do recovery involves use of FreeBSD (to make a copy of the drive, e.g. a disk image or copy of the disk onto a spare), followed by use of some commercial utilities on Windows which I trust to do a decent job. The tricky part here is that I use Windows XP (limited to 2TB partition sizes, and has no GPT support), and the fact that the partition he wants data from is above the 2TB barrier (it's a 700GB partition at the very end of the disk).

- My FreeBSD box uses a Supermicro X7SBA motherboard, which offers six (6) SATA300 ports all driven by an Intel ICH7R southbridge -- however, all of those ports are used for my own storage and data, thus I had no free SATA ports.

- To further complicate things, I do not have access to the PCIe x16 slot (which only has x8 lanes anyway) on the motherboard -- it's physically blocked by the HSF I use. That left me with PCI-X slots (I do not bother with PCI-X in general), and some 32-bit PCI slots.

- Also needed to ensure whatever SATA add-on card I went with was supported by FreeBSD natively, and used a driver that made of use CAM. Given financial data points, that left me with one choice: Silicon Image using the siis(4) driver, whose author I've talked to many times and who I trust if I was to encounter oddities.

Lots of conundrums here. The choices I made were as follows:

- To rectify the lack-of-SATA-ports situation, I went looking for a PCI-based SATA300 controller that had at least 2 internal ports. I didn't want a RAID option ROM getting in the way (since I use the ICH7R's AHCI option ROM for booting, and multiple disk option ROMs often do not work together / "stack"), but if I had to take one with an option ROM, I'd deal with that if I came to it.

- I tend to like Rosewill for "generic cheap no-frills hardware" of this sort. So I found the Rosewell RC-312 -- but nobody sells them any more; everyone is using PCIe these days.

- However, the Silicon Image 3124 chip is PCI- or PCI-X-based (other/newer chips are PCIe), so I figured all I needed to do was find a different vendors' card that used the 3124 and go with that.

- I found the Syba SY-PCI40010 at Amazon for US$40 -- more expensive than I wanted, but I wasn't going to fight over cost. I made the assumption that this thing had better "just work" and put in an order. More on that in just a moment (and is one of the reasons I posted this).

- The 3TB situation with Windows XP was a little more complex. I had to really think this one out. First of all, normally I take an image of a disk (i.e. stored as a file) and then work with that -- but with a 3TB drive as the source medium, I would need a 3TB file, and that wouldn't work because of the 2TB limitation on XP, not to mention I don't own any disks that large! :-)

- What I came up with was a combination: first, said colleague happened to have a semi-new (used for 14 hours) ST3000DM001 which he could send along with the bad drive. Okay, physical capacity issue solved -- instead of making a disk image of the source disk (as a file), I'd have to just do a disk-to-disk copy. But what about the 2TB limit, and GPT partitioning, and XP? Oh, not to mention the fact I don't have a gigantic drive (ex. 4TB) available for storing a disk image?

- The methodology I'm going to try using is virtualisation, specifically directly connecting the semi-new raw disk (not a partition!) on my XP box to a Windows 7 VM using VirtualBox. I've done this before (both with raw disks and USB sticks), but the problem that kept (well, keeps) going around in my head was this: would the underlying XP OS SATA driver (Intel's AHCI/RST driver) effectively advertise the entire LBA range to the guest (Windows 7)? My gut feeling is that yes it will, and that the "2TB limitation" is purely an XP filesystem limitation.

Rephrased: my feeling is that the underlying SATA driver on XP should support more than 2TB of addressable LBAs, so as long as that's true, the guest OS (Windows 7) under VirtualBox should actually see the entire 3TB drive.

And no, I am not installing Windows 7 for this task. There are too many reasons I stick with XP and this is not up for discussion. If the methodology I've come up with doesn't work I have other avenues of choice (such as getting a SFF PC and installing Windows 7 natively on that), but "destroying" my main workstation is not an option.

So all that would address the need for GPT, in addition to the need for 2TB+ support. Crossing fingers -- I'll deal with that when I get there.

- Both disks (herein referred to as "HELP" for the bad drive, and "NEW" for the semi-new drive) arrived a few days ago, and the SY-PCI40010 arrived today.

First problem encountered:

- Upon hooking both 3TB drives to the SY-PCI40010 (HELP on port 0, NEW on port 1), the option ROM only saw the disk on port 0. I felt the drives and sure enough both were fully spun up, and I wasn't hearing any odd noises coming from either of them.

I disconnected HELP and hooked NEW to port 0 -- the drive showed up. I then hooked HELP to port 1 -- nope, nothing.

I was using the SATA cables that came with the card, so I tried replacing the cable on port 1. Now the behaviour changed: "intermittently" the drive would show up in the option ROM. I then swapped drives (back to HELP = port 0, NEW = port 1), and the "intermittent" behaviour remained tied to port 1.

Short version: port 1 on the new SY-PCI40010 is bad in some way. Do not care to figure out what it is (cold solder joint, bad port, broken traces, whatever) -- it's unreliable.

As such, I moved NEW to port 4. Voila, both drives showed up in the option ROM and in FreeBSD (the siis(4) driver worked through CAM just like I expected). Device names, just to keep things clear:

/dev/ada6 = HELP
/dev/ada7 = NEW

- The VERY FIRST thing I did was take SMART attribute snapshots from both disks (specifically smartctl -a from both drives) and save the output to separate files.

Here is the output from the HELP drive:

smartctl 6.2 2014-02-18 r3874 [FreeBSD 9.3-BETA1 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-9YN166
Serial Number:    W1F0NMKF
LU WWN Device Id: 5 000c50 0511be4f3
Firmware Version: CC4B
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Jun 11 15:50:37 2014 PDT
 
==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en
 
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  584) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 338) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.
 
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   072   072   006    Pre-fail  Always       -       164837287
  3 Spin_Up_Time            0x0003   094   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       367
  5 Reallocated_Sector_Ct   0x0033   094   094   036    Pre-fail  Always       -       8776
  7 Seek_Error_Rate         0x000f   065   051   030    Pre-fail  Always       -       610330700323
  9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       16556
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       129
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       48116
188 Command_Timeout         0x0032   100   091   000    Old_age   Always       -       226 237 247
189 High_Fly_Writes         0x003a   066   066   000    Old_age   Always       -       34
190 Airflow_Temperature_Cel 0x0022   063   050   045    Old_age   Always       -       37 (Min/Max 31/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       85
193 Load_Cycle_Count        0x0032   050   050   000    Old_age   Always       -       100392
194 Temperature_Celsius     0x0022   037   050   000    Old_age   Always       -       37 (0 19 0 0 0)
197 Current_Pending_Sector  0x0012   076   001   000    Old_age   Always       -       4000
198 Offline_Uncorrectable   0x0010   076   001   000    Old_age   Offline      -       4000
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       15226h+46m+59.877s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       193198657458464
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       111717728336544
 
SMART Error Log Version: 1
ATA Error Count: 47744 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
 
Error 47744 occurred at disk power-on lifetime: 16556 hours (689 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.
 
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455
 
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00      00:09:43.301  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:43.200  READ LOG EXT
  60 00 00 ff ff ff 4f 00      00:09:40.452  READ FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:09:40.452  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00      00:09:40.425  FLUSH CACHE EXT
 
Error 47743 occurred at disk power-on lifetime: 16556 hours (689 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.
 
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455
 
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00      00:09:40.452  READ FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:09:40.452  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00      00:09:40.425  FLUSH CACHE EXT
  2f 00 01 10 00 00 00 00      00:09:40.349  READ LOG EXT
  60 00 00 ff ff ff 4f 00      00:09:37.460  READ FPDMA QUEUED
 
Error 47742 occurred at disk power-on lifetime: 16556 hours (689 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.
 
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455
 
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00      00:09:37.460  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:37.408  READ LOG EXT
  60 00 00 ff ff ff 4f 00      00:09:34.565  READ FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:09:34.564  WRITE FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:34.523  READ LOG EXT
 
Error 47741 occurred at disk power-on lifetime: 16556 hours (689 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.
 
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455
 
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00      00:09:34.565  READ FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:09:34.564  WRITE FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:34.523  READ LOG EXT
  60 00 00 ff ff ff 4f 00      00:09:31.751  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:31.614  READ LOG EXT
 
Error 47740 occurred at disk power-on lifetime: 16556 hours (689 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.
 
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455
 
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00      00:09:31.751  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:09:31.614  READ LOG EXT
  60 00 00 ff ff ff 4f 00      00:09:28.800  READ FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      00:09:28.800  WRITE FPDMA QUEUED
  ea 00 00 00 00 00 00 00      00:09:28.773  FLUSH CACHE EXT
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     16522         -
# 2  Short offline       Completed: read failure       90%     16518         -
# 3  Short offline       Completed: read failure       90%     16445         -
# 4  Short offline       Completed: read failure       90%     16395         -
# 5  Short offline       Completed: read failure       90%     16388         -
# 6  Short offline       Completed: read failure       90%     16388         -
# 7  Short offline       Completed: read failure       90%     16386         -
# 8  Short offline       Completed: read failure       90%     16384         -
 
SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Lots and lots to say about this awful drive:

- Attributes 1 and 7 are vendor-encoded in a way and tend to always change (Seagate F/W behaviour), so I ignored those.

- Attribute 9 indicated drive lifetime of 16556 power-on hours, so about 690 days / a little under 2 years. Not very impressive for this kind of failure.

- Attribute 5 indicated 8776 reallocated LBAs. They're 4KB LBAs, so that's minimum 67,813,376 bytes of space reallocated/data potentially lost right off the bat.

- Attribute 187 indicated 48116 LBA read or LBA write failures during the drive's lifetime, and chances are most of these were reads.

- Attribute 188 indicates some general ATA-level CDB timeout behaviours logged, but how to decode this data I'm not sure (smartmontools obviously decodes it into 3 fields, but what they represent I don't know -- I'd need to go look at the smartctl code + commit comments to figure it out).

- Attribute 189 indicated 34 events of high-fly writes, which are usually an indicator of a drive head getting too close or being too far away from the platter. The drive has three (3) 1TB platters, thus 6 heads -- which head is anyone's guess, but I don't do physical repair/recovery so knowing which head doesn't help me at all. However, what it does tell me is that it's head misalignment (either at factory or over time) possibly contributed to the failure.

- Attribute 190 indicated the highest temperature the drive ever recorded during its lifetime was 37C, and combined with attribute 194, indicated that the highest temperature was seen in my current environment -- not a surprise considering it's been absurdly hot here lately.

- Attribute 193 indicated the infamous "LCC problem", where the drive had aggressively parked its heads a total of 100392 times during its lifetime. I hate this feature so much... anyway...

- Attribute 197 indicated 4000 LBAs which were marked "suspect", i.e. unreadable. I found this number to be amusing -- 4000 exactly? Just a round, even number like that? Hmm.

- Attribute 198 indicated 4000 LBAs which were marked unremappable. Now here's where it gets tricky, because Seagate drives appear to behave differently than WD drives in this regard. From what I can tell, on this drive, combined with the value in attribute 5, effectively this drive no longer has any spare LBAs/sectors, thus cannot do any further remapping. The drive has had so many errors that it's exhausted its remapping space. One word: ouch.

- The SMART error log indicated a total of 47744 errors, with the capability to only store details of the last 5. Safe to say most of those 47744 were read errors. Of the 5 shown, they all happened during an NCQ READ request.

- The SMART self-test log showed that some program (or possibly the drive firmware itself? Unsure) had been using short tests. Not sure what my colleague may have run to do this (or as I said, drive doing it itself), but clearly the tests failed in some way. Use of smartctl -x to review the extended self-test log turned up which LBAs were unreadable (for those tests), just in case I cared (but in this case I don't):

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     16522         4937310040
# 2  Short offline       Completed: read failure       90%     16518         4937310040
# 3  Short offline       Completed: read failure       90%     16445         4937283288
# 4  Short offline       Completed: read failure       90%     16395         4937220368
# 5  Short offline       Completed: read failure       90%     16388         4937220368
# 6  Short offline       Completed: read failure       90%     16388         4937220368
# 7  Short offline       Completed: read failure       90%     16386         4937220368
# 8  Short offline       Completed: read failure       90%     16384         4937220368

As I said: what a mess!

As is always the rule when doing recovery, it's important to never make any modifications to the source material, i.e. avoid any kind of LBA remaps (if possible; if the drive does it itself on a read then we have no control over that), do not mount filesystems/partitions, yadda yadda.

I then moved on to the drive labelled NEW -- because when I'm given a drive to store data on, I test it first (record SMART stats, write zeros to every LBA, record SMART stats, read all LBAs, record SMART stats, review differences along the way), because I don't want the thing crapping out on me mid-recovery -- it just complicates and frustrates.

Here's the output from the NEW drive:

smartctl 6.2 2014-02-18 r3874 [FreeBSD 9.3-BETA1 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1CH166
Serial Number:    W1F4YCJH
LU WWN Device Id: 5 000c50 0735d6462
Firmware Version: CC29
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Jun 11 15:50:33 2014 PDT
 
==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en
 
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   89) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 335) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.
 
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   111   100   006    Pre-fail  Always       -       32370776
  3 Spin_Up_Time            0x0003   095   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       14
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       39282
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       15
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       14
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   099   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0x0022   066   053   045    Old_age   Always       -       34 (Min/Max 30/34)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       12
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       71
194 Temperature_Celsius     0x0022   034   047   000    Old_age   Always       -       34 (0 23 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       8h+55m+13.064s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       9132955008
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       24318750
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]
 
SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Everything here looked good/normal for this model of drive except for one thing: attribute 189 (high-fly writes). Okay, 1 event isn't bad, and maybe it came this way from the factory, so I'll live with it / just make mental note of it.

I then began the process of zeroing the NEW drive.

Second problem encountered:

- While zeroing a drive, since I know what I'm looking at, I tend to watch SMART stats as things progress. I immediately began seeing this (I can't wait for forum regulars to grin and smile at this one because it comes up often):

199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       12
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       15
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       16
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       19
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       24
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       26

I stopped the NEW zeroing at this point and spew quite a slew of curse words, specifically because a drive that was working fine (0 CRC errors) just suddenly began to have to do retransmissions. Everyone here probably knows my advice on this matter, so I'll recap the possibilities:

a) Bad SATA cable,
b) Bad SATA port (on either the drive, or the SATA controller),
c) Bad drive PCB,
d) Some other anomaly on the SATA controller (ex. flaky traces, badly-designed card, whatever else),
e) Extreme amounts of electronic interference.

Given that port 1 on the SY-PCI40010 was already bad, my gut feeling was that port 4 might be flaky as well. I immediately put in a replacement/RMA with Amazon because I just do not accept this kind of crap.

To rectify this problem, I did two things at once (not proper troubleshooting technique, BTW, but I really did not care to diagnose which exact thing was the source of the problem):

1) I moved NEW from port 4 to port 3
2) I replaced the SATA cable between NEW and port 3 with a cable I had personally used many times over the years with reliability (thick cable, good shielding, etc.)

I resumed the zeroing, and the CRC problem disappeared (meaning the attribute has stayed at 26).

Now comes the question: how can I reset 26 back to 0? Well, as I've stated before to others: you can't. At least not usually. The funny thing about both drives (but I WOULD NEVER do this on the HELP/bad drive given its condition!) is that they need a firmware upgrade. Firmware updates will sometimes reset SMART attributes back to zero.

So when I get done with this whole recovery nonsense, I plan on updating the firmware on the NEW drive to CC4H and cross my fingers. I've already told my colleague that if it doesn't zero, then he'll just have to remember the value was at 26. There's nothing else I can do about it. I feel awful, but I'm working with a generic crap SATA card (overpriced :P) and hoping for the best.

At this point about 409GB of the NEW drive has been zeroed, and the SMART attributes still look good, although the drive is up to 44C (again not surprised -- it's hot here, and the drive does have 3 platters).

· actions · 2014-Jun-11 8:50 pm ·

koitsu

koitsu

MVM

2014-Jun-11 8:54 pm

Oh, one other thing I wanted to point out:

The two drive models are identical (both ST3000DM001), but the "full model strings" are different:

HELP = ST3000DM001-9YN166
NEW = ST3000DM001-1CH166

Are there actual differences? YES! Will they matter/affect the data recovery? NO. However, I like to use opportunities like this to point out that vendors like to "screw around" with things without telling their customers what all they've changed/done (meaning "ST3000DM001" is not precise enough when vendors do stuff like this). Here are the differences I could determine:

HELP = supports ATA protocol revision 8 (latest standard)
NEW = supports ATA protocol revision 9 (last I checked, working draft)

HELP = firmware version CC4B
NEW = firmware version CC29

HELP = does not support PUIS (power up in standby)
NEW = supports PUIS (power up in standby) but defaults to off/disabled

Otherwise the drives are identical.

· actions · 2014-Jun-11 8:54 pm ·

psafux
Premium Member
join:2005-11-10

psafux to koitsu

Premium Member

2014-Jun-12 7:35 am

to koitsu

Lots of good info here!

· actions · 2014-Jun-12 7:35 am ·

koitsu
MVM
join:2002-07-16
Mountain View, CA

Netgear CM1000
Ubiquiti EdgeRouter X SFP
Ubiquiti Unifi UAP-AC-LITE

koitsu

MVM

2014-Jun-12 4:16 pm

People often ask me what tools/programs I use for all of this, and this situation gives me an opportunity to mention those as I go through them...

Zeroing of NEW completed this morning. Total process took several hours; 32-bit PCI bus doesn't have much bandwidth, and the SATA controller doesn't perform amazingly well -- around 71MBytes/second during zeroing. SMART stats post-zeroing look the same as pre-zeroing (aside from the obvious attributes). I didn't bother doing a full read because I want to save time (and I'll effectively be doing that anyway during the data restoration process on Windows).

I've moved on to the actual disk-to-disk copying process. The tool I use to make a disk image or a raw disk copy is GNU ddrescue (here's the documentation). Please note this is not the same as dd_rescue (note the underscore!) -- that's a similar program but the behaviour and flags differ greatly. I recommend the GNU utility.

Executing the copy was fairly simple and I'm presently waiting for errors to start (and I'm certain they will). The usual flags I use are -v -f -n, and I also specified -b 4096 which defines the sector size (otherwise the utility assumed 512 which isn't correct for these model of disks).

I'll use this opportunity to note a key piece of information here about operating systems and block / I/O behaviour:

Linux implements what's called a "buffer cache" -- more precisely, any raw device (ex. /dev/sda) will have its data cached during reads. I tend to call this a "block cache" (the term makes more sense IMO). There are pros and cons to this.

The BSDs (FreeBSD in this case) does not do this -- there is no "block cache". Instead, all caching is done at the VFS level or per-driver. Meaning: raw devices (ex. /dev/ada6) do not have any kind of caching applied to them. If I was to access a filesystem on that device (ex. /dev/ada6s1 for, say, the first GPT partition) through UFS/FFS or NTFS or some other filesystem, then the data within the filesystem would be cached.

To inhibit the "block cache" on Linux, you're supposed to use O_DIRECT during your open() call -- or in the case of ddrescue use of the flag -d. So if doing this procedure on Linux, I recommend use of that flag. (There is the dd equivalent, by the way, called direct which does the same thing; it's one of the oflag or iflag arguments).

Probably the most important argument, aside from the source and destination devices, is the last one -- the ddrescue.log file. This is what keeps track of what LBAs and/or LBA ranges were readable vs. failed. Use of this is incredibly important; the way the program works is when receiving a read I/O error (more on that later -- how long timeouts take is somewhat complex because of all the different layers involved), it skips over N number of "blocks" (LBAs) and continues reading (but makes note of what it skipped in the log file). During a 2nd run of the program -- with slightly different arguments -- what it does is go back and try to re-read those LBAs, while simultaneously decreasing the block size (e.g. instead of skipping 16 LBAs, it might skip only 8, retry, if fail skip only 4, retry, etc.) and update the log as it goes.

The idea here is to decrease how long it takes to copy from a bad disk/bad media, while simultaneously (later) going back and seeing how much, all the way down to a single LBA, it can/cannot read. In other words: reading 16 LBAs might get you an I/O error, but maybe only 1 of those 16 is the one which causes it, so why not copy over the other 15 it can read successfully? That's why ddrescue is so helpful.

I will point out that for disks/media with large amounts of linear errors this methodology doesn't help much -- which I think will be the case in my scenario here -- but it doesn't hurt. Use the log file! Always!

So here's where I stand now:

root@icarus:/home/jdc/rouge # ddrescue -v -f -n -b 4096 /dev/ada6 /dev/ada7 ddrescue.log
 
GNU ddrescue 1.17
About to copy 3000 GBytes from /dev/ada6 to /dev/ada7
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  16 sectors       Initial skip size: 16 sectors
Sector size: 4096 Bytes
 
Press Ctrl-C to interrupt
rescued:   108600 MB,  errsize:       0 B,  current rate:   38731 kB/s
   ipos:   108600 MB,   errors:       0,    average rate:   38565 kB/s
   opos:   108600 MB,    time since last successful read:       0 s
Copying non-tried blocks...

That SATA controller, sad panda. 38MBytes/sec per drive. Oh well, it's still better than the I/O rates we got 25 years ago. :-)

· actions · 2014-Jun-12 4:16 pm ·

koitsu

koitsu

MVM

2014-Jun-13 9:29 pm

Okay, latest update, but I'll be following up with another 3 posts just for easy-to-follow record keeping.

After the first pass of using ddrescue, I was presented with these numbers:

rescued:     3000 GB,  errsize:   2752 kB,  current rate:    20480 B/s
   ipos:     2249 GB,   errors:      87,    average rate:   37292 kB/s
   opos:     2249 GB,    time since last successful read:       0 s
Finished

So of all 3TB, only 2752kBytes were unreadable.

I should note that mid-pass I decreased the FreeBSD CAM timeouts and retry attempts (read: these are the timeouts and retries at the kernel level for device I/O, and have nothing to do with the retry/timeout values in ddrescue) from the defaults of 30 seconds + 6 retries (a value of 5 means try 6 times) to help speed up the first-pass recovery process.

Between each pass I also take a snapshot of SMART attributes because sometimes drives do weird things -- and this is one of those cases. "Weird" in this situation more likely means "behaved differently than koitsu expected" since I'm used to WD drives. :-) Anyway, these are the SMART statistics that changed (well, the ones that I felt were most relevant). Lines starting with - are before pass #1, and lines starting with + indicate after pass #1:

-  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       367
+  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       368
 
-  5 Reallocated_Sector_Ct   0x0033   094   094   036    Pre-fail  Always       -       8776
+  5 Reallocated_Sector_Ct   0x0033   094   094   036    Pre-fail  Always       -       8856
 
- 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       129
+ 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       130
 
-187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       48116
+187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       49014
 
-197 Current_Pending_Sector  0x0012   076   001   000    Old_age   Always       -       4000
+197 Current_Pending_Sector  0x0012   076   001   000    Old_age   Always       -       3960
 
-198 Offline_Uncorrectable   0x0010   076   001   000    Old_age   Offline      -       4000
+198 Offline_Uncorrectable   0x0010   076   001   000    Old_age   Offline      -       3960
 
-ATA Error Count: 47744 (device log contains only the most recent five errors)
+ATA Error Count: 48637 (device log contains only the most recent five errors)

Summarised, the following things appear to have happened:

- Attributes 4 and 12: The drive actually powered down and powered back up on its own at one point. Yes I am a little surprised by this, but not too much. It looks to me that the drive probably got "severely stuck" during some I/O read request and the firmware itself decided to do a full "level 0" reset.

- Attribute 5: despite the fact that the only I/O operations being executed on the drive were read, the Seagate drive clearly -- on its own -- reallocated 80 LBAs. I find this amazing because it's the first confirmed indication I've seen of a drive firmware actually reallocating LBAs on a read. WD drives do not behave this way, so this is something I have to keep in mind going forward about Seagate drives. For what it's worth, a drive remapping an LBA on a read should only happen if the drive was actually able to successfully read the data in that sector (i.e. the drive internally had to use ECC to correct the data or reached an internal threshold saying "okay after tons of retries {internal to the drive itself} I FINALLY got the data, and now that I have, I really need to move this to a spare sector").

- Attribute 187: not surprising that there were 898 uncorrectable reads returned by the drive to the controller, thus OS. I can confirm this behaviour via messages on the system console like so (just a random copy/paste example):

(ada6:siisch0:0:0:0): READ_DMA48. ACB: 25 00 10 bb d5 40 05 01 00 00 08 00
(ada6:siisch0:0:0:0): CAM status: ATA Status Error
(ada6:siisch0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada6:siisch0:0:0:0): RES: 51 40 10 bb d5 00 05 01 00 00 00
(ada6:siisch0:0:0:0): Retrying command
(ada6:siisch0:0:0:0): READ_DMA48. ACB: 25 00 10 bb d5 40 05 01 00 00 08 00
(ada6:siisch0:0:0:0): CAM status: ATA Status Error
(ada6:siisch0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada6:siisch0:0:0:0): RES: 51 40 10 bb d5 00 05 01 00 00 00
(ada6:siisch0:0:0:0): Error 5, Retries exhausted

The "UNC" shown there indicates uncorrectable status as returned by the drive itself -- meaning at the ATA protocol level the drive told the controller "sorry dude, sector can't be read". The ATA layer in FreeBSD will retried uncorrectable errors, which is why you see a "retrying command" and later "retries exhausted" message. The "Error 5" refers to errno 5 (EIO), a.k.a. "Input/output error" returned to userland, and does not indicate "retry count" or anything like that.

- Attributes 197 and 198 are the most interesting of all (to me anyway). The number of LBAs marked suspect decreased from 4000 to 3960 (so 40 LBAs). For attribute 197 this is fine, but I've never seen this happen for attribute 198 before. Quite interesting -- again a difference in Seagate vs. WD drive firmware behaviour.

- And finally, as expected, the ATA error count grew by 893 errors (from 47744 to 48637). No surprise. However I would have expected this number delta to match that of attribute 187; instead there's a 5 count difference, which is a little disconcerting. Actually, hmm, come to think of it, maybe I can explain that... but it's really nothing to do with the drive/firmware itself (I hope), so I'll exclude that for now.

· actions · 2014-Jun-13 9:29 pm ·

koitsu

koitsu

MVM

2014-Jun-13 9:42 pm

Pass #2 was issued as follows:

root@icarus:/home/jdc/rouge # ddrescue -v -f -r3 -b 4096 /dev/ada6 /dev/ada7 ddrescue.log
 
GNU ddrescue 1.17
About to copy 3000 GBytes from /dev/ada6 to /dev/ada7
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  16 sectors       Initial skip size: 16 sectors
Sector size: 4096 Bytes
 
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:     3000 GB,  errsize:   2752 kB,  errors:      87
Current status
rescued:     3000 GB,  errsize:   2752 kB,  current rate:        0 B/s
   ipos:     2249 GB,   errors:      87,    average rate:        0 B/s
   opos:     2249 GB,    time since last successful read:       0 s
Splitting failed blocks...

Note the flag differences (as mentioned, removal of -n).

The behaviour of ddrescue in this case is to basically re-try reading all the LBAs it couldn't read before (and the software itself would attempt this up to 3 times -- again that number is completely unrelated to kernel/CAM behaviour!), in hopes that it'll eventually get lucky and get more data back.

Now I will say this up front as I said before: I have never seen that work... (until now!)

I should also note before beginning pass #2 I took the liberty of adjusting the kernel/CAM timeouts and retries a bit more -- more specifically, I *increased* them, hoping that giving things more timeout time and more kernel-level retry attempts might get some more data back:

root@icarus:/home/jdc/rouge # sysctl kern.cam.ada.default_timeout=15
kern.cam.ada.default_timeout: 5 -> 15
 
root@icarus:/home/jdc/rouge # sysctl kern.cam.ada.retry_count=2
kern.cam.ada.retry_count: 1 -> 2

And the result of pass #2:

rescued:     3000 GB,  errsize:   1998 kB,  current rate:        0 B/s
   ipos:     2527 GB,   errors:     173,    average rate:       75 B/s
   opos:     2527 GB,    time since last successful read:     4.8 m
Finished

Wow, look at that. The amount of unreadable data went from 2752kBytes to 1998kBytes. So apparently increasing timeouts and so on actually helped things out quite a bit. This is the first time I've ever seen that happen. So that lead me to look at SMART attribute differences between pass #1 and pass #2. Here they are:

-187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       49014
+187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       52554
 
-197 Current_Pending_Sector  0x0012   076   001   000    Old_age   Always       -       3960
+197 Current_Pending_Sector  0x0012   077   001   000    Old_age   Always       -       3904
 
-198 Offline_Uncorrectable   0x0010   076   001   000    Old_age   Offline      -       3960
+198 Offline_Uncorrectable   0x0010   077   001   000    Old_age   Offline      -       3904
 
-ATA Error Count: 48637 (device log contains only the most recent five errors)
+ATA Error Count: 52069 (device log contains only the most recent five errors)

- Attribute 187 is what I'd expect; yup, lots of uncorrectable reads.

- But again, attributes 197 and 198 show signs of the drive determining 56 more LBAs were actually readable. But this time around, there were no remappings (attribute 5 did not change between pass #1 and pass #2). The drive literally just decided at one point "hey I finally was able to read these LBAs, they're good!" and that was that.

- And for ATA errors, there were 3432 more. No surprise there -- I increased the number of retries within the kernel, so this makes sense to me. From this point forward I won't be posting that statistic since it's not of much use.

· actions · 2014-Jun-13 9:42 pm ·

koitsu

koitsu

MVM

2014-Jun-13 9:47 pm

I then did pass #3, with the same timeouts/etc. as before. I wanted to see if it got any more data back -- and it did. Pass #3 (same arguments as #2):

root@icarus:/home/jdc/rouge # ddrescue -v -f -r3 -b 4096 /dev/ada6 /dev/ada7 ddrescue.log
 
GNU ddrescue 1.17
About to copy 3000 GBytes from /dev/ada6 to /dev/ada7
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  16 sectors       Initial skip size: 16 sectors
Sector size: 4096 Bytes
 
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:     3000 GB,  errsize:   1998 kB,  errors:     173
Current status
rescued:     3000 GB,  errsize:   1998 kB,  current rate:        0 B/s
   ipos:     2249 GB,   errors:     173,    average rate:        0 B/s
   opos:     2249 GB,    time since last successful read:       0 s
Retrying bad sectors... Retry 1

And the result of pass #3:

rescued:     3000 GB,  errsize:   1961 kB,  current rate:        0 B/s
   ipos:     2527 GB,   errors:     174,    average rate:        3 B/s
   opos:     2527 GB,    time since last successful read:       7 m
Finished

So we got 1998kBytes lost down to 1961kBytes. Great! Now it was time to check SMART attributes for changes:

-  5 Reallocated_Sector_Ct   0x0033   094   094   036    Pre-fail  Always       -       8856
+  5 Reallocated_Sector_Ct   0x0033   094   094   036    Pre-fail  Always       -       8864
 
-187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       52554
+187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       56904
 
-197 Current_Pending_Sector  0x0012   077   001   000    Old_age   Always       -       3904
+197 Current_Pending_Sector  0x0012   077   001   000    Old_age   Always       -       3832
 
-198 Offline_Uncorrectable   0x0010   077   001   000    Old_age   Offline      -       3904
+198 Offline_Uncorrectable   0x0010   077   001   000    Old_age   Offline      -       3832

Rather than analyse all the attributes like in my previous posts, it should be obvious what the results here are. What I found cute was how this time around attribute 5 incremented by 8 LBAs -- while simultaneously attributes 197 and 198 decreased again.

· actions · 2014-Jun-13 9:47 pm ·

koitsu

koitsu

MVM

2014-Jun-13 10:08 pm

Pass #4 is currently running. I won't do any more passes past this since I think this is about as good as it's going to get, but I did want to mention that I've increased the kernel/CAM timeouts and retries back to their stock defaults of 30 seconds and 6 retries:

root@icarus:/home/jdc/rouge # sysctl kern.cam.ada.default_timeout=30
kern.cam.ada.default_timeout: 15 -> 30
 
root@icarus:/home/jdc/rouge # sysctl kern.cam.ada.retry_count=4
kern.cam.ada.retry_count: 2 -> 4

Here's where things stand right now (not much of interest):

root@icarus:/home/jdc/rouge # ddrescue -v -f -r3 -b 4096 /dev/ada6 /dev/ada7 ddrescue.log
 
GNU ddrescue 1.17
About to copy 3000 GBytes from /dev/ada6 to /dev/ada7
    Starting positions: infile = 0 B,  outfile = 0 B
    Copy block size:  16 sectors       Initial skip size: 16 sectors
Sector size: 4096 Bytes
 
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:     3000 GB,  errsize:   1961 kB,  errors:     174
Current status
rescued:     3000 GB,  errsize:   1961 kB,  current rate:        0 B/s
   ipos:     2249 GB,   errors:     174,    average rate:        0 B/s
   opos:     2249 GB,    time since last successful read:    49.4 m
Retrying bad sectors... Retry 1

So I'll patiently wait until all 3 ddrescue retries are done and then haul the NEW drive up to my Windows machine + begin the whole ordeal I documented in my first post (re: using a VM with Windows 7 and all that) in hopes I can get somewhere with that. If not, I have an alternate solution that involves use of one of my spare/unused servers downstairs (will have to install Windows 7 on a spare/unrelated drive but I don't care), but I hope to be able to do this within a VM.

· actions · 2014-Jun-13 10:08 pm ·

koitsu

koitsu

MVM

2014-Jun-14 3:02 am

Pass #4 has finally completed. Looks like it was able to read another 8KBytes, so the final lost amount is 1953kB plus whatever may have gotten remapped prior to the drive reaching me.

Current status
rescued:     3000 GB,  errsize:   1953 kB,  current rate:        0 B/s
   ipos:     2527 GB,   errors:     173,    average rate:        0 B/s
   opos:     2527 GB,    time since last successful read:     3.9 h
Finished

Relevant SMART attribute differences between pass #3 and pass #4:

-187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       56904
+187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       64061
 
-197 Current_Pending_Sector  0x0012   077   001   000    Old_age   Always       -       3832
+197 Current_Pending_Sector  0x0012   077   001   000    Old_age   Always       -       3816
 
-198 Offline_Uncorrectable   0x0010   077   001   000    Old_age   Offline      -       3832
+198 Offline_Uncorrectable   0x0010   077   001   000    Old_age   Offline      -       3816

Either tomorrow or Sunday I'll attempt the Windows-7-VM-based file recovery and see what the results are. That ought to take forever; 3TB of LBAs to read, rebuild an MFT (if it can), yadda yadda... whee...

P.S. Irony: replacement SATA controller arrived in the mail today. Guess I get to test the ports on that before doing the file recovery...

· actions · 2014-Jun-14 3:02 am ·

koitsu

koitsu

MVM

2014-Jun-14 6:53 pm

Today's efforts in recovery so far haven't been good. The methodology I figured might work (using a Windows 7 VM and directly attaching the raw 3TB disk to the VM itself) didn't work. (I've done this before with USB sticks and it works fine, so this is something I can't easily deal with)

It seems some parts of the Windows XP kernel see the drive (for example it shows up in Device Manager), but some programs don't see it, while other programs (like HD Tune Pro) see it but get the capacity wrong. The Windows software I use for data recovery also sees the drive but it's limited to 2TB in capacity (argh!). Case in point:

D:\VirtualBox VMs>"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands createrawvmdk -filename new.vmdk -rawdisk \\.\PhysicalDrive2
VBoxManage.exe: error: Detected size of raw disk '\\.\PhysicalDrive2' is <NULL>, an invalid value
VBoxManage.exe: error: The raw disk vmdk file was not created
 
D:\VirtualBox VMs>"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands createrawvmdk -filename new.vmdk -rawdisk \\.\PhysicalDrive4
VBoxManage.exe: error: Cannot open the raw disk '\\.\PhysicalDrive4': VERR_FILE_NOT_FOUND
VBoxManage.exe: error: The raw disk vmdk file was not created

D:\>"D:\Util\dd for Win32\dd" --list
rawwrite dd for windows version 0.6beta3.
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL Version 2.
 
{...}
 
NT Block Device Objects
\\?\Device\CdRom0
  size is 2048 bytes
\\?\Device\CdRom1
  size is 4552669184 bytes
\\?\Device\Harddisk0\Partition0
  link to \\?\Device\Harddisk0\DR0
  Fixed hard disk media. Block size = 512
  size is 250059350016 bytes
\\?\Device\Harddisk0\Partition1
  link to \\?\Device\HarddiskVolume1
\\?\Device\Harddisk1\Partition0
  link to \\?\Device\Harddisk1\DR1
  Fixed hard disk media. Block size = 512
  size is 1000204886016 bytes
\\?\Device\Harddisk1\Partition1
  link to \\?\Device\HarddiskVolume2
\\?\Device\Harddisk2\Partition0
  link to \\?\Device\Harddisk2\DR25
  Fixed hard disk media. Block size = 512

HD Tune Pro: ST3000DM001-1CH166 Information
 
Firmware version : CC29
Serial number    :             W1F4YCJH
Capacity         : 2199.0 gB (2048.0 GB)
Buffer size      : n/a
Sector size      : 512 bytes
Standard         : SATA 3 (6.0 Gb/s)
Supported mode   : UDMA Mode 6
Current mode     : UDMA Mode 6
Avergae speed    : 109 MB/s
Rotation speed   : 7200 RPM
 
S.M.A.R.T                    : yes
48-bit Address               : yes
Read Look-Ahead              : yes
Write Cache                  : yes
Host Protected Area          : yes
Device Configuration Overlay : yes
Firmware Upgradable          : yes
Automatic Acoustic Management: no
Power Management             : yes
Advanced Power Management    : yes
Interface Power Management   : no
Power-up in Standby          : yes
Security Mode                : yes
SCT Tables                   : no
Native Command Queuing (NCQ) : yes
Trim                         : no

Why I tried both PhysicalDrive2 and PhysicalDrive4 -- because according to parts of the Windows GUI, the "Location" of the 3TB drive is "Location: 4" (it's on the last SATA port on my system). My OS drive is "Location: 0" and my secondary drive is "Location: 1", hence me trying different things. It's clearly PhysicalDrive2 though, since the error there is "I can't figure out what the capacity is", not "there's no such device/file".

In the case of dd the disk is \\?\Device\Harddisk2\Partition0 but you can see it clearly doesn't show the capacity. So my guess is Windows XP really doesn't like this and is confused quite heavily. Bummer.

So I'm going to have to resort to my other idea, which is building a temporary Windows 7 system on a spare box (server) I have downstairs in my garage. Not exactly how I wanted to spend my day, but hey that's how it goes. That should work, but complicates me restoring files to alternate media.

· actions · 2014-Jun-14 6:53 pm ·

koitsu

koitsu

MVM

2014-Jun-14 9:36 pm

So far so good -- upon installing Windows 7 on a spare SSD, I'm able to use GetDataBack for NTFS to restore my colleague's files. Software is chugging along...

I was happy to see that the GPT table was actually intact (at least enough for the recovery software to discern LBA offsets of everything), which also meant I could use the "Quick scan" option (hopefully -- it might fail later, but we'll see) and recover all the known files.

The destination drive for the restoration happens to be my WD MyPassport drive, since it was the easiest thing for me to haul around with me at the time, and had about 800GB of free space on it. The box I installed W7 on doesn't support USB 3.0 but I figure 40MBytes/second is acceptable -- besides, GetDataBack isn't exactly the fastest thing on the planet but it gets the job done in this situation.

Regarding integrity of restored data: naturally I can't guarantee this in any way (meaning some files may be partially corrupt, containing zeros where there should be data) -- at least 1953kBytes worth (assuming a file or directory happened to utilise those unreadable LBAs). There isn't anything I can do about that, sadly, but at least he'll get most of his files back. Integrity verification is something he'll have to do, and can only be done if he has valid originals to compare checksums against... but that's not my problem/responsibility.

Now I just gotta wait to see how much data gets copied over to my USB drive, then reformat + repartition that 3TB drive (since that'll be the final destination of my colleague's data + what I'll be shipping back to him) + call it a day.

· actions · 2014-Jun-14 9:36 pm ·

norwegian
Premium Member
join:2005-02-15
Outback

1 edit

norwegian

Premium Member

2014-Jun-16 7:31 am

said by koitsu:

I was happy to see that the GPT table was actually intact (at least enough for the recovery software to discern LBA offsets of everything), which also meant I could use the "Quick scan" option (hopefully -- it might fail later, but we'll see) and recover all the known files.

said by koitsu:

It seems some parts of the Windows XP kernel see the drive (for example it shows up in Device Manager), but some programs don't see it, while other programs (like HD Tune Pro) see it but get the capacity wrong. The Windows software I use for data recovery also sees the drive but it's limited to 2TB in capacity

This has been quite an interesting read, and I still have to think through a couple of parts.
However on your mention of XP kernel in the VM, I was always lead to believe it was not good to attempt XP and larger than 2.2TB drives, all for or reasons you have discussed already, and others here too, however XP is out-dated in so many ways with the newest hardware.

»support.microsoft.com/kb/2581408

Hope I haven't overstepped the mark wanting to discuss certain parts of the steps.
I did take note of the sector size of 4096 for the record for large external drives.
You never realize how important some steps are until someone points out the correct process (and as detailed as this).

· actions · 2014-Jun-16 7:31 am ·

koitsu
MVM
join:2002-07-16
Mountain View, CA

Netgear CM1000
Ubiquiti EdgeRouter X SFP
Ubiquiti Unifi UAP-AC-LITE

koitsu

MVM

2014-Jun-16 6:02 pm

To expand on my theory of using a Windows 7 VM (guest) with a raw attached disk on XP (host):

It was my hope that the underlying SATA controller + AHCI driver + parts of the kernel would still support LBA addressing that exceeded 2TB in count. I was always left with the impression that XP couldn't support partitions 2TB+ in size -- but a partition is not the same thing as a disk.

The KB article in question sort of conflates the two things. GPT/MBR are partitioning schemes and while there is a relationship between those and actual disk capacity (with regards to limits of MBR, etc.), they really don't have anything to do with the actual drive behaviour. ATA IDENTIFY, for example (see smartctl outputs I provided above), will still return all the "real" data for the drive (what the drive itself returns the controller) -- it's that the underlying OS drivers appear to make a mess of things.

The SATA controller does support such disks (i.e. if I was to install Windows 7 64-bit on the system itself, it'd work just fine with 2TB+ disks).

The AHCI driver or XP kernel (I have no idea which), however, does not appear to support such disks. Such disks show up in a very uncomfortable way -- the disk/device does show up in Windows, but the capacity information is unavailable (ex. VirtualBox bitching about NULL capacity), while other parts of the kernel "cap" the disk at 2TB (ex. how HD Tune Pro was able to actually read/write to the disk).

It was still an interesting educational experience and just goes to show that even with virtualisation you're still stuck with the underlying limitations of the OS and its drivers. I was hoping the underlying driver and OS would be "more transparent" in their passthrough of stuff to the VM, but nope...

· actions · 2014-Jun-16 6:02 pm ·

JoelC707
Premium Member
join:2002-07-09
Lanett, AL

JoelC707

Premium Member

2014-Jun-17 10:55 pm

I just found this thread and have already tagged it as a post to follow (this reply will do that too lol). I was curious if the raw disk idea would work, I had my doubts about it but interested to see that it's a host OS issue. That does lead me to a possible solution for you if you attempt something like this in the future.

Does your XP system's hardware support IOMMU or Vt-d/AMD-Vi? I don't know much about Virtualbox but I know the virtualization engine must support it as well (ESX does, and a Google search reveals that Virtualbox does and possibly vmware workstation). Assuming your hardware and virtualization engine support it, you could pass through the entire SATA card to the guest OS and any attached drives bypassing XP altogether.

· actions · 2014-Jun-17 10:55 pm ·

koitsu
MVM
join:2002-07-16
Mountain View, CA

Netgear CM1000
Ubiquiti EdgeRouter X SFP
Ubiquiti Unifi UAP-AC-LITE

1 edit

koitsu

MVM

2014-Jun-18 1:04 am

said by JoelC707:

Does your XP system's hardware support IOMMU or Vt-d/AMD-Vi? I don't know much about Virtualbox but I know the virtualization engine must support it as well (ESX does, and a Google search reveals that Virtualbox does and possibly vmware workstation). Assuming your hardware and virtualization engine support it, you could pass through the entire SATA card to the guest OS and any attached drives bypassing XP altogether.

The system does, yes. But I thought for those features the work the OS had to support some degree of passthrough (even on a generic level) for IOMMU or Intel VT-d to work. Am I wrong? (It's very possible I am, I'm still quite "new" to this virtualisation thing).

As for the SATA controller/card mentioned in the thread -- yeah, that card is intended for my FreeBSD box. I didn't think about it, but yes, I suppose I could have pulled it out and put it into my XP box and then somehow tried to get VirtualBox (or VMware Workstation (I do own a license for that but I switched to VirtualBox last year)) to actually attach to the PCI card directly + bypass XP drivers entirely. If that's possible I'd love to learn how to do it sometime!

Since the hardware used in my above recovery has been mentioned, I figured I'd just list all the stuff on each of the systems (my XP workstation and my FreeBSD box) so people know what all was involved:

Windows XP Pro SP3 box:

Case - Fractal Design Define R4
MB - Gigabyte GA-Z77-DS3H rev 1.1
CPU - Intel Core i7-2600K
HSF - Noctua NH-U9B SE2
RAM - CT2KIT102464BA1339, 8GB DDR3
RAM - CT2KIT102464BA1339, 8GB DDR3
VGA - MSI N760 TF 2GD5/OC
Sound - Creative Audigy SE (PCI)
Disk - Samsung 840 MZ-7TD250BW SSD, 250GB (C:)
Disk - WD1003FZEX (D:)
DVD - LiteOn iHAS524B (E:)
PSU - Corsair Professional Series Gold AX850

FreeBSD 9.3 box:

Case - Fractal Design Define R3
MB - Supermicro X7SBA
CPU - Intel Core 2 Quad Q9550
HSF - Noctua NH-U9B SE2
RAM - CT2KIT25672AA800, 4GB ECC
RAM - CT2KIT25672AA80E, 4GB ECC
Disk - ada0: Intel 320 SSD, 80GB
Disk - ada1: WD3003FZEX (ZFS single, backups)
Disk - ada2: WD1003FZEX (ZFS RAID 1+0, data)
Disk - ada3: WD1003FZEX (ZFS RAID 1+0, data)
Disk - ada4: WD1003FZEX (ZFS RAID 1+0, data)
Disk - ada5: WD1003FZEX (ZFS RAID 1+0, data)
HBA - Syba SY-PCI40010 (4-port SATA300 SilI 3124)
PSU - Antec TruePower New 650W

P.S. -- Yes I know the XP box has 16GB even though XP only supports 4GB. Like I said, I do have plans to upgrade sometime. I'm kinda waiting for Microsoft to pull their heads out of their ass regarding Windows 8 (including 8.1 -- yes I've tried it) and how they destroyed the taskbar + Start/Programs menu. Otherwise I'd actually use it (and I don't want third-party replacements). I just cannot stand how they botched/broke horribly bad the GUI adjustment features in Windows 7. Yes Windows 8 offers even less but at least the UI there is "flat" and semi-minimal and I could get used to that. But there's so much of the rest of the UI that is just awful (I want a workstation GUI OS, not something intended for a touchscreen!).

· actions · 2014-Jun-18 1:04 am ·

JoelC707
Premium Member
join:2002-07-09
Lanett, AL

JoelC707

Premium Member

2014-Jun-18 2:06 pm

Regarding Windows 8, I couldn't agree more. I want Aero back honestly and I'd like a real start menu back. I know you don't want 3rd party solutions but honestly Start8 is as close to Windows 7's start menu as the real thing. It's the best $5 I've spent LOL.

For XP with 16GB RAM, can you make use of PAE to address more than 4GB or has MS neutered the kernel to 4GB only? I know on 32-bit server OSes you weren't limited to 4GB, the actual memory registers on x86 hardware are 36-bit and tops out at 64GB IIRC. Might be worth looking into anyway. Of course you could always do XP 64-bit (bet I just made you shudder didn't I?) LOL!!

As for hardware passthrough, you need two things to sync: the hardware has to support it (BIOS/EFI as well as the CPU) and the virtualization engine has to support it. As long as those two are in sync, you could have passed the entire SATA card to the Windows 7 guest and bypassed XP's meddling completely. A cursory Google search showed one report that Virtualbox supports it but I've never used it so I can't help you there unfortunately. Running a 32-bit OS may hinder that, and XP itself hay still hinder that but in general that's what you need to make it work. It's worth a try IMO.

· actions · 2014-Jun-18 2:06 pm ·

Forums → Tech / Special Interest → Computer Hardware Help	« Just picked up two 4 TB WD HDDs... Smartmontools says this • (topic move) Need Help Installing A Bluetooth Dongle In A Vista Machine »