Topic 'Re: Problem with Open Solaris + disks' in forum 'All Things Unix' - dslreports.com

Re: Problem with Open Solaris + disks

Wed, 01 Jul 2009 00:24:54 EDT

galacticroot posted : I finally managed to move the files over to an Openfiler box and I just installed Debian on the system that has OpenSolaris.

I used smartmon tools to check both hard drives. Disk 1 is fine, but disk 2 has quite a few reallocated sectors (currently at 1996). I will definitely replace it if I get any more reallocated sectors. I suspect that the read failures created by disk 2 were not being handled well by OpenSolaris (or rather the SATA driver).

Linux seems to handle the errors correctly, although I haven't tried it out with the Xen VMs yet (I'm currently transferring the images).

Re: Problem with Open Solaris + disks

Fri, 29 May 2009 21:26:18 EDT

galacticroot posted : This was just a personal system which I built primarily as a NAS box, but also to run a couple VMs for various things. I was more or less running OpenSolaris strictly for the features of ZFS. I thought I would end up adding a lot more space and using some other features of ZFS which I never ended up using.

I will look at btrfs. It sounds like it could be very nice in the future.

For now, OpenFiler looks like it will work well enough for NAS, and Linux or BSD will be good for running the VMs.

Re: Problem with Open Solaris + disks

Fri, 29 May 2009 13:45:20 EDT

koitsu posted :

said by beerbum:

huh??.. maybe you missed it.. the OP is running OpenSolaris (SunOS 5.11), not Solaris (SunOS 5.10).. ...

Well colour me stupid. For the longest while now I've been under the impression that Solaris 10 (5.10) was in fact OpenSolaris. Good lord, there's something seriously wrong when an administrator of machines doesn't even know what the official title of his OS is.

I think I might save this thread to remind me of my stupid moments.

Thanks for clearing that up for me -- I appreciate it. (Damn you Sun...)
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

Re: Problem with Open Solaris + disks

Fri, 29 May 2009 07:12:40 EDT

beerbum posted :

said by koitsu:

Given that my place of employment has relied on Solaris 8 through 10 (~90% of our machines are using 10 at this point) for 5+ years now, my experience is quite the opposite.

huh??.. maybe you missed it.. the OP is running OpenSolaris (SunOS 5.11), not Solaris (SunOS 5.10).. I would not recommend anyone run OpenSolaris in a production system. Heck no admin worth anyone would recommend that.. Comparing OpenSolaris to Solaris, the SATA support is much more robust than the same in OpenSolaris.. While the new features and whatnot in OS do make it into production Solaris, one should consider OpenSolaris as a beta product.

Hell I'm pretty sure even Sun does not recommend using OpenSolaris in a production environment..

Use whatever OS gets the job done. If that's Linux, great. If that's Solaris 10, great. If that's OS/2, I'll punch you. ;-)

I started out (*nix) adminning on IBM RS2K's.. guess what I used on my peecee - yup OS/2 Warp.. In fact, my Rexx-fu is what helped me land my first gig as an admin..

Re: Problem with Open Solaris + disks

Fri, 29 May 2009 06:50:06 EDT

koitsu posted :

said by galacticroot:

Okay, I tried changing to AHCI mode, but the Open Solaris drivers seem to have an issue with that and can't mount the root filesystem. Returning to the original mode allows it to boot.

This isn't a "driver issue" -- it's probably that the device label (and all underlying filesystem slices) has changed. The disks probably won't be named "c3d0s0" any more, but could be "cXd0s0" where X is some new number, or, are labelled "sdX".

We have the same problem on FreeBSD, and Windows requires an entire reinstall. :-)

ZFS should be able to cope with the device names changing.

said by beerbum:

may I ask a stupid question.. why are you using OpenSolaris to begin with? if this is a mission critical server, heck even if it's a mission optional machine, I do not recommend running OpenSolaris.

Sun's production Solaris costs just the same and is (IMO) the more reliable route to go in a business setting.

Given that my place of employment has relied on Solaris 8 through 10 (~90% of our machines are using 10 at this point) for 5+ years now, my experience is quite the opposite. I'm talking multiple thousands of machines, all x86 (at this point), and all are mission + time-critical (all production, and are involved with VoIP + IVR; 2-3 second "stalls" or other oddities a server might encounter result in horrible caller experience, and we can't have that).

We open SunSolve cases for strange things we encounter and Sun is responsive. I'm not saying "you're wrong", I'm saying my experience is entirely different. Of course, low-level administration of devices and hardware on Solaris is significantly better on Sparc (and I do mean significantly), but x86 is standard these days.

Btrfs is the only thing on Linux that even remotely behaves like the OP's ZFS configuration. I would HIGHLY recommend the OP read the following thread (and news article!):

»Chris Mason Interview - BTRFS Founder & Lead Developer

Use whatever OS gets the job done. If that's Linux, great. If that's Solaris 10, great. If that's OS/2, I'll punch you. ;-)
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

Re: Problem with Open Solaris + disks

Thu, 28 May 2009 20:53:05 EDT

beerbum posted : may I ask a stupid question.. why are you using OpenSolaris to begin with? if this is a mission critical server, heck even if it's a mission optional machine, I do not recommend running OpenSolaris.

Sun's production Solaris costs just the same and is (IMO) the more reliable route to go in a business setting.

Re: Problem with Open Solaris + disks

Thu, 28 May 2009 20:22:00 EDT

galacticroot posted : Okay, I tried changing to AHCI mode, but the Open Solaris drivers seem to have an issue with that and can't mount the root filesystem. Returning to the original mode allows it to boot.

The hardware I find that definitely works well with Open Solaris seems to all be higher end than I can afford right now. I'm going to switch the file storage over to a separate NAS setup running OpenFiler, and replace Open Solaris with Linux on this system and run the VMs on that.

Re: Problem with Open Solaris + disks

Tue, 26 May 2009 04:00:42 EDT

koitsu posted : The errors indicate LBA read failures on Disk 0 and Disk 2. The fact that the errors occurred (according to your logs) within 5 seconds of one another is a little suspicious.

I'd recommend looking at SMART stats on both of these disks to see if you can discern what's going on. However, Solaris 10 doesn't offer an API for obtaining SMART data from ATA/SATA disks -- only SCSI. So smartmontools won't help you here.

SMART statistics will help determine if the disks themselves are actually witnessing bad blocks (and remapping them), or if the controller is responsible.

But you'll have to boot into another OS (FreeBSD, Linux, Windows) to get SMART statistics with smartmontools.

If you bought both ATA/SATA disks at the same time, it's possible both have problems. Otherwise, if both Disk 0 and Disk 2 are on the same physical controller (which they appear to be), I'd recommend the following:

1) Are these internal SATA drives? If so, replace the SATA cables. You should only have to do this once. If the problem recurs, it's not the cables.

2) If the SATA drives are external via eSATA, are you using a SATA-to-eSATA adapter bracket (e.g. cable runs between the onboard SATA controller to the backplane, with an eSATA connector)? If so, get rid of it -- chances are you're exceeding SATA cable length. Buy yourself a real PCI/PCIe-based eSATA controller.

3) If your motherboard BIOS has support for AHCI, enable it in the BIOS. I have no idea if the SB780 supports AHCI or not. You should always go with AHCI if given the chance, especially on server systems.

4) Opening a SunSolve case, as the problem could be Solaris having buggy support for the SB780 SATA controller (do not even for a moment think all SATA controllers are alike). My money is on this being the root cause.

zpool status indicates that the disks are literally falling off the SATA bus ("cannot open"). There's got to be other messages on your console to indicate that, not just sector read errors.

Regarding hardware compatibility:

I've never seen Solaris 10 behave flawlessly with SATA disks. At work we have Intel ICHx-based controllers with both SATA SSDs and standard drives. Our machines work fine -- no problems -- except during boot-up we see some CMD errors to the disks (iostat -e shows the same). There's nothing wrong with the SATA controllers here -- it's that Solaris is trying to issue SCSI commands to SATA disks, and the SCSISATA layer doesn't properly remap the commands (or remove ones not supported). Thus, the errors seen are false. We use ZFS and we don't have problems with the disks falling off the bus, though -- the only errors occur during boot.

If I had to make a recommendation, I'd say go with a motherboard that provides an on-board Intel ICHx controller. ICH7 would probably be best, possibly ICH9 (newer). I don't think Solaris 10 has decent support for ICH10 yet (too new).
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

Re: Problem with Open Solaris + disks

Mon, 25 May 2009 17:11:08 EDT

galacticroot posted : Now the other disk in rpool went offline and doesn't come back up even when I reboot for some reason. I probably have to power cycle the box, not just reboot.

I am willing to replace the controller with a PCIe one, but I'm not sure what Open Solaris supports well. Are there any PCIe SATA cards with >=4 ports that it supports well?

I am even willing to replace the motherboard if that will help. I just don't want to spend a lot of money on hardware that may not even work well with it.

Re: Problem with Open Solaris + disks

Mon, 25 May 2009 16:15:53 EDT

beerbum posted : if you were running SCSI I'd suggest replacing the cable

can you move one of the discs (Disk 2) to a different controller? or even better attach an additional drive to a different controller.

it's possible you could be overloading the controller - something easy to do with the poor sata drivers that are packaged.

Problem with Open Solaris + disks

Mon, 25 May 2009 12:20:15 EDT

galacticroot posted : I've got a server using an SB780 onboard SATA controller running Open Solaris. I am using it as a file server and to run several Xen VMs. My problems started when I recently added a VM to use as a mail server. I allocated a 40GB zfs volume on rpool for its disk image and installed Debian on it. Everything went as expected and I got the server set up, then the problems started.

Usually, what will happen is that the VM will crash, accompanied by messages like this on the host machine:

May 25 11:22:51 hydrogen xpv_psm: [ID 803547 kern.info] xVM_psm: ide (ata) instance 2 vector 0xe ioapic 0x2 intin 0xe is bound to cpu 1May 25 11:22:51 hydrogen xpv_psm: [ID 803547 kern.info] xVM_psm: ide (ata) instance 3 vector 0xf ioapic 0x2 intin 0xf is bound to cpu 0May 25 11:22:51 hydrogen genunix: [ID 408114 kern.info] /xpvd/xnb@4,0 (xnbo0) onlineMay 25 11:22:51 hydrogen xpv_psm: [ID 803547 kern.info] xVM_psm: ide (ata) instance 2 vector 0xe ioapic 0x2 intin 0xe is bound to cpu 1May 25 11:22:51 hydrogen xpv_psm: [ID 803547 kern.info] xVM_psm: ide (ata) instance 3 vector 0xf ioapic 0x2 intin 0xf is bound to cpu 0May 25 11:22:51 hydrogen mac: [ID 469746 kern.info] NOTICE: vnic1009 registeredMay 25 11:23:32 hydrogen genunix: [ID 698548 kern.notice] ata_disk_start: select failedMay 25 11:24:02 hydrogen last message repeated 6 timesMay 25 11:24:02 hydrogen scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@11/ide@0 (ata0):May 25 11:24:02 hydrogen        timeout: early timeout, target=0 lun=0May 25 11:24:02 hydrogen gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@11/ide@0/cmdk@0,0 (Disk0):May 25 11:24:02 hydrogen        Error for command 'read sector' Error Level: InformationalMay 25 11:24:02 hydrogen gda: [ID 107833 kern.notice]   Sense Key: aborted commandMay 25 11:24:02 hydrogen gda: [ID 107833 kern.notice]   Vendor 'Gen-ATA ' error code: 0x3May 25 11:24:08 hydrogen genunix: [ID 698548 kern.notice] ata_disk_start: select failedMay 25 11:24:18 hydrogen last message repeated 2 timesMay 25 11:24:18 hydrogen scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@11/ide@0 (ata0):May 25 11:24:18 hydrogen        timeout: early timeout, target=1 lun=0May 25 11:24:23 hydrogen genunix: [ID 698548 kern.notice] ata_disk_start: select failedMay 25 11:24:23 hydrogen gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@11/ide@0/cmdk@1,0 (Disk2):May 25 11:24:23 hydrogen        Error for command 'read sector' Error Level: InformationalMay 25 11:24:23 hydrogen gda: [ID 107833 kern.notice]   Sense Key: aborted commandMay 25 11:24:23 hydrogen gda: [ID 107833 kern.notice]   Vendor 'Gen-ATA ' error code: 0x3May 25 11:24:28 hydrogen genunix: [ID 698548 kern.notice] ata_disk_start: select failedMay 25 11:24:37 hydrogen scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@11/ide@0 (ata0):

Zpool will typically show problems:

  pool: rpool state: DEGRADEDstatus: One or more devices could not be opened.  Sufficient replicas exist for        the pool to continue functioning in a degraded state.action: Attach the missing device and online it using 'zpool online'.   see: http://www.sun.com/msg/ZFS-8000-2Q scrub: scrub completed after 1h48m with 0 errors on Sun May 17 22:55:01 2009config:         NAME        STATE     READ WRITE CKSUM        rpool       DEGRADED     0     0     0          mirror    DEGRADED     1     0     0            c3d0s0  UNAVAIL      8 32.7K     0  cannot open            c4d0s0  ONLINE       1     0     0 errors: No known data errors   pool: tank state: ONLINEstatus: One or more devices has experienced an unrecoverable error.  An        attempt was made to correct the error.  Applications are unaffected.action: Determine if the device needs to be replaced, and clear the errors        using 'zpool clear' or replace the device with 'zpool replace'.   see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver completed after 0h0m with 0 errors on Mon May 25 09:44:42 2009config:         NAME        STATE     READ WRITE CKSUM        tank        ONLINE       0     0     0          mirror    ONLINE       0     0     0            c3d1    ONLINE       0     4     0  85K resilvered            c4d1    ONLINE       0     0     0  54K resilvered

If I online any offline devices, clear the errors, restart the VM, and scrub the pools, everything works fine for a while, until it happens again. It seems to happen when a moderate, but not necessarily heavy IO load occurs.

I am thinking this is a problem with the disk controller, not the disks. Although the first disk gets far more errors than the others, sometimes the other disks are the ones to have failures and the first disk continues working.

This is the first time a disk actually went offline. I couldn't successfully online it without rebooting.

Has anyone had this problem before? Is there likely a software fix for this, or should I just get a new SATA controller (provided it isn't actually the disks)?