Re: [FreeBSD] Disk read corruption issues on server. in All Things Unix http://www.dslreports.com/forum/r18389651 en Fri, 27 Nov 2009 15:06:48 EDT Fri, 27 Nov 2009 15:06:48 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18524663 sporkme : Have you had the 3Ware run a full verify on the array?]]> http://www.dslreports.com/forum/remark,18524663 Mon, 18 Jun 2007 00:47:35 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18523141 galacticroot : Problem solved. There was a scratch and corrosion across some of the traces on an unused PCI riser card.]]> http://www.dslreports.com/forum/remark,18523141 Sun, 17 Jun 2007 19:00:09 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18396113 galacticroot : It is occurring at the same space in 64 bit blocks, and the PCI bus is 64 bit. I would assume that that an error on that would cause other issues, but maybe not. The RAID controller has error corrected memory, so I assume the problem would be detected if it was that.

Interestingly, by far the most common change is from f7 to ff. Assuming the /dev/urandom (which the file is from) produces reasonably distributed data, the corrupted bit seems to be highly dependent on the bits around it. This may indicate that the problem is caused by a minor voltage leakage between adjacent bus lines. Perhaps the PCI connection needs cleaning or reseating?

Here is a more extensive summary of the file changes. This is a different read and does not correspond with the XORed file above.
2416204: (d7 != df) ^= 8 BIN: 11010111 11011111
3bfbe04: (ff != f7) ^= 8 BIN: 11111111 11110111
4012304: (77 != 7f) ^= 8 BIN: 01110111 01111111
4c42804: (7f != 77) ^= 8 BIN: 01111111 01110111
51a4904: (ff != f7) ^= 8 BIN: 11111111 11110111
5f2e004: (f7 != ff) ^= 8 BIN: 11110111 11111111
6619204: (f7 != ff) ^= 8 BIN: 11110111 11111111
7cf6404: (f7 != ff) ^= 8 BIN: 11110111 11111111
7ef3f04: (f7 != ff) ^= 8 BIN: 11110111 11111111
80f0c04: (f7 != ff) ^= 8 BIN: 11110111 11111111
8c92004: (ff != f7) ^= 8 BIN: 11111111 11110111
8ce5104: (ff != f7) ^= 8 BIN: 11111111 11110111
9ec5304: (ff != f7) ^= 8 BIN: 11111111 11110111
aabcc04: (7f != 77) ^= 8 BIN: 01111111 01110111
b860594: (f7 != ff) ^= 8 BIN: 11110111 11111111
c296f04: (f7 != ff) ^= 8 BIN: 11110111 11111111
c4b5a04: (ef != e7) ^= 8 BIN: 11101111 11100111
c9ddc04: (7f != 77) ^= 8 BIN: 01111111 01110111
cc9b204: (f7 != ff) ^= 8 BIN: 11110111 11111111
cdb7204: (f7 != ff) ^= 8 BIN: 11110111 11111111
e060f04: (e7 != ef) ^= 8 BIN: 11100111 11101111
e5b9504: (d7 != df) ^= 8 BIN: 11010111 11011111
f2a570c: (df != d7) ^= 8 BIN: 11011111 11010111
f52cc04: (f7 != ff) ^= 8 BIN: 11110111 11111111
10540904: (77 != 7f) ^= 8 BIN: 01110111 01111111
107a0904: (d7 != df) ^= 8 BIN: 11010111 11011111
125c5504: (ef != e7) ^= 8 BIN: 11101111 11100111
12836604: (f6 != fe) ^= 8 BIN: 11110110 11111110
12a61204: (d7 != df) ^= 8 BIN: 11010111 11011111
12d4ce04: (ff != f7) ^= 8 BIN: 11111111 11110111
132a4e04: (ff != f7) ^= 8 BIN: 11111111 11110111
14210404: (e7 != ef) ^= 8 BIN: 11100111 11101111
14c32c04: (f5 != fd) ^= 8 BIN: 11110101 11111101
162c0504: (ff != f7) ^= 8 BIN: 11111111 11110111
16801004: (f7 != ff) ^= 8 BIN: 11110111 11111111
176fdb04: (ff != f7) ^= 8 BIN: 11111111 11110111
18904d04: (ff != f7) ^= 8 BIN: 11111111 11110111
1b649504: (f7 != ff) ^= 8 BIN: 11110111 11111111
1ca7cb04: (77 != 7f) ^= 8 BIN: 01110111 01111111
1d460004: (77 != 7f) ^= 8 BIN: 01110111 01111111
1d845604: (fb != f3) ^= 8 BIN: 11111011 11110011
1eb84d04: (ff != f7) ^= 8 BIN: 11111111 11110111
20672904: (d7 != df) ^= 8 BIN: 11010111 11011111
20df9f04: (ff != f7) ^= 8 BIN: 11111111 11110111
20f04304: (ff != f7) ^= 8 BIN: 11111111 11110111
212a4b04: (ff != f7) ^= 8 BIN: 11111111 11110111
221aa904: (77 != 7f) ^= 8 BIN: 01110111 01111111
22765104: (6f != 67) ^= 8 BIN: 01101111 01100111
23659e04: (f6 != fe) ^= 8 BIN: 11110110 11111110
236da904: (77 != 7f) ^= 8 BIN: 01110111 01111111
25744604: (ff != f7) ^= 8 BIN: 11111111 11110111
25cd1404: (e7 != ef) ^= 8 BIN: 11100111 11101111
25e8cf04: (ff != f7) ^= 8 BIN: 11111111 11110111
2603bf04: (f7 != ff) ^= 8 BIN: 11110111 11111111
262f9c04: (77 != 7f) ^= 8 BIN: 01110111 01111111
269bca04: (df != d7) ^= 8 BIN: 11011111 11010111
274b0904: (f5 != fd) ^= 8 BIN: 11110101 11111101
2750a704: (f7 != ff) ^= 8 BIN: 11110111 11111111
28082c04: (ff != f7) ^= 8 BIN: 11111111 11110111
28e7da04: (f7 != ff) ^= 8 BIN: 11110111 11111111
2a4e4f04: (df != d7) ^= 8 BIN: 11011111 11010111
2afe5904: (fe != f6) ^= 8 BIN: 11111110 11110110
2b70c904: (ff != f7) ^= 8 BIN: 11111111 11110111
2cbb3504: (d7 != df) ^= 8 BIN: 11010111 11011111
2d6637fc: (f7 != ff) ^= 8 BIN: 11110111 11111111
2dca4f04: (df != d7) ^= 8 BIN: 11011111 11010111
2dcab804: (ff != f7) ^= 8 BIN: 11111111 11110111
2f782b04: (e7 != ef) ^= 8 BIN: 11100111 11101111
2fd42604: (7f != 77) ^= 8 BIN: 01111111 01110111
2ffdd604: (ff != f7) ^= 8 BIN: 11111111 11110111
30763704: (f7 != ff) ^= 8 BIN: 11110111 11111111
3092c404: (ff != f7) ^= 8 BIN: 11111111 11110111
30cc1104: (f7 != ff) ^= 8 BIN: 11110111 11111111
30fb9804: (d7 != df) ^= 8 BIN: 11010111 11011111
31330704: (f7 != ff) ^= 8 BIN: 11110111 11111111
31d64004: (ff != f7) ^= 8 BIN: 11111111 11110111
32544304: (cf != c7) ^= 8 BIN: 11001111 11000111
32731104: (77 != 7f) ^= 8 BIN: 01110111 01111111
3309a50c: (f7 != ff) ^= 8 BIN: 11110111 11111111
335ba204: (f7 != ff) ^= 8 BIN: 11110111 11111111
34c9a404: (e7 != ef) ^= 8 BIN: 11100111 11101111
3527a604: (57 != 5f) ^= 8 BIN: 01010111 01011111
35481204: (f7 != ff) ^= 8 BIN: 11110111 11111111
35d85104: (7e != 76) ^= 8 BIN: 01111110 01110110
362f5304: (d7 != df) ^= 8 BIN: 11010111 11011111
3840da04: (ff != f7) ^= 8 BIN: 11111111 11110111
38a33004: (f5 != fd) ^= 8 BIN: 11110101 11111101
39185804: (ef != e7) ^= 8 BIN: 11101111 11100111
3a15a904: (d7 != df) ^= 8 BIN: 11010111 11011111
3a554704: (f7 != ff) ^= 8 BIN: 11110111 11111111
3c5f6f04: (d7 != df) ^= 8 BIN: 11010111 11011111
3d17ae04: (77 != 7f) ^= 8 BIN: 01110111 01111111
3d492f04: (f7 != ff) ^= 8 BIN: 11110111 11111111
3fad7604: (ff != f7) ^= 8 BIN: 11111111 11110111
3fb65104: (ff != f7) ^= 8 BIN: 11111111 11110111
3fe71e04: (57 != 5f) ^= 8 BIN: 01010111 01011111
40204404: (ff != f7) ^= 8 BIN: 11111111 11110111
40b79b04: (d7 != df) ^= 8 BIN: 11010111 11011111
4478db04: (f7 != ff) ^= 8 BIN: 11110111 11111111
45444c04: (ef != e7) ^= 8 BIN: 11101111 11100111
4544b804: (76 != 7e) ^= 8 BIN: 01110110 01111110
45745404: (ff != f7) ^= 8 BIN: 11111111 11110111
45bfaa04: (d7 != df) ^= 8 BIN: 11010111 11011111
45d12004: (77 != 7f) ^= 8 BIN: 01110111 01111111
45d20004: (55 != 5d) ^= 8 BIN: 01010101 01011101
477d88bc: (76 != 7e) ^= 8 BIN: 01110110 01111110
478d7004: (67 != 6f) ^= 8 BIN: 01100111 01101111
48945c04: (dd != d5) ^= 8 BIN: 11011101 11010101
48dc4d04: (ff != f7) ^= 8 BIN: 11111111 11110111
493cbf04: (ff != f7) ^= 8 BIN: 11111111 11110111
49afa204: (57 != 5f) ^= 8 BIN: 01010111 01011111
49d3bf04: (ff != f7) ^= 8 BIN: 11111111 11110111
49d84c04: (ff != f7) ^= 8 BIN: 11111111 11110111
4a763804: (ff != f7) ^= 8 BIN: 11111111 11110111
4a935704: (67 != 6f) ^= 8 BIN: 01100111 01101111
4c640504: (f7 != ff) ^= 8 BIN: 11110111 11111111
4c6c1d04: (f7 != ff) ^= 8 BIN: 11110111 11111111
4c7f2604: (ff != f7) ^= 8 BIN: 11111111 11110111
4d674204: (d7 != df) ^= 8 BIN: 11010111 11011111
4dc88704: (f7 != ff) ^= 8 BIN: 11110111 11111111
4ec13b04: (f7 != ff) ^= 8 BIN: 11110111 11111111
4ece0a04: (f7 != ff) ^= 8 BIN: 11110111 11111111
4fe22d04: (ff != f7) ^= 8 BIN: 11111111 11110111
501c4f04: (fd != f5) ^= 8 BIN: 11111101 11110101
503eb104: (f7 != ff) ^= 8 BIN: 11110111 11111111
51f04304: (d7 != df) ^= 8 BIN: 11010111 11011111
51f7c704: (ff != f7) ^= 8 BIN: 11111111 11110111
522e570c: (df != d7) ^= 8 BIN: 11011111 11010111
52320b04: (f7 != ff) ^= 8 BIN: 11110111 11111111
5271cc04: (7f != 77) ^= 8 BIN: 01111111 01110111
5274da04: (ef != e7) ^= 8 BIN: 11101111 11100111
53791f04: (f7 != ff) ^= 8 BIN: 11110111 11111111
538eb904: (df != d7) ^= 8 BIN: 11011111 11010111
53b9a304: (f7 != ff) ^= 8 BIN: 11110111 11111111
53d0bf04: (ff != f7) ^= 8 BIN: 11111111 11110111
555a5f04: (ff != f7) ^= 8 BIN: 11111111 11110111
560b8204: (f7 != ff) ^= 8 BIN: 11110111 11111111
560b8c04: (77 != 7f) ^= 8 BIN: 01110111 01111111
571fb704: (f7 != ff) ^= 8 BIN: 11110111 11111111
57a71404: (dd != d5) ^= 8 BIN: 11011101 11010101
587d1004: (77 != 7f) ^= 8 BIN: 01110111 01111111
590d3104: (d7 != df) ^= 8 BIN: 11010111 11011111
59b3a504: (d7 != df) ^= 8 BIN: 11010111 11011111
59f42304: (f7 != ff) ^= 8 BIN: 11110111 11111111
5bf24c04: (f7 != ff) ^= 8 BIN: 11110111 11111111
5c82bc04: (ff != f7) ^= 8 BIN: 11111111 11110111
5ed83c04: (df != d7) ^= 8 BIN: 11011111 11010111
5ee54b04: (f7 != ff) ^= 8 BIN: 11110111 11111111
5f75ba04: (f7 != ff) ^= 8 BIN: 11110111 11111111
5ff41004: (77 != 7f) ^= 8 BIN: 01110111 01111111
5ffc4504: (ff != f7) ^= 8 BIN: 11111111 11110111
6001a204: (d7 != df) ^= 8 BIN: 11010111 11011111
6086a604: (ff != f7) ^= 8 BIN: 11111111 11110111
612ec604: (ff != f7) ^= 8 BIN: 11111111 11110111
61cab504: (f7 != ff) ^= 8 BIN: 11110111 11111111
61e53204: (7f != 77) ^= 8 BIN: 01111111 01110111
62de0204: (f7 != ff) ^= 8 BIN: 11110111 11111111
638d0f04: (77 != 7f) ^= 8 BIN: 01110111 01111111
63ddb804: (f7 != ff) ^= 8 BIN: 11110111 11111111
66149304: (f7 != ff) ^= 8 BIN: 11110111 11111111
66504e04: (ff != f7) ^= 8 BIN: 11111111 11110111
66b42a04: (7f != 77) ^= 8 BIN: 01111111 01110111
693b2204: (f7 != ff) ^= 8 BIN: 11110111 11111111
69404e04: (ff != f7) ^= 8 BIN: 11111111 11110111
69592504: (f7 != ff) ^= 8 BIN: 11110111 11111111
69fb3004: (e7 != ef) ^= 8 BIN: 11100111 11101111
6a2dad04: (e7 != ef) ^= 8 BIN: 11100111 11101111
6b37bd04: (df != d7) ^= 8 BIN: 11011111 11010111
6b705b04: (ff != f7) ^= 8 BIN: 11111111 11110111
6b85fc04: (ff != f7) ^= 8 BIN: 11111111 11110111
6c724704: (7f != 77) ^= 8 BIN: 01111111 01110111
6c7b1104: (57 != 5f) ^= 8 BIN: 01010111 01011111
6d60cd04: (ff != f7) ^= 8 BIN: 11111111 11110111
6e54b004: (ff != f7) ^= 8 BIN: 11111111 11110111
6e7c2804: (f7 != ff) ^= 8 BIN: 11110111 11111111
6e932804: (57 != 5f) ^= 8 BIN: 01010111 01011111
6ed6f504: (df != d7) ^= 8 BIN: 11011111 11010111
6ef6c904: (f7 != ff) ^= 8 BIN: 11110111 11111111
6f9a9e04: (f7 != ff) ^= 8 BIN: 11110111 11111111
6fe7b204: (f7 != ff) ^= 8 BIN: 11110111 11111111
6feff704: (f7 != ff) ^= 8 BIN: 11110111 11111111
6ff2de04: (d7 != df) ^= 8 BIN: 11010111 11011111
707c0404: (ff != f7) ^= 8 BIN: 11111111 11110111
70c33504: (ff != f7) ^= 8 BIN: 11111111 11110111
7359c104: (c7 != cf) ^= 8 BIN: 11000111 11001111
74f21204: (d7 != df) ^= 8 BIN: 11010111 11011111
75033e04: (f7 != ff) ^= 8 BIN: 11110111 11111111
75905304: (af != a7) ^= 8 BIN: 10101111 10100111
75adc904: (f7 != ff) ^= 8 BIN: 11110111 11111111
761daa04: (77 != 7f) ^= 8 BIN: 01110111 01111111
7669b504: (f7 != ff) ^= 8 BIN: 11110111 11111111
76b3ac04: (d7 != df) ^= 8 BIN: 11010111 11011111
76de7504: (ff != f7) ^= 8 BIN: 11111111 11110111
76f1e404: (ff != f7) ^= 8 BIN: 11111111 11110111
78b6d404: (ff != f7) ^= 8 BIN: 11111111 11110111
79544e04: (f7 != ff) ^= 8 BIN: 11110111 11111111
79e1bf04: (ff != f7) ^= 8 BIN: 11111111 11110111
79ef3604: (ff != f7) ^= 8 BIN: 11111111 11110111
79f43b04: (77 != 7f) ^= 8 BIN: 01110111 01111111
7b330404: (f7 != ff) ^= 8 BIN: 11110111 11111111
7ba80504: (f7 != ff) ^= 8 BIN: 11110111 11111111
7bf94104: (f7 != ff) ^= 8 BIN: 11110111 11111111
7c3d3004: (f7 != ff) ^= 8 BIN: 11110111 11111111
7c7f4604: (ff != f7) ^= 8 BIN: 11111111 11110111
7d65da04: (d7 != df) ^= 8 BIN: 11010111 11011111
7d90db04: (ff != f7) ^= 8 BIN: 11111111 11110111
7daacc04: (f7 != ff) ^= 8 BIN: 11110111 11111111
7e744504: (7e != 76) ^= 8 BIN: 01111110 01110110
7f540204: (c7 != cf) ^= 8 BIN: 11000111 11001111
7f75b204: (e7 != ef) ^= 8 BIN: 11100111 11101111
7f913604: (57 != 5f) ^= 8 BIN: 01010111 01011111
]]> http://www.dslreports.com/forum/remark,18396113 Fri, 25 May 2007 18:23:35 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18392942 krod : I noticed in your previous topic it was always that 8valued bit, and that it was 7h(0x0111) changed to Fh(0x1111), but that was only with a few samples you gave and the first one didnt match that rule(in increased in size?? or was that a typo?)]]> http://www.dslreports.com/forum/remark,18392942 Fri, 25 May 2007 03:35:17 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18392810 leibold : That is what I was looking for. It is always the same bit in a 128bit/16byte word (it would be an even larger address space if it wasn't for that one error at 0xb860592). As elegant as your xor trick is in highlighting the defect bit, it hides whether it is always the same kind of change (0 to 1 or 1 to 0) or if it is random (however my guess would be that it is always the same change). If the corruption was happening in a serial bus (such as the sata cables to your disk drives) or in a narrow parallel bus (e.g.: 32-bit PCI bus) then the defect would show up in other positions as well.

This is very typical for a single bad memory cell and it would have to be an area where you have a wide parallel bus (such as a dual-channel memory interface which is 128-bit wide) for it to be otherwise. However if it was the main memory interface or one of the cpu caches I would expect more serious problems in keeping the system running. I would also expect memtest86/memtest86+ to detect those errors.

My guess is either the memory on the raid controller or a harddisk cache memory chip (none of which can be tested with memtest). I don't think you will be able to further narrow it down without swapping parts.

P.S.: rereading your posts I don't see how I got the wrong impression on what your conclusions were. Sorry!
--
Got some spare cpu cycles ? Join Team Helix or Team Starfire!]]> http://www.dslreports.com/forum/remark,18392810 Fri, 25 May 2007 02:19:10 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18392361 galacticroot : First, my conclusion was that the corruption was occurring before it reaches the cache in main memory, which is what I did that test to show. Once a file is cached, it doesn't change. The 2 gig file is too big for the cache, so it is read differently each time and has changes. The smaller file is cached completely and isn't re-read.

Second, as for the difference in the files, I noted that only the occasional byte was being changed in the post about tripwire. The corrupt bytes seemed random, since they were different each time.

However, I decided to look more closely at the difference itself and found something VERY interesting. I wrote a small program to XOR files, one byte at a time:
$ cp random_2gig.dat random_2gig-2.dat
$ ./fxor random_2gig.dat random_2gig-2.dat xor.dat
$ hexdump xor.dat
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
2416200 0000 0800 0000 0000 0000 0000 0000 0000
2416210 0000 0000 0000 0000 0000 0000 0000 0000
*
3525700 0000 0800 0000 0000 0000 0000 0000 0000
3525710 0000 0000 0000 0000 0000 0000 0000 0000
*
41fb300 0000 0800 0000 0000 0000 0000 0000 0000
41fb310 0000 0000 0000 0000 0000 0000 0000 0000
*
6619200 0000 0800 0000 0000 0000 0000 0000 0000
6619210 0000 0000 0000 0000 0000 0000 0000 0000
*
7ef3f00 0000 0800 0000 0000 0000 0000 0000 0000
7ef3f10 0000 0000 0000 0000 0000 0000 0000 0000
*
80f0c00 0000 0800 0000 0000 0000 0000 0000 0000
80f0c10 0000 0000 0000 0000 0000 0000 0000 0000
*
922c800 0000 0800 0000 0000 0000 0000 0000 0000
922c810 0000 0000 0000 0000 0000 0000 0000 0000
*
b84df00 0000 0800 0000 0000 0000 0000 0000 0000
b84df10 0000 0000 0000 0000 0000 0000 0000 0000
*
b860590 0000 0800 0000 0000 0000 0000 0000 0000
b8605a0 0000 0000 0000 0000 0000 0000 0000 0000
*
cdb7200 0000 0800 0000 0000 0000 0000 0000 0000
cdb7210 0000 0000 0000 0000 0000 0000 0000 0000
*
d1d1300 0000 0800 0000 0000 0000 0000 0000 0000
d1d1310 0000 0000 0000 0000 0000 0000 0000 0000
*
d7ce200 0000 0800 0000 0000 0000 0000 0000 0000
d7ce210 0000 0000 0000 0000 0000 0000 0000 0000
*
e5b9500 0000 0800 0000 0000 0000 0000 0000 0000
e5b9510 0000 0000 0000 0000 0000 0000 0000 0000
*
fcf5a00 0000 0800 0000 0000 0000 0000 0000 0000
fcf5a10 0000 0000 0000 0000 0000 0000 0000 0000
*
10d67c00 0000 0800 0000 0000 0000 0000 0000 0000
10d67c10 0000 0000 0000 0000 0000 0000 0000 0000
*
11b02900 0000 0800 0000 0000 0000 0000 0000 0000
11b02910 0000 0000 0000 0000 0000 0000 0000 0000
*
12836600 0000 0800 0000 0000 0000 0000 0000 0000
12836610 0000 0000 0000 0000 0000 0000 0000 0000
*
12b82f00 0000 0800 0000 0000 0000 0000 0000 0000
12b82f10 0000 0000 0000 0000 0000 0000 0000 0000
*
13aa2400 0000 0800 0000 0000 0000 0000 0000 0000
13aa2410 0000 0000 0000 0000 0000 0000 0000 0000
*
14210400 0000 0800 0000 0000 0000 0000 0000 0000
14210410 0000 0000 0000 0000 0000 0000 0000 0000
*
14bc7d00 0000 0800 0000 0000 0000 0000 0000 0000
14bc7d10 0000 0000 0000 0000 0000 0000 0000 0000
*
14c32c00 0000 0800 0000 0000 0000 0000 0000 0000
14c32c10 0000 0000 0000 0000 0000 0000 0000 0000
*
19fb0e00 0000 0800 0000 0000 0000 0000 0000 0000
19fb0e10 0000 0000 0000 0000 0000 0000 0000 0000
*
1be4e100 0000 0800 0000 0000 0000 0000 0000 0000
1be4e110 0000 0000 0000 0000 0000 0000 0000 0000
*
1ca7cb00 0000 0800 0000 0000 0000 0000 0000 0000
1ca7cb10 0000 0000 0000 0000 0000 0000 0000 0000
*
1d460000 0000 0800 0000 0000 0000 0000 0000 0000
1d460010 0000 0000 0000 0000 0000 0000 0000 0000
*
1def1b00 0000 0800 0000 0000 0000 0000 0000 0000
1def1b10 0000 0000 0000 0000 0000 0000 0000 0000
*
21b00100 0000 0800 0000 0000 0000 0000 0000 0000
21b00110 0000 0000 0000 0000 0000 0000 0000 0000
*
2239de00 0000 0800 0000 0000 0000 0000 0000 0000
2239de10 0000 0000 0000 0000 0000 0000 0000 0000
*
23659e00 0000 0800 0000 0000 0000 0000 0000 0000
23659e10 0000 0000 0000 0000 0000 0000 0000 0000
*
236da900 0000 0800 0000 0000 0000 0000 0000 0000
236da910 0000 0000 0000 0000 0000 0000 0000 0000
*
252f0600 0000 0800 0000 0000 0000 0000 0000 0000
252f0610 0000 0000 0000 0000 0000 0000 0000 0000
*
2561e300 0000 0800 0000 0000 0000 0000 0000 0000
2561e310 0000 0000 0000 0000 0000 0000 0000 0000
*
25cd1400 0000 0800 0000 0000 0000 0000 0000 0000
25cd1410 0000 0000 0000 0000 0000 0000 0000 0000
*
262f9c00 0000 0800 0000 0000 0000 0000 0000 0000
262f9c10 0000 0000 0000 0000 0000 0000 0000 0000
*
2660d100 0000 0800 0000 0000 0000 0000 0000 0000
2660d110 0000 0000 0000 0000 0000 0000 0000 0000
*
26c7fd00 0000 0800 0000 0000 0000 0000 0000 0000
26c7fd10 0000 0000 0000 0000 0000 0000 0000 0000
*
26ed2a00 0000 0800 0000 0000 0000 0000 0000 0000
26ed2a10 0000 0000 0000 0000 0000 0000 0000 0000
*
272c8900 0000 0800 0000 0000 0000 0000 0000 0000
272c8910 0000 0000 0000 0000 0000 0000 0000 0000
*
274b0900 0000 0800 0000 0000 0000 0000 0000 0000
274b0910 0000 0000 0000 0000 0000 0000 0000 0000
*
28bc0d00 0000 0800 0000 0000 0000 0000 0000 0000
28bc0d10 0000 0000 0000 0000 0000 0000 0000 0000
*
29087c00 0000 0800 0000 0000 0000 0000 0000 0000
29087c10 0000 0000 0000 0000 0000 0000 0000 0000
*
2cb71500 0000 0800 0000 0000 0000 0000 0000 0000
2cb71510 0000 0000 0000 0000 0000 0000 0000 0000
*
2cbb3500 0000 0800 0000 0000 0000 0000 0000 0000
2cbb3510 0000 0000 0000 0000 0000 0000 0000 0000
*
2d6637f0 0000 0000 0000 0000 0000 0800 0000 0000
2d663800 0000 0000 0000 0000 0000 0000 0000 0000
*
2e0ef600 0000 0800 0000 0000 0000 0000 0000 0000
2e0ef610 0000 0000 0000 0000 0000 0000 0000 0000
*
2e2ac600 0000 0800 0000 0000 0000 0000 0000 0000
2e2ac610 0000 0000 0000 0000 0000 0000 0000 0000
*
2e543a00 0000 0800 0000 0000 0000 0000 0000 0000
2e543a10 0000 0000 0000 0000 0000 0000 0000 0000
*
31330700 0000 0800 0000 0000 0000 0000 0000 0000
31330710 0000 0000 0000 0000 0000 0000 0000 0000
*
32731100 0000 0800 0000 0000 0000 0000 0000 0000
32731110 0000 0000 0000 0000 0000 0000 0000 0000
*
3309a500 0000 0000 0000 0000 0000 0800 0000 0000
3309a510 0000 0000 0000 0000 0000 0000 0000 0000
*
335ba200 0000 0800 0000 0000 0000 0000 0000 0000
335ba210 0000 0000 0000 0000 0000 0000 0000 0000
*
3527a600 0000 0800 0000 0000 0000 0000 0000 0000
3527a610 0000 0000 0000 0000 0000 0000 0000 0000
*
35481200 0000 0800 0000 0000 0000 0000 0000 0000
35481210 0000 0000 0000 0000 0000 0000 0000 0000
*
362f5300 0000 0800 0000 0000 0000 0000 0000 0000
362f5310 0000 0000 0000 0000 0000 0000 0000 0000
*
366d5400 0000 0800 0000 0000 0000 0000 0000 0000
366d5410 0000 0000 0000 0000 0000 0000 0000 0000
*
36fb0500 0000 0800 0000 0000 0000 0000 0000 0000
36fb0510 0000 0000 0000 0000 0000 0000 0000 0000
*
372ab600 0000 0800 0000 0000 0000 0000 0000 0000
372ab610 0000 0000 0000 0000 0000 0000 0000 0000
*
372cfd00 0000 0800 0000 0000 0000 0000 0000 0000
372cfd10 0000 0000 0000 0000 0000 0000 0000 0000
*
38a33000 0000 0800 0000 0000 0000 0000 0000 0000
38a33010 0000 0000 0000 0000 0000 0000 0000 0000
*
39f75e00 0000 0800 0000 0000 0000 0000 0000 0000
39f75e10 0000 0000 0000 0000 0000 0000 0000 0000
*
3a15a900 0000 0800 0000 0000 0000 0000 0000 0000
3a15a910 0000 0000 0000 0000 0000 0000 0000 0000
*
3a554700 0000 0800 0000 0000 0000 0000 0000 0000
3a554710 0000 0000 0000 0000 0000 0000 0000 0000
*
3aaadb00 0000 0800 0000 0000 0000 0000 0000 0000
3aaadb10 0000 0000 0000 0000 0000 0000 0000 0000
*
3c5f6f00 0000 0800 0000 0000 0000 0000 0000 0000
3c5f6f10 0000 0000 0000 0000 0000 0000 0000 0000
*
3d492f00 0000 0800 0000 0000 0000 0000 0000 0000
3d492f10 0000 0000 0000 0000 0000 0000 0000 0000
*
3d4cd200 0000 0800 0000 0000 0000 0000 0000 0000
3d4cd210 0000 0000 0000 0000 0000 0000 0000 0000
*
3fceda00 0000 0800 0000 0000 0000 0000 0000 0000
3fceda10 0000 0000 0000 0000 0000 0000 0000 0000
*
3fe71e00 0000 0800 0000 0000 0000 0000 0000 0000
3fe71e10 0000 0000 0000 0000 0000 0000 0000 0000
*
4068e200 0000 0800 0000 0000 0000 0000 0000 0000
4068e210 0000 0000 0000 0000 0000 0000 0000 0000
*
408d1600 0000 0000 0000 0000 0000 0800 0000 0000
408d1610 0000 0000 0000 0000 0000 0000 0000 0000
*
42999300 0000 0800 0000 0000 0000 0000 0000 0000
42999310 0000 0000 0000 0000 0000 0000 0000 0000
*
44540200 0000 0800 0000 0000 0000 0000 0000 0000
44540210 0000 0000 0000 0000 0000 0000 0000 0000
*
4544b800 0000 0800 0000 0000 0000 0000 0000 0000
4544b810 0000 0000 0000 0000 0000 0000 0000 0000
*
458f3300 0000 0800 0000 0000 0000 0000 0000 0000
458f3310 0000 0000 0000 0000 0000 0000 0000 0000
*
45928d00 0000 0800 0000 0000 0000 0000 0000 0000
45928d10 0000 0000 0000 0000 0000 0000 0000 0000
*
459c8300 0000 0800 0000 0000 0000 0000 0000 0000
459c8310 0000 0000 0000 0000 0000 0000 0000 0000
*
45b2f800 0000 0800 0000 0000 0000 0000 0000 0000
45b2f810 0000 0000 0000 0000 0000 0000 0000 0000
*
45bfaa00 0000 0800 0000 0000 0000 0000 0000 0000
45bfaa10 0000 0000 0000 0000 0000 0000 0000 0000
*
45d12000 0000 0800 0000 0000 0000 0000 0000 0000
45d12010 0000 0000 0000 0000 0000 0000 0000 0000
*
46022500 0000 0800 0000 0000 0000 0000 0000 0000
46022510 0000 0000 0000 0000 0000 0000 0000 0000
*
477d88b0 0000 0000 0000 0000 0000 0800 0000 0000
477d88c0 0000 0000 0000 0000 0000 0000 0000 0000
*
478d7000 0000 0800 0000 0000 0000 0000 0000 0000
478d7010 0000 0000 0000 0000 0000 0000 0000 0000
*
49051700 0000 0800 0000 0000 0000 0000 0000 0000
49051710 0000 0000 0000 0000 0000 0000 0000 0000
*
49afa200 0000 0800 0000 0000 0000 0000 0000 0000
49afa210 0000 0000 0000 0000 0000 0000 0000 0000
*
4a74d100 0000 0800 0000 0000 0000 0000 0000 0000
4a74d110 0000 0000 0000 0000 0000 0000 0000 0000
*
4a935700 0000 0800 0000 0000 0000 0000 0000 0000
4a935710 0000 0000 0000 0000 0000 0000 0000 0000
*
4b3f0a00 0000 0800 0000 0000 0000 0000 0000 0000
4b3f0a10 0000 0000 0000 0000 0000 0000 0000 0000
*
4c6c1d00 0000 0800 0000 0000 0000 0000 0000 0000
4c6c1d10 0000 0000 0000 0000 0000 0000 0000 0000
*
4cb75f00 0000 0800 0000 0000 0000 0000 0000 0000
4cb75f10 0000 0000 0000 0000 0000 0000 0000 0000
*
4dc88700 0000 0800 0000 0000 0000 0000 0000 0000
4dc88710 0000 0000 0000 0000 0000 0000 0000 0000
*
4ec13b00 0000 0800 0000 0000 0000 0000 0000 0000
4ec13b10 0000 0000 0000 0000 0000 0000 0000 0000
*
503eb100 0000 0800 0000 0000 0000 0000 0000 0000
503eb110 0000 0000 0000 0000 0000 0000 0000 0000
*
508f2000 0000 0800 0000 0000 0000 0000 0000 0000
508f2010 0000 0000 0000 0000 0000 0000 0000 0000
*
50f6ed00 0000 0800 0000 0000 0000 0000 0000 0000
50f6ed10 0000 0000 0000 0000 0000 0000 0000 0000
*
51978800 0000 0800 0000 0000 0000 0000 0000 0000
51978810 0000 0000 0000 0000 0000 0000 0000 0000
*
51bca200 0000 0800 0000 0000 0000 0000 0000 0000
51bca210 0000 0000 0000 0000 0000 0000 0000 0000
*
534b5000 0000 0800 0000 0000 0000 0000 0000 0000
534b5010 0000 0000 0000 0000 0000 0000 0000 0000
*
53791f00 0000 0800 0000 0000 0000 0000 0000 0000
53791f10 0000 0000 0000 0000 0000 0000 0000 0000
*
53b9a300 0000 0800 0000 0000 0000 0000 0000 0000
53b9a310 0000 0000 0000 0000 0000 0000 0000 0000
*
541a0600 0000 0800 0000 0000 0000 0000 0000 0000
541a0610 0000 0000 0000 0000 0000 0000 0000 0000
*
560b8200 0000 0800 0000 0000 0000 0000 0000 0000
560b8210 0000 0000 0000 0000 0000 0000 0000 0000
*
560b8c00 0000 0800 0000 0000 0000 0000 0000 0000
560b8c10 0000 0000 0000 0000 0000 0000 0000 0000
*
5796da00 0000 0800 0000 0000 0000 0000 0000 0000
5796da10 0000 0000 0000 0000 0000 0000 0000 0000
*
57d29200 0000 0800 0000 0000 0000 0000 0000 0000
57d29210 0000 0000 0000 0000 0000 0000 0000 0000
*
587d1000 0000 0800 0000 0000 0000 0000 0000 0000
587d1010 0000 0000 0000 0000 0000 0000 0000 0000
*
590d3100 0000 0800 0000 0000 0000 0000 0000 0000
590d3110 0000 0000 0000 0000 0000 0000 0000 0000
*
59b3a500 0000 0800 0000 0000 0000 0000 0000 0000
59b3a510 0000 0000 0000 0000 0000 0000 0000 0000
*
5db62600 0000 0800 0000 0000 0000 0000 0000 0000
5db62610 0000 0000 0000 0000 0000 0000 0000 0000
*
5dfce200 0000 0800 0000 0000 0000 0000 0000 0000
5dfce210 0000 0000 0000 0000 0000 0000 0000 0000
*
5ee54b00 0000 0800 0000 0000 0000 0000 0000 0000
5ee54b10 0000 0000 0000 0000 0000 0000 0000 0000
*
5f102300 0000 0800 0000 0000 0000 0000 0000 0000
5f102310 0000 0000 0000 0000 0000 0000 0000 0000
*
5f31d300 0000 0800 0000 0000 0000 0000 0000 0000
5f31d310 0000 0000 0000 0000 0000 0000 0000 0000
*
5f75ba00 0000 0800 0000 0000 0000 0000 0000 0000
5f75ba10 0000 0000 0000 0000 0000 0000 0000 0000
*
6001a200 0000 0800 0000 0000 0000 0000 0000 0000
6001a210 0000 0000 0000 0000 0000 0000 0000 0000
*
638d0f00 0000 0800 0000 0000 0000 0000 0000 0000
638d0f10 0000 0000 0000 0000 0000 0000 0000 0000
*
63ddb800 0000 0800 0000 0000 0000 0000 0000 0000
63ddb810 0000 0000 0000 0000 0000 0000 0000 0000
*
64ce0e00 0000 0800 0000 0000 0000 0000 0000 0000
64ce0e10 0000 0000 0000 0000 0000 0000 0000 0000
*
68611400 0000 0800 0000 0000 0000 0000 0000 0000
68611410 0000 0000 0000 0000 0000 0000 0000 0000
*
69231a00 0000 0800 0000 0000 0000 0000 0000 0000
69231a10 0000 0000 0000 0000 0000 0000 0000 0000
*
693b2200 0000 0800 0000 0000 0000 0000 0000 0000
693b2210 0000 0000 0000 0000 0000 0000 0000 0000
*
69592500 0000 0800 0000 0000 0000 0000 0000 0000
69592510 0000 0000 0000 0000 0000 0000 0000 0000
*
69fb3000 0000 0800 0000 0000 0000 0000 0000 0000
69fb3010 0000 0000 0000 0000 0000 0000 0000 0000
*
6b2f2200 0000 0800 0000 0000 0000 0000 0000 0000
6b2f2210 0000 0000 0000 0000 0000 0000 0000 0000
*
6baa0200 0000 0800 0000 0000 0000 0000 0000 0000
6baa0210 0000 0000 0000 0000 0000 0000 0000 0000
*
6c7b1100 0000 0800 0000 0000 0000 0000 0000 0000
6c7b1110 0000 0000 0000 0000 0000 0000 0000 0000
*
6dd8aa00 0000 0800 0000 0000 0000 0000 0000 0000
6dd8aa10 0000 0000 0000 0000 0000 0000 0000 0000
*
6e34d500 0000 0800 0000 0000 0000 0000 0000 0000
6e34d510 0000 0000 0000 0000 0000 0000 0000 0000
*
6e932800 0000 0800 0000 0000 0000 0000 0000 0000
6e932810 0000 0000 0000 0000 0000 0000 0000 0000
*
6fe7b200 0000 0800 0000 0000 0000 0000 0000 0000
6fe7b210 0000 0000 0000 0000 0000 0000 0000 0000
*
6feff700 0000 0800 0000 0000 0000 0000 0000 0000
6feff710 0000 0000 0000 0000 0000 0000 0000 0000
*
7447ec00 0000 0800 0000 0000 0000 0000 0000 0000
7447ec10 0000 0000 0000 0000 0000 0000 0000 0000
*
75033e00 0000 0800 0000 0000 0000 0000 0000 0000
75033e10 0000 0000 0000 0000 0000 0000 0000 0000
*
755f7100 0000 0800 0000 0000 0000 0000 0000 0000
755f7110 0000 0000 0000 0000 0000 0000 0000 0000
*
761daa00 0000 0800 0000 0000 0000 0000 0000 0000
761daa10 0000 0000 0000 0000 0000 0000 0000 0000
*
7662de00 0000 0800 0000 0000 0000 0000 0000 0000
7662de10 0000 0000 0000 0000 0000 0000 0000 0000
*
76be0500 0000 0800 0000 0000 0000 0000 0000 0000
76be0510 0000 0000 0000 0000 0000 0000 0000 0000
*
78162a00 0000 0800 0000 0000 0000 0000 0000 0000
78162a10 0000 0000 0000 0000 0000 0000 0000 0000
*
7a42d600 0000 0800 0000 0000 0000 0000 0000 0000
7a42d610 0000 0000 0000 0000 0000 0000 0000 0000
*
7b330400 0000 0800 0000 0000 0000 0000 0000 0000
7b330410 0000 0000 0000 0000 0000 0000 0000 0000
*
7bf94100 0000 0800 0000 0000 0000 0000 0000 0000
7bf94110 0000 0000 0000 0000 0000 0000 0000 0000
*
7c3d3000 0000 0800 0000 0000 0000 0000 0000 0000
7c3d3010 0000 0000 0000 0000 0000 0000 0000 0000
*
7d65da00 0000 0800 0000 0000 0000 0000 0000 0000
7d65da10 0000 0000 0000 0000 0000 0000 0000 0000
*
7f75b200 0000 0800 0000 0000 0000 0000 0000 0000
7f75b210 0000 0000 0000 0000 0000 0000 0000 0000
*
7f870800 0000 0800 0000 0000 0000 0000 0000 0000
7f870810 0000 0000 0000 0000 0000 0000 0000 0000
*
7f913600 0000 0800 0000 0000 0000 0000 0000 0000
7f913610 0000 0000 0000 0000 0000 0000 0000 0000
*
80000000

Note that this files has been corrupted three times, once on copy, and twice when reading the two files.
]]> http://www.dslreports.com/forum/remark,18392361 Thu, 24 May 2007 23:41:59 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18391864 leibold :
said by galacticroot See Profile :

Oddly, the same thing does NOT happen with 2 gigs of zeros.
Sounds like a bit that is either stuck at 0 or pulled down to 0 if certain conditions are fulfilled (e.g. neighboring bits are also 0).

I disagree with your conclusion that the corruption is happening to the data in the cache (mainmemory of your system). Once cached the data doesn't change and memtest doesn't find any problems in main memory either. While that is not sufficient evidence to completely eliminate mainmemory it does make it more likely that the data is being corrupted on the way from one of the disk drives via the harddisk cache, the raid controller cache until it ends up corrupted in mainmemory.

You said the server is in a datacenter, so incremental hardware changes can be a hassle. Do you have the ability to break the mirror and test the system with one drive at a time ? This can eliminate the memory cache on the harddisk itself as a possible source if the problem still happens with each of the drives.

md5sum doesn't give us any clues as to the extend of the differences. What happens when you do a binary comparison of the random_2gig.dat file with itself ? (Probably no difference, but still worth testing).
What about when you make a copy of the random_2gig.dat file and then keep comparing the two ? Which data bits are different and how many in each comparison ? Are low-order address bits always the same when data bytes are different ?
--
Got some spare cpu cycles ? Join Team Helix or Team Starfire!]]> http://www.dslreports.com/forum/remark,18391864 Thu, 24 May 2007 21:59:54 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18391457 galacticroot : Got that done, and everything is upgraded, but the problem still exists on that server.]]> http://www.dslreports.com/forum/remark,18391457 Thu, 24 May 2007 20:52:38 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18390638 galacticroot : I'm actually upgrading all my FreeBSD systems to RELENG_6 now, largely because of the recent vulnerability in "file".]]> http://www.dslreports.com/forum/remark,18390638 Thu, 24 May 2007 17:57:04 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389827 koitsu : Interesting.

Have you tried cvsup'ing to the most recent copy of RELENG_6 and rebuilding world/kernel to see if any kernel changes between -RELEASE and -STABLE fix it? Might be worth a shot if you have the time.

Otherwise, post to freebsd-stable with what you've tried here, and let someone respond. I'm thinking Scott Long will be able to help, since he's worked on twa(4) as of late.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,18389827 Thu, 24 May 2007 15:27:18 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389651 galacticroot : I've disabled the 3ware cache already and it hasn't had any affect. This is i386 FreeBSD.]]> http://www.dslreports.com/forum/remark,18389651 Thu, 24 May 2007 14:53:47 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389506 koitsu : Most of the time when I've gotten back different MD5 checksums from a system, it turns out to be bad RAM (in the host system).

memtest86 isn't infalliable (hardware RAM testers actually do things like increase voltage and send amounts of current through all the banks), but I would guess it _would_ pick up this type of problem if it were RAM-related.

Is this i386 FreeBSD, by the way, or amd64? Yes, it matters. :-)

Can you disable the hardware cache on the 3ware device temporarily and see if the problem goes away?

Either way, please post this problem to freebsd-stable@freebsd.org. People there should be able to help narrow it down to a specific piece of hardware.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,18389506 Thu, 24 May 2007 14:23:04 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389444 galacticroot : Ok. I upgraded the firmware to the latest version, but that did not affect the problem.]]> http://www.dslreports.com/forum/remark,18389444 Thu, 24 May 2007 14:10:44 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389426 galacticroot : Hmm. I'll look into that. I've never upgraded the firmware on the card, so it may be old.

Good example of the problem. A 2 gig file won't fit in the cache and must be re-read from disk each time:
$ dd if=/dev/urandom of=random_2gig.dat bs=1m count=2048
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 158.899191 secs (13514755 bytes/sec)
$ md5 random_2gig.dat
MD5 (random_2gig.dat) = 6e457ee2cc38ac1786311a0bd699c451
$ md5 random_2gig.dat
MD5 (random_2gig.dat) = 67700e8af9cba93890864605b8f43093
$ md5 random_2gig.dat
MD5 (random_2gig.dat) = 93b207b3a3cddd0b1920ea5b4e656efa
$ md5 random_2gig.dat
MD5 (random_2gig.dat) = bf903326c940b9be0ab5b46daee5cefe
$ md5 random_2gig.dat
MD5 (random_2gig.dat) = ea4fc8e8e12ff8c13746fd92032ff37c

A 256 meg file, on the other hand, is cached:
$ dd if=/dev/urandom of=random_256meg.dat bs=1m count=256
256+0 records in
256+0 records out
268435456 bytes transferred in 19.859751 secs (13516557 bytes/sec)
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771
$ md5 random_256meg.dat
MD5 (random_256meg.dat) = 731500d84022bb254b1ab027629c8771

Oddly, the same thing does NOT happen with 2 gigs of zeros.
$ dd if=/dev/zero of=zero_2gig.dat bs=1m count=2048
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 58.706760 secs (36579836 bytes/sec)
$ md5 zero_2gig.dat
MD5 (zero_2gig.dat) = a981130cf2b7e09f4686dc273cf7187e
$ md5 zero_2gig.dat
MD5 (zero_2gig.dat) = a981130cf2b7e09f4686dc273cf7187e
$ md5 zero_2gig.dat
MD5 (zero_2gig.dat) = a981130cf2b7e09f4686dc273cf7187e
$ md5 zero_2gig.dat
MD5 (zero_2gig.dat) = a981130cf2b7e09f4686dc273cf7187e
$ md5 zero_2gig.dat
MD5 (zero_2gig.dat) = a981130cf2b7e09f4686dc273cf7187e
]]> http://www.dslreports.com/forum/remark,18389426 Thu, 24 May 2007 14:07:55 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18389118 koitsu : Have you brought this up on the freebsd-stable list? 3ware cards have a history of being very finicky, particularly each firmware release induces entirely new bugs and fixes old ones.

I'm voting that your 3ware card either has bad cache (on the card itself, not in the host system/PC) or some hardware-level problem.

Scott Long should be able to help. Again, please mail freebsd-stable about this matter. :-)

EDIT: There was some recent work done on the twa driver in RELENG_6 (read: after 6.2-RELEASE) :

»www.freebsd.org/cgi/cvsweb.cgi/s···dev/twa/
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,18389118 Thu, 24 May 2007 13:06:51 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18387247 galacticroot : Both the server and the RAID card use ECC memory. The server ram has been tested for 24 hours with memtest86. The RAID card has no errors/alarms beyond the one I mentioned above and passes all internal tests.

Changing the cables is a good idea, although that may take a while since the server is in a data center.]]> http://www.dslreports.com/forum/remark,18387247 Thu, 24 May 2007 02:03:29 EDT Re: [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18387029 billaustin : The first thing I would do is to change all the cables running between the drives and the raid card.

The next would be to change the ram module on the raid card, if it has a removeable one. Otherwise, it would be to change the raid card.

Based on your post, the drives seem to be okay. You need to determine if the data is being corrupted on the way to the cache, or after being put in the cache.

Are you using ECC ram in the server? Is the ram on the raid card ECC?]]> http://www.dslreports.com/forum/remark,18387029 Thu, 24 May 2007 00:40:33 EDT [FreeBSD] Disk read corruption issues on server. http://www.dslreports.com/forum/remark,18385716 galacticroot : Note: I'm making a new thread for this since the old thread is misleading and the problem doesn't have anything to do with tripwire (other than being detected by it).

OK. I've got a serious problem on on a FreeBSD server. The server is and SMP system running FreeBSD 6.2-RELEASE. It has a 3ware 9500S-4LP raid card with two 250GB drives in a RAID-1 configuration.

Simply put, under heavy loads, data is being read from the array, corrupted occasionally, and then cached.

The result is that random files will often become temporarily corrupted in the cache until they are actually read again.

I have verified the RAID array several times and no errors have been found on it.

The actual level of corruption is minimal, maybe one byte in every billion is altered, so I only noticed it when I started running tripwire. Every time tripwire was run, the large amount of reading would refresh the cache and different files would be corrupted. See the previous post.

So, basically, I don't know how long this has been occurring, where it is occurring, or why it is occurring. I haven't gotten any errors about read failures, so I assume it is occurring at someplace where the data is not error corrected.

The data is NOT corrupted on disk, since the files often revert to the correct copies when they are actually re-read.

It could be a software bug, or a hardware issue. I'm not sure. It is occurring somewhere between the disk platter and the OS read cache, but I can't figure out where.

Additionally, I have been getting the following on the RAID controller diagnostic log:
Error, Unit 64: Logical unit not present
(EC:0x10a, SK=0x05, ASC=0x25, ASCQ=0x00, SEV=01, Type=0x70)
unit=64

It may not be important, though. There are no other related errors. Any help would be appreciated.]]> http://www.dslreports.com/forum/remark,18385716 Wed, 23 May 2007 20:37:35 EDT