 krod meh
join:2007-04-08 1 edit | reply to galacticroot Re: [FreeBSD] Disk read corruption issues on server.
I noticed in your previous topic it was always that 8valued bit, and that it was 7h(0x0111) changed to Fh(0x1111), but that was only with a few samples you gave and the first one didnt match that rule(in increased in size?? or was that a typo?) |
|
 galacticroot
join:2004-05-17 clubs:
| It is occurring at the same space in 64 bit blocks, and the PCI bus is 64 bit. I would assume that that an error on that would cause other issues, but maybe not. The RAID controller has error corrected memory, so I assume the problem would be detected if it was that.
Interestingly, by far the most common change is from f7 to ff. Assuming the /dev/urandom (which the file is from) produces reasonably distributed data, the corrupted bit seems to be highly dependent on the bits around it. This may indicate that the problem is caused by a minor voltage leakage between adjacent bus lines. Perhaps the PCI connection needs cleaning or reseating?
Here is a more extensive summary of the file changes. This is a different read and does not correspond with the XORed file above.
2416204: (d7 != df) ^= 8 BIN: 11010111 11011111 3bfbe04: (ff != f7) ^= 8 BIN: 11111111 11110111 4012304: (77 != 7f) ^= 8 BIN: 01110111 01111111 4c42804: (7f != 77) ^= 8 BIN: 01111111 01110111 51a4904: (ff != f7) ^= 8 BIN: 11111111 11110111 5f2e004: (f7 != ff) ^= 8 BIN: 11110111 11111111 6619204: (f7 != ff) ^= 8 BIN: 11110111 11111111 7cf6404: (f7 != ff) ^= 8 BIN: 11110111 11111111 7ef3f04: (f7 != ff) ^= 8 BIN: 11110111 11111111 80f0c04: (f7 != ff) ^= 8 BIN: 11110111 11111111 8c92004: (ff != f7) ^= 8 BIN: 11111111 11110111 8ce5104: (ff != f7) ^= 8 BIN: 11111111 11110111 9ec5304: (ff != f7) ^= 8 BIN: 11111111 11110111 aabcc04: (7f != 77) ^= 8 BIN: 01111111 01110111 b860594: (f7 != ff) ^= 8 BIN: 11110111 11111111 c296f04: (f7 != ff) ^= 8 BIN: 11110111 11111111 c4b5a04: (ef != e7) ^= 8 BIN: 11101111 11100111 c9ddc04: (7f != 77) ^= 8 BIN: 01111111 01110111 cc9b204: (f7 != ff) ^= 8 BIN: 11110111 11111111 cdb7204: (f7 != ff) ^= 8 BIN: 11110111 11111111 e060f04: (e7 != ef) ^= 8 BIN: 11100111 11101111 e5b9504: (d7 != df) ^= 8 BIN: 11010111 11011111 f2a570c: (df != d7) ^= 8 BIN: 11011111 11010111 f52cc04: (f7 != ff) ^= 8 BIN: 11110111 11111111 10540904: (77 != 7f) ^= 8 BIN: 01110111 01111111 107a0904: (d7 != df) ^= 8 BIN: 11010111 11011111 125c5504: (ef != e7) ^= 8 BIN: 11101111 11100111 12836604: (f6 != fe) ^= 8 BIN: 11110110 11111110 12a61204: (d7 != df) ^= 8 BIN: 11010111 11011111 12d4ce04: (ff != f7) ^= 8 BIN: 11111111 11110111 132a4e04: (ff != f7) ^= 8 BIN: 11111111 11110111 14210404: (e7 != ef) ^= 8 BIN: 11100111 11101111 14c32c04: (f5 != fd) ^= 8 BIN: 11110101 11111101 162c0504: (ff != f7) ^= 8 BIN: 11111111 11110111 16801004: (f7 != ff) ^= 8 BIN: 11110111 11111111 176fdb04: (ff != f7) ^= 8 BIN: 11111111 11110111 18904d04: (ff != f7) ^= 8 BIN: 11111111 11110111 1b649504: (f7 != ff) ^= 8 BIN: 11110111 11111111 1ca7cb04: (77 != 7f) ^= 8 BIN: 01110111 01111111 1d460004: (77 != 7f) ^= 8 BIN: 01110111 01111111 1d845604: (fb != f3) ^= 8 BIN: 11111011 11110011 1eb84d04: (ff != f7) ^= 8 BIN: 11111111 11110111 20672904: (d7 != df) ^= 8 BIN: 11010111 11011111 20df9f04: (ff != f7) ^= 8 BIN: 11111111 11110111 20f04304: (ff != f7) ^= 8 BIN: 11111111 11110111 212a4b04: (ff != f7) ^= 8 BIN: 11111111 11110111 221aa904: (77 != 7f) ^= 8 BIN: 01110111 01111111 22765104: (6f != 67) ^= 8 BIN: 01101111 01100111 23659e04: (f6 != fe) ^= 8 BIN: 11110110 11111110 236da904: (77 != 7f) ^= 8 BIN: 01110111 01111111 25744604: (ff != f7) ^= 8 BIN: 11111111 11110111 25cd1404: (e7 != ef) ^= 8 BIN: 11100111 11101111 25e8cf04: (ff != f7) ^= 8 BIN: 11111111 11110111 2603bf04: (f7 != ff) ^= 8 BIN: 11110111 11111111 262f9c04: (77 != 7f) ^= 8 BIN: 01110111 01111111 269bca04: (df != d7) ^= 8 BIN: 11011111 11010111 274b0904: (f5 != fd) ^= 8 BIN: 11110101 11111101 2750a704: (f7 != ff) ^= 8 BIN: 11110111 11111111 28082c04: (ff != f7) ^= 8 BIN: 11111111 11110111 28e7da04: (f7 != ff) ^= 8 BIN: 11110111 11111111 2a4e4f04: (df != d7) ^= 8 BIN: 11011111 11010111 2afe5904: (fe != f6) ^= 8 BIN: 11111110 11110110 2b70c904: (ff != f7) ^= 8 BIN: 11111111 11110111 2cbb3504: (d7 != df) ^= 8 BIN: 11010111 11011111 2d6637fc: (f7 != ff) ^= 8 BIN: 11110111 11111111 2dca4f04: (df != d7) ^= 8 BIN: 11011111 11010111 2dcab804: (ff != f7) ^= 8 BIN: 11111111 11110111 2f782b04: (e7 != ef) ^= 8 BIN: 11100111 11101111 2fd42604: (7f != 77) ^= 8 BIN: 01111111 01110111 2ffdd604: (ff != f7) ^= 8 BIN: 11111111 11110111 30763704: (f7 != ff) ^= 8 BIN: 11110111 11111111 3092c404: (ff != f7) ^= 8 BIN: 11111111 11110111 30cc1104: (f7 != ff) ^= 8 BIN: 11110111 11111111 30fb9804: (d7 != df) ^= 8 BIN: 11010111 11011111 31330704: (f7 != ff) ^= 8 BIN: 11110111 11111111 31d64004: (ff != f7) ^= 8 BIN: 11111111 11110111 32544304: (cf != c7) ^= 8 BIN: 11001111 11000111 32731104: (77 != 7f) ^= 8 BIN: 01110111 01111111 3309a50c: (f7 != ff) ^= 8 BIN: 11110111 11111111 335ba204: (f7 != ff) ^= 8 BIN: 11110111 11111111 34c9a404: (e7 != ef) ^= 8 BIN: 11100111 11101111 3527a604: (57 != 5f) ^= 8 BIN: 01010111 01011111 35481204: (f7 != ff) ^= 8 BIN: 11110111 11111111 35d85104: (7e != 76) ^= 8 BIN: 01111110 01110110 362f5304: (d7 != df) ^= 8 BIN: 11010111 11011111 3840da04: (ff != f7) ^= 8 BIN: 11111111 11110111 38a33004: (f5 != fd) ^= 8 BIN: 11110101 11111101 39185804: (ef != e7) ^= 8 BIN: 11101111 11100111 3a15a904: (d7 != df) ^= 8 BIN: 11010111 11011111 3a554704: (f7 != ff) ^= 8 BIN: 11110111 11111111 3c5f6f04: (d7 != df) ^= 8 BIN: 11010111 11011111 3d17ae04: (77 != 7f) ^= 8 BIN: 01110111 01111111 3d492f04: (f7 != ff) ^= 8 BIN: 11110111 11111111 3fad7604: (ff != f7) ^= 8 BIN: 11111111 11110111 3fb65104: (ff != f7) ^= 8 BIN: 11111111 11110111 3fe71e04: (57 != 5f) ^= 8 BIN: 01010111 01011111 40204404: (ff != f7) ^= 8 BIN: 11111111 11110111 40b79b04: (d7 != df) ^= 8 BIN: 11010111 11011111 4478db04: (f7 != ff) ^= 8 BIN: 11110111 11111111 45444c04: (ef != e7) ^= 8 BIN: 11101111 11100111 4544b804: (76 != 7e) ^= 8 BIN: 01110110 01111110 45745404: (ff != f7) ^= 8 BIN: 11111111 11110111 45bfaa04: (d7 != df) ^= 8 BIN: 11010111 11011111 45d12004: (77 != 7f) ^= 8 BIN: 01110111 01111111 45d20004: (55 != 5d) ^= 8 BIN: 01010101 01011101 477d88bc: (76 != 7e) ^= 8 BIN: 01110110 01111110 478d7004: (67 != 6f) ^= 8 BIN: 01100111 01101111 48945c04: (dd != d5) ^= 8 BIN: 11011101 11010101 48dc4d04: (ff != f7) ^= 8 BIN: 11111111 11110111 493cbf04: (ff != f7) ^= 8 BIN: 11111111 11110111 49afa204: (57 != 5f) ^= 8 BIN: 01010111 01011111 49d3bf04: (ff != f7) ^= 8 BIN: 11111111 11110111 49d84c04: (ff != f7) ^= 8 BIN: 11111111 11110111 4a763804: (ff != f7) ^= 8 BIN: 11111111 11110111 4a935704: (67 != 6f) ^= 8 BIN: 01100111 01101111 4c640504: (f7 != ff) ^= 8 BIN: 11110111 11111111 4c6c1d04: (f7 != ff) ^= 8 BIN: 11110111 11111111 4c7f2604: (ff != f7) ^= 8 BIN: 11111111 11110111 4d674204: (d7 != df) ^= 8 BIN: 11010111 11011111 4dc88704: (f7 != ff) ^= 8 BIN: 11110111 11111111 4ec13b04: (f7 != ff) ^= 8 BIN: 11110111 11111111 4ece0a04: (f7 != ff) ^= 8 BIN: 11110111 11111111 4fe22d04: (ff != f7) ^= 8 BIN: 11111111 11110111 501c4f04: (fd != f5) ^= 8 BIN: 11111101 11110101 503eb104: (f7 != ff) ^= 8 BIN: 11110111 11111111 51f04304: (d7 != df) ^= 8 BIN: 11010111 11011111 51f7c704: (ff != f7) ^= 8 BIN: 11111111 11110111 522e570c: (df != d7) ^= 8 BIN: 11011111 11010111 52320b04: (f7 != ff) ^= 8 BIN: 11110111 11111111 5271cc04: (7f != 77) ^= 8 BIN: 01111111 01110111 5274da04: (ef != e7) ^= 8 BIN: 11101111 11100111 53791f04: (f7 != ff) ^= 8 BIN: 11110111 11111111 538eb904: (df != d7) ^= 8 BIN: 11011111 11010111 53b9a304: (f7 != ff) ^= 8 BIN: 11110111 11111111 53d0bf04: (ff != f7) ^= 8 BIN: 11111111 11110111 555a5f04: (ff != f7) ^= 8 BIN: 11111111 11110111 560b8204: (f7 != ff) ^= 8 BIN: 11110111 11111111 560b8c04: (77 != 7f) ^= 8 BIN: 01110111 01111111 571fb704: (f7 != ff) ^= 8 BIN: 11110111 11111111 57a71404: (dd != d5) ^= 8 BIN: 11011101 11010101 587d1004: (77 != 7f) ^= 8 BIN: 01110111 01111111 590d3104: (d7 != df) ^= 8 BIN: 11010111 11011111 59b3a504: (d7 != df) ^= 8 BIN: 11010111 11011111 59f42304: (f7 != ff) ^= 8 BIN: 11110111 11111111 5bf24c04: (f7 != ff) ^= 8 BIN: 11110111 11111111 5c82bc04: (ff != f7) ^= 8 BIN: 11111111 11110111 5ed83c04: (df != d7) ^= 8 BIN: 11011111 11010111 5ee54b04: (f7 != ff) ^= 8 BIN: 11110111 11111111 5f75ba04: (f7 != ff) ^= 8 BIN: 11110111 11111111 5ff41004: (77 != 7f) ^= 8 BIN: 01110111 01111111 5ffc4504: (ff != f7) ^= 8 BIN: 11111111 11110111 6001a204: (d7 != df) ^= 8 BIN: 11010111 11011111 6086a604: (ff != f7) ^= 8 BIN: 11111111 11110111 612ec604: (ff != f7) ^= 8 BIN: 11111111 11110111 61cab504: (f7 != ff) ^= 8 BIN: 11110111 11111111 61e53204: (7f != 77) ^= 8 BIN: 01111111 01110111 62de0204: (f7 != ff) ^= 8 BIN: 11110111 11111111 638d0f04: (77 != 7f) ^= 8 BIN: 01110111 01111111 63ddb804: (f7 != ff) ^= 8 BIN: 11110111 11111111 66149304: (f7 != ff) ^= 8 BIN: 11110111 11111111 66504e04: (ff != f7) ^= 8 BIN: 11111111 11110111 66b42a04: (7f != 77) ^= 8 BIN: 01111111 01110111 693b2204: (f7 != ff) ^= 8 BIN: 11110111 11111111 69404e04: (ff != f7) ^= 8 BIN: 11111111 11110111 69592504: (f7 != ff) ^= 8 BIN: 11110111 11111111 69fb3004: (e7 != ef) ^= 8 BIN: 11100111 11101111 6a2dad04: (e7 != ef) ^= 8 BIN: 11100111 11101111 6b37bd04: (df != d7) ^= 8 BIN: 11011111 11010111 6b705b04: (ff != f7) ^= 8 BIN: 11111111 11110111 6b85fc04: (ff != f7) ^= 8 BIN: 11111111 11110111 6c724704: (7f != 77) ^= 8 BIN: 01111111 01110111 6c7b1104: (57 != 5f) ^= 8 BIN: 01010111 01011111 6d60cd04: (ff != f7) ^= 8 BIN: 11111111 11110111 6e54b004: (ff != f7) ^= 8 BIN: 11111111 11110111 6e7c2804: (f7 != ff) ^= 8 BIN: 11110111 11111111 6e932804: (57 != 5f) ^= 8 BIN: 01010111 01011111 6ed6f504: (df != d7) ^= 8 BIN: 11011111 11010111 6ef6c904: (f7 != ff) ^= 8 BIN: 11110111 11111111 6f9a9e04: (f7 != ff) ^= 8 BIN: 11110111 11111111 6fe7b204: (f7 != ff) ^= 8 BIN: 11110111 11111111 6feff704: (f7 != ff) ^= 8 BIN: 11110111 11111111 6ff2de04: (d7 != df) ^= 8 BIN: 11010111 11011111 707c0404: (ff != f7) ^= 8 BIN: 11111111 11110111 70c33504: (ff != f7) ^= 8 BIN: 11111111 11110111 7359c104: (c7 != cf) ^= 8 BIN: 11000111 11001111 74f21204: (d7 != df) ^= 8 BIN: 11010111 11011111 75033e04: (f7 != ff) ^= 8 BIN: 11110111 11111111 75905304: (af != a7) ^= 8 BIN: 10101111 10100111 75adc904: (f7 != ff) ^= 8 BIN: 11110111 11111111 761daa04: (77 != 7f) ^= 8 BIN: 01110111 01111111 7669b504: (f7 != ff) ^= 8 BIN: 11110111 11111111 76b3ac04: (d7 != df) ^= 8 BIN: 11010111 11011111 76de7504: (ff != f7) ^= 8 BIN: 11111111 11110111 76f1e404: (ff != f7) ^= 8 BIN: 11111111 11110111 78b6d404: (ff != f7) ^= 8 BIN: 11111111 11110111 79544e04: (f7 != ff) ^= 8 BIN: 11110111 11111111 79e1bf04: (ff != f7) ^= 8 BIN: 11111111 11110111 79ef3604: (ff != f7) ^= 8 BIN: 11111111 11110111 79f43b04: (77 != 7f) ^= 8 BIN: 01110111 01111111 7b330404: (f7 != ff) ^= 8 BIN: 11110111 11111111 7ba80504: (f7 != ff) ^= 8 BIN: 11110111 11111111 7bf94104: (f7 != ff) ^= 8 BIN: 11110111 11111111 7c3d3004: (f7 != ff) ^= 8 BIN: 11110111 11111111 7c7f4604: (ff != f7) ^= 8 BIN: 11111111 11110111 7d65da04: (d7 != df) ^= 8 BIN: 11010111 11011111 7d90db04: (ff != f7) ^= 8 BIN: 11111111 11110111 7daacc04: (f7 != ff) ^= 8 BIN: 11110111 11111111 7e744504: (7e != 76) ^= 8 BIN: 01111110 01110110 7f540204: (c7 != cf) ^= 8 BIN: 11000111 11001111 7f75b204: (e7 != ef) ^= 8 BIN: 11100111 11101111 7f913604: (57 != 5f) ^= 8 BIN: 01010111 01011111 |
|
 galacticroot
join:2004-05-17 clubs: | Problem solved. There was a scratch and corrosion across some of the traces on an unused PCI riser card. |
|