bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-17 11:47 pm
Re: Sporadic corrupt downloadsI thought perhaps disabling my anti-virus (nod32) would be a good test since it inserts itself into http flows. It didn't help either. I wonder if the modem is bad. Could that be possible? |
|
Bob4Account deleted join:2012-07-22 New Jersey |
Bob4
Member
2012-Oct-18 6:37 am
How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?
1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.
2. Contact the site and ask if they've been having trouble with corrupted files recently. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 9:06 am
said by Bob4:How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?
1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.
2. Contact the site and ask if they've been having trouble with corrupted files recently. 1. I did exactly thatg. I downloaded 5 copies of the same 1GB video all one right after the other. The first 3 copies were fine. The 4th and 5th were corrupt. 2. It happens with two completely different sites so I don't think it has to do with the sites themselves, especially in view of #1. -- bc |
|
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
How are you downloading, UDP or TCP? Corrupt packets are normal on the Internet but there are mechanisms in place which normally deal with most of them. TCP has its own error control built in, though it is far from perfect. If you are downloading via UDP you might be better off if it can be done with TCP. » en.wikipedia.org/wiki/Tr ··· etection |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 9:17 am
I presume TCP as I initiate the downloads from a webpage (right click the file, choose save-as and store on my desktop). |
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
I believe that it would be TCP. You could try running wireshark and see if you can spot any differences in the packets captures on the good/bad downloads. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 9:28 am
One thing I did was to use compare tools to compare a good copy of one of the videos to a bad one. That particular file was identical all the way out to about 70% through it. At the point where it diverged, it is not as if there were just a few bytes different. The bytes past that point in each file were totally different for a long way (I did not exhaustively compare all the remaining data to see if perhaps they became identical again at some point, which I suppose might be possible if, for instance, some data was just dropped).
I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ... -- bc |
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
said by bcronin:I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ... -- bc Yeah I suspect it would be a major undertaking. |
|
Bob4Account deleted join:2012-07-22 New Jersey |
to bcronin
Next logical step would be to try a download on a different PC. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 9:43 am
I've tried it on two different PC's and get the same kind of results. Its sporadic but it does happen on both. -- bc |
|
bcronin |
bcronin
Premium Member
2012-Oct-18 9:51 am
I just replicated it again. This time the good file and bad file were different sizes. The good file had exactly 8k more data than the bad. That's got to be significant somehow. Ideas? -- bc |
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
While it is 2 different sites, there is some minor possibility that they use the same CDN. You could run wireshark for the beginning of the downloads to see exactly where the files originate. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 10:17 am
Possibly, twit.tv definitely uses cachefly, but its not clear whether revision3 does or not. None of their URL's reference it, anyway ... -- bc |
|
bcronin |
bcronin
Premium Member
2012-Oct-18 11:12 am
For what its worth I used xvi32 to find the location in the bad copy of the video where the 8k block is missing. The two files are identical out to offset 187CD25D, at which point there is an 8K block of data that is entirely missing from the bad file. The data following that 8K block in the good file is identical to the data in the bad file from offset 187CD25E through the end of the file.
So, it looks like blocks of data (in multiples of 1K) are being randomly dropped.
Now, what would cause that? -- bc |
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
TCP should not allow that to happen. » packetlife.net/blog/2010 ··· numbers/The client on either side of a TCP session maintains a 32-bit sequence number it uses to keep track of how much data it has sent. This sequence number is included on each transmitted packet, and acknowledged by the opposite host as an acknowledgement number to inform the sending host that the transmitted data was received successfully.
When a host initiates a TCP session, its initial sequence number is effectively random; it may be any value between 0 and 4,294,967,295, inclusive. However, protocol analyzers like Wireshark will typically display relative sequence and acknowledgement number in place of the field's actual value. These values are relative to the initial sequence number of that stream. This is handy, as it is much easier to keep track of relatively small, predictable numbers rather than the actual numbers sent on the wire. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 11:43 am
Maybe its filesystem related. I notice my drives have 4K per cluster, If somehow 2 of them got lost, that might be an explanation. But its happening on two different PC's, so I don't think hardware is the issue. Maybe I'll uninstall my defragger and see if anything changes (both PC's use the same one, PerfectDisk 12.5). |
|
bcronin |
bcronin
Premium Member
2012-Oct-18 12:18 pm
Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.
Now to try to convince Raxco ... -- bc |
|
|
said by bcronin:Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.
Now to try to convince Raxco ... -- bc You did WHAT exactly to eliminate the problem? What in the world is "PD" ? |
|
|
said by cablewizzard:said by bcronin:Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.
Now to try to convince Raxco ... -- bc You did WHAT exactly to eliminate the problem? What in the world is "PD" ? Perfect Disk I assume, since he mentioned it in earlier post. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 2:51 pm
Correct, sorry for the confusion. Preliminary indications are that uninstalling it may have solved it, but so far the evidence is circumstantial at best. If I don't get any more corrupted downloads over the next 4-5 days, then I'll have more confidence that this was indeed the answer. In the meantime I've alerted the support staff at Raxco that there may be a problem with PerfectDisk. They're skeptical of course (as I suppose I would be too), but time will tell. |
|
|
Is using different computer an option? Something with fairly different software setup (OS, etc)? If you still have problems with another PC that probably means the problem is not related to specific PC.
If you don't have another PC it may even be worth trying some virtual machines that you can rent from Amazon EC2 on hourly basis. If problem still occurs on VM then more than likely the root cause is in the source for your files because with Amazon VM you would be using different network to access the source. |
|
bcronin Premium Member join:2004-03-27 Hyde Park, NY |
bcronin
Premium Member
2012-Oct-18 8:42 pm
More testing revealed that the problem was still happening even after PerfectDisk was uninstalled. So next I focused on nod32. I uninstalled that and downloaded 4-5GB worth of videos and didn't have one error. Then I reinstalled it and downloaded the same set of videos again and 9 of the 11 files were corrupt. Searching through the nod32 options led me to a feature called "protocol filtering" whereby nod32 claims to scan incoming http and pop3 data streams looking for trouble. That seemed to me to be a likely thing to try to turn off. After I did and re-downloaded all the files again (I'm glad I don't have a data cap , they were all clean. So I've concluded now that the problem lies in nod32's protocol filtering feature. Apologies to Raxco for a hasty conclusion earlier today based on not-enough-testing. -- bc |
|
bcronin |
bcronin
Premium Member
2012-Oct-19 10:01 am
Quick follow-up. It turns out this issue was triggered by a recent Windows Update, KB2735855. See this thread for more information: » www.wilderssecurity.com/ ··· t=332920-- bc |
|
TheWiseGuyDog And Butterfly MVM join:2002-07-04 East Stroudsburg, PA |
I'm glad you found the cause. I must admit, I thought it would be difficult if not impossible to solve. Kudos |
|
|
Bob4Account deleted join:2012-07-22 New Jersey |
to bcronin
The Windows hooks that the anti-virus vendors use seem to cause a lot of problems. McAfee has had a non-paged pool memory leak for weeks, caused by the interaction of their code and the Windows hooks. We've been waiting for McAfee to release a fix. Meanwhile, I have to reboot every day. |
|