dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
share rss forum feed


bcronin
Premium
join:2004-03-27
Hyde Park, NY
reply to frdrizzt

Re: Sporadic corrupt downloads

Two different sources (twit.tv and revision3.com).
--
bc


bcronin
Premium
join:2004-03-27
Hyde Park, NY
It occurred to me just now that perhaps a good place to start would be to replace the cable running from the cable modem to the router. I'll do that and see if anything changes.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Didn't help. Downloaded 4 copies of a nearly 1GB video file. The first 3 were fine. The 4th, corrupt. Hmmm.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Click for full size
For what its worth, here's the modem status page.

UofMiamiGrad
Premium
join:2001-02-03
Great Neck, NY
How about your upstream signal levels?


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Click for full size
Oops, sorry. Here they are.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
I thought perhaps disabling my anti-virus (nod32) would be a good test since it inserts itself into http flows. It didn't help either. I wonder if the modem is bad. Could that be possible?

Bob4
Account deleted

join:2012-07-22
New Jersey
Reviews:
·Optimum Online
How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?

1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.

2. Contact the site and ask if they've been having trouble with corrupted files recently.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
said by Bob4:

How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?

1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.

2. Contact the site and ask if they've been having trouble with corrupted files recently.

1. I did exactly thatg. I downloaded 5 copies of the same 1GB video all one right after the other. The first 3 copies were fine. The 4th and 5th were corrupt.

2. It happens with two completely different sites so I don't think it has to do with the sites themselves, especially in view of #1.
--
bc

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
How are you downloading, UDP or TCP? Corrupt packets are normal on the Internet but there are mechanisms in place which normally deal with most of them. TCP has its own error control built in, though it is far from perfect. If you are downloading via UDP you might be better off if it can be done with TCP.

»en.wikipedia.org/wiki/Transmissi···etection
--
Warning, If you post nonsense and use misinformation and are here to argue based on those methods, you will be put on ignore.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
I presume TCP as I initiate the downloads from a webpage (right click the file, choose save-as and store on my desktop).

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
I believe that it would be TCP. You could try running wireshark and see if you can spot any differences in the packets captures on the good/bad downloads.
--
Warning, If you post nonsense and use misinformation and are here to argue based on those methods, you will be put on ignore.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
One thing I did was to use compare tools to compare a good copy of one of the videos to a bad one. That particular file was identical all the way out to about 70% through it. At the point where it diverged, it is not as if there were just a few bytes different. The bytes past that point in each file were totally different for a long way (I did not exhaustively compare all the remaining data to see if perhaps they became identical again at some point, which I suppose might be possible if, for instance, some data was just dropped).

I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ...
--
bc

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
said by bcronin:

I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ...
--
bc

Yeah I suspect it would be a major undertaking.
--
Warning, If you post nonsense and use misinformation and are here to argue based on those methods, you will be put on ignore.

Bob4
Account deleted

join:2012-07-22
New Jersey
reply to bcronin
Next logical step would be to try a download on a different PC.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
I've tried it on two different PC's and get the same kind of results. Its sporadic but it does happen on both.
--
bc


bcronin
Premium
join:2004-03-27
Hyde Park, NY
I just replicated it again. This time the good file and bad file were different sizes. The good file had exactly 8k more data than the bad. That's got to be significant somehow. Ideas?
--
bc

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
While it is 2 different sites, there is some minor possibility that they use the same CDN. You could run wireshark for the beginning of the downloads to see exactly where the files originate.
--
Warning, If you post nonsense and use misinformation and are here to argue based on those methods, you will be put on ignore.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Possibly, twit.tv definitely uses cachefly, but its not clear whether revision3 does or not. None of their URL's reference it, anyway ...
--
bc


bcronin
Premium
join:2004-03-27
Hyde Park, NY
For what its worth I used xvi32 to find the location in the bad copy of the video where the 8k block is missing. The two files are identical out to offset 187CD25D, at which point there is an 8K block of data that is entirely missing from the bad file. The data following that 8K block in the good file is identical to the data in the bad file from offset 187CD25E through the end of the file.

So, it looks like blocks of data (in multiples of 1K) are being randomly dropped.

Now, what would cause that?
--
bc

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
TCP should not allow that to happen.

»packetlife.net/blog/2010/jun/7/u···numbers/

The client on either side of a TCP session maintains a 32-bit sequence number it uses to keep track of how much data it has sent. This sequence number is included on each transmitted packet, and acknowledged by the opposite host as an acknowledgement number to inform the sending host that the transmitted data was received successfully.

When a host initiates a TCP session, its initial sequence number is effectively random; it may be any value between 0 and 4,294,967,295, inclusive. However, protocol analyzers like Wireshark will typically display relative sequence and acknowledgement number in place of the field's actual value. These values are relative to the initial sequence number of that stream. This is handy, as it is much easier to keep track of relatively small, predictable numbers rather than the actual numbers sent on the wire.


--
Warning, If you post nonsense and use misinformation and are here to argue based on those methods, you will be put on ignore.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Maybe its filesystem related. I notice my drives have 4K per cluster, If somehow 2 of them got lost, that might be an explanation. But its happening on two different PC's, so I don't think hardware is the issue. Maybe I'll uninstall my defragger and see if anything changes (both PC's use the same one, PerfectDisk 12.5).


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc

cablewizzard

join:2009-06-14
Hicksville, NY
kudos:1
said by bcronin:

Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc

You did WHAT exactly to eliminate the problem?
What in the world is "PD" ?

UofMiamiGrad
Premium
join:2001-02-03
Great Neck, NY
said by cablewizzard:

said by bcronin:

Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc

You did WHAT exactly to eliminate the problem?
What in the world is "PD" ?

Perfect Disk I assume, since he mentioned it in earlier post.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Correct, sorry for the confusion. Preliminary indications are that uninstalling it may have solved it, but so far the evidence is circumstantial at best. If I don't get any more corrupted downloads over the next 4-5 days, then I'll have more confidence that this was indeed the answer. In the meantime I've alerted the support staff at Raxco that there may be a problem with PerfectDisk. They're skeptical of course (as I suppose I would be too), but time will tell.

JoeSchmoe007
Premium
join:2003-01-19
Brooklyn, NY
Reviews:
·Optimum Online
Is using different computer an option? Something with fairly different software setup (OS, etc)? If you still have problems with another PC that probably means the problem is not related to specific PC.

If you don't have another PC it may even be worth trying some virtual machines that you can rent from Amazon EC2 on hourly basis. If problem still occurs on VM then more than likely the root cause is in the source for your files because with Amazon VM you would be using different network to access the source.


bcronin
Premium
join:2004-03-27
Hyde Park, NY
reply to bcronin
More testing revealed that the problem was still happening even after PerfectDisk was uninstalled. So next I focused on nod32. I uninstalled that and downloaded 4-5GB worth of videos and didn't have one error. Then I reinstalled it and downloaded the same set of videos again and 9 of the 11 files were corrupt. Searching through the nod32 options led me to a feature called "protocol filtering" whereby nod32 claims to scan incoming http and pop3 data streams looking for trouble. That seemed to me to be a likely thing to try to turn off. After I did and re-downloaded all the files again (I'm glad I don't have a data cap , they were all clean. So I've concluded now that the problem lies in nod32's protocol filtering feature. Apologies to Raxco for a hasty conclusion earlier today based on not-enough-testing.
--
bc


bcronin
Premium
join:2004-03-27
Hyde Park, NY
Quick follow-up. It turns out this issue was triggered by a recent Windows Update, KB2735855. See this thread for more information:
»www.wilderssecurity.com/showthre···t=332920
--
bc

TheWiseGuy
Dog And Butterfly
Premium,MVM
join:2002-07-04
East Stroudsburg, PA
kudos:3
I'm glad you found the cause. I must admit, I thought it would be difficult if not impossible to solve. Kudos