dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
1379

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Sporadic corrupt downloads

I download a number of large video podcasts every day (that get copied to one of my TiVo's using TiVo desktop). Lately I have noticed that some of them don't play correctly. They'll be OK for a while, but some minutes into them they'll freeze.

At first I thought it was some problem with the TiVo and/or the PC they're downloaded to. But then I noticed that I could manually download a specific episode and try to simply play it in Windows Media Player and it would behave the same way. So it is clearly not the TiVo. The file was corrupt to begin with.

Then I erased the corrupt episode and re-downloaded it and the new copy would play fine all the way to the end. I then tried the same thing on a different PC and got similar results. Sometimes the downloads were corrupt. Sometimes they were not. So I am thinking its not a PC problem.

It kinda smells like a network issue, but I don't know what to look for next. I did check the cable modem log and did not see any errors around the time of the corrupt downloads. Further, the corrected/uncorrectable counts on each of the 8 downstream channels look really good (very low correcteds, zero uncorrectables). SNR's are all 37 or 38. Power levels range from about to to 5 dbmv depending on the channel (is that good?). This is all very frustrating/worrying and I don't know what to do next.

Over the span of time since the problems began I've used two different routers (so I don't think its related to the router either).

Any advice on how to proceed?
--
bc
frdrizzt
join:2008-05-03
Ronkonkoma, NY

frdrizzt

Member

Do you download the podcasts from the same place? In other words, does this happen if you download from multiple sources?

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Two different sources (twit.tv and revision3.com).
--
bc
bcronin

bcronin

Premium Member

It occurred to me just now that perhaps a good place to start would be to replace the cable running from the cable modem to the router. I'll do that and see if anything changes.
bcronin

bcronin

Premium Member

Didn't help. Downloaded 4 copies of a nearly 1GB video file. The first 3 were fine. The 4th, corrupt. Hmmm.
bcronin

bcronin

Premium Member

Click for full size
For what its worth, here's the modem status page.
UofMiamiGrad
Premium Member
join:2001-02-03
Syosset, NY

UofMiamiGrad

Premium Member

How about your upstream signal levels?

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Click for full size
Oops, sorry. Here they are.
bcronin

bcronin

Premium Member

I thought perhaps disabling my anti-virus (nod32) would be a good test since it inserts itself into http flows. It didn't help either. I wonder if the modem is bad. Could that be possible?
Bob4
Account deleted
join:2012-07-22
New Jersey

Bob4

Member

How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?

1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.

2. Contact the site and ask if they've been having trouble with corrupted files recently.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

said by Bob4:

How do you know that the authors didn't produce a corrupt file (which you downloaded), then fix it later? How do you know that the source site isn't corrupting the data?

1. Take a file which was downloaded correctly and download it several more times and see if any of the subsequent downloads are corrupted.

2. Contact the site and ask if they've been having trouble with corrupted files recently.

1. I did exactly thatg. I downloaded 5 copies of the same 1GB video all one right after the other. The first 3 copies were fine. The 4th and 5th were corrupt.

2. It happens with two completely different sites so I don't think it has to do with the sites themselves, especially in view of #1.
--
bc
TheWiseGuy
Dog And Butterfly
MVM
join:2002-07-04
East Stroudsburg, PA

TheWiseGuy

MVM

How are you downloading, UDP or TCP? Corrupt packets are normal on the Internet but there are mechanisms in place which normally deal with most of them. TCP has its own error control built in, though it is far from perfect. If you are downloading via UDP you might be better off if it can be done with TCP.

»en.wikipedia.org/wiki/Tr ··· etection

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

I presume TCP as I initiate the downloads from a webpage (right click the file, choose save-as and store on my desktop).
TheWiseGuy
Dog And Butterfly
MVM
join:2002-07-04
East Stroudsburg, PA

TheWiseGuy

MVM

I believe that it would be TCP. You could try running wireshark and see if you can spot any differences in the packets captures on the good/bad downloads.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

One thing I did was to use compare tools to compare a good copy of one of the videos to a bad one. That particular file was identical all the way out to about 70% through it. At the point where it diverged, it is not as if there were just a few bytes different. The bytes past that point in each file were totally different for a long way (I did not exhaustively compare all the remaining data to see if perhaps they became identical again at some point, which I suppose might be possible if, for instance, some data was just dropped).

I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ...
--
bc
TheWiseGuy
Dog And Butterfly
MVM
join:2002-07-04
East Stroudsburg, PA

TheWiseGuy

MVM

said by bcronin:

I do have wireshark, but I shudder at the thought of collecting a trace of several copies of a 1GB download ...
--
bc

Yeah I suspect it would be a major undertaking.
Bob4
Account deleted
join:2012-07-22
New Jersey

Bob4 to bcronin

Member

to bcronin
Next logical step would be to try a download on a different PC.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

I've tried it on two different PC's and get the same kind of results. Its sporadic but it does happen on both.
--
bc
bcronin

bcronin

Premium Member

I just replicated it again. This time the good file and bad file were different sizes. The good file had exactly 8k more data than the bad. That's got to be significant somehow. Ideas?
--
bc
TheWiseGuy
Dog And Butterfly
MVM
join:2002-07-04
East Stroudsburg, PA

TheWiseGuy

MVM

While it is 2 different sites, there is some minor possibility that they use the same CDN. You could run wireshark for the beginning of the downloads to see exactly where the files originate.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Possibly, twit.tv definitely uses cachefly, but its not clear whether revision3 does or not. None of their URL's reference it, anyway ...
--
bc
bcronin

bcronin

Premium Member

For what its worth I used xvi32 to find the location in the bad copy of the video where the 8k block is missing. The two files are identical out to offset 187CD25D, at which point there is an 8K block of data that is entirely missing from the bad file. The data following that 8K block in the good file is identical to the data in the bad file from offset 187CD25E through the end of the file.

So, it looks like blocks of data (in multiples of 1K) are being randomly dropped.

Now, what would cause that?
--
bc
TheWiseGuy
Dog And Butterfly
MVM
join:2002-07-04
East Stroudsburg, PA

TheWiseGuy

MVM

TCP should not allow that to happen.

»packetlife.net/blog/2010 ··· numbers/

The client on either side of a TCP session maintains a 32-bit sequence number it uses to keep track of how much data it has sent. This sequence number is included on each transmitted packet, and acknowledged by the opposite host as an acknowledgement number to inform the sending host that the transmitted data was received successfully.

When a host initiates a TCP session, its initial sequence number is effectively random; it may be any value between 0 and 4,294,967,295, inclusive. However, protocol analyzers like Wireshark will typically display relative sequence and acknowledgement number in place of the field's actual value. These values are relative to the initial sequence number of that stream. This is handy, as it is much easier to keep track of relatively small, predictable numbers rather than the actual numbers sent on the wire.


bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Maybe its filesystem related. I notice my drives have 4K per cluster, If somehow 2 of them got lost, that might be an explanation. But its happening on two different PC's, so I don't think hardware is the issue. Maybe I'll uninstall my defragger and see if anything changes (both PC's use the same one, PerfectDisk 12.5).
bcronin

bcronin

Premium Member

Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc
cablewizzard
join:2009-06-14
Woodbury, NY

cablewizzard

Member

said by bcronin:

Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc

You did WHAT exactly to eliminate the problem?
What in the world is "PD" ?
UofMiamiGrad
Premium Member
join:2001-02-03
Syosset, NY

UofMiamiGrad

Premium Member

said by cablewizzard:

said by bcronin:

Hmmm, ok, so I uninstalled PD and downloaded 10 copies of the 1GB video and all of them are fine. Previously I had not gotten more than 4 or 5 good ones in a row. So, I think I may have found the cause. Thanks for your help in confirming that TCP cound not be that cause of the sort of corruption I observed. The exactly-8k-missing-bytes was a key clue.

Now to try to convince Raxco ...
--
bc

You did WHAT exactly to eliminate the problem?
What in the world is "PD" ?

Perfect Disk I assume, since he mentioned it in earlier post.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

Correct, sorry for the confusion. Preliminary indications are that uninstalling it may have solved it, but so far the evidence is circumstantial at best. If I don't get any more corrupted downloads over the next 4-5 days, then I'll have more confidence that this was indeed the answer. In the meantime I've alerted the support staff at Raxco that there may be a problem with PerfectDisk. They're skeptical of course (as I suppose I would be too), but time will tell.
JoeSchmoe007
Premium Member
join:2003-01-19
Brooklyn, NY

JoeSchmoe007

Premium Member

Is using different computer an option? Something with fairly different software setup (OS, etc)? If you still have problems with another PC that probably means the problem is not related to specific PC.

If you don't have another PC it may even be worth trying some virtual machines that you can rent from Amazon EC2 on hourly basis. If problem still occurs on VM then more than likely the root cause is in the source for your files because with Amazon VM you would be using different network to access the source.

bcronin
Premium Member
join:2004-03-27
Hyde Park, NY

bcronin

Premium Member

More testing revealed that the problem was still happening even after PerfectDisk was uninstalled. So next I focused on nod32. I uninstalled that and downloaded 4-5GB worth of videos and didn't have one error. Then I reinstalled it and downloaded the same set of videos again and 9 of the 11 files were corrupt. Searching through the nod32 options led me to a feature called "protocol filtering" whereby nod32 claims to scan incoming http and pop3 data streams looking for trouble. That seemed to me to be a likely thing to try to turn off. After I did and re-downloaded all the files again (I'm glad I don't have a data cap , they were all clean. So I've concluded now that the problem lies in nod32's protocol filtering feature. Apologies to Raxco for a hasty conclusion earlier today based on not-enough-testing.
--
bc