Broadband Tech > Filesharing Software > How frequesnt do NZB sites update?

How frequesnt do NZB sites update?

I think if you knew the mechanics of how an indexer actually work, you would find that some of your questions don't really make sense.

The process of newsgroups and how you accessed their content has matured over the last two decades. I've been using them back when alt.binaries was for GIFs, before we had jpegs, and back before the GIFs were animated.

Anyways, newsgroups have headers. These are like looking at your e-mail client. You see the subject, some information about the e-mail, but not the e-mail contents or the attachments. Alt.binaries is where attachments can go. Anything outside of alt.binaries are text only.

Pulling down headers is the process of pulling down hundreds of thousands or millions of these headers, these descriptors. You could then decide to pull down the entire message if you want, similar to opening up an e-mail and simultaneously bringing down the attachment.

Because of the sheer volume of posts, indexers came about as a search engine mechanism for the public. They continuously pull down these headers. These posts have a file-size limit, so larger files like TV shows, movies, etc., are broken into dozens or hundreds of posts. You would have to grab all of these posts and then unrar all the pieces into a single file. If you had issues, you then had to feed them par files which would repair corrupt components. Downloaders like sabnzbd automate the entire process.

An .nzb file is an XML file that collects all of the information of certain posts and then places it into a single .nzb file explaining the group and the various posts that make up whatever specific content you're looking for. This gets fed into your reader (sabnzbd), which pulls down the content.

A news indexer has to go out and sift through millions of posts and dozens of groups, pull down all of those headers and then run them through certain scripts (RegEx's) and create a .nzb file. This .nzb file is then presented online and indexed in the only search engine for you or your sabnzbd (via an API key) to download.

This consumes quite a bit of processing power and quite a bit of bandwidth, especially if you're a public tracker that has to cater to a wide audience of people who want anything and everything under the sun.

Newzbin has a threaded updater which will pull down headers for 10 groups at a time. As one group finishes, it puts a new group in the hopper. Once all the groups update, it runs through various processes of creating the nzb files, updating the website, pulling down tvdb info, etc.

Depending on the equipment and the bandwidth, this process could take a while. At bare minimum, with a handful of groups, it's taking me 45 minutes to cycle through the entire thing. Now that I've added significantly more groups AND I'm trying to backfill while keeping up with the current stuff, the process is taking 4-6 hours per cycle. Once I'm done backfilling, I hope to get the process back down to under 2 hours.

Depending on the hardware behind the indexer and how aggressive their processing is, you could catch a release early in a cycle or later in a cycle. Pulling down and updating content as soon as it is out is a difficult task, especially if you're indexing dozens or hundreds of groups.
--
A government big enough to give you everything you want, is strong enough to take everything you have. -MLK

thanks for the nice description.

at one time, i too, was getting files the old fashion way. Sifting through all of the headers with old clients. ahhh! the old days (circa ~1999 for me)

So from what im gathering from your post, there is some delay until things pop up. Am i understanding that correct?

There has to be. The rate of delay varies by several factors.

reply to seamore
Just to add, for the specific example you mentioned, it was posted with a coded filename last night. But usually tv shows will be available within 20 minutes of airing in my experience.

Based on how automated the DMCA takedown process has become, "coded filenames" may be the future of usenet's survival for TV shows and movies.

Oh boy go back to Napster days when all the files were misnamed. I still have a few I never bothered to change

reply to zzzzzzzzag
Not if people post the nzb on a site monitored by you know who.

There are easy ways to find content even with "coded filenames".