Dealing with incompletes

Technical support and discussion of Newsbin Version 6 series.

Dealing with incompletes

Postby bruce73 » Fri Jun 10, 2011 10:02 am

I have the v.6.0RC1 on a Win 7/64 machine and am having problems with large files and how it manages incompletes. I understand that it doesn't auto-assemble relatively new files and understand the rationale for that. But the issue I'm having is when the program is closed and what happens when it is re-opened and the download list still has this file.

Here's an example: this morning after an all-night download of a 40GB file, there were ~20 incompletes. Before I could manually assemble them, I needed to reboot for some other issue. When I restarted NB, the program re-downloaded all of those files (about 1GB worth) -- only then was I able to manually assemble. IIRC, v.5.xx would simply have the incompletes listed, ready for assembly. Am I missing something somewhere? Is this how it's supposed to work?

FWIW, I don't have auto-RAR enabled, because I'd rather pick the time to do that, and not have it be automatic. That shouldn't make a difference, should it?
User avatar
bruce73
Seasoned User
Seasoned User
 
Posts: 101
Joined: Sat Jun 26, 2004 10:34 am
Location: Los Angeles

Registered Newsbin User since: 06/23/04

Re: Dealing with incompletes

Postby Quade » Fri Jun 10, 2011 10:13 am

What do you have the "MemCacheLimit" set to? If you don't know then it's probably the default which I believe is 15 megs. Files that are cached in memory probably don't get saved on restart. I'll check the large files out and report back.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43660
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Dealing with incompletes

Postby bruce73 » Fri Jun 10, 2011 10:51 am

I assume that setting is in the .ini file, which I haven't edited, so it would be at the default. So, since my first post -- after assembling all the incompletes, the program just sat there. So I ran the PAR2 file and saw how many blocks I needed (44). Added that many to the download list, they downloaded but just sat in the list. So, long story short, I uninstalled v.6 and re-installed v.5.59 and I finally got the file to repair.

I really like v.6 -- the GUI, the improvements with headers, etc. -- but I don't like the way incompletes work. Maybe it's just me and that I'm used to doing things a certain way, but for large files I still prefer v.5.59.
User avatar
bruce73
Seasoned User
Seasoned User
 
Posts: 101
Joined: Sat Jun 26, 2004 10:34 am
Location: Los Angeles

Registered Newsbin User since: 06/23/04

Re: Dealing with incompletes

Postby Moondawgie » Fri Jun 10, 2011 2:43 pm

This issue, and one involving the deletion of message parts by an nntp provider ( :shock: ) before Newsbin has finished downloading and assembling all of the parts into a complete file, are the remaining big problems which I encounter with my use of NewsBin 6.

The problem with files being stored in memory, rather than being written to disk, when my computer being automatically shutdown during the night by my backup power supply when the power has gone out, is what has led to my receipt of two warnings from Charter Communications that I'd exceeded my monthly 100GB allotment of "bandwidth". With the last warning they informed me that the next time will result in my loss of their cable internet service for 6 months. Unfortunately for me, Charter is the only broadband internet service available at my location.

In those situations with Newsbin 5 where the nntp provider would delete files, I would contact friends and acquaintances who use other nntp providers from whose nntp servers the parts might not yet have been deleted, asking them to retrieve and send me the missing parts, which I'd add to those downloaded and saved to my computer by Newsbin 5, and use to assemble a complete file. Some might find it interesting that some such files sometimes are deleted from the nntp servers at Easynews, and Giganews, but remain, at least for awhile longer, on the http server at Easynews. I've not yet figured out how to save the individual parts of multi-part postings for use in such situations, but I've a feeling that it can be done. :?

Such problems have led to my resorting to the use of my old (s l o w) newsreader, Forte Agent, to download such files, so that far less is lost in the event of a loss of power, or computer crash.
MS Windows 10 Pro 64-Bit Insider Preview - ASUS P8Z77-V LE; Intel i7-3770K 3.5 GHz; 16.0GB DDR3; 300 Mbps cable internet.
User avatar
Moondawgie
Seasoned User
Seasoned User
 
Posts: 404
Joined: Mon Aug 13, 2001 5:17 pm

Registered Newsbin User since: 05/17/03

Re: Dealing with incompletes

Postby itimpi » Fri Jun 10, 2011 2:49 pm

KingFish wrote:The problem with files being stored in memory, rather than being written to disk, when my computer being automatically shutdown during the night by my backup power supply when the power has gone out, is what has led to my receipt of two warnings from Charter Communications that I'd exceeded my monthly 100GB allotment of "bandwidth". With the last warning they informed me that the next time will result in my loss of their cable internet service for 6 months. Unfortunately for me, Charter is the only broadband internet service available at my location.

You can turn off the caching of files in RAM if you do not ming the performance hit by setting the size for the memory cache to 0 in the .nbi file. You can also set the upper size for files that will be cached by a nbi file entry. Sounds as if these settings might be worthwhile to you.
The Newsbin Online documentation
The Usenettools for tutorials, useful information and links
User avatar
itimpi
Elite NewsBin User
Elite NewsBin User
 
Posts: 12591
Joined: Sat Mar 16, 2002 7:11 am
Location: UK

Registered Newsbin User since: 03/28/03

Re: Dealing with incompletes

Postby Moondawgie » Fri Jun 10, 2011 2:50 pm

I'll certainly give this suggestion a shot.

Thank you.
MS Windows 10 Pro 64-Bit Insider Preview - ASUS P8Z77-V LE; Intel i7-3770K 3.5 GHz; 16.0GB DDR3; 300 Mbps cable internet.
User avatar
Moondawgie
Seasoned User
Seasoned User
 
Posts: 404
Joined: Mon Aug 13, 2001 5:17 pm

Registered Newsbin User since: 05/17/03

Re: Dealing with incompletes

Postby Quade » Fri Jun 10, 2011 6:33 pm

Ok, did find something with saving partials. A single missing block was enough to make it miss the rest of the downloaded blocks. Fixed. Be in the next RC.

All the stuff about memory mode is still true though.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43660
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Dealing with incompletes

Postby bruce73 » Fri Jun 10, 2011 8:56 pm

So the thing about the Memory Cache is that partials are not written to disc at all? Only when they are combined? And that increases performance? I gather this is new to v.6, right?
User avatar
bruce73
Seasoned User
Seasoned User
 
Posts: 101
Joined: Sat Jun 26, 2004 10:34 am
Location: Los Angeles

Registered Newsbin User since: 06/23/04

Re: Dealing with incompletes

Postby Quade » Fri Jun 10, 2011 9:30 pm

Yeah, memory mode caches the entire file in memory and writes it out in one write. It'll cache files up to the size you set. Then when the file is finished, you manually assemble or it assembles itself, it's written out to disk in a single write.

New. The problem is most disks are the bottlenecks these days. Even SSD's are too slow to handle simultaneous, writing, unraring and repairing.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43660
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: No registered users and 1 guest