Slow header d/l after v5.59 updgrade to v6

Technical support and discussion of Newsbin Version 6 series.

Slow header d/l after v5.59 updgrade to v6

Postby PSPDWB » Mon Jul 25, 2011 12:43 am

I upgraded from 5.59 to 6 and then to 6.01B1. Since upgrading to 6, it takes many hours to update my groups using the "download latest" option, even if I "download latest" almost every day. I did the "download all headers" to completion after upgrading, so it's not like it's trying to do that, but it takes about as long as if it was. I do subscribe to some large groups, like a.b.dvd, but with version 5.59, it took maybe an hour max to complete "download latest" whereas now it takes several hours. My dl speeds fluctuate, but can get up to 120Mpbs, so that doesn't seem to be the problem, and nothing much else has changed with my pc, ISP, or news server, other that upgrading to v6.

Any ideas what might be wrong?
PSPDWB
n00b
n00b
 
Posts: 1
Joined: Sun Jul 24, 2011 6:38 pm

Registered Newsbin User since: 02/18/04

Re: Slow header d/l after v5.59 updgrade to v6

Postby itimpi » Mon Jul 25, 2011 2:11 am

You should expect v6 to take longer to download and save headers.

With v5 when headers were downloaded they were basically just written to disk.

With v6 after actually downloading the headers there is then considerable extra work done to process these headers into the .db3 file that is used for loading into post lists. It is this additional processing that allows v6 to load groups into post lists much after, and run with a much smaller memory footprint.

Having said that I do not know if the times you are experiencing are typical. If you tend to leave Newsbin running you could set NB to automatically download headers at more frequent intervals to spread the load. If doing this there are techniques you can use to limit the number of connections allowed to download headers thus minimizing the impact on file downloads running in parallel.
The Newsbin Online documentation
The Usenettools for tutorials, useful information and links
User avatar
itimpi
Elite NewsBin User
Elite NewsBin User
 
Posts: 12604
Joined: Sat Mar 16, 2002 7:11 am
Location: UK

Registered Newsbin User since: 03/28/03

Re: Slow header d/l after v5.59 updgrade to v6

Postby Quade » Mon Jul 25, 2011 8:12 am

In the options, you can turn on "Background Update". This will speed up header downloads but, it'll take longer for the new records to become visible in the post lists.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Slow header d/l after v5.59 updgrade to v6

Postby rh » Mon Jul 25, 2011 10:22 pm

Turning on that option made my v6 header downloads work just like v5.59. Much better. Not sure why this is slower in v6 than v5.59 without the option.

As far as taking longer to appear in the post lists, I recall a thread that mentioned that. I think there was a desire for some sort of progress indicator. Is there one? How would one know when the background update is complete?
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Slow header d/l after v5.59 updgrade to v6

Postby DThor » Tue Jul 26, 2011 7:49 am

No matter what, v6 takes longer than v5 to process spools simply because 5 was more of a raw dump to disk. The upside with v6 is that with that processing you get a much more condensed spool that lets you load significantly more spool a lot faster with a lot less memory use. It's assumed if you want to bother storing spools locally in this world of nzb and Internet search like the kind newsbin offers, searching and loading spools fast is the repeated process you want optimized. The foreground / background thing is really just there to give an option for those that use newsbin differently. I like it in the foreground because that is effectively your feedback to progress, background means 'just get to it when you can, I'm doing something else'.
I agree it wouldn't hurt to have some sort of indication of how it's doing, though.

DT
V6 Troubleshooting FAQ . V6 docs. Usenet info at Usenet Tools. Thanks!
User avatar
DThor
Elite NewsBin User
Elite NewsBin User
 
Posts: 5943
Joined: Mon Jul 01, 2002 9:50 am

Registered Newsbin User since: 04/01/03

Re: Slow header d/l after v5.59 updgrade to v6

Postby rh » Tue Jul 26, 2011 10:51 pm

Thanks for the explanation! That certainly explains why search is so much faster. I just finished a typical search session and the difference from 5.59 is enormous.

Yeah, I'm old school for now, everything locally. Definitely need some sort of way (in my use case) to know when it's safe (ready) to load a recently updated spool.

Maybe my machine is quick enough that there's not much lag. I download headers and go do something else. When I come back, it's done and there's no out of the ordinary disk i/o present on any drives.
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Slow header d/l after v5.59 updgrade to v6

Postby Quade » Wed Jul 27, 2011 7:57 am

On my machine, background mode's pretty quick too for normal update. If you do something like "Download all Headers" on a significant number of groups AND you have a fast connection, it's pretty easy to get pretty far behind. With compression, my header downloads hit 400 Mbps. Trying to insert that in real time is pretty unrealistic.

I've added an indicator for import state. Not sure it it's enough or if it's in the correct place but, as a prototype, it's doing what I expect.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Slow header d/l after v5.59 updgrade to v6

Postby rh » Fri Jul 29, 2011 8:21 pm

My connection is only 20Mb, not "fast" compared to others I've seen mentioned here. I only download about 20 groups at any one time and almost never "Download All Headers".

So I'm probably safe. In the next update I'll observe the indicator and it will likely verify that I'm not pushing any envelope.
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Slow header d/l after v5.59 updgrade to v6

Postby Quade » Fri Jul 29, 2011 9:25 pm

With 250 groups, I see it queue 150-160 blocks of data for insert. When I just update a couple groups, like a double-click on a topic it's pretty much real time.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: No registered users and 3 guests