Header Import Tying Up Entire Application

Technical support and discussion of Newsbin Version 6 series.

Header Import Tying Up Entire Application

Postby thuvia » Sat Jul 02, 2011 2:10 pm

It's going to take me several days at least to import nearly 100GB of RV4 format group data (I like my local searches!). It would be nice if the silly progress bar didn't prevent access to any other part of NewsBin (even the minimize button!!).
thuvia
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Sat Jul 02, 2011 7:59 am
Location: Scotland

Registered Newsbin User since: 07/01/06

Re: Header Import Tying Up Entire Application

Postby rh » Sat Jul 02, 2011 5:43 pm

I did a few, saw how long it was taking, saved the rest for bed time. Woke up, all done. Will never need it again. Glad it's there. No need to put more work into it.
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Header Import Tying Up Entire Application

Postby thuvia » Sat Jul 02, 2011 6:05 pm

I wouldn't have minded so much either if it would all be done in one night, but having seen it struggle for hours through a single group, that's not going to be the case for me.

Also, I find this use of a modal dialog to be very sub-optimal UI design, and remember seeing it elsewhere in a later revision of NewsBin 5, but at least there was an option to disable it (post deletion progress).
thuvia
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Sat Jul 02, 2011 7:59 am
Location: Scotland

Registered Newsbin User since: 07/01/06

Re: Header Import Tying Up Entire Application

Postby rh » Sat Jul 02, 2011 6:28 pm

Granted, not an optimal design, but it's a one shot utility - not something you'll use every day.

Besides, you were warned :)

http://www.newsbin.com/beta.php

"A progress screen will pop up and you'll have to wait for it to finish before continuing."
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Header Import Tying Up Entire Application

Postby thuvia » Sat Jul 02, 2011 6:45 pm

Yes, it probably should have said "you may have to wait a few days for it to finish before continuing"... :roll:
thuvia
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Sat Jul 02, 2011 7:59 am
Location: Scotland

Registered Newsbin User since: 07/01/06

Re: Header Import Tying Up Entire Application

Postby DThor » Sun Jul 03, 2011 9:21 am

Personally, I can't be bothered exercising my disk that much, but then I'm not a hard core spool downloader. I just redownload.

Btw, a tip once you're done, I found that with all six of my connections downloading headers every day, the disk IO was getting pretty crazy. There was obviously a lot of thrashing going on, and that simply increases the overall time to insert new spool. I found the time to update them lessened significantly when I only used two connections. I copied that server, and set it to be a slave server, which won't attempt to get spools, so the upshot was I had the same number of download connections, but only two trying to insert spool at a given time, overall time to update spools lessened.

DT
V6 Troubleshooting FAQ . V6 docs. Usenet info at Usenet Tools. Thanks!
User avatar
DThor
Elite NewsBin User
Elite NewsBin User
 
Posts: 5943
Joined: Mon Jul 01, 2002 9:50 am

Registered Newsbin User since: 04/01/03

Re: Header Import Tying Up Entire Application

Postby rh » Sun Jul 03, 2011 2:05 pm

DThor wrote:Btw, a tip once you're done
DT


Thanks for that tip! I was trying last night to figure out how to ask a question about the thrashing I was seeing - that I could not ever recall experiencing 5.x, who's headers would come down quickly. I decided to use 6.x for a while to see if it was persistent or just a 1st time occurrence.
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Header Import Tying Up Entire Application

Postby rh » Mon Jul 04, 2011 1:53 am

Let me see if I have this right.

Starting with a single server and 10 connections. Change that to a slave with 8 connections. Then create a second master server with 2 connections?

So header downloads will use the master and article downloads with use ?both?
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Header Import Tying Up Entire Application

Postby itimpi » Mon Jul 04, 2011 3:40 am

Correct.
The Newsbin Online documentation
The Usenettools for tutorials, useful information and links
User avatar
itimpi
Elite NewsBin User
Elite NewsBin User
 
Posts: 12604
Joined: Sat Mar 16, 2002 7:11 am
Location: UK

Registered Newsbin User since: 03/28/03

Re: Header Import Tying Up Entire Application

Postby DThor » Mon Jul 04, 2011 8:54 am

Yeah, sorry I wasn't totally clear.

There was some discussion on irc that perhaps in future this could be folded into Newsbin so that in a server configuration, you could set, say, 'maximum number of spool insert threads ' or something perhaps a little less klingon-ese, to save having to hack around this. Two threads still keeps the IO pretty constant on a decent pipe, frankly I can't imagine why anyone would want more unless they're running a raid or ssd. Hmm, or perhaps if they're running a really slow connection too, I guess.

DT
V6 Troubleshooting FAQ . V6 docs. Usenet info at Usenet Tools. Thanks!
User avatar
DThor
Elite NewsBin User
Elite NewsBin User
 
Posts: 5943
Joined: Mon Jul 01, 2002 9:50 am

Registered Newsbin User since: 04/01/03

Re: Header Import Tying Up Entire Application

Postby Quade » Mon Jul 04, 2011 9:35 am

You can also use "Background Mode" which can download headers with all connections but, uses a single thread to insert the records in the DB. This will also reduce disk IO load.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Header Import Tying Up Entire Application

Postby rh » Mon Jul 04, 2011 8:54 pm

Newsbin Options / Switches / Background Update

http://forums.newsbin.com/viewtopic.php?f=43&t=28453&p=174288&hilit=background+update#p174219

I'll try that too.

Thanks!
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04


Return to V6 Technical Support

Who is online

Users browsing this forum: No registered users and 3 guests