Database update on header download

General NewsBin and Usenet related discussions. Make recommendations for additional specific discussion groups.

Moderators: Quade, dexter

Database update on header download

Postby malhusky » Thu Mar 19, 2020 3:17 pm

Is there a way to speed up the process of downloading new headers for an existing group. For some popular groups I find it can take 15 minutes or more to process the headers after only 3 days worth have been downloaded. I suppose lowering the Storage Age might be one option. What about compressing the database? Cache is currently set at 400.

Another issue is that when updating multiple groups it's not always obvious which groups have finished until the cache clears. Is there a way to determine that while the update is in progress?
Thx
malhusky
Occasional Contributor
Occasional Contributor
 
Posts: 47
Joined: Sat Apr 28, 2012 1:01 am

Registered Newsbin User since: 12/13/10

Re: Database update on header download

Postby Quade » Fri Mar 20, 2020 1:58 pm

Some groups simply don't work for headers anymore. Instead of posting files, they're just posting posts with random subjects. These groups, the header database gets 100 times more writes than a group with proper files. These posts are only usable with NZB Files.

One way to tell is to load the group, disable any size filters and see if it's full of ungrouped 300-750K posts.

I'm thinking about putting out a new beta that simply tosses these posts so they don't clutter the header database.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43776
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Database update on header download

Postby stavros » Tue Apr 28, 2020 3:07 am

Hi Quade,

I'm thinking about putting out a new beta that simply tosses these posts so they don't clutter the header database.

Is there a away to achieve this at the moment, by using an import filter? I've tried to do this, but couldn't quite get it right to focus only on the dodgy headers, ended up casting too wide a net.

I definately think it would be a good option to have on a per group and GOG basis, if at all possible.

Would there also be a way to retrospectively apply this option to an existing message base, to sanitise polluted groups, without re-downloading headers?

regards
Stavros.
stavros
Active Participant
Active Participant
 
Posts: 69
Joined: Wed Dec 24, 2003 2:07 am

Registered Newsbin User since: 12/24/03


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest