Any way to speed up Header importing?

Technical support and discussion of Newsbin Version 6 series.

Any way to speed up Header importing?

Postby Lurker » Mon Apr 23, 2018 12:02 pm

I try to download headers from even these largely spammed groups and the download of the headers seems fast, but it sits and imports the downloaded headers into the newsbin header database incredibly slow. I am talking 6-8 hours long. Is there any way to speed this up? I am on a 16core processor and while I watch with the system basically idle, task manager says newsbin only occasionally uses 0.1% of my CPU power and negligible amount of system ram. So I am thinking newsbin is not really using the available resources effectively.

Worst part of it is it interferes with the usage of newsbin. So for example if I want to pull the headers from a large group like a.b.ath I right click on the group name select download latest headers and newsbin downloads it relatively quickly. Then it takes a while to actually bring the downloaded header into newsbin's database (it can take hours). Now while newsbin is still processing the header files say I want to check another group immediately like a.b.multimedia.<something> so I double click on the group name to see the headers. Newsbin will just sits there after displaying the non updated headers and does nothing even after it downloaded the headers because a.b.m.<something> is alphabetically after a.b.ath. So till a.b.ath finishes processing it's headers, a.b.m.<something> won't import the headers so the display won't update. The only way to kind of get around it is to actually go into the newbinpro_data\import folder and delete the files ahead of a.b.m.<something>.txt temporarily (ie to the recycle bin) except the very first file which newsbin has open and is processing then wait for it to finish that one file then it imports the a.b.m.<something> headers and updates the display. I can then restore the files in the recycle bin and newsbin will process the remaining files. It seems a very sequential process, so is there a way to multi-thread this so it can process multiple groups at once or maybe have newsbin give priority to processing the group whole posts are being displayed currently?
Lurker
n00b
n00b
 
Posts: 5
Joined: Sun Jan 29, 2017 12:18 pm

Registered Newsbin User since: 07/12/09

Re: Any way to speed up Header importing?

Postby ddaniel51 » Sun Apr 29, 2018 1:45 am

I use an NVME SSD for my OS and additional NVME SSD's for newsbin data and downloads. Doing even a large header update only takes a minute or so (I Luv Newsbin). Intel 7900x, 64GB, Win 10, no page files, 9 TB Samsung 960 Pro SSD's, no mechanical HDD's allowed. Sounds like a badly fragmented HDD churning page files and data.
ddaniel51
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Fri Aug 15, 2003 10:44 pm
Location: Arizona

Registered Newsbin User since: 04/05/03

Re: Any way to speed up Header importing?

Postby Quade » Sun Apr 29, 2018 2:23 am

Sorry I forgot to reply.

Large header imports are a bit problematic. If you load and display a group. The headers for the displayed groups get displayed before other headers. Basically there's an import priority system based on loaded groups.

Once 6.80 is out, I can consider making it multi-threaded. The reason I didn't before was because it was already relatively intensive but with 32 bit systems essentially going away, multi-threading it makes more sense.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43027
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Any way to speed up Header importing?

Postby syshog » Sat May 19, 2018 1:49 am

Quade wrote:Sorry I forgot to reply.

Large header imports are a bit problematic. If you load and display a group. The headers for the displayed groups get displayed before other headers. Basically there's an import priority system based on loaded groups.

Once 6.80 is out, I can consider making it multi-threaded. The reason I didn't before was because it was already relatively intensive but with 32 bit systems essentially going away, multi-threading it makes more sense.


If you are going to enable multi-threaded support for header imports could you at least give us the option of using GZ compression again? I was unimpressed with compressing the Import folder. The txt files get out of control when there are a lot of updates.
system:
xeon with 64GB of ram and a lot of storage.
User avatar
syshog
Seasoned User
Seasoned User
 
Posts: 106
Joined: Sun Jun 18, 2006 7:26 pm

Registered Newsbin User since: 06/17/06

Re: Any way to speed up Header importing?

Postby Quade » Sat May 19, 2018 8:02 am

The txt files get out of control when there are a lot of updates.


What's "out of control" mean? People were complaining about stuck imports and that's mostly gone away with the switch to text only header imports. Virus scanners don't typically touch text files but they do mess with GZ Files. Import is faster too without the compression.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43027
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: Google [Bot] and 3 guests