6.80 RC7 - Header download - *.txt files instead of *.gz

This is the place to help test and discuss Version 6 Beta releases.

6.80 RC7 - Header download - *.txt files instead of *.gz

Postby Calahan » Sat Mar 17, 2018 10:31 am

I'm now testing RC7 and saw that instead of creating *.gz files in the import folder, RC7 now creates *.txt files.
I only updated from B11 to RC7, no settings changed. Is that by (new) design or is the header download not compressed anymore?

Here's a part of my import folder:
old files:
group.name-Newsservername-000039034397-000039548100.gz
group.name-Newsservername-000039547100-000039548144.gz
new file after updating:
group.name-Newsservername-000039547144-000039548156.txt
Calahan
Active Participant
Active Participant
 
Posts: 93
Joined: Wed May 15, 2013 11:14 am

Registered Newsbin User since: 01/14/12

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby Quade » Sat Mar 17, 2018 10:56 am

Yeah, we believe it imports faster as txt than as GZ. They get deleted after import so, they don't stick around. You could turn on folder compression for the import folder.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43314
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby Calahan » Sat Mar 17, 2018 11:42 am

Good to know. Thanks for the info, Quade.
Calahan
Active Participant
Active Participant
 
Posts: 93
Joined: Wed May 15, 2013 11:14 am

Registered Newsbin User since: 01/14/12

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby stavros » Sat Mar 17, 2018 2:06 pm

Hi Quade,

something I have noticed since the switch to *.txt is that the disc seems to be getting fragmented a lot more (and more quickly) now.

I have the whole newsbin directory on a single 2TB hard drive (with the data folder on another physical drive and no other activity than newsbin), that puts the Import and Spool_v6 directories on the same drive.
With the huge increase in file size of the *.txt files in the import directory, I suspect that it is the cause of the fragmentation of both the *.txt files and the spool *.db3 files.

I took the time to completely defragment the drive, but within a day, nearly half of the storage*.db3 files (over 150 of them) had more than 100 fragments each. This is with an auto header download every 1hour of 360 groups. The -wal files are also getting fragmented during the import processing , especially of the larger, more volatile, groups. And, of course, the fragmentation is occurring at the hot spots of the *.db3 files, where subsequent search and update activity is going to occur once the import is completed.

This may be less of an issue for installations on an SSD, but for a hard drive it is not so good.

Do you have any ideas on how to improve the situation (other than defragging daily) ?

regards
Stavros.
stavros
Active Participant
Active Participant
 
Posts: 63
Joined: Wed Dec 24, 2003 2:07 am

Registered Newsbin User since: 12/24/03

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby Quade » Sat Mar 17, 2018 4:17 pm

DB3's will fragment. They get extended in dribs and drabs so, you're not going to get contiguous runs of disk space. It's just the nature of the beast.

As I mentioned before, you could turn on folder compression for the import folder if you want some compression of these files.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43314
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby stavros » Sat Mar 17, 2018 7:14 pm

Thanks, I'll try compressing the import folder and see if that helps.

Is there a variable I can set in the NBI that will allow me to move the spool_v6 directory tree on to a different drive than the rest of the newsbin installation - similar to the data and download?

I've seen a SpoolPath variable, but that is pointing to my newsbin directory, while the data path is pointing to newsbin\data, which in turn contains the spool_v6....

[SETTINGS]
...
DataPath=N:\Newsbin\Data\
...
DownloadPath=F:\DL\$(GROUP)\
...
PosterFile=N:\Newsbin\POSTERS.DAT
...
SpoolPath=N:\Newsbin\

Then, of course, there's the [AUTORAR] bits as well, so I'm not sure if things will break if I change SpoolPath.
That would limit fragmentation to *.db3 growth and -wal & -journal temporary storage (I don't think it is possible to relocate these elsewhere than the *.db3 directory?).

regards
Stavros.
stavros
Active Participant
Active Participant
 
Posts: 63
Joined: Wed Dec 24, 2003 2:07 am

Registered Newsbin User since: 12/24/03

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby dexter » Sat Mar 17, 2018 7:23 pm

You can move the entire Newsbin Data folder but not just the spool folder. That's the largest part anyways. The procedure is here: http://help.newsbin.com/index.php/V660- ... r_location
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9441
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby Quade » Sat Mar 17, 2018 7:31 pm

That would limit fragmentation to *.db3 growth and -wal & -journal temporary storage (I don't think it is possible to relocate these elsewhere than the *.db3 directory?).


Alternatively you could just ignore this issue since it's always been this ways since DB3 first started being used (which is a long time now).
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43314
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby stavros » Sat Mar 17, 2018 7:48 pm

True about the -wal & -journal files, but volume of data and usenet has chnaged a lot since 'the old days' :-)

I can't currently compress the import folder as the drive has a cluster size of 64K (folder compression is only available on <= 4K).

I'll try the suggested info, thanks.
stavros
Active Participant
Active Participant
 
Posts: 63
Joined: Wed Dec 24, 2003 2:07 am

Registered Newsbin User since: 12/24/03

Re: 6.80 RC7 - Header download - *.txt files instead of *.gz

Postby syshog » Sun Mar 18, 2018 1:58 am

Quade wrote:DB3's will fragment. They get extended in dribs and drabs so, you're not going to get contiguous runs of disk space. It's just the nature of the beast.

As I mentioned before, you could turn on folder compression for the import folder if you want some compression of these files.



I wanted to say didn't even think of enabling folder compression that's the good one for the import folder. I haven't experienced any weird processor usage with RC6 like i did RC5. On to RC7 I'll bitch and moan if I run into a problem ;D Good work work will RC6.
system:
xeon with 64GB of ram and a lot of storage.
User avatar
syshog
Seasoned User
Seasoned User
 
Posts: 107
Joined: Sun Jun 18, 2006 7:26 pm

Registered Newsbin User since: 06/17/06


Return to Newsbin Version 6 Beta Support

Who is online

Users browsing this forum: No registered users and 10 guests