6.8 final HDD load

Technical support and discussion of Newsbin Version 6 series.

6.8 final HDD load

Postby jonas » Fri May 04, 2018 6:37 pm

Hi,

I just upgraded to 6.8 final from a previous beta (not quite sure which one) and I noticed that since updating, the load on my HDD where the Newsbin profile is stored and the chunks get written to before the actual files are created (the download folder is on a different HDD) goes to 100% while downloading and my download speed drops by almost half. I didn't have this problem before so I assume it might be related to the new RAR processing? Anything I can do on my end?
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Fri May 04, 2018 10:35 pm

You mean the amount of disk space used? How much free space do you have on this drive?
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Sat May 05, 2018 5:38 am

No, I don't mean the amount of disk space (the HDD has around 200GB free if that still matters). I'm talking about simultaneous read/write operations. I did some further tests last night and I noticed, that the problem does not occur with every file I tried. Sometimes Newsbin behaves as I would expect: checking in Task Manager the activity on HDD 1 (where the chunks are stored) goes to below 20%, activity on HDD 2 (where the actual files are created) goes to around 40% and I get a speed of around 400Mbit downloading.
But with a lot of files, I get the problem I described above: Activity on HDD 1 spikes and the speed drops to around 250Mbit for the entire download. It is definitely not random, since re-downloading the files I tested always leads to the the same behaviour.

EDIT:
I did some more tests and it seems it is actually the minority of files it happens with, so it is quite possible, that the problem is not new but that I simply did not notice it until yesterday.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Sat May 05, 2018 11:12 am

Newsbin buffers chunks in memory but if the chunk count of a file > memory chunks the chunks get flushed out to disk. The chunk count is down where it says "Cache" at the bottom. The default is 400 chunks in 6.80. If you're downloading files that are > 400 chunks per file, some of that partial file will hit disk.

Your symptoms suggest high chunk count files.


[PERFORMANCE]
ChunkCacheSize=X

If you add this to the NBI file and set the chunk cache to larger than the per file chunk size, you might make this go away. It's about 1 meg per chunk block worst case. Some people set it to 1000 or 2000. There's a point of diminishing returns too. If you're using an SSD for this drive, you probably don't need to change this.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Sun May 06, 2018 9:47 am

Thanks for the suggestion, I've set it to 2000 and performance seems to be improved while still not perfect. Setting it to a higher value makes no more difference.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby jonas » Thu May 10, 2018 9:04 am

Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Thu May 10, 2018 10:30 am

You might try playing with the URL for the download. I think Dex leaves the older ones on the site.

I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby Quade » Thu May 10, 2018 12:09 pm

One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Thu May 10, 2018 12:33 pm

Quade wrote:You might try playing with the URL for the download. I think Dex leaves the older ones on the site.


I've tried that, didn't work unfortunately.

Quade wrote:I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.


There is no security software apart from windows defender installed on my machine.

Quade wrote:One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.


I'm sure if you have a faster HDD or SSD it won't be as much of a bottleneck as it is for me, but now that I've watched performance the last couple of days I am sure that the problem is definitely new for me. Might be related to something other than a change in newsbin but without the previous betas I can't test it...
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby dexter » Thu May 10, 2018 5:21 pm

jonas wrote:Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.


Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Archive Page
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9424
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Thu May 10, 2018 8:39 pm

dexter wrote:
jonas wrote:Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.


Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Links not allowed for unregistered users


ok, so I installed version 6.73 as suggested and infact I don't experience any unusual activity on my HDD and get a consistent 450 mbps. Installed 6.8 final again and HDD activity spikes and speeds go down to about 250 mbps. Something in the latest version does not play nice with my setup Image link not allowed for unregistered users
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Thu May 10, 2018 10:18 pm

You could use "Procmon" to examine disk IO.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Fri May 11, 2018 10:16 am

Quade wrote:You could use "Procmon" to examine disk IO.


The only thing that jumps out at me is, that in both versions I'm seeing a lot of "QueryOpen" operations on chunk files that return "NAME NOT FOUND" and while the old version followed by performing a single "WriteFile" operation on the Logfile.txt, the new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Fri May 11, 2018 10:27 am

he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?


Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Fri May 11, 2018 11:08 am

Quade wrote:
he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?


Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.


Setting logging to normal under advanced options makes no difference. With disable logging server commands you mean the "show server commands" option? Other than that I don't see an option to control logging in the network tab and that option was disabled to begin with.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Fri May 11, 2018 12:28 pm

Yeah "Show server commands". On a high speed connection logging that might consume more disk and CPU than you want.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby jonas » Tue May 15, 2018 11:49 am

So I moved the DataPath from one of my HDDs to my system drive, which is an SSD, and that eliminates the bottleneck I was experiencing. Somewhere between version 6.73 and 6.8 Newbin got a lot less efficient and it seems to download at high speeds you now need to use an SSD at least for the profile folder.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby LaRokka » Fri May 18, 2018 10:31 am

hello

pretty similar issue. just updated from 6.72 (release) to 6.80 (release) and my hard drive is getting hammered. in 6.72 all was smooth.
using a quite normal samsung 7200rpm sata 3 hdd as my download drive.
my download speed is officially 150mbit, usualy sits between 150 and 160mbit.

since 6.80 the hdd is constantly hammering when downloading. the drive LED is permanetly on, just stops for nanosecs every few secs.
the larger the files it is downloading are, the worse it gets.

1-3gb i am still getting 150-160 mbits.
3-6gb the harddrive is hit so hard, download goes down to 120ish mbit.
8gb down to 100ish mbit.
didnt test bigger files yet as i just installed 6.80.

i have "pre-allocate files for spinnign discs" enabled. tried to disable it, does not change behaviour.
it somehow seems as 6.80 is not caching (just from belly feeling, cant say for sure). but i know it wasnt this bad at any time before this release in the past years.

help greatly appreciated ! :)

no native speaker, plz excuse me :)
LaRokka
n00b
n00b
 
Posts: 6
Joined: Fri Dec 02, 2011 6:36 pm

Registered Newsbin User since: 12/03/06

Re: 6.8 final HDD load

Postby Quade » Fri May 18, 2018 10:39 am

I suspect it has something to do with the logging doing "FlushFileBuffers". I've disabled that in code but you won't see it till 6.81ish.

It's looking like we need a 6.81 because some people are reporting "Add to download list and it stalls" problems.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby LaRokka » Fri May 18, 2018 2:05 pm

i just downloaded 6.73 from the archive. can i install it over 6.80 like a normal update, with my settings kept ?
i would then continue to use 6.73 until 6.81 is out. didnt have issues with the older version anyhow, just updated to be up-to-date :)
LaRokka
n00b
n00b
 
Posts: 6
Joined: Fri Dec 02, 2011 6:36 pm

Registered Newsbin User since: 12/03/06

Re: 6.8 final HDD load

Postby dexter » Fri May 18, 2018 2:11 pm

Yes, you can install 6.73 over 6.80 with no issues. I've done it myself during the testing phase. If you are a header downloader, I'd make sure the import folder is clean before downgrading since they are no longer compressed in 6.80 so 6.73 won't process them.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9424
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: 6.8 final HDD load

Postby LaRokka » Fri May 18, 2018 2:52 pm

ok guys, thank you for your quick replys.

see you in 6.81 :)
LaRokka
n00b
n00b
 
Posts: 6
Joined: Fri Dec 02, 2011 6:36 pm

Registered Newsbin User since: 12/03/06

Re: 6.8 final HDD load

Postby jonas » Thu May 31, 2018 10:02 pm

Looks like the just released 6.81 beta fixes the issue.
jonas
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sun May 08, 2016 9:32 am

Re: 6.8 final HDD load

Postby Quade » Thu May 31, 2018 11:12 pm

Thanks for the feedback. I removed some "flush file buffer" calls.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43229
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: No registered users and 2 guests