Allow downloads to be assigned to server groups, or server

Make suggestions for new features.

Moderator: Quade

Allow downloads to be assigned to server groups, or server

Postby Morrolan » Thu Sep 14, 2017 1:45 pm

If a user has multiple servers setup, there is a priority setup here. Each nzb grab goes one at a time down these priorities. If my nzb grab is not available on priority 2 server, I would like my 2nd nzb grab to use the available priority 2 server. No reason that I can't have both nzb grabs simultaneously grabbed right? I would like to have servers in groups, so that I can assign an nzb grab to a specific group. Then each group should download independent of each other (unless a post is missing in 1 group, it can try 2nd group to recover this post). I should be able to right click my download and assign it to a server or server group. Currently if I want to do something like this, I have to launch multiple apps.
Morrolan
n00b
n00b
 
Posts: 3
Joined: Mon Nov 09, 2015 12:04 am

Registered Newsbin User since: 12/24/12

Re: Allow downloads to be assigned to server groups, or serv

Postby Quade » Thu Sep 14, 2017 2:45 pm

Why not just set all the servers to the same priority and let them all work at the same time?

The whole point of a priority system is to prevent some servers from being used. If you want all the servers to be used, just set them all to the same priority and they'll all be used during all downloads.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42400
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Allow downloads to be assigned to server groups, or serv

Postby Morrolan » Thu Sep 14, 2017 2:50 pm

Newsbin only downloads one file at a time like this right? Most of the time, this is fine as it will utilize all servers. If I know that 1 of my downloads is not on one of the servers, I don't want it to keep retrying that server and wasting time. So I have to set that server to higher priority, so that it does not get used. Now it sits idle, but it could be used for the other files queued for download. Does that make sense?
Morrolan
n00b
n00b
 
Posts: 3
Joined: Mon Nov 09, 2015 12:04 am

Registered Newsbin User since: 12/24/12

Re: Allow downloads to be assigned to server groups, or serv

Postby Quade » Thu Sep 14, 2017 10:55 pm

Newsbin only downloads one file at a time like this right?


1 - Files are made of chunks which are downloaded one per connection. Each chunk us 500-750K

2 - If you have a server set to 10 connections, 10 chunks will download at the same time.

3 - If you have 2 servers set for 10 connections each, 20 chunks will download at a time..

4 - You hit a point of diminishing returns where more connections won't get any faster.
My connection is 120 Mbps and with 10 connections, I get 120 Mbps so, there's no point to using more than 10 connections.

So Newsbin is constantly using 100% of the connections, depending on the server priority. To me it's pointless to download lets say 2 NZB's at the same time, if each one downloads 1/2 as fast. Instead Newsbin downloads the top set of files as fast as the connection and PC permits before it moves on to the next set of files.

How fast is your connection? If it's under 100 Mbps, It's kind of pointless to mess with the connections if you're already getting full speed. If you have a 300 Mbps connection, running two servers in parallel is probably a good idea. It works fine as slower speeds to, it just doesn't improve downloading much. The reason torrents download multiple sets of files in parallel is because they don't have a central server so it's much harder to achieve full speed. Usenet doesn't need or benefit from downloading that way.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42400
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Allow downloads to be assigned to server groups, or serv

Postby Morrolan » Thu Sep 14, 2017 11:43 pm

Sorry, if I am not clear. Let me try this again.

1. I have 3 files in my download queue.
2. I have 4 servers configured as priority 1
3. 1st download starts and shows that only 1 server has parts (chunks), the other 3 servers show errors because the chunks are not available on these servers.
4. I set the priority to 2 on the 3 servers showing errors.
5. My 1st file in the queue is now downloading on 1 server only. This is a slower overseas server.
6. My 2 other files in download queue are waiting for my 1st file to finish downloading.
7. The 3 servers on priority 2 are sitting idle.
8. I have 100MB connection with bandwidth to spare.
9. I would like to direct Newsbin to start downloading the next file in my download queue against 1 or more of my idle available servers sitting in priority 2. This way, I can utilize servers sitting idle, and I can use bandwidth that is available to me to obtain all 3 files faster.

I hope that makes sense. What I do now, is I load up nzbget, I configure it to only use the idle servers, and I send the 2nd and/or 3rd nzb file downloads over there depending on what priority I want for these files. I then delete them from the newsbin download queue while newsbin is busy working on the slow download of file 1.
Morrolan
n00b
n00b
 
Posts: 3
Joined: Mon Nov 09, 2015 12:04 am

Registered Newsbin User since: 12/24/12

Re: Allow downloads to be assigned to server groups, or serv

Postby itimpi » Fri Sep 15, 2017 4:01 am

It sounds as if you have misunderstood the priority system! A low number in the priority field puts servers ahead of those with a higher number. The way you are trying to set servers those with a higher priority number are only used if the ones with a lower number (I.e. your slow server) turn out to not have part of a post.

If you do not want them all at the same priority level then you want your preferred servers to have a lower value for priority, and your slow server to have a higher one so that it is only used for parts that are not on the fast servers. That way a server with a higher value in the priority field only gets used for items that are not on the other servers which will start looking ahead for items that they CAN download.
The Newsbin Online documentation
The Usenettools for tutorials, useful information and links
User avatar
itimpi
Elite NewsBin User
Elite NewsBin User
 
Posts: 12546
Joined: Sat Mar 16, 2002 7:11 am
Location: UK

Registered Newsbin User since: 03/28/03

Re: Allow downloads to be assigned to server groups, or serv

Postby Quade » Fri Sep 15, 2017 12:35 pm

You have 3 servers that don't deliver any parts for an NZB and then one server that does.

In this case, after the servers all get a chance at the top set of files, they should be moving on to the set of files below it. The issue is that most people using multiple servers have one server that costs more to use than the others so, they only want to use it if the other servers don't have parts of the files. Since that's the case, the cheap servers get checked first. Meaning every single chunk of the file gets checked on the 3 cheap servers before the expensive server is checked. That's a relatively slow process.

The cheap servers should then move on to the next set of files once they finish checking the top file set.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42400
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Allow downloads to be assigned to server groups, or serv

Postby jackholexxxx » Wed Sep 20, 2017 9:31 pm

What about an error threshold per download. A user configurable value. If a download gets more than <value> 430 errors on a particular server, that server will de-prioritize itself and move on to other downloads. If all servers get de-prioritized, and the download is not finished or in a repairable state, then we circle the wagons and try to download the missing chunks the old fashioned way before failing the download.
jackholexxxx
Seasoned User
Seasoned User
 
Posts: 196
Joined: Wed Apr 15, 2009 6:18 pm

Registered Newsbin User since: 11/01/04


Return to Feature Requests

Who is online

Users browsing this forum: No registered users and 1 guest