Compressed Headers

Technical Support for Astraweb. Moderated by a real live Astraweb representative.

Moderator: stevef

Compressed Headers

Postby stevef » Fri Nov 21, 2008 12:01 am

In case anyone has missed it, you can now download compressed headers from our servers using Newsbin.
You should see a substantial speed increase anywhere from 5x to 15x.

Compressed header downloading is available in Newsbin version 5.50 B7 and above. It might be available since B6.
The option can be activated in the server options screen. Make sure the "Header Compression" box is checked.


Enjoy!
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Postby richy99 » Fri Nov 21, 2008 8:06 am

works fantastically well too :D
User avatar
richy99
Elite NewsBin User
Elite NewsBin User
 
Posts: 6353
Joined: Fri Nov 21, 2003 8:04 pm
Location: Wales

Registered Newsbin User since: 12/31/03

Postby ozzii » Fri Nov 21, 2008 11:29 am

Works for me. Thanks :wink:
ozzii
Seasoned User
Seasoned User
 
Posts: 410
Joined: Thu Feb 23, 2006 6:10 pm
Location: France

Registered Newsbin User since: 02/23/06

Postby hugzzz » Thu Nov 27, 2008 8:23 pm

Just gone over to Astraweb from Powerusenet and WOW does the compressed headers make a hell of a difference.

On a 10Mb cable it's giving me a 95Mb header download :)
User avatar
hugzzz
Occasional Contributor
Occasional Contributor
 
Posts: 48
Joined: Tue Mar 04, 2003 11:23 am
Location: Luton, United Kingdom

Registered Newsbin User since: 12/21/02

Postby Quade » Thu Nov 27, 2008 11:51 pm

Yeah, it's pretty sweet.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby obvious » Fri Nov 28, 2008 10:22 am

Been trying to download all headers for a.b.multimedia using header compression. There were gaps in the data folders which newsbin 9246 seemed to be trying to fill but it just kept coming up with:-

"[12:24:36] Level:ERROR SERVER=eu.secure.news.astraweb.com, MSG=NNTPServer: Compressed Headers - No data after decompression"

..so I ended up deleting the a.b.m data folder and trying to 'download all'. It starts off with 49 million headers to download then after 15 million headers have downloaded it fails and restarts with 33million headers to go then promptly gets stuck repeating the same "No data after decompression" message.

If I cancel the failing/looping 'download all' with 33 million to go and do a 'download latest', it starts again from 49 million
C:\ net helpmsg 4006
User avatar
obvious
Seasoned User
Seasoned User
 
Posts: 584
Joined: Wed Jun 06, 2001 8:30 pm

Registered Newsbin User since: 04/13/03

Postby stevef » Fri Nov 28, 2008 10:47 am

Quade's aware of the "No data after decompression" loop. I believe it is fixed in the latest beta (might be IRC private beta).
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Postby obvious » Fri Nov 28, 2008 11:14 am

It works fine (and still accelerated) if I uncheck the 'yEnc XZVER Support' box.
C:\ net helpmsg 4006
User avatar
obvious
Seasoned User
Seasoned User
 
Posts: 584
Joined: Wed Jun 06, 2001 8:30 pm

Registered Newsbin User since: 04/13/03

Postby Quade » Fri Nov 28, 2008 12:21 pm

If you don't run the most current beta's,you should. I assume you're running B7 which was the first version with compression. Steve brought this to my attention and I fixed it in B8. Pretty sure B8 has the fix anyway. My todo list doesn't track versions, just days.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby richy99 » Fri Nov 28, 2008 12:34 pm

i can confirm its still has this issue in b9
User avatar
richy99
Elite NewsBin User
Elite NewsBin User
 
Posts: 6353
Joined: Fri Nov 21, 2003 8:04 pm
Location: Wales

Registered Newsbin User since: 12/31/03

Postby Quade » Fri Nov 28, 2008 12:51 pm

The message is still there. Newsbin doesn't abandon the download now though.

Edit: Apparently this isn't true. I'm looking at it.
Last edited by Quade on Fri Nov 28, 2008 2:27 pm, edited 1 time in total.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby richy99 » Fri Nov 28, 2008 1:00 pm

it stopped for me the same as it did for obvious above, it got to a certain point then just repeated the error message and the download latest just was flashing at 0 as the error repeated

its almost as tho when it hits a certain area it just chokes and wont go any further
User avatar
richy99
Elite NewsBin User
Elite NewsBin User
 
Posts: 6353
Joined: Fri Nov 21, 2003 8:04 pm
Location: Wales

Registered Newsbin User since: 12/31/03

Postby obvious » Fri Nov 28, 2008 1:28 pm

Quade wrote:If you don't run the most current beta's,you should. I assume you're running B7 which was the first version with compression. Steve brought this to my attention and I fixed it in B8. Pretty sure B8 has the fix anyway. My todo list doesn't track versions, just days.


Been running x64 B8 since a couple of hours after it was posted.
C:\ net helpmsg 4006
User avatar
obvious
Seasoned User
Seasoned User
 
Posts: 584
Joined: Wed Jun 06, 2001 8:30 pm

Registered Newsbin User since: 04/13/03

Postby Quade » Fri Nov 28, 2008 2:28 pm

Yeah, none of the internal testers reported this. It's a little different than Steve's original issue. I'm looking at it.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby obvious » Fri Nov 28, 2008 2:29 pm

:)
C:\ net helpmsg 4006
User avatar
obvious
Seasoned User
Seasoned User
 
Posts: 584
Joined: Wed Jun 06, 2001 8:30 pm

Registered Newsbin User since: 04/13/03

Great!

Postby huanhuan19 » Fri Aug 07, 2009 7:38 pm

Works beautifully!

But I have a question to limit the size of the hearder downloads. It is too much to download all of them even if the speed is over 5 mbps.
huanhuan19
Occasional Contributor
Occasional Contributor
 
Posts: 36
Joined: Fri Aug 07, 2009 7:07 pm
Location: Charlottesville,Virginia

Registered Newsbin User since: 08/07/09

Re: Compressed Headers

Postby mojo-chan » Wed Oct 05, 2011 7:37 pm

Even with compressed headers they are coming down really slowly for me. Newsbin 6.10RC2, ssl-eu in use. They seem to go in spurts, fast for a few seconds then stall for a few seconds.
mojo-chan
Occasional Contributor
Occasional Contributor
 
Posts: 45
Joined: Mon Aug 14, 2006 2:49 pm

Registered Newsbin User since: 08/11/06

Re: Compressed Headers

Postby Quade » Wed Oct 05, 2011 10:52 pm

Maybe try the US server and try it without SSL? If not using SSL works, you can actually set it up to download headers without SSL and use SSL just for the file parts.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Compressed Headers

Postby hemz » Mon Nov 21, 2011 8:20 pm

I've had Astraweb over 2 years and never knew about this, how do i set it up on Newsbin.

Am a newbie to the site any help would be appreciated. using the 6.11 build.

Thanks

Craig
hemz
n00b
n00b
 
Posts: 5
Joined: Mon Nov 21, 2011 7:15 pm

Re: Compressed Headers

Postby Quade » Mon Nov 21, 2011 9:38 pm

It's automatic these days. You don't have to do anything.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Compressed Headers

Postby Plankton » Wed Aug 29, 2012 6:05 am

stevef anychance you could fix header compressions across all your server farms and give it traffic priority seeing as it's only 1 connection so users can actually download all the headers of a large top group without it timing out or dropping to low speeds for days?

Thanks
Plankton
Active Participant
Active Participant
 
Posts: 59
Joined: Mon Jan 16, 2012 9:07 pm

Re: Compressed Headers

Postby stevef » Wed Aug 29, 2012 7:14 am

There is no artificial priority difference between header and body downloads.

Any slow header download in the last 5 days can be attributed to a header server problem. This is no longer an issue.
If you're having slow header downloads at this very instant, and it is reproducible, then send helpdesk a ticket for testing.
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Re: Compressed Headers

Postby Quade » Wed Aug 29, 2012 10:05 pm

How many connections are you running? Lots of people are running too many connections and causing issues. Just for grins, I'd run it back to 10 connections and try again. I get crazy fast speeds to AW when everything is working.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Compressed Headers

Postby stevef » Wed Aug 29, 2012 10:17 pm

I wouldn't call 40mbps header download speeds "slow".
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Re: Compressed Headers

Postby qubin » Wed Aug 29, 2012 11:22 pm

Quade wrote:How many connections are you running? Lots of people are running too many connections and causing issues. Just for grins, I'd run it back to 10 connections and try again. I get crazy fast speeds to AW when everything is working.


he only has 1 connection for header fetch?
qubin
Occasional Contributor
Occasional Contributor
 
Posts: 19
Joined: Wed Apr 18, 2012 8:42 pm

Re: Compressed Headers

Postby qubin » Wed Aug 29, 2012 11:27 pm

stevef wrote:I wouldn't call 40mbps header download speeds "slow".


So as users we can download binaries at 120mbps but it's perfectly acceptable to download headers at a fake 40mbps compressed speed for hours and days and get to about 10% and then it times out without completing?
qubin
Occasional Contributor
Occasional Contributor
 
Posts: 19
Joined: Wed Apr 18, 2012 8:42 pm

Re: Compressed Headers

Postby stevef » Wed Aug 29, 2012 11:36 pm

You are comparing article downloads using multiple connections, whereas header download is using a single connection per group.
Article download there is no server processing. Header download there is heavy parsing of the article to give you that single overview line.
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Re: Compressed Headers

Postby Quade » Wed Aug 29, 2012 11:37 pm

I always update multiple groups at a time so, all my connections are used for header downloads. I've seen speeds close to 1 Gb/sec to AW. More typical is 500-600 Mbps. "Per connection" speed of 40 Mbps isn't really that bad. Giganews will barely do that, typically they're slower "per connection" than AW.

Newsbin has to work much harder downloading headers than downloading files. I wouldn't be surprised if Newsbin processing plays a role in that 40 Mbps per connection speed. I mean if nothing else it has to decompress a multi-mega-bit/sec data stream. When you use multiple connections, you get multiple cores doing the decompression.

If you just let Newsbin update your groups automatically once an hour, you'd always have mostly current headers too.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Compressed Headers

Postby qubin » Wed Aug 29, 2012 11:48 pm

You can't know what is in a newsgroup and choose what to download unless you can download and view all the headers.

You can't download all the headers from boneless without it timing out out so we are paying for a service we can't use as we can't view the complete content.

Please don't says NZB groups as most of us have no interest in others choosing a small selection for us on our behalf or paying for a subscription search as we have already paid for access.
qubin
Occasional Contributor
Occasional Contributor
 
Posts: 19
Joined: Wed Apr 18, 2012 8:42 pm

Re: Compressed Headers

Postby Quade » Thu Aug 30, 2012 12:06 am

You can't download all the headers from boneless without it timing out out so we are paying for a service we can't use as we can't view the complete content.


Just restart the header download. Newsbin will continue where it left off. Just don't "download all headers" a second time because then you're starting over again and won't make any progress. Even if you completely exit Newsbin, restart and download headers again, it'll still download from where it left off till it gets all the headers.

I can only guess you "download all headers" again and throw away the headers you've already downloaded.

The problem you're having might just be you. Your PC or network. With Newsbin, it's easy to recover from a connections dropping out or disconnecting, just start the header download going again.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42031
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Compressed Headers

Postby Cocha » Thu Aug 30, 2012 5:30 am

I don't understand this crazy idea of repeatedly compressing such a large database over and over again per customer.

There should be a system that auto prunes any headers from the server database that is older than 30 days and compresses them once into monthly header archives that are then served to the customer as a multi part binary files so they can max out there connection like any other file.
Cocha
Occasional Contributor
Occasional Contributor
 
Posts: 26
Joined: Sat Dec 10, 2011 6:16 am

Re: Compressed Headers

Postby stevef » Thu Aug 30, 2012 5:57 am

Sounds like a good idea.
Now, how will you serve me the data when I request a range that is a fraction of the multipart, or a request that begins in the middle of one multipart and ends in the middle of another multipart? And remember, I don't want any extra data, because I've got a download cap.
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Re: Compressed Headers

Postby Cocha » Thu Aug 30, 2012 6:19 am

Couldn't there be a system coded that runs in parallel so if they need a fraction of a multiplart they get switched back over to the current way of serving the headers for that header range until they are back in sync with the monthly compressed snapshots?

Or even a system for those that request all headers from a group to send a new server command so they are treated differently and use a compressed archive for most part to take the load off the server and give them max speed?

Just a thought.
Cocha
Occasional Contributor
Occasional Contributor
 
Posts: 26
Joined: Sat Dec 10, 2011 6:16 am

Re: Compressed Headers

Postby stevef » Thu Aug 30, 2012 6:31 am

If you look at the pattern of requests, nearly ALL requests will be partial multiparts.
And introducing new commands would require the co-operation of all USENET providers and NNTP readers. Just introducing compressed headers without breaking the existing NNTP specification was a headache in itself.
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03

Re: Compressed Headers

Postby Cocha » Thu Aug 30, 2012 6:48 am

Ok my last attempt at an idea to make things easier for users. =)

Could you do a virgin install of newsbin at the Astraweb network and download all the headers of say the top 10 or 20 groups and when it has finished leave the SPOOL_V6 folder from newsbin in the members area for us to download and copy to our own newsbin folder as that is a compressed database of the headers and all we need to do is then update the headers from when it was done.?

Most of the other groups are not a problem for getting all the headers so it would only need to be the top ones like boneless.

Thanks
Cocha
Occasional Contributor
Occasional Contributor
 
Posts: 26
Joined: Sat Dec 10, 2011 6:16 am

Re: Compressed Headers

Postby stevef » Thu Aug 30, 2012 7:24 am

Cocha wrote:Could you do a virgin install of newsbin at the Astraweb network and download all the headers of say the top 10 or 20 groups and...

No, that's not going to be workable. Every time there's an update, you will all be downloading the entire SPOOL_V6 folder again. Plus the infrastructure at the moment has you downloading across a load balanced server farm. In your scenario, we would have to additionally make sure the web servers a load balanced just to serve out the SPOOL_V6 folder.
stevef
Official Astraweb Representative
Official Astraweb Representative
 
Posts: 602
Joined: Fri Jun 20, 2003 1:30 am

Registered Newsbin User since: 10/22/03


Return to Astraweb

Who is online

Users browsing this forum: No registered users and 1 guest