How to finish downloading all headers?

[RayMark]

I' m not sure if it is an issue with 6.0 build 1019, perhaps with 5 also.
Or perhaps an issue with my ISP (yet to try a different ISP).

But when switching to 6.0 I decided to make a fresh start and to download all the headers from scratch (at least for some groups).
So far, I tried "Download All Headers" with a.b.multimedia 3 times, and each time after several hours I get this:

[02:54:18] ERROR - SSL Read Failed: 10054 An existing connection was forcibly closed by the remote host.
[02:54:18] ERROR NNTPSocket - SendCommand Worker - Server:news-europe.giganews.com SendCommand - failed to Read Response From server: Socket - Read From Server Failed
[02:54:18] ERROR NNTPSocket - Socket HandleError - Server:news-europe.giganews.com SendCommand - Server Disabled Service on connection:

And the header download is aborted/ends. If "Download All Headers" is immediately started again - it is working, if there was a problem with the internet connection, it was a very short one.
With 5.0 I don't remember having this problem.

So my questions:

1. Could you make NewsBin more tolerant, make it keep retrying instead of aborting the header download?

2. How to finish downloading all the headers?
Do I need to try again "Download All Headers" or it is enough to use "Download Latest" (perhaps as many times as needed, if there are further aborts)?
Can I be sure that "Download Latest" will download all the remaining headers, all the older or newer ones, whichever are needed, wherever the "Dowload All Headers" was aborted?

3. Can I download new headers with 6.0 first and then additionally import version 5 headers? Will they be merged correctly, in the case that some of the new 6.0 headers are missing?

[itimpi]

The error message appears to be a case where the remote end has closed the connection for some reason.

If it happens do NOT do a Download All Headers again as that wipes any existing headers and restarts from the beginning. Instead use the Download Latest option and Newsbin will continue from the point it reached before the error occurred.

In terms of retires, I am not sure whether Newsbin has automatic retries on headers or not at the moment.

[DThor]

I suspect #3 would be a real mess. Really, ultimately the issue here is why you got disconnected. I certainly haven't seen that, except in the scenario where you're downloading crapload of headers and the remote server has a time limit on single activities on a connection. This may be the case here. However, inconvenience aside, once you've said 'download all', you should just be able to pick up where you left off later using 'download latest'.

DT

DOH - simulpost!

[Quade]

You just need to start the header download again. It'll start up where it left off. Some servers disconnect the connection when you've transferred X bytes of data.

If you

- Download all

Subsequent downloads should be

- Download Latest.

Eventually you'll get them all.

[RayMark]

Thanks, I finished with Download Latest.
Maybe I will trust you or maybe I will try once more after moving the current V6 headers to a backup location and try to compare.

[Quade]

You can force it to re-download without first purging by deleting the "Range.db3" for the group and setting the proper "Download age". That resets the "Latest" and tells it to try to download "Download age" worth of headers. It'll just integrate the old records with the new. You could use the same technique if you add a new server.

[RayMark]

Thanks, it is a useful technique.
I think I used it with V.5 to re-download all the headers but at the same time to retain the already downloaded posts marked as downloaded.
I think in such a case it is better to preserve the Range.db3 file somewhere and after re-downloading to move it back.

[thuvia]

I notice that the V550 Reference "Inner Workings" section mentions the use of the Range.db3 file, but the V600 Reference doesn't. That kind of detail is very useful when trying to hack around a problem not necessarily of NewsBin's doing.

[e10kstarfire]

You just need to start the header download again. It'll start up where it left off. Some servers disconnect the connection when you've transferred X bytes of data.

If you

- Download all

Subsequent downloads should be

- Download Latest.

Eventually you'll get them all.

Sorry for jumping but I'm having the same issue with newshosting.

I am doing a Download all, it kicks off but clearly at some point is failing as when I go to show all posts and sort by date I am only getting about 60 days worth.

I then try download latest but that hardly downloads anything and finishes successfully.

I do not see any obvious error messages in logging unless I need to turn something on? I only see:-
[17:30:56] HIGH InterSocket - SSL Zlib Compression Added
[23:22:26] HIGH InterSocket - SSL Zlib Compression Added
[23:24:15] HIGH InterSocket - SSL Zlib Compression Added
[21:24:12] HIGH InterSocket - SSL Zlib Compression Added
[21:24:18] HIGH InterSocket - SSL Zlib Compression Added
[21:27:03] HIGH InterSocket - SSL Zlib Compression Added


Could you help please?

[Quade]

What's your "Storage Age" set to?

Easynews only keeps 120 million posts max in their groups. That means some of their groups don't go back in time with headers all the way to full retention. If you "Download all" then "Show All" and only get 60 days worth AND your storage age > 60. I suspect 60 is all the server can deliver. I don't know if Newshosting is like this but, it wouldn't surprise me.

If you go into the network options and select "Show Server Commands" then update the group.

You see something like this for that group

[16:44:06] HIGH NNTPSocket - GROUP alt.binaries.testes
[16:44:06] HIGH NNTPSocket - 211 173271 2143 175413 alt.binaries.testes

The second number is the number of posts in the group.

173271 in this case.


Edit: A trick you could do ,is sign up for a free trial with Giganews, download all the headers you want from them and use these headers with Newshosting.

[e10kstarfire]

Thanks for your help.

Storage Age is set to 1000.

Here is the Show Server Commands output (I also tried Verbose option under Advanced Status Tab Logging)

[23:39:40] HIGH NNTPSocket - 201 NNRP unlimited.newshosting.com Service Ready - support@newshosting.com (fe17.ams2) (No Posting)
[23:39:40] HIGH NNTPSocket - XFEATURE COMPRESS GZIP

[23:39:40] HIGH NNTPSocket - 290 Feature Enabled
[23:39:40] HIGH NNTPSocket - GROUP alt.binaries.hdtv

[23:39:40] HIGH NNTPSocket - 211 120000001 1369517933 1489517933 alt.binaries.hdtv

So guess that's 120 million also.

I have tried download all headers again, will see how I get on, it seems to have got half way through so far on the green Current Progress column.

[Quade]

[19:25:10] HIGH NNTPSocket - GROUP alt.binaries.hdtv
[19:25:11] HIGH NNTPSocket - 211 1126678108 358091328 1484769435 alt.binaries.hdtv

Here you go, giganews. 1,126,678,108

1.2 billion posts.

[e10kstarfire]

Yep thats very good - but it is double the cost of what I pay so guess it should be.

Seems newshosting provide 500 days of headers retention, I will be happy if I can just download those.

On other newshosting groups newsbin has successfully downloaded 500 days, just seems it keeps failing for some reason on the hdtv ones?

Anyway it's strangely still working now (up to 96 million), only thing I changed was the verbose logging and show server commands you suggested

[Quade]

Seems newshosting provide 500 days of headers retention, I will be happy if I can just download those.

Except they don't. Their groups contain no more than 120 million headers, which for some groups, is 500 days and for more active groups less. a.b.hdtv isn't even the largest group. So, giganews has 10 times as many headers and 1000 days of retention. That means 120 million is 1/10th the headers of giganews. So, on a good day, I'd guess newhosting has no more than 100 days worth of headers in this group. Considering how volume has gone up, I'm not surprised you can only see 60 days worth. Best case, they have no more than 100 days and your experience suggests real world retention of headers is 60 days.

You understand, with a hard cap of 120 million headers, header retention will be different for each group, the more active the group, the lower the header retention. If they claim 500 days worth of headers in this group, they're lying. I'm not suggesting you change servers, I'm telling you though, you've probably got all the headers you can possibly get with this server for this group. I suggested a cheap way to get the rest of the headers too.

Giganews has a $10 plan with a 50 gig limit but, free headers so, you could download headers from giga, make giga a fill server so, it only fills in parts that are missing on newhosting and then you stop downloading headers from Newshosting but, still download most of your files from them. I'm not really promoting giga, I get nothing if you sign up but, they do have the best header retention of any server. Alternatively, you sign up for a month, download all the headers you want and then cancel, while maintaining the headers you downloaded on your PC and simply topping them off every day.

[e10kstarfire]

Ok, I understand now, thanks for the explanation!

Bit misleading when they state all this retention but you cannot download the headers to access it! (only got 64 and 43 days for two hdtv groups!)

Guess I could use .nzb's or use the workaround you suggested but think its probably best to look at a new provider.

Thanks for your help

[e10kstarfire]

Just trying out Astraweb and that seems much better, over 1 billion headers but after a period of hours the header download keeps stopping with:

[16:16:10] ERROR NNTPSocket - Server:ssl-eu.astraweb.com NNTPServer: Compressed Headers - Timeout Waiting for Server to send Data: Socket - Read or Write Timeout
[16:16:10] ERROR NNTPSocket - Server:ssl-eu.astraweb.com NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write Timeout

As per recommendations I am then doing Download Latest.


Could something be added to newsbin to just do an automatic Download Latest again? - it takes some time before noticing it has failed and manually restarting it (as I don't obviously want to sit there for hours watching) ?

Thanks.

[itimpi]

Under Options->Switches you can set how often Newsbin should automatically attempt to download headers. This is not a retry but it might satisfy your need. It does mean that you do not need to be sittings at the PC to get headers downloaded.

[e10kstarfire]

Good workaround, thanks.