Newsbin Beta 6.90B6 - Build 5303

[Quade]

http://downloads.newsbin.com/downloads/beta/nb690B6-5303.exe


- AutoPAR: When unraring many small files, the GUI is hit with an update flood. Triggers a cache flush.
- Display: Really large file downloads will show twice as much downloaded in the Size column as what’s actually downloaded.
- Download: Added “RAWNZBFILE” which assigns the unmodified and unshorted NZB Filename to the path segment. Will make really long paths.
- Report that Newsbin can create paths too long for Sonarr to pull from. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. NZBFilenames in path are restricted to 80 chars now.
- Base: Automode for groups is working. Automode for watch-list was working in the previous build.
- Display: Auto-Old isn’t kicking in.
- Display: If you have hide-old enabled when using the search tab, nothing shows up.
- Display: Search results are coming up pre-”old”
- Search: Added a search to download and the annotated subject didn’t display.
- Display: Local Search in post lists apparently doesn’t pop the progress bar. - no repro
- NZB: Modify NZB reader to feed meta as key value pair of strings.
- NZB: Fixed Problem with cleaner bashing UTF8 - prevented load of some NZBs
- NZB: Fixed a problem with incomplete amp detectors reading 1 byte two far into dead data. In rare cases caused a load failure.
- Display: Lightning button doesn’t work in files list.
- Display: Unrar and up/down don’t work in files list.
- Display: Local GOG search doesn’t seem to pop the progress bar consistently.
- Display: Delete filter profile causes a crash.
- Display: Report that files list loses one file per-restart - It happened with many files in the list. It doesn't seem to happen now.
- Watch: Watch counters don’t seem to increment - Works for local for sure.
- Watch: Re-work scan dialog internals. Really needs a thread.
- Watch: Do local watches pull from header downloads - verified
- Watch: Verify auto-download from local headers.
- Display: Report that download list column sorting on size doesn’t work. - Unresolved files in download list don’t know their size so they don’t sort.
- Download: Made Obscured file renaming optional. [AutoRAR] RenameObscured = 0/1 defaults to 1.
- Download: Verify making NZBs from unresolved files in the download list.
- ConfigFile: In some cases, not removing spaces from options when reading config file. Not an issue normally but manually entered config options might not get picked up.
- ConfigFile: Improved performance by removing some strings and replacing with string_views
- Display: Restart isn’t purging files list down to size.
- Display: Report that auto-shutdown happens prematurely.
- Display: Files list isn’t self-purging on insert - files list purges down to 5000 items on startup and shrinks by 100 items when it grows > 5000 elements
- Base: Auto-shutdown is gated both by the download list emptying and the header imports finishing.

[fidodkk]

Link is not working

[Quade]

Try it now. Thanks.

Edit: I had to right click "Open in New Tab" to get it to download for some reason.

[ccarlin]

With both B5 and now B6 I'm seeing a lot of obfuscated downloads not getting (re-)named correctly. Ballpark w/ B5 and B6 maybe 5 of 12 downloads didn't get renamed right. With prior 6.90B? versions I saw very few - maybe less than 5 not renamed correctly with 100(+) downloads (note, I used 6.90B1 the vast majority of the time, but don't remember an unusual # with B3 or B4). Maybe this is just a coincidence? For what it's worth I always have the files for each single .nzb downloaded to their own directory, and I don't remember ever seeing an incorrect directory name with obfuscated files with 6.90x code (thus, Newsbin 6.90x always creates a correct directory name). However, as noted, the file names themselves can be wrong, the % wrong based on the version of newsbin used (?). Thinking back, I'm pretty 6.82's directory names were also always correct too, with obfuscated files. I always used the directory name to correct the created file name, thinking it was odd one would always be right but the other sometimes wrong. Hopefully I'm being clear here .... The message is the logic used for directory names works great. Chris.

[Quade]

If you email some nzb's to ts@newsbin.com, I can check them out. I don't typically download obscured files so, it's your NZB's I test against.

[RayMark]

A secret beta?
The latest available on the beta page is still beta 2 - not a great beta, if I may say so.

Beta 2 shows all the new posts as old ones.
Beta 1 and beta 6 seem to show ok, as new posts.

Here is what happens when attempting to download posts from nzb - putting nzb in the special folder - the recommended way, I forgot how it is called, if anything.

Beta 1: downloads all the unnecessary pars too, repairs if needed (?) and extracts files from rars.
but when par2 clearly shows also a (missing) .nfo file - it does not re-create the .nfo file from par2s. The .nfo file is missing in the result. At least when rars do not require repair? When rars do require repair, not sure, perhaps .nfo is created after all.

Beta 2: does not download enough par2 files to repair and does not repair rars / does not extract files from rars.
Even if forcing to download enough par2s - still it does nothing after that (I did not try retry download).
But if I download enough par2s completely separately, then repair/extraction kicks in automatically and everything is done, except that the result is put in an entirely different folder. Also .nfo files are created and extracted correctly.

Beta 6: downloads all available par2 files (unnecessarily) but still does not repair and does not extract.

So beta 1 seems to work best, if you don't care about .nfo files.
If you do, beta 6 downloads enough par2s, and then repair and extraction can be done manually. For a dozen of nzbs. But not so much for hundreds.


As to the length of the nzb file name and the auto-renaming.
Is this option to use the whole nzb still available?
RAWNZBFILE
I rarely use Sonarr.
I want the full length of nzb, not truncated to 80 characters. And I also want 100% renaming. I might understand when the names are truncated when they are very long. But I cannot understand why very short obfuscated names are left not renamed.
What is the reason not to rename short names?
At least an option allowing to rename everything (short names)? If something goes wrong, this option then can be switched off.

[Quade]

A secret beta?

First it goes to IRC. Then it goes to the forum. At some point I decide whether it's good enough to be announced. I asked Dex to announce B6 yesterday or the day before.

Beta 6: downloads all available par2 files (unnecessarily) but still does not repair and does not extract.

Are you feeding the NZB's directly to the download list? If not, you need to change how you feed them so they go to the download list. The symptoms you report suggest you're loading them to the post list then feeding them to the download list. That's the least reliable way to handle modern NZB's.

I've had no issued with repair or unrar with B6. On Windows or Linux.

If you email me an NZB that has these symptoms I'll be happy to check it.

[RayMark]

I was feeding directly to the download list.
I am yet to find a nzb file that is repaired/extracted with beta 6. Or beta 2.

Beta 1 unrars, but not so sure if it repairs first, because if .nfo file is not included inside the rar but is parallel to the rar and is missing, it is not created from the par2 set, as it should.

Beta 2 does not repair and does not unrar. It stops with no par2 blocks downloaded. Not even 1. Just the main par2 file.
Even when par2 blocks are needed for repairing.
If some par2 files are downloaded manually (the post expanded in the download list and par2 files are manually un-paused) - nothing happens. Repair / unrar does not kick in even now when there are enough of par2 blocks.
But if par2 block files are added to the download list separately, outside of the nzb of the post, then repair/unrar kicks in - unfortunately, the unrarred files are placed not in the folder of the nzb but in the folder specified for unrarred files.
If it so happens that the unrarred file is left with the obfuscated short name, then downloading more than one post at a time may cause confusion, unless the unrar path includes the name of the individual post.
Other problems with beta 2: all new posts showed as old, descript.ion file is missing.

Beta 6 downloads all the available par2 blocks (so, most often - too many) but still does not proceed to repair / unrar.
I did not try to add par2 blocks to the download list separately, because they would be dupes, but perhaps I should have tried, to see if repair/unrar can be activated that way. I found beta 6 accidentally (when I wanted to write feedback about beta 2) only today and did not test it too much yet. But descript.ion file is again present. And new posts are shown as new.

I was not able to find any info about newsbin on irc. Even though I spent probably more than a minute - you need to fix the website.
Probably it would be quicker to google "newsbin irc" rather than search on the newsbin website.

[Quade]

Beta 6: downloads all available par2 files (unnecessarily) but still does not repair and does not extract.

In a weird coincidence, I believe I ran into this today. The problem was extra files in the NZB the failed. I mean it's a bug.

You ought to be able to work around this for now by adding an NZB Loading filter. Hit the + on the download and see what files didn't download, probably an NFO or something like that and then make the NZB Loading filter reject NFO files. I imagine something like subject reject "[.]nfo" would do it.

My fix will probably be two fold.
1 - Filter these files out of the compaction
2 - Prevent a single NFO from preventing unrar and repair.

[RayMark]

Come to think of it - it is possible that with beta 6 (and possibly, with beta 2 too - I also downloaded it only yesterday) I was trying only with nzbs where .nfo files were included in par2 sets but missing.
With beta 1 I did not even know it - everything was extracted OK, and if .nfo files were missing I just assumed there were no .nfo file posted.

The problem, then, seems to be in this specific case:
An .nfo file is included into par2 file set but not included into the nzb file.
Still, I don't understand why NewsBin just stops. It should run par2 check, it should find a missing .nfo file and it should create it from par2 blocks even if that file is not included into the nzb file.
What happens now - it seems that it is not only wrong but also a more complicated implementation.
It would be simpler just to repair all the files without checking if they are included into nzb or not.
Especially that when they are included into par2 but not included into nzb - it is usually a wrong incomplete nzb.
Somehow that .nfo file was missed when generating nzb.
No reason to stick to it, better to fix it by still producing missing in it files.

BTW, .nfo files are not what they used to be. Except for pr0n, most of .nfo files today are just generated with no additional information. I mean, release groups probably include correct .nfo files but whoever posts to usenet, substitutes them with generated ones, as if doing a good deed. Evil posters.
It is very important to me to have .nfo files in the cases of PROPER, REPOST, RERIP, etc. to see the reason for it.
Sometimes the reason shows that the original is better: because the original exceeded some size or bitrate, or had some additional languages included, for example.
Unfortunately, such .nfo files are usually no longer included into the posts, so who cares about .nfo files.

[Quade]

Still, I don't understand why NewsBin just stops. It should run par2 check, it should find a missing .nfo file and it should create it from par2 blocks even if that file is not included into the nzb file.

I did say it's a bug. The reason is that a compacted set doesn't get declared "done" till All the non-PARS either download or fail and the NFO wasn't being failed properly.

[RayMark]

That .nfo is not listed in the nzb.
Its lack can be determined only by running par2 check - which is done, isn't it?
I think, incrementally. That is probably the problem - the incremental par2 check.
You check only the downloaded files, and the .nfo file is not in nzb and therefore it is not downloaded and the incremental check is not run on it.
But still, even an incremental check should end up with the totals showing that one file is missing.
There is a contradiction here somewhere: the repair is not run because the .nfo file is not checked incrementally, but unrar is also not run because there seems to be some flag showing that par2 check result is not OK and repair is required.

Notice that even beta 1 does not handle this case correctly: it does less than beta 6. It does not even notice that the final result of par2 check is not OK when .nfo is missing, because it proceeds with unrar.

[Quade]

That .nfo is not listed in the nzb.

I don't know what you mean then. Either there was an NFO in the grouped files or there wasn't. Your symptoms and mine suggest that in addition to the RARS and PARS there was some other smaller file in the grouping and that since it didn't fail properly, Newsbin didn't attempt repair and unrar.

The fact the PAR doesn't know about the NFO has nothing to do with whether the NFO was listed in the grouped files in the download list.

You check only the downloaded files, and the .nfo file is not in nzb and therefore it is not downloaded and the incremental check is not run on it.

I check all sorts of things during download, you really have no way of knowing what the logic is unless I explain it. For example, the reason I don't attempt repair and unrar till all files finish or fail downloading, is because I'm avoiding a premature repair from triggering when I have enough good blocks and repair blocks but haven't finished downloading the last non-PAR file in the group. Repair is the most expensive process so it's worth avoiding when possible.

I have to handle cases where the PARS download before the RARs, where the PARS download after the RARS, where the files downloading aren't grouped and only slowly accumulate valid blocks with Newsbin not knowing if they'll ever finish. I have to handle the case where the NFO is included in the PAR set and when it's not.

If you're saying there's only rars and pars in the download list grouping for a set and it's not repairing then this is completely different then the failure mode I saw yesterday.

[RayMark]

Yes, I am saying this:
.nfo file is not included into the nzb file, so when the nzb is loaded into the download list, the .nfo file is not listed there.
But that .nfo file is included in the set of par2 protected files, if I ran par2, then I see that .nfo file in MultiPar tool, for example. And it is created if I run repair with MultiPar. That is why it looks strange to me: your par check should also detect missing files and create them. OK, you check incrementally. But when everything is checked and you are about to unrar, you could double-check if all files listed in par2 are present, if not - create them before running unrar.

You know what, I think that this problem might be limited to DOGnzb, I downloaded the same releases from other places, and the .nfo file was inside nzb files. The nzb files (some of them?) from DOGnzb are incomplete in that respect.
So OK, I will avoid using DOGnzb. I will use it only when I cannot find the same thing elsewhere.
Problem solved.
But still a bug in NewsBin. In the sense that NewsBin should be able to handle incomplete (according to par2) nzb files better.

Look at this scenario: somebody could post only par2s and leave the files to be created from par2s.
Quite possibly such a posting strategy is already used?
You post 130% of par2 blocks with obfuscated names. And nothing else. Or just some file with a key or something.
Perhaps some usenet indexer is able to generate the nzb file for your post.
So NewsBin should be able to repair and extract the non-existing files correctly when that nzb is loaded.
Maybe it already can handle such posts. Bravo to you then.

[BZee]

Trying 6.90B6
2 problems
ONE:
using headers
Main Download folder:
Y:\U\$(SEARCH)\$(GROUP)\

Step 1) search GOG "CARS" for "Chevy" -- shows many posts

Step 2) select say 10 posts "Chevy-1957 Red" that are in "alt.binaries.pictures.cars" and download to new folder (name folder say "1957 Chevy")
as expected downloaded files are in Y:\U\CARS\1957 Chevy\

Step 3) select say 10 posts "Chevy-1957 White" and download to existing folder (use same as above)
downloaded files were added to Y:\U\CARS\1957 Chevy\

Step 4) select say 10 posts "Chevy-1957 silver" and do "Download - Bypass Filters"
now they are downloaded to Y:\U\CARS\NewsBin Pro Data\spool_v6\alt.binaries.pictures.cars\Storage.db3
(folder Storage.db3 has the 10 downloaded JPGs and descript.ion)

TWO:
Under settings - network I have pause set to resume after 30 Minutes. I paused Newsbin and it resumed quickly (I didn't time it but maybe after 30 seconds or less)

[Quade]

But that .nfo file is included in the set of par2 protected files, if I ran par2, then I see that .nfo file in MultiPar tool, for example. And it is created if I run repair with MultiPar. That is why it looks strange to me: your par check should also detect missing files and create them. OK, you check incrementally.

Yes it'll generate the NFO if it's included in the PAR. Not having the NFO doesn't prevent repair and unrar. Sounds like we're talking about two different failure modes and that I can't reproduce yours. If you want to send me some NZB's that fail, I'll try them.

Step 4) select say 10 posts "Chevy-1957 silver" and do "Download - Bypass Filters"
now they are downloaded to Y:\U\CARS\NewsBin Pro Data\spool_v6\alt.binaries.pictures.cars\Storage.db3
(folder Storage.db3 has the 10 downloaded JPGs and descript.ion)

Let me give that a try thanks.

[alanstarr]

Seems like "Purge to Display Age" isn't working properly. Open the popup box, but nothing happens after that.

[kirm]

I have one group set under properties toggled to "Disable UnRAR for this Group".

This is with B6, first one used since B2 due to hide old issue. The operation is working as it always had, leaving the rar files alone and placed in folder named E:\Downloads\$(GROUP)\ which uses the group folder name a.b.group in the Downloads folder. This is still working properly.

Since using B6, there is now an additional folder created alongside the a.b.group folder called $(GROUP) which contains only the extracted rar files from this a.b.group. This should not be.

Hope I explained this clearly. "a.b.group" referring to the actual group name.

[Calahan]

6.90B6 - Build 5303 is using much more virtual memory than 6.82 and Windows 7 wants to close Newsbin very often. Sometimes Newsbin crashed when there wasn't enough memory available.
This is caused by excessive use of memory for any connection. I used Process Explorer, from Sysinternals, to check the memory usage.

6.82 with 155 connections used: "Working Set" = 942 MB and "Private Bytes" = 1265 MB.

6.90 beta 6 with 155 connections used: "Working Set" = 1843 MB and "Private Bytes" = 14125 MB. When I had some more connections, Newsbin used more than 16 GB of virtual memory.
When I start Newsbin and all connections are disabled, then "Working Set" = 39 MB and "Private Bytes" = 24 MB.
Tried some other connection settings: "Private Bytes": 15 connections = 2878 MB, 30 connections = 3550 MB, 60 connections = 5921 MB, 90 connections = 9048 MB.

[Quade]

I've written a note to check RAM usage.

Since using B6, there is now an additional folder created alongside the a.b.group folder called $(GROUP) which contains only the extracted rar files from this a.b.group. This should not be.

This is with you manual extracting or are you saying disabling unrar doesn't disable it?

Thanks.

[wiggins09]

Thanks for fixing the watch topics feature Quade
Couple of oddities emerged:-
---Selecting files found by watch topic & naming a folder = named folder (empty or with .nzb) + separate [$Subject] folder where the unRAR'd files actually go.
---Selecting files from headers in the same group (right-click & 'download') created named folder but files also went to the [$Subject] folder.[/list]

[kirm]

This is with you manual extracting or are you saying disabling unrar doesn't disable it?

There is only one group that I have disabled unrar. It does disable unrar in the folder created for the a.b.group as always. What I am saying with this build, it takes those rar files and unrars them into a separate folder, whereas previously if you did not disable unrar, they would get extracted to the same folder a.b.group. It appears disabling unrar doesn't disable it, just places unrared files in another separate folder.

Regards,

[EvanVanVan2]

I think the server tabs has changed recently? I like how you can expand them and see all of the individual connections. But it also appears you can't collapse them without disabling them? Could you add the option to collapse them and could you make the colors visible somewhere when the sever is collapsed? It was easier to quickly tell if my unlimited server is downloading or one of the block servers without having to scroll.

[Calahan]

Seems the servers are not sorted by priority anymore too.

[BZee]

Newsbin 6.90B6 no longer writes the descript.ion data to DownloadedFiles.txt.
I periodically move this file to a storage folder and have several years of data. Every so often I need to search through them for one thing or another.
Can this functionality be restored? I'll revert back to NB 6.82 for now.

[Calahan]

Yesterday the "Auto Mode" feature failed downloading a complete grouped post.
It downloaded until *.r85, all files after that(r86-r99, s00-s14 and all par2 block files) were ignored.
Files s00-s14 were commented in debug Logging as unknown extension.

[Calahan]

Now Newsbin forgot to fill $(GROUP) and $(RAWSUBJECT) with the groupname and subject and created the path O:\!G\$(GROUP)\$(RAWSUBJECT).
The Auto Mode group has a property "UnRAR folder" which contains R:\!D\$(RAWSUBJECT)\.
But Newsbin extracted the files to the download folder O:\!G\$(GROUP)\$(RAWSUBJECT). In exact that folder.
(So as above, Newsbin forgot to fill $(GROUP) and $(RAWSUBJECT) with the groupname and subject.)

It worked some days ago. The only thing I've done is to downgrade to 6.82 Build 5142 to check the memory usage and then updated to beta 6 again.
I will stop and start Newsbin to see if that helps.

[Calahan]

Just re-started Newsbin and there is always? a nearly empty line in the "Files List".
On the left there is a green icon, but fields "Download Descriptions", "Filename" and "Poster" are empty.
"Size" shows 0 KB and "Date" shows 01.01.1970 02:00.
If I click on the "Remove files that no longer exist on disk" - button, the line will be deleted.

Everytime I start Newsbin an entry to a file, that is still there, is deleted.

[Calahan]

Some uploader are packing their nzb files in a single passworded rar.
I don't know if that could be the reason, but Winrar-Info shows: Version to extract -> 5.0, Host OS -> Unix.
The rar file is downloaded, but is not extracted. No entry is set in the "Failed Files" list.
I mailed the nzb file and the debug logging lines.

[Stuart264]

Quade, caught one for the "fix list", if you go to add a new group. Hitting enter at the end of the search box no longer works.

In other news my problematic network issues we discussed are sorted, Virgin finally decided to shut me up by sending 2 engineers round to check everything. The issue, the Hub3 overheating and faulting out. Which is exactly the issue I have been telling them since the start of December, bloody f***wits.

[Quade]

Sounds good. Thanks for the feedback all.

[kenr]

+1 on the $(GOG) not filling this under wine.
Couldn't find the files downloaded till I looked at the folder structure

[Quade]

I believe this is already fixed. I'm shooting for another beta Friday.

[kenr]

another werdie, both sfv & par2 files available, for mp3 files
NB downloads the sfv, par2 and all the files but leaves the sfv & par2 in the download window.

quickpar checks both the sfv & par2 as 100%

all under wine

[Calahan]

I've mailed a nzb file were the unpacking fails.

[Quade]

Looking at it, it's an obscured format and there are no PAR files to force a rename. All the files download but there's no connection between one rar and the next so, the parless decoder doesn't even see these as being from the same RAR grouping.

A fix is possible but not in B7 beta. Thanks for the NZB.

[BZee]

Trying 6.90B6
2 problems
ONE:
using headers
Main Download folder:
Y:\U\$(SEARCH)\$(GROUP)\

Step 1) search GOG "CARS" for "Chevy" -- shows many posts

Step 2) select say 10 posts "Chevy-1957 Red" that are in "alt.binaries.pictures.cars" and download to new folder (name folder say "1957 Chevy")
as expected downloaded files are in Y:\U\CARS\1957 Chevy\

Step 3) select say 10 posts "Chevy-1957 White" and download to existing folder (use same as above)
downloaded files were added to Y:\U\CARS\1957 Chevy\

Step 4) select say 10 posts "Chevy-1957 silver" and do "Download - Bypass Filters"
now they are downloaded to Y:\U\CARS\NewsBin Pro Data\spool_v6\alt.binaries.pictures.cars\Storage.db3
(folder Storage.db3 has the 10 downloaded JPGs and descript.ion)

TWO:
Under settings - network I have pause set to resume after 30 Minutes. I paused Newsbin and it resumed quickly (I didn't time it but maybe after 30 seconds or less)

Problem 2: Still resumes downloading after a few seconds (even if "Use Pause Time-out" is not checked)
Problem 1: fixed but entries like Y:\U\CARS\NewsBin Pro Data\spool_v6\alt.binaries.pictures.cars\Storage.db3 are still listed in the descript.ion file

[Quade]

Is the scheduler enabled? It'll override manual pause too.

B7 is up.

[BZee]

My bad - twice!

The scheduler was on - must have accidentally clicked the wrong area of the status bar.

I posted to the wrong area. This was using B7. I went to the B6 area to get my quote and didn't switch back.