Duplicate Detection when it isn't a duplicate

This is the place to help test and discuss Version 6 Beta releases.

Duplicate Detection when it isn't a duplicate

Postby zforray » Thu Jun 11, 2015 10:34 pm

Lately, I have been seen this issue, a lot. Running current beta.

I just started it up to download a TV show I just heard of so no way it is a duplicate. All of the download pieces are listed as "Downloaded" but the directory listed under the "Download Path" is empty. I am waiting for it to time-out and then I will force it to "Download Load - Bypass Filters". I have done this many times recently and it always works.

So, what gives?

I am also seeing lots of incidents where it simply doesn't run AutoPar and then when I run QuickPar manually on the par files it properly fixes the missing pieces and then I manually unrar with 7-zip with no problems.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby Quade » Thu Jun 11, 2015 11:42 pm

What version are you using?

1 - Any file that has identical prefixes can trigger the dup detector. You can disable it in the advanced options. I'm not getting any false positives but I might not be downloading what you download.

2 - Don't know about this one either. Haven't had a problem download in 6.60 in a long time. During download, you can note how many PAR blocks Newsbin is finding in the files. Because PAR says they're good, that suggests the unrar is failing.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Fri Jun 12, 2015 6:57 pm

660B15 - anything I can help to diagnose this? I just got another one. Just tried to download "the.auction.house.s03e02.720p.hdtv.x264-c4tv" via binsearch.info early this morning before going to work. The downloading has stopped/hung with almost everything marked as "paused" except for a .par file which is listed as downloading (for the past 10+ hours) and sitting in "Failed" as a duplicate.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Fri Jun 12, 2015 6:57 pm

From the "About" 6.60B15: 00 0B 0A 2A 06 8C
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Sat Jun 13, 2015 9:05 pm

Just happened again. I downloaded about 20-episodes of a show and notices the rars still remaining for at least 2-episodes. There is nothing in "Downloading" or "Failed Files" for these. I ran QuickPar on one of the par files and it successfully repaired the 6-blocks missing from this episode and now I successfully, manually unrared using 7zip. I have been seeing this more and more since using the beta version.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Sun Jun 14, 2015 9:14 am

and another 2-stuck that I have to manually quickpar/unrar.

Again, what can I do to help debug this?
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby Quade » Sun Jun 14, 2015 11:25 am

I'm puzzled by the whole thing. I've not had one fail in a long assed time. I'd probably check the logging tab for errors. In the folder opened by options, "Open Data Folder". There should be a file called "Logfile.txt". It gets overwritten each time you start Newsbin but you could wait till it fails. Exit. Then copy and examine this file. I'd suggest disabling "Show Server Commands" in the network options and I'd probably set the logging to "Normal" in the advanced tab of the options.

I'm wondering if you're bumping into space limitations. You really need 3 times as much space free on the disk as the size of the file to ensure a successful unrar/repair (3 times is worst case, 2-2.5 times is typical).

Are you just unraring to a local hard disk?
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Sun Jun 14, 2015 5:19 pm

Yes unraring to local drive.

I have captured everything in the AppData\Local\Newsbin directory on my machine and put it in Dropbox so you can download, here:



The original problem/files left behind/not unrared was for <some shit> <Some Files - don't mention specific files> Both of these directories have all of the rars/par still in them. When I manually unrar with 7zip it shows a CRC error. When I click on the first par file, QuickPar scans and finds 21-blocks missing which it promptly corrects (chunk r06) and then I can unrar just fine.

I have put an untouched/unpar'd copy of the files downloaded for s01e02, here, but since it is 2.4GB, it will take a while to upload (Comcast fast download but slow upload) :
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby mimauk » Sun Jun 14, 2015 5:31 pm

I've had this happen a few times as well with a tv download - the odd thing about it is that in the failed file tab there is usually a reason for failing eg. spam or exe or duplicate, but there was no reason given at all just a blank space.
mimauk
Seasoned User
Seasoned User
 
Posts: 324
Joined: Sun Aug 20, 2006 12:35 pm

Registered Newsbin User since: 03/29/06

Re: Duplicate Detection when it isn't a duplicate

Postby TMM » Sun Jun 14, 2015 6:35 pm

I have also encountered this only happens to me when downloading audio files
copious amounts of free space on my drive
TMM
Seasoned User
Seasoned User
 
Posts: 110
Joined: Sat Feb 05, 2011 12:03 am

Registered Newsbin User since: 08/04/10

Re: Duplicate Detection when it isn't a duplicate

Postby Quade » Sun Jun 14, 2015 7:20 pm

I have also encountered this only happens to me when downloading audio files
copious amounts of free space on my drive


What version are you using? Are you talking about files in RARS or naked files?

I've had this happen a few times as well with a tv download - the odd thing about it is that in the failed file tab there is usually a reason for failing eg. spam or exe or duplicate, but there was no reason given at all just a blank space.


I don't believe he's seeing them in the failed list (might be wrong). I think he's reporting the download completes, leaves the download list but doesn't unrar. OP if they're in the failed list, then that's more like something I can diagnose.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Thu Jun 25, 2015 9:08 am

Currently using 660rc1 - just downloaded 660rc2 so not sure if this is related. Just did a download where it moved it to "Failed Files" with no explanation in the FF section as to why it failed.
<Image Removed>

However, when checking the Logging tab, it says "[07:58:12] ALWAYS Chunk Filter: Chunk Filter Plugin HFC:Duplicate Detector Match: storage.hunters.uk.s02e04.720p.hdtv.x264-c4tv.nfo" but I don't recall downloading this before. Is there any way to look at the list of what has been downloaded, before? The current "File List" doesn't have this file in it.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby Quade » Thu Jun 25, 2015 10:08 am

The duplicate detector doesn't store filenames. It just stores a number that represents the first 23K of the file. The problem is any file with the same prefix will be dup checked out. I haven't seen any false positives recently but I'm sure it happens. You just need to either disable the dup detector altogether or add the files with "Bypass Filter".
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Thu Jun 25, 2015 10:29 pm

Thanks for clarifying that. Most of the time it is pretty accurate but lately it seems to be catching more and more.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby zforray » Sun Jul 05, 2015 9:18 am

Quade wrote:What version are you using?

2 - Don't know about this one either. Haven't had a problem download in 6.60 in a long time. During download, you can note how many PAR blocks Newsbin is finding in the files. Because PAR says they're good, that suggests the unrar is failing.


FYI - FWIW, I am still seeing lots of occasions where it doesn't autopar and when I manually autopar, it is cleanly repaired and unrar-able. Just happened on the last 2-downloads using rc2. Just installed rc3 so I don't know if you addressed this in rc3.
zforray
Occasional Contributor
Occasional Contributor
 
Posts: 34
Joined: Thu May 21, 2009 4:46 pm

Registered Newsbin User since: 11/04/06

Re: Duplicate Detection when it isn't a duplicate

Postby Quade » Sun Jul 05, 2015 10:49 am

It never happens to me so, it's kinda hard for me to address it. Logs should be telling you what's happening. Without more to go on it's going to be hard to trouble-shoot.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44866
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to Newsbin Version 6 Beta Support

Who is online

Users browsing this forum: Google [Bot] and 2 guests