Cache header backlog using filters - all files fail

Technical support and discussion of Newsbin Version 6 series.

Cache header backlog using filters - all files fail

Postby Bexley » Tue Jun 06, 2017 8:34 am

Hi, I am using positive filters for a group (accept certain users only). I continually end up with blocked cache. For example now I have 21 files on Import for a group. Following your procedure to clear this I remove the locked file by copying the Import files elsewhere to see what is locked. The problem is Newsbin fails to process on every single file so I end up doing stop and restart 21 times, or I can just delete all the Import files and not get an update. Since these are filtered updates - no spam, much lower volume - why is Newsbin failing consistently here? Any thoughts on what to do? Regards
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 28
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Cache header backlog using filters - all files fail

Postby Quade » Tue Jun 06, 2017 10:11 am

I'm skeptical the filters have anything to do with the fact your GZ files are stalling. It's more likely they're corrupt on disk and that's what's causing them to stall.

You could try to un-GZ them with WinRAR. That might tell you if they're damaged or not. If you're running out of disk space while doing header downloads, that could corrupt them too.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42106
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Cache header backlog using filters - all files fail

Postby Bexley » Wed Jun 07, 2017 9:03 am

Thanks, I will check. Only mentioned filters to indicate lower volumes, nothing to do with them I think. Regards
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 28
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Cache header backlog using filters - all files fail

Postby Bexley » Sun Jun 11, 2017 1:20 pm

Hi, there appeared to be nothing wrong with the gz files, able to open with 7z no problem. Disk is fast SSD with download and decoding sent to other disks. Over 100GB free. Regards
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 28
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Cache header backlog using filters - all files fail

Postby Quade » Mon Jun 12, 2017 12:15 am

Maybe we should revisit your filtering idea then.

The question is whether the GZ files are really getting stuck or whether they filters are slowing things down enough to make them appear stuck.

How complicated are the filters? Filtering the header imports is much more expensive than filtering later.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42106
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Cache header backlog using filters - all files fail

Postby Bexley » Mon Jun 12, 2017 9:11 am

Hi, thanks as always for the prompt reply. First, I opened and tested the gz, it opened but the crc failed. This is 100MB download but the SSD is fast, etc (gaming hardware). The filter is accept if string match (no regex). I can try poster match? So today there were 6 chunks 40MB, on test all failed crc check. I will try poster match filter.
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 28
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Cache header backlog using filters - all files fail

Postby Bexley » Mon Jun 12, 2017 9:17 am

Negatory on poster filter, 4 small chunks, 1.4m, 10.5kb, 1.56m, 141kb all fail crc. Really it seems to fail with even the smallest update, mystified here.
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 28
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Cache header backlog using filters - all files fail

Postby Quade » Mon Jun 12, 2017 5:32 pm

Hi, thanks as always for the prompt reply. First, I opened and tested the gz, it opened but the crc failed. This is 100MB download but the SSD is fast, etc (gaming hardware). The filter is accept if string match (no regex). I can try poster match? So today there were 6 chunks 40MB, on test all failed crc check. I will try poster match filter.


If you have bad GZ files that probably explains the stalling. The main question then is why your GZ files get corrupted but mine don't. GZ files aren't that complicated. Newsbin reads data from the internet and writes it out compressed as GZ files. There's not a bunch of room for it to get corrupted. Typically I suspect either something like bad RAM or your security software damaging the files.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 42106
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: Google [Bot] and 2 guests