Filtering out header downloads.

There is no such thing as a stupid question here as long as it pertains to using NewsBin. Newbies feel free to get help getting started or asking questions that may be obvious. Regular users are asked to be gentle. :-)

Moderators: Quade, dexter

Re: Filtering out header downloads.

Postby zorro6204 » Sun Jan 29, 2017 3:38 am

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

I tried that as a subject filter, didn't work for me. When I restarted Newsbin, it's still trying to download 8M headers, and it can't, even if I left it all night it would crash.

"Even with my filter working, the initial download takes an eternity now. Sigh."

Yeah, well, that's the problem now, I can't even get to the point where I can see if the filter is working. Why is the software ignoring the filter and trying to download all those headers?

It seems to me the solution would be to include as an option in the setup, a choice whether to ignore all posters who have put up X number of files in the last 24 hours (say). Put in a pull-down option bar with numbers like 10,000, 50,000, 100,000, etc. That would include all the good guys and exclude the bad, even if they change their name every few hours.

But for something like that to work, it has to work immediately, as soon as you add the group, or you can never load the group to use it at all.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby StrangeLove » Sun Jan 29, 2017 5:16 am

Shouldn't the Storage.db3 file be getting smaller as I delete all these spam files or is there a separate command to resize it ?

Thanks
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby OMG2011 » Sun Jan 29, 2017 5:50 am

I which we could filter by msgID...
Newsbin Pro 64bit 6.90 Build 5524 (on Win 10 Pro for Workstations)
User avatar
OMG2011
Occasional Contributor
Occasional Contributor
 
Posts: 47
Joined: Sat May 21, 2011 6:50 am

Registered Newsbin User since: 06/06/09

Re: Filtering out header downloads.

Postby JayPea » Sun Jan 29, 2017 6:26 am

This current onslaught of spam in the groups is making Newsbin quite unusable. The headers are posted in such volume that even on a i7-6700k@4.6GHz, 32GB DDR4, NVME SSD the application is just sitting as not responding when trying to delete these spam posts from the group. I can't imagine what it's like on a mechanical drive connected to a low-spec machine. I know there's a tendency to think of pulling headers as old fashioned but some of us still do it. Something's gotta change :x
JayPea
Active Participant
Active Participant
 
Posts: 92
Joined: Tue Dec 12, 2006 2:48 pm

Registered Newsbin User since: 12/11/06

Re: Filtering out header downloads.

Postby PCZ » Sun Jan 29, 2017 6:52 am

The latest batch of spam in ab.teevee can be filtered out quite easily.
Removing it if you do get the headers in your database is anything but.
You have to delete a few at a time otherwise NB will just hang up.

To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.
PCZ
n00b
n00b
 
Posts: 7
Joined: Sun Jul 04, 2010 5:51 pm

Registered Newsbin User since: 09/28/05

Re: Filtering out header downloads.

Postby alxnet » Sun Jan 29, 2017 7:01 am

PCZ wrote:To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.


Thumbs up for this. :D
alxnet
n00b
n00b
 
Posts: 5
Joined: Tue Jan 24, 2017 3:03 pm

Registered Newsbin User since: 08/15/05

Re: Filtering out header downloads.

Postby JayPea » Sun Jan 29, 2017 7:07 am

Well, the fix for me was to add a header filter to reject if poster contains [0-9a-f]{9}@[0-9a-f]{9}.[0-9a-f]{9} then purge the message stores and then pull the headers down again. Now NB seems to process the group without having a meltdown.

God damn these monstrous pricks ruining usenet. A pox on them all.

PS. will also add the :: filter now. Cheers.
JayPea
Active Participant
Active Participant
 
Posts: 92
Joined: Tue Dec 12, 2006 2:48 pm

Registered Newsbin User since: 12/11/06

Re: Filtering out header downloads.

Postby zorro6204 » Sun Jan 29, 2017 10:04 am

I tried that, but I can't even get started, no matter what I do the software crawls, displays a memory error and crashes. For example, when I reinstate teevee I tried just downloading 50K headers, and that's all that shows on the download tab. But the blue bar still crawls and it crashes, so apparently the software is still having to process all those millions of headers.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby MarvinGardens » Sun Jan 29, 2017 11:50 am

zorro6204 wrote:It seems to me the solution would be to include as an option in the setup, a choice whether to ignore all posters who have put up X number of files in the last 24 hours (say). Put in a pull-down option bar with numbers like 10,000, 50,000, 100,000, etc. That would include all the good guys and exclude the bad, even if they change their name every few hours.


If we can determine that a single poster has put up multiple files, even though he is using different names, can't we filter on that?

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?
MarvinGardens
Occasional Contributor
Occasional Contributor
 
Posts: 25
Joined: Sat May 20, 2006 9:53 pm

Re: Filtering out header downloads.

Postby jacob733 » Sun Jan 29, 2017 12:10 pm

MarvinGardens wrote:I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?


No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby shawn » Sun Jan 29, 2017 12:14 pm

Has anyone else noticed something odd with the Test function for the filter? It seems that once it rejects a filter it gets stuck and
won't test new filters. I thought it might be due the amount of memory Newsbin Pro was using at the time so I restarted NBP and did
a quick test with a filter that it would reject and then tested a simple filter of just the letter a. It kept showing it as rejected even as I
tried testing it multiple times. Only a restart would allow NBP to accept the filter. This is with the RC2 candidate.
shawn
Seasoned User
Seasoned User
 
Posts: 202
Joined: Wed May 22, 2002 7:56 pm

Registered Newsbin User since: 04/02/03

Re: Filtering out header downloads.

Postby shawn » Sun Jan 29, 2017 12:16 pm

He may be selling a service but essentially it is a DOS for everyone else trying to use the newsgroups he is flooding. Any responsible
news server owner should be rejecting him and if they don't then the upstream provider should reject that news server until such
time as he stops flooding the news groups as it makes them very hard to use for everyone else.
shawn
Seasoned User
Seasoned User
 
Posts: 202
Joined: Wed May 22, 2002 7:56 pm

Registered Newsbin User since: 04/02/03

Re: Filtering out header downloads.

Postby StrangeLove » Sun Jan 29, 2017 12:17 pm

zorro6204 wrote:I tried that, but I can't even get started, no matter what I do the software crawls, displays a memory error and crashes. For example, when I reinstate teevee I tried just downloading 50K headers, and that's all that shows on the download tab. But the blue bar still crawls and it crashes, so apparently the software is still having to process all those millions of headers.
You've been following this thread? You need to update to the newest version to have the new drop down. Follow the instructions you will find on page 3 of this thread.

Once you delete teevee, you need to close Newsbin. Go to your install folder, then /SPOOL_V6/ and make sure you don't see a /alt.binaries.teevee folder, if so delete it. Reopen Newsbin, add new groups alt.binaries.teevee and I close here (may not be necessary). Reopen, make sure you to add your new filter to the group. Once you get it all done right click on your new alt.binaries.teevee download special 100k. You should be good to go.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby chimmyjim » Sun Jan 29, 2017 12:37 pm

FWIW, you can get pretty specific with the following:
Code: Select all
\[[0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f]\]

This will only target the spam files and leave most everything else untouched.

Jim
chimmyjim
Seasoned User
Seasoned User
 
Posts: 178
Joined: Thu Dec 18, 2008 7:16 pm
Location: Seattle, WA

Registered Newsbin User since: 12/18/08

Re: Filtering out header downloads.

Postby MarvinGardens » Sun Jan 29, 2017 1:59 pm

jacob733 wrote:
MarvinGardens wrote:I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?


No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb


Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.
MarvinGardens
Occasional Contributor
Occasional Contributor
 
Posts: 25
Joined: Sat May 20, 2006 9:53 pm

Re: Filtering out header downloads.

Postby unyquy » Sun Jan 29, 2017 4:20 pm

Does what PCZ wrote, namely

"To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon."

work for version 6.62. Can someone please let me know. Thanks. unyquy
unyquy
Occasional Contributor
Occasional Contributor
 
Posts: 16
Joined: Sat Apr 25, 2009 1:15 pm

Registered Newsbin User since: 11/17/07

Re: Filtering out header downloads.

Postby sunishun » Sun Jan 29, 2017 6:52 pm

What a mess. But, adding this reject filter seems to be working for me so far. Thanks for all the tips and help filtering out these bastard's posts.

toryin40 wrote:I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark
User avatar
sunishun
Occasional Contributor
Occasional Contributor
 
Posts: 21
Joined: Tue Jul 05, 2011 8:27 pm

Registered Newsbin User since: 04/13/07

Re: Filtering out header downloads.

Postby Rilly » Sun Jan 29, 2017 8:07 pm

After adding these new filters, I have many headers that say Subject Reject under status - but I'm still seeing those headers. What am I doing wrong?
Rilly
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sat Dec 14, 2013 4:10 pm

Registered Newsbin User since: 11/09/13

Re: Filtering out header downloads.

Postby shawn » Sun Jan 29, 2017 8:36 pm

Maybe those headers were already in your database? As others have said the filter feature won't get rid of the rejected from your database if they are already there. It just serves to block new headers from being added to the database if they match the filters.
shawn
Seasoned User
Seasoned User
 
Posts: 202
Joined: Wed May 22, 2002 7:56 pm

Registered Newsbin User since: 04/02/03

Re: Filtering out header downloads.

Postby dr_dx » Sun Jan 29, 2017 9:16 pm

PCZ wrote:To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.


Currently, this is working for me with 6.73RC2. Out of 10mil headers DL in teevee, 1692 legit headers came through.
dr_dx
n00b
n00b
 
Posts: 9
Joined: Wed Feb 18, 2004 2:21 am

Registered Newsbin User since: 02/18/04

Re: Filtering out header downloads.

Postby salparadise » Sun Jan 29, 2017 9:20 pm

Setting up this filter in 6.56 was working perfectly. But whatever this spammer has changed over the last day or two has messed it up again.

From the affected groups it downloads the headers but chokes and never makes progress importing them. :cry:

I understand this is a problem for the new 6.73 RC2 as well, correct?
salparadise
Occasional Contributor
Occasional Contributor
 
Posts: 16
Joined: Sat May 16, 2009 1:49 am

Registered Newsbin User since: 07/11/08

Re: Filtering out header downloads.

Postby Rilly » Sun Jan 29, 2017 9:33 pm

shawn wrote:Maybe those headers were already in your database? As others have said the filter feature won't get rid of the rejected from your database if they are already there. It just serves to block new headers from being added to the database if they match the filters.


I started with deleting the group using the "purge storage" option. Shouldn't that start it from scratch or does that not delete the headers in the database?
Rilly
Occasional Contributor
Occasional Contributor
 
Posts: 11
Joined: Sat Dec 14, 2013 4:10 pm

Registered Newsbin User since: 11/09/13

Re: Filtering out header downloads.

Postby lincsat » Sun Jan 29, 2017 10:09 pm

MarvinGardens wrote:
jacob733 wrote:
MarvinGardens wrote:I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?


No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb


Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.


I've always suspected that this sort of spam flooding is something to do with the Media industry. DMCA orders don't work, so make the usenet unusable
lincsat
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Fri Jul 18, 2003 4:44 pm
Location: UK

Registered Newsbin User since: 04/05/03

Re: Filtering out header downloads.

Postby jlindquist » Sun Jan 29, 2017 10:52 pm

Could also be Mike Pence cleaning up America's morals.
jlindquist
Occasional Contributor
Occasional Contributor
 
Posts: 48
Joined: Thu Nov 29, 2007 4:24 am

Registered Newsbin User since: 11/22/07

Re: Filtering out header downloads.

Postby stephane » Mon Jan 30, 2017 1:52 am

My config allowed Newsbin to "update" the groups on startup. When your connection to the Internet is fast, that leads to thousands and thousands unwanted headers downloaded.
My only option was to delete the group and start fresh, with the latest filters.

Works well, but - tell me if I'm wrong - I cannot load more than 10 000 000 headers. Today that's roughly the last 2 days of a spammed group.

The other choice is to reload all the headers.

I suggest there is a few new steps to the Download Special menu, maybe 50 000 000 or 100 000 000 posts ?
stephane
Occasional Contributor
Occasional Contributor
 
Posts: 15
Joined: Sun Apr 19, 2009 3:34 pm

Re: Filtering out header downloads.

Postby alxnet » Mon Jan 30, 2017 3:24 am

stephane wrote:I suggest there is a few new steps to the Download Special menu, maybe 50 000 000 or 100 000 000 posts ?


Great idea. I'd like these options too.
alxnet
n00b
n00b
 
Posts: 5
Joined: Tue Jan 24, 2017 3:03 pm

Registered Newsbin User since: 08/15/05

Re: Filtering out header downloads.

Postby johnisme » Mon Jan 30, 2017 4:31 am

I can't be bothered to use filters to block spammers,takes too much time,cant even load all the headers from teevee since it eats all my ram(13gb+ out of 16).All i use now is the search engine.
User avatar
johnisme
Active Participant
Active Participant
 
Posts: 92
Joined: Mon Sep 24, 2007 12:00 am

Registered Newsbin User since: 09/22/07

Re: Filtering out header downloads.

Postby Zen_Rebel » Mon Jan 30, 2017 7:58 am

I was focused on rejecting headers with more or less acceptable results using a filter.

I now created a filter with just: accept if....subject contains...#a.b.teevee@EFNet and the processing of todays 9 million headers took 20 seconds and i retrieved all the legit headers from the spider. Of course you won't have the personal posts and/or commentaries but i never opened them anyways. Don't forget to retrace your steps if you edited the .nbi file manually. Good luck to all!
Zen_Rebel
Occasional Contributor
Occasional Contributor
 
Posts: 37
Joined: Wed Dec 02, 2009 5:49 am

Registered Newsbin User since: 12/01/09

Re: Filtering out header downloads.

Postby technogeek » Mon Jan 30, 2017 1:40 pm

MarvinGardens wrote:
jacob733 wrote:
MarvinGardens wrote:I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?


No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb


Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.



I've always suspected that this sort of spam flooding is something to do with the Media industry. DMCA orders don't work, so make the usenet unusable
==========================================

Nah, it's that their customers don't use NG readers or deal with headers at all. The spammers who are selling access to the data they are parking there just need a NG that has good volume and activity which in turn cause the Providers to up the retention and disk size for it. The chaos they cause for traditional users likely never even reaches their notice. Very scammy.
TANSTAAFL
User avatar
technogeek
Eternal n00b
Eternal n00b
 
Posts: 49
Joined: Tue Oct 22, 2002 4:38 pm

Registered Newsbin User since: 04/24/03

Re: Filtering out header downloads.

Postby zorro6204 » Mon Jan 30, 2017 1:47 pm

"so make the usenet unusable"

But all they've done is to make one feature of usenet software unusable for one or more groups, that is, display all the headers posted over a given time. That was useful, as it hipped me to new shows I was unaware of, but there are other ways to gather that information, websites on new shows in the US, UK and elsewhere. In the meantime, posting has gone on as usual, everything I wanted I found, either using Binsearch, if the show was posted using its name, or by scanning thumbnails with Easynews, for shows posted using gibberish file names (like a particularly icy one).

So if this is a grandiose effort to shut everything down, it failed. But it doesn't feel like that, it feels like the rantings of a child.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby StrangeLove » Mon Jan 30, 2017 2:32 pm

If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.

Btw is it over for the moment, only 100k headers in the last 4 hours? That's download not, after filter header count.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby Stuart264 » Tue Jan 31, 2017 7:37 am

Quade

Just an idea but would it be possible to add an option to use the "killthisspam" (or whatever you have called it) filter in compact newsgroups, that way you can add a filter retrospectively and clean up the fertilizer without having to delete all posts and download the headers again with an updated filter in place?

Stuart.
User avatar
Stuart264
Seasoned User
Seasoned User
 
Posts: 227
Joined: Sun Jun 13, 2004 8:25 am
Location: United Kingdom

Registered Newsbin User since: 04/10/04

Re: Filtering out header downloads.

Postby Usual Suspect » Tue Jan 31, 2017 10:41 am

StrangeLove wrote:If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.


If he was "just" posting for his customers he wouldn't have felt the need to change it up when other people learned how to block him out. And the volume he is posting is beyond any reasonable size, even for warez.
Usual Suspect
n00b
n00b
 
Posts: 7
Joined: Mon Jul 14, 2003 6:03 pm

Registered Newsbin User since: 04/13/03

Re: Filtering out header downloads.

Postby Quade » Tue Jan 31, 2017 10:54 am

Just an idea but would it be possible to add an option to use the "killthisspam" (or whatever you have called it) filter in compact newsgroups, that way you can add a filter retrospectively and clean up the fertilizer without having to delete all posts and download the headers again with an updated filter in place?


I was wondering about this the other day. Whether there needs to be a method to apply a filter to stored headers. I'm still thinking about it. It won't be trivial.

But all they've done is to make one feature of usenet software unusable for one or more groups, that is, display all the headers posted over a given time. That was useful, as it hipped me to new shows I was unaware of, but there are other ways to gather that information, websites on new shows in the US, UK and elsewhere. In the meantime, posting has gone on as usual, everything I wanted I found, either using Binse


There's no need for them to post to popular groups. If they posted one time to some obscure groups, it would still work for them but not annoy other people.

I now created a filter with just: accept if....subject contains...#a.b.teevee@EFNet and the processing of todays 9 million headers took 20 seconds and i retrieved all the legit headers from the spider. Of course you won't have the personal posts and/or commentaries but i never opened them anyways. Don't forget to retrace your steps if you edited the .nbi file manually. Good luck to all!


To me this is the best way to filter. Only filter in the posts you want. It saves the most disk space and should make import the fastest.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44865
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Tue Jan 31, 2017 11:04 am

Usual Suspect wrote:
StrangeLove wrote:If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.


If he was "just" posting for his customers he wouldn't have felt the need to change it up when other people learned how to block him out. And the volume he is posting is beyond any reasonable size, even for warez.
Had no idea he changed it up but yes, another good reason he's a fraud. I actually thought it was a different guy, since I couldn't still see the original spam once I got my filters working.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby robe36 » Tue Jan 31, 2017 1:10 pm

Any thought of adding the ability to filter message-id?
robe36
Occasional Contributor
Occasional Contributor
 
Posts: 10
Joined: Tue Mar 19, 2013 12:14 pm

Registered Newsbin User since: 03/14/13

Re: Filtering out header downloads.

Postby ItsMe » Tue Jan 31, 2017 9:29 pm

StrangeLove wrote:Shouldn't the Storage.db3 file be getting smaller as I delete all these spam files or is there a separate command to resize it ?

Thanks


To shrink the storage.db3 file you need to right click the group go to post storage and then click compact group, then sit back and prepare to wait, best to do just before going to bed....
ItsMe
Active Participant
Active Participant
 
Posts: 61
Joined: Wed Jan 18, 2006 12:02 am

Registered Newsbin User since: 12/29/05

Re: Filtering out header downloads.

Postby rh » Fri Feb 10, 2017 10:37 pm

So I guess there's no way to test the filter without downloading headers? For instance, doing some sort of search through posts I've already downloaded to make sure my regex is correct.
rh
Seasoned User
Seasoned User
 
Posts: 301
Joined: Sat Nov 27, 2004 4:02 pm

Registered Newsbin User since: 11/27/04

Re: Filtering out header downloads.

Postby dexter » Fri Feb 10, 2017 11:46 pm

rh wrote:So I guess there's no way to test the filter without downloading headers? For instance, doing some sort of search through posts I've already downloaded to make sure my regex is correct.


Sure there is, just create the filter profile then load a group that has some of the posts you want filtered and apply the filter profile then see if they disappear. You just have to turn the filters on in the filter bar first by clicking the icon that looks like a power button. Then select the filter profile in the last field to the right.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby dexter » Fri Feb 10, 2017 11:47 pm

robe36 wrote:Any thought of adding the ability to filter message-id?


It exists already. Quade documented it here.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby johnisme » Sat Feb 11, 2017 2:27 am

There no reason to use header filtering now since the poster has stopped flooding the newsgroups,been working good with no issues for the last week.
User avatar
johnisme
Active Participant
Active Participant
 
Posts: 92
Joined: Mon Sep 24, 2007 12:00 am

Registered Newsbin User since: 09/22/07

Re: Filtering out header downloads.

Postby dexter » Sat Feb 11, 2017 2:02 pm

johnisme wrote:There no reason to use header filtering now since the poster has stopped flooding the newsgroups,been working good with no issues for the last week.


Unless you are downloading headers from January 2017. Then you'll definitely need it. For example, the default download age is 30 days, which will definitely (from mid Feb 2017 at the time of this writing) dip in to the spam flood period.

This is only for people who add the group or reset the group. If you have been downloading headers all along and are just doing updates then yeah, probably don't need filters at the moment.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby Mark Allie » Sun Feb 12, 2017 2:33 am

I am going to try filtering using accept instead of reject. Do you know how to setup filtering to reject all accept what is in the accept filters. I tried accept filters and they don't actually reject things which are not accepted. Is there a reject all accept type statement or not? What do you use?

Mark
Mark Allie
Occasional Contributor
Occasional Contributor
 
Posts: 15
Joined: Mon Nov 19, 2012 10:38 pm

Re: Filtering out header downloads.

Postby dexter » Sun Feb 12, 2017 10:59 am

Mark Allie wrote:Do you know how to setup filtering to reject all accept what is in the accept filters.Mark


That's the definition of "Accept If". It will only bring in posts that match the regular expression of posts you wish to accept. If it isn't working then it must not be configured correctly.

You can test your filter profile by loading a few days of a group you want to use this on then apply the filter profile in that post list. If you can't figure it out, tell us what regular expression you are using and maybe we can see what's wrong with it. If you are not familiar with regular expressions, it's easy to enter something that will match everything. Also if there is a syntax error in the expression you wrote, it will appear to match everything.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby dexter » Sun Feb 12, 2017 11:00 am

dexter wrote:
Mark Allie wrote:Do you know how to setup filtering to reject all accept what is in the accept filters.Mark


That's the definition of "Accept If". It will only bring in posts that match the regular expression you want to apply to posts you wish to accept. If it isn't working then it must not be configured correctly.

You can test your filter profile by loading a few days of a group you want to use this on then apply the filter profile in that post list. If you can't figure it out, tell us what regular expression you are using and maybe we can see what's wrong with it. If you are not familiar with regular expressions, it's easy to enter something that will match everything. Also if there is a syntax error in the expression you wrote, it will appear to match everything.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Tue Feb 14, 2017 4:35 pm

Ooc, is there a way to update your current database files to the new filters, or is Deleting the group and starting over the only way, atm?

The reason I ask, it would be nice to keep all the data I collected on another group. I lost all my past data from the teevee situation. Att, I really didn't know how to back them up properly or use filters. This other group is getting a much more sophisticated attack, where the title and poster name is randomly generated for each post. There is no common factor to create a 'Reject If' parameter. So my only real choice is to 'Accept If' parameter.


Thanks, again for all your work on this great program!
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby dexter » Tue Feb 14, 2017 5:51 pm

There currently is no way within Newsbin to apply filters to delete items already in the database except to load items in a post list, apply the filter there, then ctrl-a to highlight them and shift-delete to remove them from the database. If it is a spammed group, the system resources required to do this may be prohibitive but you could try to load the group a week at a time and attack it that way.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby pjcamp » Tue Feb 14, 2017 7:58 pm

Would someone mind saying what, specifically, your filters are that will make Newsbin usable again? I'm not having a great deal of luck.
pjcamp
Active Participant
Active Participant
 
Posts: 67
Joined: Sat Sep 19, 2009 3:58 pm

Registered Newsbin User since: 08/25/07

Re: Filtering out header downloads.

Postby StrangeLove » Tue Feb 14, 2017 9:02 pm

Thanks for the reply!

Pjcamp, it really depends on the group your trying to filter and who/which your trying to filter. You could list the group and the poster/technique or read the thread and/or the wiki. All the answers are here.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby Quade » Wed Feb 15, 2017 1:59 pm

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

Seems to be the best one with a "subject reject". If you group is already contaminated, you might need to set this filter and then purge the group and re-download headers.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44865
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

PreviousNext

Return to Newbie Forum

Who is online

Users browsing this forum: No registered users and 2 guests