Filtering out header downloads.

There is no such thing as a stupid question here as long as it pertains to using NewsBin. Newbies feel free to get help getting started or asking questions that may be obvious. Regular users are asked to be gentle. :-)

Moderators: Quade, dexter

Re: Filtering out header downloads.

Postby Quade » Wed Jan 25, 2017 4:42 pm

I will investigate the search option - I had no idea that that would let you avoid downloading headers.


You don't need a subscription to test it. You can go into the search tab and if it's set for "Internet" do some searches. I recommend setting the minimum file size to a reasonable number to filter out junk posts. You can also tell search to only return results from specific groups too.

You only need a subscription to download from it. It's useful for seeing what's in the groups even if you do download headers.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Wed Jan 25, 2017 4:50 pm

I changed my filter name from Spam to HeaderSpam, since I noticed Newsbin had a Spam filter already in the nbi file. I thought maybe there was some possibility of conflict, in naming. Deleted teevee again and tried to download new headers. Set for 1 day, double click caused it to start downloading 8+ million headers for the last 24 hours. So failed again, stopped it and retried with Special 100k, right click command. It still downloaded the spam I'm trying to filter. I even added a second condition;
Reject if Subject \[Private\]
Reject if Poster pr3d@NET.world

Still no luck. It also seems to leave the spam in the Storage.db , since it gets too big too quick, afai can see. Also I can look in the top panels and see Unsorted (Filter Profile) HeaderSpam and a.b.teevee (Filter Profile) HeaderSpam. I actually tried adding it to both, in case one had an issue the other might register.

Thanks for any tips or help.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby dexter » Wed Jan 25, 2017 4:56 pm

Send us a screenshot of your filter options screen with your "HeaderSpam" filter profile selected and send your .nbi file to ts at newsbin dot com and I'll take a look.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Wed Jan 25, 2017 9:03 pm

Sent, thanks for your help and work on this great program. :)
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby dexter » Wed Jan 25, 2017 10:55 pm

FYI, Version 6.73RC2 now allows you to configure a header download filter from the GUI so you don't have to mess with the .nbi file.

viewtopic.php?f=44&t=41375

New instructions (including pictures!) are in the Newsbin Help Wiki.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Wed Jan 25, 2017 11:03 pm

Firstly, I did find an older version of Newsbin, pre v6 where the nbi is in the /data folder. I've double checked the install path and there doesn't seem to be anyway for it to be reading off my I: drive. It's certainly not pathed in the .nbi or even this install of windows. So no idea how it could read it, it would be looking in a different unregistered drive (to the newsbin install and windows), it's just a storage drive.

Installed, found the drop down and added just fine. Deleted teevee, closed and reopened, updated all my groups with the new tab. Closed and reopened, added teevee, added settings, closed and reopened. Double clicked update, it did cut the 8+ million headers in half, now it's only trying to download 4.5 million. It still seems a lot, just for 24 hours. Maybe it has download them before it can remove them from the group? I was tempted not to let it continue at 4.5 million and just do 100k again but I'll let it run. It's up to 2 million downloaded, will check back in about 20 min.

Thanks
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby dexter » Wed Jan 25, 2017 11:13 pm

It is still going to download all the headers in the range, it will then throw away anything matching your filter before it inserts it into the database. I downloaded 2 days of a.b.teevee yesterday and it was 9 million headers. Filtered down represented like 28,000 real posts.

Let it run, wait for Newsbin to process the headers (including filtering out stuff you don't want) and then load the group and see what you have. Remember to watch the Cache display in the Status bar to tell when the headers are done processing. The number in the parenthesis, i.e. "Cache 90/100 (xx)", will go to 0.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby StrangeLove » Wed Jan 25, 2017 11:18 pm

YES! It worked, thanks again for all your hard work and help! 8)
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby Valsidur » Thu Jan 26, 2017 1:43 am

A critical instruction that might be missing is once newsbin is running, to right click on the group or group of groups, and check the "Use Profile" and select the filter name you created for the group of groups you are applying it too.
Valsidur
Occasional Contributor
Occasional Contributor
 
Posts: 15
Joined: Mon Sep 13, 2004 8:29 pm

Registered Newsbin User since: 10/17/03

Re: Filtering out header downloads.

Postby dexter » Thu Jan 26, 2017 10:01 am

No, that is not critical, that turns on the filter profile to be applied when you load headers from the database. It has nothing to do with setting the filter to apply to headers *before* they are in the database.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby ack9 » Thu Jan 26, 2017 1:35 pm

Bravo guys on adding the additional setting in the new RC2.
ack9
Active Participant
Active Participant
 
Posts: 69
Joined: Thu Nov 11, 2004 3:15 pm

Registered Newsbin User since: 11/11/04

Re: Filtering out header downloads.

Postby jacob733 » Thu Jan 26, 2017 5:30 pm

A little tip for how to identify the spammers if newsbin locks up on importing headers in the DB if you happen to run Linux:

Navigate to the import folder in a terminal and zcat the files. This will output the raw headers and it should be relatively easy to spot the big spammers by starting and stopping a couple of times, the spam headers will show up most of the time then.

If there are multiple spammers then you can use grep -v to filter out known spammers one by one so you can spot the new ones.
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby Brad#4 » Thu Jan 26, 2017 6:28 pm

Now that I picked up RC2 that makes it a tad bit easier to filter out the bulk spamers.

I have to prune/sweep up the db3 that has all of that spam in it.

I downloaded the current version of sqlite3

Tried this script under cmd shell with no luck (I did remember to exit the newsbin program first)

set SPOOLDIR=C:\Users\brad#4\AppData\Local\Newsbin\spool_v6

for /F %%x in ('dir /b/d/ad %SPOOLDIR%\alt*') do (
echo %%x
C:\Users\brad#4\AppData\Local\Newsbin\sqlite3.exe %SPOOLDIR%\%%x\Storage.db3 "select count(*) from Compact_V4 where NN_From='nEwZ[NZB] <pr3d@NET.world>';"
)

but the offending poster is still in the database files. - right click purge all posts from user looks like it is doing something for 5+ mins - then hangs on a wait.

highlight one message and sift del fails to remove one message.

I rather not remove the full database because the download times suck due to how much spam has been stuffed in the impacted groups.
Brad#4
n00b
n00b
 
Posts: 1
Joined: Thu Jan 26, 2017 6:08 pm

Registered Newsbin User since: 08/23/09

Re: Filtering out header downloads.

Postby dexter » Thu Jan 26, 2017 9:15 pm

Right-click and delete all posts from poster is the way to go. Do it before you go to bed and let it run overnight. It could take hours depending on how long you've been downloading this spam.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby alxnet » Fri Jan 27, 2017 12:53 am

dexter wrote:No, that is not critical, that turns on the filter profile to be applied when you load headers from the database. It has nothing to do with setting the filter to apply to headers *before* they are in the database.


Good to know. Thanks.
alxnet
n00b
n00b
 
Posts: 5
Joined: Tue Jan 24, 2017 3:03 pm

Registered Newsbin User since: 08/15/05

Re: Filtering out header downloads.

Postby zorro6204 » Fri Jan 27, 2017 12:53 am

How do you get 6.73? I updated and it only gave me 6.72. I tried the instructions anyway, exactly as they pictured it, but no go, it's currently trying to download 2.5 million headers . . .
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby ack9 » Fri Jan 27, 2017 1:50 am

Brad#4 wrote:Now that I picked up RC2 that makes it a tad bit easier to filter out the bulk spamers.

...
C:\Users\brad#4\AppData\Local\Newsbin\sqlite3.exe %SPOOLDIR%\%%x\Storage.db3 "select count(*) from Compact_V4 where NN_From='nEwZ[NZB] <pr3d@NET.world>';"
)



You're using the wrong part of my message: "here's a quick batch script to -->automate finding<--- the affected groups:"

The DELETE command is up above that section.
Last edited by ack9 on Fri Jan 27, 2017 2:05 am, edited 1 time in total.
ack9
Active Participant
Active Participant
 
Posts: 69
Joined: Thu Nov 11, 2004 3:15 pm

Registered Newsbin User since: 11/11/04

Re: Filtering out header downloads.

Postby ack9 » Fri Jan 27, 2017 1:57 am

jacob733 wrote:A little tip for how to identify the spammers if newsbin locks up on importing headers in the DB if you happen to run Linux:

Navigate to the import folder in a terminal and zcat the files. This will output the raw headers and it should be relatively easy to spot the big spammers by starting and stopping a couple of times, the spam headers will show up most of the time then.

If there are multiple spammers then you can use grep -v to filter out known spammers one by one so you can spot the new ones.


Or use SQLite for this. The following will find posters, in a given Storage.db3 file, with more than 500 posts (change to whatever value you want) and print their Poster value (NN_From) and total records count, sorted by smallest number of records to largest:

From SQLite's command prompt after you have opened a .db3 file:
# select NN_From, count(*) AS CountOf FROM Compact_V4 GROUP BY NN_From HAVING count(*)>500 ORDER BY count(*);
ack9
Active Participant
Active Participant
 
Posts: 69
Joined: Thu Nov 11, 2004 3:15 pm

Registered Newsbin User since: 11/11/04

Re: Filtering out header downloads.

Postby jacob733 » Fri Jan 27, 2017 5:09 am

ack9 wrote:Or use SQLite for this.


But if Newsbin locks up while importing headers then the database might not have the headers yet. That's why it is worth checking the import folder.
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby dexter » Fri Jan 27, 2017 11:53 am

zorro6204 wrote:How do you get 6.73? I updated and it only gave me 6.72. I tried the instructions anyway, exactly as they pictured it, but no go, it's currently trying to download 2.5 million headers . . .


It's on the Beta page, not yet released as a production version.

Even with the filters in place, you will still see the complete header count in the header download task in the download list. Fewer headers will be inserted into the header database on your hard drive though. And 2.5 million is nothing during this spam flood. a.b.teevee gets that much every day now.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby zorro6204 » Sat Jan 28, 2017 3:46 am

Thanks, I'll check it out. I had the max days set to one, so 2.5M is about right, there was a lull that day. Still, it crashed Newsbin and froze my PC.

I'm getting more comfortable using a combination of Binsearch and Easynews thumbnails for searching and just using Newsbin to handle the nzb's during this mess (which, if the past is any clue, will end). It's not like I don't know what I'm looking for, I don't need to see everything posted. Directly accessing the troubked group through Newsbin was faster than those two, no delay, but one can learn to be little patient.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby lamekuma » Sat Jan 28, 2017 4:16 am

I had a quick question, is there a tutorial anywhere so I can add additional posters?

For example, in your tutorial, you have the following example:
nEwZ[NZB] Links not allowed for unregistered users = nEwZ\[NZB\].*prd3@net.world

However, I had added it into my lockout poster list and it turned out to be: nEwZ\[NZB\] \<pr3d@NET\.world\>

Thanks for all your assistance so far!
lamekuma
n00b
n00b
 
Posts: 2
Joined: Sat Jan 28, 2017 4:11 am

Re: Filtering out header downloads.

Postby wiggins09 » Sat Jan 28, 2017 8:28 am

I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820
Awesome FREE file manager - http://multicommander.com/
wiggins09
Seasoned User
Seasoned User
 
Posts: 142
Joined: Fri Feb 07, 2014 6:55 pm

Registered Newsbin User since: 02/08/14

Re: Filtering out header downloads.

Postby JesseKnows » Sat Jan 28, 2017 9:48 am

Still on 6.72.

Question about the filters:
In the Filter Profiles dialog I see that the filter named "KillThisPoster" contains "reject if Poster contains pr3d\@NET\.world".
I put "ccc" in the box next to "Enter Test Text" and after clicking Test I get the Test Results "Test text was accepted - Will be included." I put "pr3d@NET.world" in the box and I get the same Test Results. Same results with "pr3d\@NET\.world". Is the test supposed to reflect the same matching as the actual filtering?
Last edited by JesseKnows on Sat Jan 28, 2017 10:08 am, edited 1 time in total.
JesseKnows
Occasional Contributor
Occasional Contributor
 
Posts: 18
Joined: Thu Jun 09, 2005 5:58 pm

Re: Filtering out header downloads.

Postby dexter » Sat Jan 28, 2017 10:02 am

lamekuma wrote:I had a quick question, is there a tutorial anywhere so I can add additional posters?

For example, in your tutorial, you have the following example:
nEwZ[NZB] pr3d@NET.world = nEwZ\[NZB\].*prd3@net.world

However, I had added it into my lockout poster list and it turned out to be: nEwZ\[NZB\] \<pr3d@NET\.world\>

Thanks for all your assistance so far!


Either will work. There are several ways to write a regular expression that will work on that string. The first one is more broad, the 2nd one was generated by Newsbin to match the exact term. If it is working for you I wouldn't touch it.

Keep in mind that each additional item you add to your filter profile will start to degrade performance so choose wisely.
If the regular expressions are targeting different matching items, you can just add an additional rule to the same filter profile.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby dexter » Sat Jan 28, 2017 10:12 am

wiggins09 wrote:I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820


Are you sure nothing changed with your filters? I just ran 1 day of a.b.teevee and there were almost 5 million posts but the header filter was still effective.

You aren't going to see the latest headers until the backlog clears out. You can monitor it by watching the Cache display at the bottom of the Newsbin window. It looks something like "Cache 200/200 (xx)". As long as there are numbers in the parenthesis, it means Newsbin is still processing headers into the database.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Filtering out header downloads.

Postby wiggins09 » Sat Jan 28, 2017 10:47 am

Thanks Dexter

AFAIK my filters/settings were unchanged & I did go thru a check to ensure everything was as it should be.
Ended up deleting the <22gb & starting again with a new batch of <42m headers
It is still processing the cache but not using so much RAM since adding the 'accept if' statement
Awesome FREE file manager - http://multicommander.com/
wiggins09
Seasoned User
Seasoned User
 
Posts: 142
Joined: Fri Feb 07, 2014 6:55 pm

Registered Newsbin User since: 02/08/14

Re: Filtering out header downloads.

Postby dward51 » Sat Jan 28, 2017 12:43 pm

Thanks to all who have contributed to this thread!!!!

The filter poster during header download is working perfectly for me (although I see there is now a patch in the program update to make the process easier). Even with 32gb of ram and an octa-core processor on SSD drives, all that spam basically was putting the affected groups in an endless sort for me.
dward51
n00b
n00b
 
Posts: 1
Joined: Sat Dec 17, 2016 2:11 pm

Registered Newsbin User since: 06/27/08

Re: Filtering out header downloads.

Postby zorro6204 » Sat Jan 28, 2017 2:04 pm

I tried again, downloaded the new version, did exactly and precisely what the "wiki" said, clicked on the teevee group, and about an hour later Newsbin crashed with an "out of memory" error. That left my PC running all messed up and slow, so I cleared all the junk out with Cyber Scrub and rebooted, tried again. The download said it was only trying to load 60K headers, but the blue bar has been crawling across the bottom of the screen for 15 minutes, and I'm going to abort.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby jacob733 » Sat Jan 28, 2017 4:09 pm

zorro6204 wrote:I tried again, downloaded the new version, did exactly and precisely what the "wiki" said, clicked on the teevee group...


You should probably wait to actually open the group until the headers finish downloading and importing. Newsbin is much faster once the download and import is complete.
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby robe36 » Sat Jan 28, 2017 5:16 pm

Is it possible to add message-id to filter gui? Today they are posting the generated names, thousands of different names, in the a.b.teevee and a.b.moovee.

I do notice one thing they share in common is the message id showing @PRIVATE at the end of each message ID.

I would copy an example here, but when rt clicking a header and selecting properties, I have no ability to copy that info to the clipboard.

See this same trend on another poster in a.b.misc and a.b.etc with message ID ending with @reader.easyusenet.nl

Thanks
Last edited by robe36 on Sat Jan 28, 2017 5:37 pm, edited 1 time in total.
robe36
Occasional Contributor
Occasional Contributor
 
Posts: 10
Joined: Tue Mar 19, 2013 12:14 pm

Registered Newsbin User since: 03/14/13

Re: Filtering out header downloads.

Postby BillFrugge » Sat Jan 28, 2017 5:37 pm

wiggins09 wrote:I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820

I'm seeing the same thing. Suddenly I'm stalling again during header download. Nothing on the search engines.

I killed the download and looked in on the headers. I don't know if it's the same clown, but someone figured out a new angle. Every post has a unique email address, and there is nothing common in any of the headers that can be filtered.

Best bet is to reverse your filter as was suggested earlier. Change it to accept only posts containing '\[FULL\]\-\[\#a.b.teevee\@EFNet\]' in the subject line. You'll want to do this in a unique folder because this will prevent anything else that does not include '[FULL]-[#a.b.teevee@EFNet]' in the subject from coming through. I tried it on my setup, and I was finally able to load headers. 2 Million in the last 3 hours!
Last edited by BillFrugge on Sat Jan 28, 2017 5:39 pm, edited 1 time in total.
BillFrugge
Occasional Contributor
Occasional Contributor
 
Posts: 10
Joined: Sun Aug 03, 2003 4:32 pm

Registered Newsbin User since: 04/12/03

Re: Filtering out header downloads.

Postby stephane » Sat Jan 28, 2017 5:38 pm

I guess that
Links not allowed for unregistered users (RIPPER)
should be filtered out too.

Groups:
a.b.comp
a.b.erotica
a.b.movies
a.b.multimedia
a.b.warez

Update sent to https://t.me/NewsbinUsers
stephane
Occasional Contributor
Occasional Contributor
 
Posts: 15
Joined: Sun Apr 19, 2009 3:34 pm

Re: Filtering out header downloads.

Postby robe36 » Sat Jan 28, 2017 5:54 pm

Hey BillFrugge,

I'm confused on what you mentioned by adding this as an accept filter.. \[FULL\]\-\[\#a.b.teevee\@EFNet\] .. I usually search several groups I'm subscribed to, unknowing what I will get from them. Are you saying to add exceptions for each poster that I usually get the downloads from? otherwise that filter to add may cause me to miss alot of posts.

Would be nice to get the message ID filter added so we can reject from that most likely spammer PRIVATE

I sent 2 emails to their usenet provider abuse email address, but they don't respond to emails, at least from me. abuse@snellerdownloaden.com
robe36
Occasional Contributor
Occasional Contributor
 
Posts: 10
Joined: Tue Mar 19, 2013 12:14 pm

Registered Newsbin User since: 03/14/13

Re: Filtering out header downloads.

Postby BillFrugge » Sat Jan 28, 2017 6:06 pm

robe36 wrote:Hey BillFrugge,

I'm confused on what you mentioned by adding this as an accept filter.. \[FULL\]\-\[\#a.b.teevee\@EFNet\] .. I usually search several groups I'm subscribed to, unknowing what I will get from them. Are you saying to add exceptions for each poster that I usually get the downloads from? otherwise that filter to add may cause me to miss alot of posts.

Would be nice to get the message ID filter added so we can reject from that most likely spammer PRIVATE

I sent 2 emails to their usenet provider abuse email address, but they don't respond to emails, at least from me. abuse@snellerdownloaden.com

Whereas you have a filter now that says something to the effect of "reject if poster contains," or "reject if subject contains," my DownloadFilter now says "Accept if subject contains" and used the above string for the criteria.

I added to my message that you'll want to move teevee to it's own folder in your group list (and add the DownloadFilter line) so that this filter is only applied to the teevee group. It will filter out everyone else so that you only see the posts by the bots. I hate doing that as well, but it's the only way I see of getting only the posts that I want.

Somehow, I don't think their usenet provider is going to pay attention. They must be aware of all of the traffic that they've suddenly received and they haven't addressed it yet.
BillFrugge
Occasional Contributor
Occasional Contributor
 
Posts: 10
Joined: Sun Aug 03, 2003 4:32 pm

Registered Newsbin User since: 04/12/03

Re: Filtering out header downloads.

Postby toryin40 » Sat Jan 28, 2017 7:09 pm

I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark
toryin40
n00b
n00b
 
Posts: 2
Joined: Tue Feb 02, 2010 9:48 am

Registered Newsbin User since: 10/17/08

Re: Filtering out header downloads.

Postby jacob733 » Sat Jan 28, 2017 8:05 pm

I took a closer look at the data. It is not spam as such. It looks like it is on-topic data. It is just obfuscated and possibly encrypted. I guess someone grew tired of the stuff getting DMCAed.

Bad news is that this would mean that the posts are not going away again. And they will probably just increase in volume.

They have a special header named "Verify-Signature-Newznzb", so if it was possible to filter on existence of a header, we could target this.

In the mean time, I will try to make a regex that matches the subject. The characters look hexadecimal, so I will work on that. How advanced is the regex support in Newsbin?
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby Taylor » Sat Jan 28, 2017 8:36 pm

Now flooding with random header and posting fields.

As he is using an unusual usenet provider can't that be blocked or an ipaddress from his posts?
Taylor
n00b
n00b
 
Posts: 5
Joined: Mon Jan 23, 2017 5:14 am

Re: Filtering out header downloads.

Postby zorro6204 » Sat Jan 28, 2017 8:54 pm

"Now flooding with random header and posting fields."

Maybe that explains why the procedure didn't work?


"You should probably wait to actually open the group until the headers finish downloading and importing. Newsbin is much faster once the download and import is complete."

But, isn't the purpose of the filter to kill the bad posts? On a normal day Newsbin would load the teevee headers in seconds.


Also, I doubt the poster is well intentioned, you don't fill multiple newsgroups with old posts to the point where the software blows up to be doing a service. He knows he's hated and is revels in it, it's the only explanation.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby MichaelW_ » Sat Jan 28, 2017 9:23 pm

well, the asshole in teevee has switched to random characters now, with random poster id's. Every post seems to include in the subject a ::/ and/or a \:: so i tried trapping a filter for subject contains one of those, but it doesn't seem to be working. Here is a sample subject line:

[ee8cea4558] \775a118142\::3660a589b05a25.e7c07822a7ff1bdf548456da991610.b901d0e7::/2970e1e8293c/ [a441a920a9] [1/2] - yEnc 26342209536

is there any reason this shouldn't work?


Update - I then tried trapping for subject contains ] \ and that doesnt work either
Last edited by MichaelW_ on Sat Jan 28, 2017 10:32 pm, edited 2 times in total.
MichaelW_
Active Participant
Active Participant
 
Posts: 58
Joined: Sat May 02, 2009 7:54 pm

Registered Newsbin User since: 11/10/09

Re: Filtering out header downloads.

Postby lamekuma » Sat Jan 28, 2017 9:28 pm

toryin40 wrote:I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark


Okay, I'll try to use the reject subject header if it contains \[..........\]

I just hope it doesn't block unnecessary stuff
lamekuma
n00b
n00b
 
Posts: 2
Joined: Sat Jan 28, 2017 4:11 am

Re: Filtering out header downloads.

Postby zorro6204 » Sat Jan 28, 2017 9:29 pm

Maybe the solution is to abandon teevee for . . . alt.binaries.cute.cat.pictures or something and cross post to teevee so it takes asshat awhile to catch on.
zorro6204
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Mon Mar 09, 2009 12:03 pm

Re: Filtering out header downloads.

Postby jacob733 » Sat Jan 28, 2017 9:46 pm

This subject reject filter seems to work for me:
\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]

zorro6204 wrote:Also, I doubt the poster is well intentioned, you don't fill multiple newsgroups with old posts to the point where the software blows up to be doing a service. He knows he's hated and is revels in it, it's the only explanation.


Actually, he probably doesn't care. He just charges people $10 a month to give them a DMCA free warez service, and this is his data store. At least that is what I get from https://usenetreviewz.com/newznzb-review/ and https://newznzb.info/

I guess we can hope that he switches groups soon, otherwise this will never end.
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby fcorreia666 » Sat Jan 28, 2017 10:45 pm

I wish I knew how to kill the spamm. But I do not know, unfortunately.
But the idea to abandon teevee, moovee and other groups may not function as the spammer may be reading the posts in this forum.
So, everyone goes to alt.binaries.something and the follows. The only thing he needs to do is add another NG.
I am not tech savvy but I suppose he is using automated tools to do the spamming.
Maybe there are other tools to kill the tools he uses to do his spamming.
I leave that to the guys who know about software.
Just for the record. I sent a complaint to Giganews about this person and never received an answer.
fcorreia666
n00b
n00b
 
Posts: 1
Joined: Fri Nov 25, 2011 7:20 am

Registered Newsbin User since: 11/27/09

Re: Filtering out header downloads.

Postby jacob733 » Sat Jan 28, 2017 11:51 pm

fcorreia666 wrote:But the idea to abandon teevee, moovee and other groups may not function as the spammer may be reading the posts in this forum.


I pretty much guarantee that he doesn't care. He just needs somewhere to store his obfuscated files. Or I should probably say they, since it looks like a group of some sorts. Perhaps the Russian mafia or another organisation dealing with cyber crime.

Anyway, I looked at some of the files and my guess is that if you have some special newzNZB software and an account to someplace then you can get real videos from those posts. Unfortunately that probably means that the poster will need to mutate the format regularly to stay ahead of DMCA requests, and at the same time cause regular annoyance for us when we need to update our filters.
jacob733
Occasional Contributor
Occasional Contributor
 
Posts: 38
Joined: Wed Jan 09, 2002 12:44 pm
Location: Denmark

Registered Newsbin User since: 04/26/03

Re: Filtering out header downloads.

Postby StrangeLove » Sun Jan 29, 2017 12:19 am

Of course they are reading the posts, hell there probably reading here. It's obviously an attack on the server. There's tons of places they could dump files if they are some secret russian mafia files. There's no need to pick the most popular newsgroup.


Thanks again for all your work on this great program! :)
How would you guys feel about adding some functionality that would allow, bypass of Delete Poster and Approval popup. I.e. what if we could say hold down CTL, then right click on a post, automatically deleting all posts from that spammer? Then there's only 2 clicks or 1 (if you don't count holding CTL), instead of 4.
StrangeLove
Occasional Contributor
Occasional Contributor
 
Posts: 42
Joined: Mon Apr 04, 2011 2:07 am

Registered Newsbin User since: 02/08/13

Re: Filtering out header downloads.

Postby ytaxq889 » Sun Jan 29, 2017 1:08 am

When I saw the irritating posts I noticed they were often of a uniform size.

Perhaps a download filter could be created along the lines of "if post greater than XX bytes and less than YY bytes, then don't download". That might not get them all, but enough to make the Group usable. A Message of the Day could be sent to suggest the range if it becomes a variable. Obviously if he makes the post sizes large, then the quantity of posts will decline.

Also this person must be being paid to do this. E.g., if Giganews et al stop this poster, then that proves they are able to exclude posters. In turn that may result in "requests" to stop the good posters. If this person is selling some service, I would avoid giving any personal information to him.

Also it would be nice to have a finer gradation in Min and Max sizes.
ytaxq889
n00b
n00b
 
Posts: 1
Joined: Sun Jan 29, 2017 12:53 am

Registered Newsbin User since: 05/23/07

Re: Filtering out header downloads.

Postby shadowmn » Sun Jan 29, 2017 2:09 am

dexter wrote:FYI, Version 6.73RC2 now allows you to configure a header download filter from the GUI so you don't have to mess with the .nbi file.

viewtopic.php?f=44&t=41375

New instructions (including pictures!) are in the Newsbin Help Wiki.


I think the instructions need to include that you need to restart Newsbin after you add the Header Filter before it will begin taking effect.

shadowmn
shadowmn
Occasional Contributor
Occasional Contributor
 
Posts: 14
Joined: Fri Apr 11, 2003 3:08 pm

Registered Newsbin User since: 05/05/03

Re: Filtering out header downloads.

Postby bnf » Sun Jan 29, 2017 2:56 am

Eight million new headers in twelve hours. Even with my filter working, the initial download takes an eternity now. Sigh.
User avatar
bnf
n00b
n00b
 
Posts: 7
Joined: Fri Sep 24, 2004 9:52 pm

Registered Newsbin User since: 09/12/04

Re: Filtering out header downloads.

Postby shawn » Sun Jan 29, 2017 3:09 am

I don't think there's any choice now but to set up filters based on what you will accept and not what you will reject.
That way you can filter any new iteration that the spammer uses (at least for now.) His new headers are just too different
to try adding them all in to any reject filter.
shawn
Seasoned User
Seasoned User
 
Posts: 202
Joined: Wed May 22, 2002 7:56 pm

Registered Newsbin User since: 04/02/03

PreviousNext

Return to Newbie Forum

Who is online

Users browsing this forum: No registered users and 2 guests