Batch Filter additions?

Tips on writing regular expressions for searching the post list

Moderators: Quade, dexter

Batch Filter additions?

Postby gardenzombie » Mon Feb 04, 2008 4:48 pm

Does anyone know how to batch-add a whole funk-locker full of filters? For example, let's say I want to filter, either deny or accept, for about 1000 bands. Do I really have to add each one manually in the filter dialog? I think the filters are in XML format, no? If so - has anyone created a script or tool to, say, parse a text/csv/etc document with a bunch of items, and automatically populate the correct filter list?
gardenzombie
n00b
n00b
 
Posts: 4
Joined: Mon Feb 04, 2008 4:42 pm

Registered Newsbin User since: 03/24/06

Postby nzzz » Mon Feb 04, 2008 5:03 pm

Build as few well thought out Regular Expressions as possible.
Using too many filters just slows you down.

RegExBuddy is a great way to learn, build and test before you add.
nzzz
Seasoned User
Seasoned User
 
Posts: 280
Joined: Thu Nov 10, 2005 9:50 pm

Registered Newsbin User since: 11/11/05

Batch Filter additions?

Postby gardenzombie » Mon Feb 04, 2008 6:03 pm

I hear ya - and thanks very much for the response - but this is mainly for filtering out =many= unwanted person-names.

For example - if I have a newsgroup that lists a short Biography of every recording artist, and the group retains around 300,000 post entries ... I'd like to not have to read through that many names, when, for example, I know I'll never want to see the bio for "Prince" or "Ella Fitzgerald" (just examples - I've got nothing against those two artists).

This is an ongoing, growing, filter list. I had hundreds of entries in the filter, all manually added. Then I lost the filter list, and am slowly recreating it. However, I'd rather find a way to add multiple entries in one shot, rather than adding them one at a time.
gardenzombie
n00b
n00b
 
Posts: 4
Joined: Mon Feb 04, 2008 4:42 pm

Registered Newsbin User since: 03/24/06

Postby Quade » Mon Feb 04, 2008 7:16 pm

He's right about performance. You'll eventutually find newsbin slowing down to the point it's unusable. You're better off selecting the files you don't want and just hitting "Shift-Delete" to permenantly delete them from the DB or RV2 files (depending on your version).

Filtering's great for finding things you want, not so hot for trying to filter all of usenet.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44972
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby gardenzombie » Mon Feb 04, 2008 8:12 pm

I understand your point fully. However, neither yours nor the previous response addresses my original question, which was whether there was a way to batch-add entries.


What I'm filtering isn't the issue - but rather how to add the filters I'm going to use. I do, however, truly appreciate your condescending remark about filtering all of usenet.
gardenzombie
n00b
n00b
 
Posts: 4
Joined: Mon Feb 04, 2008 4:42 pm

Registered Newsbin User since: 03/24/06

Postby richy99 » Mon Feb 04, 2008 8:32 pm

the filters are now stored in a db3 file rather than xml, there is no facility to batch import
User avatar
richy99
Elite NewsBin User
Elite NewsBin User
 
Posts: 6353
Joined: Fri Nov 21, 2003 8:04 pm
Location: Wales

Registered Newsbin User since: 12/31/03

Postby Quade » Mon Feb 04, 2008 8:45 pm

If you know SQL you can add filters pretty easily.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44972
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Postby gardenzombie » Fri Feb 08, 2008 2:06 am

Ah, great, thanks.
gardenzombie
n00b
n00b
 
Posts: 4
Joined: Mon Feb 04, 2008 4:42 pm

Registered Newsbin User since: 03/24/06

Postby bobkoure » Thu Feb 21, 2008 2:26 pm

If you go to the SqLite web site, there are some free utilities you can download - or if you're comfortable with writing code, there are Sqlite "connectors". For instance, when I'm writing something where I care more about implementation time than execution time, I use Python, which is easy. In that environment, I'd use pysqlite for easy access to sqlite dbs

To get a notion of the schema NBP is using, download sqlite3 and try the following command, which will list the contents.
sqlite3 "filters.db3" "SELECT FL_FilterName,FL_FilterText FROM FilterList x, FilterMatchSubjects y WHERE x.FL_FilterId = y.FL_FilterId UNION SELECT FL_FilterName,FL_FilterText FROM FilterList x, FilterMatchFiles y WHERE x.FL_FilterId = y.FL_FilterId;"
Note that sqlite3.exe and filters.db need to be in the same directory for this to work.

It seems like a batch import would be straightforward to build. Just use the windows "ini-file" format, which python and perl and vb have built in parsers for (well, in perl, it's tacked on, but so is everything else). I asked some while back (when I didn't have a couple of deadlines bearing down on me) about interest in such a utility. Interest was basically zero, so I built the easiest thing that satisfied my own needs, which was a batch file that uses the above statement to list filters.db contents.

newsgroup that lists a short Biography of every recording artist

What newsgroup might that be? Don't answer unless there's some non-copyrighted info in there, so as not to get yourself in trouble. But a NG like that might actually prompt me to write something that looks for all the artists I have CDs for.
bobkoure
 


Return to Regular Expressions

Who is online

Users browsing this forum: No registered users and 2 guests