Signature Generator

[Moondawgie]

Will the Signature Generator version 2.0 (2.1?) utility work with the signature.db3 file utilized by Newsbin 6.00?

If not, will an updated version of the Signature Generator utility be released for use with Newsbin 6?

Alternatively, if the Newsbin 6 signature.db3 file were replaced with a Newsbin 5 signature.db3 file, would Newsbin 6 recognize the file as being from Newsbin 5 and convert it for use with Newsbin 6?

[Quade]

I would say my answer is "I don't know". It's not something dex or I've talked about.

[Ace]

I tried searching to find an answer to my question and I didn't find much except this thread which I think partially answers it as of earlier this year.

Has any more thought been given to a signature generator for version 6? Or any of the other questions Kingfish asked? I had the same questions.

I finally installed version 6 and see that it does have the duplicate signature option, but I don't know how to catch that up with the files I already downloaded.

Also, I was looking on the Newsbin website for the signature generators for version 5, the ones Kingfish referred to.

I didn't find them, can someone post a download link please? I know I saw them there before, but the website looks different.

Thanks

[Quade]

The signature file from 5.x is compatible with 6 so, there is no upgrade path. Nor is one needed. I suppose I could build something like "siggen" into Newsbin easily enough. There hasn't been much call for it as you can see.

I'll write myself a note and put it on the todo.

[Ace]

Thanks for the reply!

I still need the siggen for version 5 though, is that still on your website or do I have to make a special request for it?

It was there, but I can't find it anymore.

[Quade]

I don't have a copy handy. I'd have to write one again (which isn't that hard). I'll ask dex if he has the old siggen around. It's not clear to me if it'll feed a 5/6 Signature.db3.

const uint32_t C_SIGDATASIZE = 24 * 1024;

byte pHash[RMD_DIGEST_LENGTH * 2];
RMD(&rWorkOrder.m_vDownloadBuffer[0],min(C_SIGDATASIZE,static_cast<uint32_t>(rWorkOrder.m_vDownloadBuffer.size())),pHash);

Basically it's the RMD hash of the first N bytes up to 24K.

char pszBuffer[128];
uint32_t nLength = sizeof(pszBuffer);
CMimeCode::Encode(pszData,RMD_DIGEST_LENGTH,(byte*)pszBuffer,nLength);

Then it's mime encoded before insert into the DB. That's how old the signature is. Mime was needed back in the day. Now I'd just use a blob.

RMD is a common hash function. Not sure why I chose it over MD5.

[Ace]

Wow, I hate to put you to that much trouble since there doesn't seem to be a lot of demand for it. I hope Dex Kept a copy or maybe another poster here who downloaded it still has it, like the first poster in this thread might have it?

Or maybe it's not all that hard to re-do for a person of your talents? You make it sound not too hard but it wouldn't be easy for me. The hints you just gave me would go a long way though i I had to write it.

In any case, I'd sure appreciate it. It can be a tremendous help.

Even where I have signature files for certain stuff in v5, some of it I downloaded as RARs and unrared the individual files.

Now people are posting the individual files, some of which were posted as RARs and some weren't, so even though I already have some files, the signature filter isn't rejecting the files I have, since it apparently doesn't look at the individual files inside the RAR, right? The siggen tool would fix that by scanning the unrared files, and the updated signature file could reject them, so it would be very helpful.

Thanks very much for any help you can offer on this Quade. I appreciate it.

[Quade]

Posted a new one to Usenet.

Encrypted Test. Download at own risk. "SigGenerator.rar" yEnc 902148 Bytes,Idle/New [1 files,0 pars],1.2 MB,0d:00h:00m,Hmm <Hmm@facebook.com>,a.b.test

I haven't tested much beyond making sure it didn't trash the input files and making sure something appeared in the DB.

PM'd you the password. Anyone else wants it, just PM me.

[Moondawgie]

I tried searching to find an answer to my question and I didn't find much except this thread which I think partially answers it as of earlier this year.

Has any more thought been given to a signature generator for version 6? Or any of the other questions Kingfish asked? I had the same questions.

I finally installed version 6 and see that it does have the duplicate signature option, but I don't know how to catch that up with the files I already downloaded.

Also, I was looking on the Newsbin website for the signature generators for version 5, the ones Kingfish referred to.

I didn't find them, can someone post a download link please? I know I saw them there before, but the website looks different.

Thanks

An unmodified copy of the distribution file, siggen5-2.1.zip, downloaded from newsbin.com on September 21, 2009, can be downloaded from FileSwap.com using the following link:

siggen5-2.1.zip - 1014.93 KB

Offhand, I don't know the period of time during which the file will remain available for download.

[Ace]

@Kingfish,

Thank you very much!

@Quade,

Thank you! When you said you posted it to usenet, I searched for "Newsbin" in the newsgroups search, and I found 2 of them:

alt.binaries.newsbin (Giganews tells me the group is empty)
alt.comp.software.newsreaders.newsbin There's only one post in 2011, about a month ago. Interesting to see there are posts going back to 2003, I didn't realize retention was that long. One of the last posts in that newsgroup redirects people to this forum

Is there another usenet group that you posted it to?

[DThor]

It's in alt.binaries.test

DT

[Ace]

Thanks DThor, I found it there, and it does need to be tested so I guess that makes sense, I just didn't think to look there!

That's a huge group, I only downloaded 4 days worth of headers and it still took a while, there were 0ver 600,000 I think!

Now it's time to test it!

Kingfish, have you tried the old Siggen for v5 with Newsbin v6? Any issues?

[Quade]

One change between old and new is that new includes files smaller than 24K and new doesn't include the filename in the DB.

[Ace]

Fantastic!

That sounds more like two changes, and both of them sound like fantastic improvements!

I had no idea the filename was in there, that wasn't really necessary for my purposes.

How small will it go below 24k, or is there a lower limit?

[Quade]

Newsbin doesn't use a lower limit so, I removed the lower limit from Siggen. New siggen is also Unicode and supports > 4 GB files. Mostly from, just rebuilding it but, it too a few tweeks.

[Moondawgie]

Thanks DThor, I found it there, and it does need to be tested so I guess that makes sense, I just didn't think to look there!

That's a huge group, I only downloaded 4 days worth of headers and it still took a while, there were 0ver 600,000 I think!

Now it's time to test it!

Kingfish, have you tried the old Siggen for v5 with Newsbin v6? Any issues?

No. It's bigger than I care to admit (I know... I know...) I was waiting for the final release of Version 6. And now that there is a new version I doubt that I'll bother working with the old version.

I uploaded the old file thinking that Quade might find it useful, or something of interest to add to his private museum.

[Quade]

Haha thanks!

[Ace]

Posted a new one to Usenet.
I haven't tested much beyond making sure it didn't trash the input files and making sure something appeared in the DB.

I just ran a test on it.

Out of a group of 100 files I randomly downloaded 50 of them using v4 so it wouldn't affect the v6 signature.

I then ran the siggen utility and it seemed to work.

Then I tried to download those 100 files with version 6 and it rejected exactly the 50 it was supposed to and downloaded the other 50.

Not a large test, but it sure looks like it's working as intended to me! You're good!

Thanks very much Quade!

[Quade]

Sounds good. Thanks for testing it.

[flat_beer]

2 questions:

1. Can this be used to add more while newsbin is running?
2. If I add files that are already in the signature database do they end up being in it twice?

[Quade]

Hmm

1 - probably. You'll be the first one to try it though so, you can let me know.

2 - Duplicates won't happen.

[IdontexistM8]

How exactly do I set this up in v6? I've got the zip, extracted to the Data folder, run the exe, chosen database but then I get nothing after I've selected a folder. Do I need to put the extracted files somewhere else?

I haven't used the Sig Gen since v4 so perhaps I'm missing something blindingly obvious. . .

[Quade]

You pick a folder where the files you want to generate a signature exist. You should get a list and you should be able to click the go button (or whatever I called it) and watch it scan the files.

[IdontexistM8]

You pick a folder where the files you want to generate a signature exist. You should get a list and you should be able to click the go button (or whatever I called it) and watch it scan the files.

I think I've got a handle on what's happening. It's only showing an added folder in the list if there's a sub-folder(s) within it. For example, if I add a single folder called 'Random Pictures' that has no sub folders you can't see it. But if I added 'More Random Pictures' which has a sub called 'Extras' then you can.

However this doesn't see to matter as the Sig Gen does still scan and say 'done' whether you can see it or not, and the database is updated.

[IdontexistM8]

Quick update. . . I found the V6 Sig Gen in the wiki, now working as expected.

http://www.newsbin.com/downloads/SigGenV6.exe