ReadFullDatabase: Invalid packet Format: Storage.db3

This is the place to help test and discuss Version 6 Beta releases.

ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Mon Aug 23, 2021 6:54 pm

I just got ERROR ReadFullDatabase: Invalid packet Format: Storage.db3 and all downloads halted. Any idea what I should do about it?

I do have extensive headers as I tend to search old pictures newsgroups so want to go back full retention. Its not the end of the world if I have to download all headers again but it would be somewhat annoying if I lost records of what I have downloaded already.

If its relevant I'm on 6.90RC1 Build 5424.
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Tue Aug 24, 2021 12:58 am

I'm not completely sure. There's an issue in RC1 where it won't read old header formats but, I don't believe it would give that specific error.

RC2 will be up soon and you can try it. Headers changed from a 2 file setup to a one file and RC1 only knows how to read the new one (changed some time ago).
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Tue Aug 24, 2021 4:38 pm

Oh I found the log files, is there any way the view the DEBUG entries in the application?

After closing and reopening the application:

Code: Select all
[08/24 20:31:05]  DEBUG Periodic Search:  Executing!
[08/24 20:31:05]  DEBUG Periodic: 0 Watch List Items found
[08/24 20:31:05]  DEBUG Periodic Search: Waiting for next Period 1800 Secs
[08/24 20:31:15]  DEBUG Periodic Search: Waiting for next Period 1790 Secs
[08/24 20:31:23]  DEBUG NZB File Load Checking
[08/24 20:31:25]  DEBUG Periodic Search: Waiting for next Period 1780 Secs
[08/24 20:31:35]  DEBUG Periodic Search: Waiting for next Period 1770 Secs
[08/24 20:31:43]  DEBUG NZB File Load Checking
[08/24 20:31:45]  DEBUG Periodic Search: Waiting for next Period 1760 Secs
[08/24 20:31:49]  ERROR Downloader:  Socket Exception NEWS SERVER ERROR: Tweaknews Unable to Connect to news server  Tweaknews
[08/24 20:31:55]  DEBUG Periodic Search: Waiting for next Period 1750 Secs
[08/24 20:32:03]  DEBUG NZB File Load Checking
[08/24 20:32:05]  DEBUG Periodic Search: Waiting for next Period 1740 Secs
[08/24 20:32:15]  DEBUG Periodic Search: Waiting for next Period 1730 Secs
[08/24 20:32:23]  DEBUG NZB File Load Checking
[08/24 20:32:25]  DEBUG Periodic Search: Waiting for next Period 1720 Secs
[08/24 20:32:35]  DEBUG Periodic Search: Waiting for next Period 1710 Secs
[08/24 20:32:43]  DEBUG NZB File Load Checking
[08/24 20:32:45]  DEBUG Periodic Search: Waiting for next Period 1700 Secs
[08/24 20:32:55]  DEBUG Periodic Search: Waiting for next Period 1690 Secs
[08/24 20:33:03]  DEBUG NZB File Load Checking
[08/24 20:33:05]  DEBUG Periodic Search: Waiting for next Period 1680 Secs
[08/24 20:33:15]  DEBUG Periodic Search: Waiting for next Period 1670 Secs
[08/24 20:33:23]  DEBUG NZB File Load Checking
[08/24 20:33:25]  DEBUG Periodic Search: Waiting for next Period 1660 Secs
[08/24 20:33:35]  DEBUG Periodic Search: Waiting for next Period 1650 Secs
[08/24 20:33:43]  DEBUG NZB File Load Checking
[08/24 20:33:45]  DEBUG Periodic Search: Waiting for next Period 1640 Secs
[08/24 20:33:55]  DEBUG Periodic Search: Waiting for next Period 1630 Secs
[08/24 20:34:03]  DEBUG NZB File Load Checking
[08/24 20:34:05]  DEBUG Periodic Search: Waiting for next Period 1620 Secs
[08/24 20:34:15]  DEBUG Periodic Search: Waiting for next Period 1610 Secs
[08/24 20:34:23]  DEBUG NZB File Load Checking
[08/24 20:34:25]  DEBUG Periodic Search: Waiting for next Period 1600 Secs
[08/24 20:34:35]  DEBUG Periodic Search: Waiting for next Period 1590 Secs
[08/24 20:34:43]  DEBUG NZB File Load Checking
[08/24 20:34:45]  DEBUG Periodic Search: Waiting for next Period 1580 Secs
[08/24 20:34:55]  DEBUG Periodic Search: Waiting for next Period 1570 Secs
[08/24 20:35:03]  DEBUG NZB File Load Checking
[08/24 20:35:05]  DEBUG Periodic Search: Waiting for next Period 1560 Secs
[08/24 20:35:15]  DEBUG Periodic Search: Waiting for next Period 1550 Secs
[08/24 20:35:23]  DEBUG NZB File Load Checking
[08/24 20:35:25]  DEBUG Periodic Search: Waiting for next Period 1540 Secs
[08/24 20:35:35]  DEBUG Periodic Search: Waiting for next Period 1530 Secs
[08/24 20:35:43]  DEBUG NZB File Load Checking
[08/24 20:35:45]  DEBUG Periodic Search: Waiting for next Period 1520 Secs
[08/24 20:35:55]  DEBUG Periodic Search: Waiting for next Period 1510 Secs
[08/24 20:36:03]  DEBUG NZB File Load Checking
[08/24 20:36:05]  DEBUG Periodic Search: Waiting for next Period 1500 Secs
[08/24 20:36:07]  DEBUG Pausing Downloaders


On attempting to update all headers, had to close and reopen the application as it was stuck paused:

Code: Select all
[08/24 21:12:15]  DEBUG CNNTPDbImportThread -Ninja-nl-000003810968-000004010968.txt
[08/24 21:12:15]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG Periodic Search: Waiting for next Period 1740 Secs
[08/24 21:12:17]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:17]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:18]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts Finished
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:20]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:20]  DEBUG CNNTPDbImportThread -Ninja-nl-000004010968-000004210968.txt
[08/24 21:12:20]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:20]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts Finished
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:22]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:23]  DEBUG CNNTPDbImportThread -Ninja-nl-000004210968-000004410968.txt
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
Last edited by Alex Atkin UK on Tue Aug 24, 2021 5:50 pm, edited 2 times in total.
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Tue Aug 24, 2021 5:18 pm

How many connections are you using to tweak? How many are you allowed to use? If you use too many connections, you'll get that connection error. It could also imply you account has lapsed. Seems unrelated to the original report.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Tue Aug 24, 2021 5:30 pm

Quade wrote:How many connections are you using to tweak? How many are you allowed to use? If you use too many connections, you'll get that connection error. It could also imply you account has lapsed. Seems unrelated to the original report.


Yes the connection error is unrelated as it was a bandwidth based account used for fills only so likely just used up my quota.

It was the pausing downloaders I couldn't understand, as the app showed unpaused and there is no reason for anything to pause that I can tell.

Despite the above in the logs, headers ARE updating but they're going very very very slowly. This didn't happen the last time I downloaded all headers, so something is definitely not right.

Oh and just a reminder, this is under wine-5.8 (Staging) on Fedora.
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Tue Aug 24, 2021 6:08 pm

I'm just going to assume it's old versus new header DB format thing that I already know about. You can try again in rc2.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Tue Aug 24, 2021 8:45 pm

The joy of using Beta versions. :roll: It DOES seem to be downloading now after refreshing all the headers despite the error messages. More than that, its hitting 400Mbit which as you recall I was struggling with before.

I have also noticed 7z support seems to be much more stable lately, a few versions back I was still having some not extract. So were definitely getting there, and considering how long ago I bought this product its amazing you still keep improving it without charging an upgrade fee.

The one thing that puzzles me is how much CPU seems to be used just idling with a group open. In fact a lot of the issues tends to stem from seeing CPU activity but nothing seemingly happening in the download list or logs (in the application). I will have to keep an eye on the log files themselves I think as I do wonder if that CPU thing is a database issue.
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Tue Aug 24, 2021 11:38 pm

The one thing that puzzles me is how much CPU seems to be used just idling with a group open


I sort of suspect you're seeing this error because you forced it to re-download a bunch of headers you already have. Maybe you added another server and let it download headers (which isn't required). So you have a bunch of header blocks back-logged waiting to import. Or maybe you used "Use download age" on the group to triggered new header downloads of headers you already have.

If you look down where it says "Cache X/Y (ZZ) " ZZ represents the number of header blocks awaiting import. Until that hits zero, Newsbin will be banging away at the disk and using CPU.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Wed Aug 25, 2021 4:57 am

I meant in general, I see spikes as high as 50% (though usually more like 25%) of a CPU core, just with Newsbin open and nothing in the queue at all, no group open. It causes the CPU to boost regularly if Newsbin is running and a very lazy/forgetful so Newsbin is usually running.

Its constantly jumping to the top of the list of CPU usage on my box, even though it has a constant Firefox window open showing server stats and ktorrent running.

Should I just blame WINE? :wink:
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Wed Aug 25, 2021 9:16 am

I test it under wine but I don't watch what it's doing that closely. It seems to run as well under Wine as under windows. I see download speeds that are the same as my Windows box. Someone reported they were seeing stuttering while moving the mouse. Makes me wonder if Newsbin is too dependent on having multiple cores at the ready? So it stutters on single or 2 core boxes. I'll see what it looks like it's doing when it's idle. You do apparently have periodic search firing off in the background.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Alex Atkin UK » Wed Aug 25, 2021 6:32 pm

Yeah I don't understand what that periodic search is about, I have nothing configured to do that.
User avatar
Alex Atkin UK
Occasional Contributor
Occasional Contributor
 
Posts: 46
Joined: Wed Sep 27, 2006 10:22 pm

Registered Newsbin User since: 06/12/06

Re: ReadFullDatabase: Invalid packet Format: Storage.db3

Postby Quade » Thu Aug 26, 2021 11:20 am

The periodic search thread is running waiting for a periodic search to be defined. I guess it's waking up, sees it's got nothing to do then goes back to sleep for awhile.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44365
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to Newsbin Version 6 Beta Support

Who is online

Users browsing this forum: Bing [Bot] and 3 guests