Obfuscated/Differently named par2 files causing failures

General NewsBin and Usenet related discussions. Make recommendations for additional specific discussion groups.

Moderators: Quade, dexter

Obfuscated/Differently named par2 files causing failures

Postby shirfenlag » Sun Feb 25, 2018 2:16 am

There's an issue with more recent SpotWeb downloads that causes download failures in Newsbin.

Files are named stuff like
rB44jdmakdoiEnfDJAned39MDN.DDBBJK21.par2
rB44jdmakdoiEnfDJAned39MDN.r00
...
rB44jdmakdoiEnfDJAned39MDN.rar

The par2 is actually looking for differently named files like burgertime2.NES.fanmade.full.r00 and because of this it fails

If I rename the par2 to rB44jdmakdoiEnfDJAned39MDN.par2 and open it with quickpar it will rename the files appropriately even though it thinks it's missing a block.

I felt like messing around with golang since I haven't used it for anything so I wrote a quick tool that verifies the integrity of these files and extracts them then deletes the files. It seems to work, but it's crap code since I got pretty lazy halfway through.

If anyone feels like messing with it, you need 7-zip and to have the 7z.exe in your environment path. You have to have go installed. Once you've got that, just name this file something like obfucsate_extractor.go and using powershell or cmd.exe navigate to the directory where you put obfuscate_extractor.go then type "go build". It'll build an executable and then you put that in the directory where all your annoying obfuscated files are. I recommend running the exe from a command line in case there are any failures, otherwise it'll just silently quit.

WARNING: This deletes files so there's a potential for something to go wrong. No implied warranty or support, use at your own risk. MIT/BSD style permissive license.

Code: Select all
package main

import (
   "bytes"
   "fmt"
   "os"
   "os/exec"
   "path/filepath"
   "regexp"
)

// @TODO - not finished
func runParForFile(fileName string) {
   cmd := "phpar2"
   args := []string{"verify", "testfile1", "testfile2"}
   if err := exec.Command(cmd, args...).Run(); err != nil {
      fmt.Fprintln(os.Stderr, err)
      os.Exit(1)
   }
}

// only needed for future - if we automate pulling from nzbindex.nl, phpar2 needs full list of obfuscated files since they dont match what it expects
func getRarFilesMatching(fullDir string, localDir string, fileName string) []string {
   runes := []rune(fileName)
   fileNameNoEx := string(runes[:len(runes)-5])

   rars := make([]string, 1)
   r, err2 := regexp.Compile(fileNameNoEx + `.[a-z0-9][a-z0-9][a-z0-9]$`)
   if err2 != nil {
      panic(err2)
   }

   err := filepath.Walk(fullDir, func(path string, info os.FileInfo, err error) error {
      if err != nil {
         fmt.Printf("halt execution, rar file walk issue")
         panic(err)
      }
      if info.IsDir() && info.Name() != localDir {
         return filepath.SkipDir
      }
      if r.MatchString(info.Name()) && info.Name() != fileName {
         rars = append(rars, info.Name())
      }
      return nil
   })
   if err != nil {
      fmt.Printf("error walking the path %q: %v\n", fullDir, err)
   }

   return rars
}

func rarFileIsVerified(fullDir string, fileName string) bool {
   cmd := "7z"
   args := []string{"t", fullDir + "\\" + fileName}

   unrarCommand := exec.Command(cmd, args...)
   var outb, errb bytes.Buffer
   unrarCommand.Stdout = &outb
   unrarCommand.Stderr = &errb
   if err := unrarCommand.Run(); err != nil {
      fmt.Fprintln(os.Stderr, err)
      os.Exit(1)
   }

   r, err2 := regexp.Compile(`Everything is Ok`)
   if err2 != nil {
      panic(err2)
   }
   if r.MatchString(outb.String()) {
      return true
   }
   return false
}

func extractRar(fullDir string, fileName string) {
   cmd := "7z"
   args := []string{"x", fullDir + "\\" + fileName}

   unrarCommand := exec.Command(cmd, args...)
   if err := unrarCommand.Run(); err != nil {
      fmt.Fprintln(os.Stderr, err)
      os.Exit(1)
   }
}

func deleteFilesMatchingParName(fullDir string, localDir string, fileName string) {
   runes := []rune(fileName)
   fileNameNoEx := string(runes[:len(runes)-5])

   r, err2 := regexp.Compile(fileNameNoEx + `.([a-z0-9][a-z0-9][a-z0-9]$|[a-z0-9][a-z0-9][a-z0-9][a-z0-9]$)`)
   if err2 != nil {
      panic(err2)
   }

   err := filepath.Walk(fullDir, func(path string, info os.FileInfo, err error) error {
      if err != nil {
         fmt.Printf("halt execution, delete file walk issue")
         panic(err)
      }
      if info.IsDir() && info.Name() != localDir {
         return filepath.SkipDir
      }
      if r.MatchString(info.Name()) {
         err := os.Remove(path)
         if err != nil {
            fmt.Println(err)
            return nil
         }
      }
      return nil
   })
   if err != nil {
      fmt.Printf("error walking the path %q: %v\n", fullDir, err)
   }
}

func main() {
   ex, errEx := os.Executable()
   if errEx != nil {
      panic(errEx)
   }
   dir := filepath.Dir(ex)
   localDir := filepath.Base(dir)

   r, err2 := regexp.Compile(`[a-zA-Z0-9]+\.[a-zA-Z0-9]+\.par2`)
   if err2 != nil {
      panic(err2)
   }

   re := regexp.MustCompile(`[a-zA-Z0-9]+\.par2`)

   err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
      if err != nil {
         // ignore, going to get a bunch of this since we're deleting stuff
         return nil
      }
      if info.IsDir() && info.Name() != localDir {
         return filepath.SkipDir
      }
      if r.MatchString(info.Name()) {
         newFileName := re.ReplaceAllString(info.Name(), "par2")
         fmt.Println(newFileName)

         err3 := os.Rename(dir+"\\"+info.Name(), dir+"\\"+newFileName)

         if err3 != nil {
            fmt.Println(err3)
            panic(err3)
         }

         parFileRunes := []rune(newFileName)
         fileNameNoEx := string(parFileRunes[:len(parFileRunes)-4])
         fmt.Println("Verifying " + fileNameNoEx + "rar ...")

         if rarFileIsVerified(dir, fileNameNoEx+"rar") {
            fmt.Println("Extracting " + fileNameNoEx + "rar ...")
            extractRar(dir, fileNameNoEx+"rar")
            fmt.Println("Cleaning up " + fileNameNoEx + "rar files.")
            deleteFilesMatchingParName(dir, localDir, newFileName)
         } else {
            fmt.Printf(fileNameNoEx + "rar file incomplete.  Download more par2s in the future")
         }

      }
      return nil
   })

   if err != nil {
      fmt.Printf("error walking the path %q: %v\n", dir, err)
   }
}
shirfenlag
n00b
n00b
 
Posts: 2
Joined: Sun Feb 25, 2018 1:53 am

Registered Newsbin User since: 12/17/11

Re: Obfuscated/Differently named par2 files causing failures

Postby Quade » Sun Feb 25, 2018 7:59 am

What version are you using? Newsbin shouldn't really care what the PAR filenames are called in order to process them. Is it the case where Newsbin seems to handle them but the repair itself fails?

If the PAR block counts seem to count up then Newsbin is handling the PARS. If the subsequent repair fails, this is something I can look into. I already rename the files so, renaming the PARS as part of repair wouldn't be too hard.

If you have any NZB's that demonstrate this, if you email them to ts@newsbin.com, I'd be happy to look into it.

Edit: Re-reading your post. Newsbin should already be renaming those file to the filenames used inside the PAR. This is the first thing Newsbin does when it starts to repair the files.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43149
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Obfuscated/Differently named par2 files causing failures

Postby shirfenlag » Sun Feb 25, 2018 2:45 pm

Quade wrote:What version are you using? Newsbin shouldn't really care what the PAR filenames are called in order to process them. Is it the case where Newsbin seems to handle them but the repair itself fails?

If the PAR block counts seem to count up then Newsbin is handling the PARS. If the subsequent repair fails, this is something I can look into. I already rename the files so, renaming the PARS as part of repair wouldn't be too hard.

If you have any NZB's that demonstrate this, if you email them to ts@newsbin.com, I'd be happy to look into it.

Edit: Re-reading your post. Newsbin should already be renaming those file to the filenames used inside the PAR. This is the first thing Newsbin does when it starts to repair the files.


6.80 RC3, I'll upgrade to RC5.

My mistake, I missed describing a key part in this. The NZB files that have this problem do NOT inclue all of the PARs. There's only the initial PAR2. The other PARs with recovery blocks are posted but they're not part of the NZB. The initial nzb is always missing 1 block (the nfo) so they can't be reconstructed.

I realize that makes the problem quite a bit trickier.
shirfenlag
n00b
n00b
 
Posts: 2
Joined: Sun Feb 25, 2018 1:53 am

Registered Newsbin User since: 12/17/11

Re: Obfuscated/Differently named par2 files causing failures

Postby Quade » Sun Feb 25, 2018 4:27 pm

Newsbin used to rename as the files were downloaded but this made it difficult to restart the file downloads after a restart. I might have to go back to that if this becomes pretty common. I can rename the files from the small PAR2.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 43149
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest