TestDisk: *the* cross-platform data recovery tool

A few months ago, I lost both disks at the same time in the RAID 1 mirror running on my QNAP TS-209!

I get the feeling hardware manufacturers that extol the virtues of RAID would like you to believe that this sort of event is extremely uncommon. However, judging by how hot the disks were running (I couldn't hold them for more than a few seconds comfortably) the likelihood of both failing was probably quite high. The QNAP's cooling fan obviously wasn't able to provide sufficient airflow over the disks in question (a couple of Hiatchi Deskstar 1Tb 7K1000's) which probably had a lot to do with the failures and data corruption. In hindsight, I'm amazed they lasted as long as they did.

I had been running an rsync job between the QNAP and my main PC fairly frequently. Just prior to the disk fatalities I'd been running short of disk space on my main machine and hadn't run the sync job for about six weeks. Annoyingly, when it failed there were still some files on the QNAP that I wanted.

At the time I had a brief search around for some free data recovery tools using all of the keyword variations you might expect to pull up something useful via a search engine. I tried a few I came across and was left distinctly unimpressed by what was on offer. So, I decided to shelve the disks until I could find the time and a suitable tool to try and get at least some of the data back.

Then, last week, I came across Victor Stinner's excellent library, hachoir. On that project's wiki I noticed the rather intriguing entry simply titled 'forensics' which in turn led me to the TestDisk homepage.

What a find!

I haven't been more impressed with a bit of software since someone introduced me to VLC several years ago. TestDisk runs on an impressive array of platforms (Linux, BSD, Solaris, Mac OS X, Windows) and supports the drive formats you'd expect to come across - FAT12/16/32, NTFS, EXT2/3, ReiserFS etc. I was running TestDisk on Windows Vista 64-bit retrieving data from an Ext3 partition.

Even better, it's free (GPL licensed). I think you'd be hardpressed to find a better free tool. Even paid for tools would likely struggle to match TestDisk's impressive list of features. I'm frankly astounded at how hard it is to find such a good tool via a search engine unless you know it by name (which is ridiculous).

This tale does have a happy ending. I managed to retrieve about 80% of the files for what I believe is only some partition corruption. I've yet to ascertain if there is any physical damage to the disks. TestDisk was able to dig up the partition table that the QNAP managed to trash somehow and even allowed me to get back copies of files that had previous been deleted when the disks were still functioning. Truly amazing. I'm also hoping there is some life left in my Hitachi Deskstar disks. Mind you, I intend to give them a thorough soak test before trusting them with any serious data.

I really recommend checking out the TestDisk web page and downloading it if you have spare 5 mins.

A big thank you to Christophe Grenier for all his efforts in creating a truly great piece of software!

netaddr 0.6 released

For those who are interested netaddr 0.6 has just been released.

A couple of nights ago, I hit a very important personal milestone.

One of the killer features I've wanted since starting on netaddr back in January of 2008 is now in the bag! It ended up being one of the more tricky problems I've had to deal with but has been worth the effort. It will also prove extremely useful in my day-to-day work.

Essentially, you can now take an arbitrary start and end IP address (versions 4 and 6), and formulate a list of intervening CIDRs that exactly bridge the two with no overlaps.

A lot of infrastructure code had to be built before I was at a point where this could be implemented elegantly. The wait is (finally) over. It has also allowed me to kill some really ugly code based on math.frexp() which didn't work for IPv6 as it died somewhere around 2^54 due to rounding issues.

So, here is the result of the (worst case) scenario in the IPv4 address space (IPv6 would take up too much screen space) :-

In [1]: from netaddr import *

In [2]: cidrs = IPRange('0.0.0.1', '255.255.255.254').cidrs()

In [3]: print "\n".join(["%-18s: %-15s -> %-15s" % (str(c), c[0], c[-1]) for c in cidrs])
0.0.0.1/32        : 0.0.0.1         -> 0.0.0.1
0.0.0.2/31        : 0.0.0.2         -> 0.0.0.3
0.0.0.4/30        : 0.0.0.4         -> 0.0.0.7
0.0.0.8/29        : 0.0.0.8         -> 0.0.0.15
0.0.0.16/28       : 0.0.0.16        -> 0.0.0.31
0.0.0.32/27       : 0.0.0.32        -> 0.0.0.63
0.0.0.64/26       : 0.0.0.64        -> 0.0.0.127
0.0.0.128/25      : 0.0.0.128       -> 0.0.0.255
0.0.1.0/24        : 0.0.1.0         -> 0.0.1.255
0.0.2.0/23        : 0.0.2.0         -> 0.0.3.255
0.0.4.0/22        : 0.0.4.0         -> 0.0.7.255
0.0.8.0/21        : 0.0.8.0         -> 0.0.15.255
0.0.16.0/20       : 0.0.16.0        -> 0.0.31.255
0.0.32.0/19       : 0.0.32.0        -> 0.0.63.255
0.0.64.0/18       : 0.0.64.0        -> 0.0.127.255
0.0.128.0/17      : 0.0.128.0       -> 0.0.255.255
0.1.0.0/16        : 0.1.0.0         -> 0.1.255.255
0.2.0.0/15        : 0.2.0.0         -> 0.3.255.255
0.4.0.0/14        : 0.4.0.0         -> 0.7.255.255
0.8.0.0/13        : 0.8.0.0         -> 0.15.255.255
0.16.0.0/12       : 0.16.0.0        -> 0.31.255.255
0.32.0.0/11       : 0.32.0.0        -> 0.63.255.255
0.64.0.0/10       : 0.64.0.0        -> 0.127.255.255
0.128.0.0/9       : 0.128.0.0       -> 0.255.255.255
1.0.0.0/8         : 1.0.0.0         -> 1.255.255.255
2.0.0.0/7         : 2.0.0.0         -> 3.255.255.255
4.0.0.0/6         : 4.0.0.0         -> 7.255.255.255
8.0.0.0/5         : 8.0.0.0         -> 15.255.255.255
16.0.0.0/4        : 16.0.0.0        -> 31.255.255.255
32.0.0.0/3        : 32.0.0.0        -> 63.255.255.255
64.0.0.0/2        : 64.0.0.0        -> 127.255.255.255
128.0.0.0/2       : 128.0.0.0       -> 191.255.255.255
192.0.0.0/3       : 192.0.0.0       -> 223.255.255.255
224.0.0.0/4       : 224.0.0.0       -> 239.255.255.255
240.0.0.0/5       : 240.0.0.0       -> 247.255.255.255
248.0.0.0/6       : 248.0.0.0       -> 251.255.255.255
252.0.0.0/7       : 252.0.0.0       -> 253.255.255.255
254.0.0.0/8       : 254.0.0.0       -> 254.255.255.255
255.0.0.0/9       : 255.0.0.0       -> 255.127.255.255
255.128.0.0/10    : 255.128.0.0     -> 255.191.255.255
255.192.0.0/11    : 255.192.0.0     -> 255.223.255.255
255.224.0.0/12    : 255.224.0.0     -> 255.239.255.255
255.240.0.0/13    : 255.240.0.0     -> 255.247.255.255
255.248.0.0/14    : 255.248.0.0     -> 255.251.255.255
255.252.0.0/15    : 255.252.0.0     -> 255.253.255.255
255.254.0.0/16    : 255.254.0.0     -> 255.254.255.255
255.255.0.0/17    : 255.255.0.0     -> 255.255.127.255
255.255.128.0/18  : 255.255.128.0   -> 255.255.191.255
255.255.192.0/19  : 255.255.192.0   -> 255.255.223.255
255.255.224.0/20  : 255.255.224.0   -> 255.255.239.255
255.255.240.0/21  : 255.255.240.0   -> 255.255.247.255
255.255.248.0/22  : 255.255.248.0   -> 255.255.251.255
255.255.252.0/23  : 255.255.252.0   -> 255.255.253.255
255.255.254.0/24  : 255.255.254.0   -> 255.255.254.255
255.255.255.0/25  : 255.255.255.0   -> 255.255.255.127
255.255.255.128/26: 255.255.255.128 -> 255.255.255.191
255.255.255.192/27: 255.255.255.192 -> 255.255.255.223
255.255.255.224/28: 255.255.255.224 -> 255.255.255.239
255.255.255.240/29: 255.255.255.240 -> 255.255.255.247
255.255.255.248/30: 255.255.255.248 -> 255.255.255.251
255.255.255.252/31: 255.255.255.252 -> 255.255.255.253
255.255.255.254/32: 255.255.255.254 -> 255.255.255.254

I am now officially proud of this project ;-)

"Not Invented Here" is a powerful force

A link to this 2006 blog post came up on my feeds recently :-

http://blog.pythonisito.com/2006/01/three-reasons-why-you-shouldnt-write.html

It's a great article (with excellent commentary) on reasons for (and against) starting your own code project versus joining an existing one. Things are certainly not as clear cut as they might appear at first. It uses the umpteenth Python web frameworks spawned (prior to the arrival of Django) as a basis for discussion but is good general advice for those thinking about contributing to the pool of open source software.

Certainly food for thought.

Sony Customer Service; second to none?

Recently, I got caught in the whole Sony Vaio TZ fiasco and had to send off my machine for inspection and repair. While annoying that I should be subject to this at all, my experience with the resolution to the problem is actually worth blogging about ;-)

I am extremely impressed with Sony's approach to customer service. They did (almost) everything right :-

• the 3 working day turnaround (they could easily have made me wait weeks)
  
• an apologetic letter enclosed with the returned hardware
• complimentary 2Gb Sony Microvault memory stick (nice)
  
• new Vaio branded screen cleaning cloth (very nice as the one that came with it has seen better days)
• friendly, helpful and local call centre staff on hand when I called up to check on progress with the repair
  
• and last but not least, a nice clean machine and screen with problems resolved (laptops do get gunky after a while)
  

The only negative thing about the whole process was having to read about the problem in the press rather than being contacted directly by Sony themselves, even though they have my contact details that were registered when I bought the product. OK, it's a voluntary programme so I guess I can let this last point slide.

As far as I am concerned, this is the benchmark by which other companies should do business with their customers. OK, so the hardware isn't cheap, but I think this proves that you do get what you pay for (at least some of the time). It also shows that Sony actually care about and have respect for their customers feelings and opinions.

All this of course, means that Sony will keep their existing customers with the added bonus of the word of mouth effect working in their favour. It is so refreshing to see a company of this size still realising the importantance of customer satisfaction and having a duty of care *after* the sale, rather than stomping all over them or even worse, just ignoring them altogether.

Large unnamed corporates in competing hardware businesses would do well to take note of Sony's impressive example.

Time for a real blog

I've been hobbling along with an extremely basic Google Pages 'blog' (I use the term lightly) for a little while now.

Sadly, it doesn't have any of the features that I'll probably find rather useful going forward. Time for a change! So here I am, with a proper blog this time. The back-dated post feature on Blogger is great!

netaddr release 0.4 is available

All the features I wanted are now in and tested, so I've packaged them up and uploaded the various packages to PyPI and the hosting site download page.

A decent number of changes have made their way into this release. Please see the CHANGELOG for full details.

netaddr release 0.3.1

I ran pylint over my project and fixed a several errors and warnings. I decided not to get upset when it told me off for having module files that were greater than 1000 lines! So what?"

I also squashed my first very small officially reported bug :-)

Build It, and They Will Come"

Well, I guess we'll have to put that theory to the test. I've now created a project page on PyPI, sent an announcement to comp.lang.python.announce and created a NetAddr Google Group to discuss all things related to this shiny new library.