Friday (23-August-2013) — New Jersey

Technology – Never Enough Storage Space.

Although the capacity of hard drives has increased, and the cost/(MB, GB, TB) has gone down, technology lets us collect more and more data. Each image I take with the 36 MP Nikon D800 and save in RAW format takes up ~40 MB of storage space. I’m the type of person that saves every image (other than the ones that are totally white or black, totally out of focus). I like to go back and review the images to learn from my mistakes, and also get ideas of things that I want to do better when returning to a site I have photographed before. As the image processing software has improved, I find that I can reprocess images to get something good/interesting out of a previously rejected/not-interesting image. (Just ask the photographer that had images of Monica with Bill).

Well, this year with all of my travel I hit the wall again with my data storage. I started off a few years ago with one (Lenovo ThinkPad W700ds) then a 2nd (Lenovo ThinkPad W510) computer at home that I use to process images, each with a collection of disks with images. I also keep a 3rd set of the original images offsite. I first used a collection of Western Digital MyBook 2-disk arrays – but ran out of USB, Firewire, and eSata ports on my computers. I then upgraded to a Drobo 800i (8 x 2 TB disk array) that was connected to my primary computer via gigabyte Ethernet. I finally had all of my images in one place. Life was good until I needed to expand the storage space. To upgrade to a (8 x 3 TB disk array), I had to replace the disks in the Drobo one at a time and then wait 2-3 days for until I could replace the next drive. The same thing happened when the Drobo indicated that a disk had failed – and unfortunately the log files on the Drobo are encrypted so you can’t tell what caused the problem. One of the “failed” disks that I replaced, I ran a full analysis using the Seagate SeaTools – and found no errors/problems with the disk (I ultimately repurposed the hard drive and have been using it for a year with no problems). I had also been reading other reports on the net about Drobo’s that became bricks, and issues with lack of support after the warranty ended. I decided to investigate other large capacity storage systems, and ultimately decided to try out a Synology DS1812+ system initially with the 8 x 2 TB drives that I just removed from the Drobo 800i.

The Synology disk arrays runs on Linux using a program they call DSM. Much more flexible than the Drobo, and a lot more real-time diagnostic information about the health of the system. The other neat thing about the Synology DS1812+ is that it has two eSata ports that you can add Synology DX513 (five disk arrays). As soon as the Seagate 4TB drives became available at a reasonable cost I added a Synology DX513 with 5 x 4TB Seagate hard drives. You plug it in, and all of a sudden you have another 14.3 TB (one disk fault tolerance) of storage space. This was working well through the beginning of this year – my primary image processing computer (Lenovo W700-ds) using the older Drobo 800i and the secondary image processing computer (Lenovo W510) using the Synology DS1812+ and DX513. I would keep the two disk arrays synchronized using sneaker-net and FolderMatch. (Future project – home server and networking).

After the Semester at Sea Spring 2013 Enrichment Voyage, and my recent trip to Iceland I was getting close to running out of space on my primary system. Then I got a warning light on the Drobo that another drive failed. I ordered a new Seagate 4TB drive and when it arrived (Amazon overnight is great) it took 6 days before I had all green lights on the Drobo again. The other problem that I had was that the Drobo array was not recognized after rebooting the computer. I thought I had a Drobo brick. If I was really patient, the computer would finally see the Drobo array after 30-60 minutes. I can see where others with this problem were not patient enough, gave up on Drobo – and published there rants on the internet. I was beyond the warranty and didn’t want to sit on the phone and pay for support. Enough was enough and I decided it was time to give up on the Drobo, and ordered the newer Synology DS1813+ which I populated with 8 x 4TB Seagate NAS drives and a DX513 with 4 x 4TB Seagate NAS drives. Each array is set up with one disk fault tolerance, and gives me a total of 25.3 + 14.3 = 39.6 TB of capacity for my primary image processing computer. This should get me through another year (or two or three) unless the Nikon D4x comes out at 54 MP 🙂 The Drobo 800i will probably become part of my offsite data storage.

Saturday (10-November-2012) — New Jersey

Technology Update: Drobo Good Bad and Ugly.

I’ve had a love/hate relationship with my Drobo disk arrays ever since I got them. I have two Drobo’s, an eight disk Dobro B800i (configured with eight 2 TB drives) as my primary working array and a 5 disk Drobo S (configured with five 3 TB drives) as an offline backup. Although some glitches when setting them up, when working they are fast and have been happy. Well, mostly happy other than the >10 days it took to add more drives: see my 13-November-2011 post.

When I got back from the SAS 2012 Fall Semester voyage, I realized that my storage space was getting tight, and after the Gone to See America 2012 Road Trip/Nikonians ANPAT 12 Trip I was running out of space. The B800i drive had a message that I was >85% full, and the one of the hard drives had a yellow light indicating that it should be replaced with a higher capacity hard drive.

One of the reasons that I bought the Drobo disk arrays was that I was supposed to be able to a hot swap upgrade if I a drive went bad, or I needed to increase drive space on the disk-array. Well, the first bit of bad news I found out that if you are using a fully populated Drobo array and all of the drives are the same size, you need to upgrade with two higher capacity drives before you get access to more space. The second bit of bad news was that I got back to New Jersey just in time for Hurricane Sandy.

Since I had configured my Drobo in the default (single drive failure mode) this meant that I could only add the higher capacity drives one at a time. And each time it takes 24+ hours for the data protection to complete before adding the next drive. This gets to be a long, long process especially during Hurricane Sandy when the power kept going down. In my case the power was on and off four times over 10 days. A UPS works for a couple of hours, but not for days. As such, I ended up having to restart the data protection process several times. This was all on the Drobo B800i that was initially set up with eight 2 TB hard drives, that I wanted to upgrade to eight 3 TB hard drives.

The upgrade process on the Drobo S system populated with five 3 TB drives was more complex. In this case the system was also configured in the default mode (single drive failure). I was at the point that there was less than 10% free space, and the top drive was red – indicating that it needed to be upgraded. The availability of 4 TB hard drives is limited, but I was finally able to find some. Again, not well documented by Drobo, but you need to replace two or more drives to get more space and get back to a normal green operating space. When I replaced the “red” 3 TB drive with a 4 TB drive you have to be very patient. First it takes several minutes to indicate that the new 4 TB drive was installed, and then it takes almost half an hour to go from red to flashing green/yellow and indicate that data protection is in progress. Really bad news – 56 hours to complete data protection.

When the data completion was finally complete (less than 56 hours, but more than 36 hours), the system flashed red on the same 4 TB drive that I just replaced – indicating that the drive that I needed to replace was the drive that I just replaced. OUCH – that was the drive that I installed and waited 36+ hours for the data protection process to complete. Was the new drive bad – or was there a software/hardware problem with the Drobo S unit. Well, I took a chance (since I have 2 other on & off site copies of the data) and the unit did indicate that the data protection process was complete. I did a hot swap of the #2 hard drive with a new 4 TB drive. It took several minutes to 1) get an indication that the drive was removed; 2) many more minutes to indicate that a new 4 TB drive was installed; and after waiting for 20 minutes that the data protection process was underway (all drives flashing green and yellow). This time it indicated 69 hours to complete (dropped to 36 hours in a couple of hours). We will see in another day or two if this works.

I have been concerned about the Drobo arrays since the posts by Scott Kelby in June. As a noted Photographer, he had four Drobo disk array fail and ultimately decided on another solution for storage of his images. Right after his posts, I started getting e-mail messages from Drobo that I needed to purchase an extended warranty. Do I stay with Drobo or find a less proprietary solution? With the amount of time it has been taking me to upgrade my storage over the last couple of weeks I wonder if it is time to evaluate a Synology NAS disk array.

Sunday (04-November-2012) — New Jersey

Hurricane Sandy Aftermath: Day 6 — Power Restored.

I am happy! Power to the house was restored yesterday. When I woke up this morning, power was still on and the house was warm. With the time change, the sun was just coming up through my bedroom window rather than getting up in the dark. I’m thinking of having a Sunday roast chicken dinner with lots of garlic.

With the power restored, I restarted the backup of my server and image collection. I was finally able to start reviewing and process images from the New Mexico trip. One teaser image is of Noah’s Ark from the Albuquerque Balloon Fiesta.

The Drobo disk array where I store my images started beeping. The error message told me the storage space was nearly full and that I needed to add some bigger hard drives. We will see how long this takes, and hope that the power stays on for the upgrade to process. I followed the instructions and did a hot swap of the 2 TB drive with a yellow light with a 3 TB drive. After many, many minutes I finally got a message that the new hard drive was recognized — but that the Drobo disk array will require ~38 hours for the data protection process to complete. In the mean time I am making other backups. I am looking forward to the Seagate 4 TB hard drives becoming available at a reasonable cost. With these I could double the size of my image collection.

Noah's Ark. Albuquerque 2012 Balloon Fiesta Final Day. Nikonians ANPAT 12. Image taken with a Nikon D4 and 70-200 mm f/2.8 VRII lens (ISO 100, 155 mm, f/7.1, 1/200 sec). (David J Mathre)
Noah’s Ark. Albuquerque 2012 Balloon Fiesta Final Day. Image taken with a Nikon D4 camera and 70-200 mm f/2.8 VRII lens (ISO 100, 155 mm, f/7.1, 1/200 sec). (David J Mathre)

Sunday (13-November-2011) — New Jersey

Technology Update: Drobo Storage Device Restored!

I’d previously posted that it was taking a long time for my Drobo B800i storage device to update after adding two additional 3 TB drives. The update never completed (after 12+ days), and appeared to need to restart after the multiple power failure after the early snow storm (and ultimately 42 hours of outage). I finally gave up, and decided to reset the device and restore my image database from backups. For the most part this went well, just took time. However, one of my primary backup devices, a Western Digital 2 TB MyBook devices failed. During the restore process, the process stalled. At first, I thought it was an overheating problem, so let the device cool before restarting. This was repeated several times. I even used the Western Digital “Data Lifeguard” program to check the device – and the device passed with no errors. Ultimately, I got a message that I needed to format the device before I could use it. OUCH!!!

OK — no problem, I have other backups. I went to the other backup that I keep in the house, the original RAW files stored on Western Digital Passport drives. These were my original external storage devices (160 GB, 240 GB, 500 GB, 750 GB, 1 TB) that I used since I started getting into digital photography. Unfortunately, one of the drives containing the data lost on the MyBook device was not recognized. STARTING TO PANIC!! — this included images from ANPAT 9 including image from a once in a lifetime trip to Alaska.

I guess that I am paranoid about data storage. I keep three, and sometimes four backups of key data, and some of the data is stored off-site (away from home). The third backup, I keep at work. I went and picked it up this weekend (3 hour round trip) and I was able to restore the data to my primary Drobo device. I am still running a full comparison (using Program Match — 32 bit CRC mode) since some of the earlier data restored from the failed drive may have been corrupted.

The thing that I did loose was any current metadata, or processing data for the images since I was going back to the original RAW files saved from the camera.

Bottom Line — One backup is not enough!! Primary media can fail, backup media can fail (hard drives fail). Primary media can be lost (fire, theft, other disasters etc). Technology moves forward, and old backup devices become obsolete. I am currently dealing with nearly 500 K (7 TB) images, and need to future proof my storage and back-up work-flow. Need to keep all of this in focus when devising a backup storage protocol.

I plan to get back to reviewing and processing images soon, so I can share images again!!


Day 5 and Counting

My Drobo disk-array is rebuilding after adding a new drive. Don’t know how long it will take. New images (including a meteor caught last night with two cameras) to follow. The down side to large storage solutions such as Drobo is the time required for backup and restore. Not sure if this will be hours or days!!!!

I did post some images on Google+ for the following themes: #SongBirdSaturday  #RuralSaturday  #SunsetSaturday  #SepiaSaturday  #SaturdayNightLight  #Caturday

For a list of all of the Daily Google+ Photography Themes

Also, check out my post from 5 years ago today of a Turkey Vulture soaring over New Jersey: 22-October-2006