«

»

Jun 22

Enterprise vs….not

I have a cousin. Very well-to-do man, owns a company that does something with storing and providing stock data to other users. I don’t pretent do know the details of the business, but what I do know is that it’s storage and bandwidth intensive.

He’s building his infrastructure on a home-grown storage solution – Tyan motherboards, Areca SATA controllers, infiniband back-end, etc. Probably screaming fast but I don’t have any hard-numbers on what kind of performance he’s getting.

Now I understand people like me not wanting to invest a quarter-mil on “enterprise-class” storage, but why would someone who’se complete and total livelihood depends on their storage infrastructure rely on an open-source, unsupported architecture?

One of the things you get with the Symmetrix is the 24×7 monitored support. One of the stories I tell people was about my first experience with EMC. When I worked at Intuit I was on the graveyeard operations shift. (The grunt shift, that most of us have been subjected to at least once in their lives) About 4am one morning I got a call from EMC saying that a hard-disk in our old Symmetrix-3 array had failed, and that the tech would be onsite in about 20 minutes (I guess they gave him the head-start) to replace it. I asked them if there was anything I needed to do and they told me that it was transparent and that the hosts wouldn’t notice the difference.

I was in love.

People ask what the “Enterprise” money gets you, and that’s it. You get the security of knowing that it doesn’t matter when, where, or how a failure happens, they are on top of it and have it dealt with before you even know the problem exists most of the time.

My second great EMC story – I was working at the Library of Congress on a tech-refresh, they had four Symm4 and 2 Symm5 arrays that were being upgraded to a pair of DMX’s. About two weeks before we were to have decomissioned one of the Symm4’s, it started experiencing problems. It seemd that 2 of the three power supplies had failed. The Symm4 was at least 7 years old at the time, and was designed for n+1 redundancy.

Even with two-thirds of it’s power gone, the thing kept running for almost 7 hours, tapping the internal batteries as needed. (Unfortunately it took only slightly longer to locate a replacement power-supply for such an antiquated peice of hardware, but at least it gave us the chance to gracefully power-down the last remaining hosts and gracefully power-off the Symm.

I’ve heard other stories, one in particular of a Symm in California that, after an earthquake, ran laying on it’s side until the hardware could be replaced and the data-migrated off it. (But having no first-hand knowledge of this, I will consider this an urban ledgend until someone who witnessed it tells me it really happened)

*THAT* is what you get for enterprise money.

Of course another relative from the same branch of the family is the one who told me “I have RAID, why do I need backups?”

4 comments

Skip to comment form

  1. Anyone who’s ever had a presentation from an EMC salesman has seen the piccie of the symmetrix that fell through the raised floor and kept running (think it might have been at Nestle)

    My favourite EMC story…

    An engineer came to install some new FAs in one of our 8730s. He was following the “script” (back then the engineers had very rigid procedures they followed for everything – don’t know if thats still the case), and ran into problems… the devices wouldn’t install correctly. Next day the local regional specialist was onsite reviewing the problem with the engineer… the eventual upshot was – we can’t install the FAs and we don’t know why… so what did EMC do? Well 2 weeks later they rolled a whole new 8730 into the data centre and SRDF’d everything across from the old one! Our project was held up a couple of weeks, and we had to take some short outages on some of our production systems, but the point was that they did what was required to get us around the issue.

  2. Jesse

    They definitely do break their back when they need to. When I did a tech-refresh / move last year and some of the Symms were old Symm 4.0 models. When it was discovered that the RA boards that were shipped weren’t supported by the particular model, they moved heaven and earth to get the correct boards overnighted to New Jersey.

  3. Nakarti

    He doesn’t need backups because he doesn’t know what its like to be missing all his data. I used to be the same way (I am a server guy after all.)

    Tell him two hard drives failed because of a short circuit and were replaced, he has no data. For one day. He will ask what you’d use for backups.

    1. Jesse

      Here’s the type of anal-retentive I am.

      My farm runs in vmware.

      I use VCB (don’t judge, I’ve got it working like clockwork and backing up 15 VM’s in about 90 minutes) to take a full copy of every VM, every night. I then *ALSO* use a filesystem backup to capture the data on the drives so that I can protect customers against the ‘oops I deleted x’ Those backups are replicated offsite. (The amount of bandwidth it would take to replicate the block-image-backups every night is too expensive, sadly)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>