Quick note…

Transitioning “dial-home” away from modem has got to be the crappiest idea there is.

When the network is down, remote support is down.

I’m sitting in a datacenter that was powered off this weekend (don’t ask) On recovering, one Symm failed to IML.

The network is still down.

I can’t even START troubleshooting the Symm until it’s back up.

Start of a craptastic weekend in my book.

Lesson Learned – keep the modems JUST IN CASE.

8 comments

Skip to comment form

    • Phil on November 20, 2010 at 12:16 pm
    • Reply

    I feel your pain. We have kept ours in for EXACTLY that reason. Modems are SO last century but thank feck they are….

    Hope everything goes well for you.

  1. We came out of it.

    Two servers down. (As in gone, won’t even post)

    14 hours to replace the SP and an FA card in one Symm.

    And a whopping total of 5 failed drives. (Luckily enough not in the same raid groups)

    Not bad in the grand scheme of things, though the Symmetrix outage was totally unacceptable and avoidable.

    1. Sounds painful, though honestly I personally believe that ESRS is the right answer, but also in an environment where you are deploying SYMMs, the network should NEVER be down. Any quality datacenter, should have redundant network connections to the outside world, redundant server switches at the distribution layer, redundant routers at the core, and even when scheduled maintenance is being performed, some basic level of functionality should try and be maintained. Even when large scale power work is being done, such as replacing a UPS or a generator, only 1 leg should be worked on at a time, and each rack should feed from 2 different sources. That way the DC should never go down except for wherever redundant power supplies happen to fail. But of course that’s just my rant on the matter.

      1. Only one power leg in the building.

        Only one battery stack.

        I totally agree with you. I’ve worked for a crappy non-profit that *STILL* managed to have two sets of batteries, even if they both were running off the same street feed and generator.

        • John Doe on February 3, 2011 at 5:42 am
        • Reply

        That doesn’t work when you see customers put the ESRS appliance on a VMWare host that boots from the array… And the array or host has issues…

        1. Yes, it’s always a bad idea to monitor from any system that has any dependency on the SAN.

          Our EMC ControlCenter infrastructure runs on SAN disks, but ESRS is on local disks only for that very reason.

    • John on February 14, 2011 at 6:19 pm
    • Reply

    Good feedback. As soon as we migrated from an old DMX to VMAX over a year ago, we made the switch to ESRS (which runs on a VMWare host on EMC LUNs). ESRS works great and is much faster for the 99.99+% of the time when our network and SAN infrastructure is up and running. For the times that the network and/or SAN are down, we installed a state-of-the-art 56k dial-up modem and had it configured so if ESRS is down for any reason, EMC can still dial in via the modems. I HIGHLY recommend going with a dual ESRS + dial-up modem config if at all possible.

    1. The problem is dial out… I don’t think the symm sp can have a secondary dial out. If esrs is down, they’ll never get paged.

      (though a severity 1 ticket is opened if esrs is down for more than 60 minutes..so it might be a moot point.)

Leave a Reply

Your email address will not be published.