How to tell if your sales rep hates you….

I just got the following job posting and it made me, literally, laugh out loud, spitting latte all over my laptop.

If your sales rep allows you to do something like this, it’s a fair bet that s/he hates you (or is planning to buy your company out of bankruptcy later).

WANTED: VMWare 1-month resident to assist with new deployment/planning around 200VM’s and new Celerra NS480’s being purchased by client. Will probably end up primarily being VM’s using NFS on NS Celerra Replication will be enabled between (2) NS480’s.”

The key points are:






Did I mention NFS?

Someone actually sold this?  Even if the customer comes to you direct and says “this is what I want…” the answer should be “In the interests of protecting you from yourself, I can’t allow you to do this.”

I don’t care how much the deal is worth.


Skip to comment form

    • Andrew Storrs on May 22, 2009 at 11:13 pm
    • Reply

    I’m confused Jesse… the issue is NFS and VMware in your mind? or just that it has anything to do with a Celerra… 😉

  1. Heh heh. Guess I had that one coming.

    No, I love Celerra, hell I own one.

    Its the 200 VM’s over NFS that would scare the crap out of me walking into a new site.

    Better have a *REALLY* good IP switch.

    • Andrew Storrs on May 23, 2009 at 7:21 pm
    • Reply

    Lol, it’s not actually that uncommon to use NFS with VMware. I’ve seen implementations with around 30-40 ESX hosts and 800-1K VMs running of NetApp storage over NFS.

    In fact by using NFS instead of VMFS storage they were able to get great dedup with ASIS since they weren’t limited to 2TB per datastore.

    But yeah… you’ll want to be using something other than D-Link. 😉

    1. You’re still limiting your bandwidth dramatically. VMWare 3.5 doesn’t really have a solid multipathing IP stack (IE they don’t use LACP, etc) so you’re still only going to receive at 1Gig (Until 10G gets widespread adoption that is)

      I tried NFS into a server running quad-gig-e a while back, and at about 6 VM’s it started getting unstable… Not something I would want to do enterprise-wide. (Then again, I didn’t have the Celerra in-house at that point, maybe it’s something to revisit)

    • Andrew Storrs on May 25, 2009 at 1:56 pm
    • Reply

    If what you’re saying was true, iSCSI would be equally as useless and everyone would be using FC with ESX. There’s a good blog post from a couple years ago that should clear up your confusion ( Note: this guy works for NetApp, but what he says is still valid.

    What are you missing from ESX’s networking stack (especially with 4.0)?

    LACP isn’t necessary since all connections to ESX hosts should ideally be trunk ports, and you can take advantage of the aggregate bandwidth automatically by associating the various pNICs with the same vSwitch.

    • Andrew Storrs on May 25, 2009 at 1:59 pm
    • Reply

    Rich Brambley over at VM/ETC also wrote a great post covering the same subject earlier in the month:

  2. The only person more clueless than the sales rep was the IT manager who signed the purchase order. It would be an extremely desperate, and thus almost as clueless, wannabe SAN admin who would take on the residency.

    • Andrew Storrs on May 27, 2009 at 3:39 am
    • Reply

    Again I don’t see the issue CyberED…

    • william bishop on May 31, 2009 at 3:20 pm
    • Reply

    Sorry andy, but you’re not going to get much traction. Most people who use FC don’t like or want to go to the ethernet based storage methods…mostly because of the latency increase and the reliability(glass, once laid is fairly set and forget) issues. Why do I want to send several packets for every single FC packet? FC is a porsche compared to the IP ford focus.

  3. Ethernet is a protocol designed around imperfection. it’s designed to be resilient at the expense of speed, which is 90% of why it’s the best for what it does. When routes across the country come and go, the protocol by design uses alternate paths to get to it’s destination.

    Such a protocol is, while perfect for SMTP and Web traffic, is ill-suited to storage, where timing is everything and dropped packets absolutely can-not be tolerated.

    Recently I sat in on a customer that had bought a Celerra NX4 to handle a pretty sizable VMWare implementation on.

    While the NX4 is a great box, they customer skimped on everythin ELSE that mattered. No Dedicated network, (and then when they were told they needed a dedicated network tried to build it on Dell PowerConnect switches (fairly low-end, 5324’s I think)

    (Don’t get me wrong, I use a 5324 for my primary network switch in my rack here, but the one time I tried to put iSCSI on it it barfed every time I tried to run a backup across the same swtich. (Mental note – this is where seperate switches would have come in handy.)

    When told they needed to upgrade their switch to something more medium-high range Cisco-ish, the proverbial shit hit the fan.

    (of course this particular customer had issues with not having had sized the environment correctly, and came up short on storage too)

    The best use for iSCSI, IMHO, is using something like Symmetrix or Clariion with a mix of ports, iSCSI and FC, where you can use the iSCSI to present cloned / snapped copies of production data to development servers where it’s more about the data being available than having any kind of performance.

    And I’d *STILL* put a seperate network behind it, just ’cause.

    • Andrew Storrs on May 31, 2009 at 10:51 pm
    • Reply

    I don’t disagree with you in that FC is faster, cleaner and all around nicer than Ethernet based storage (block or file). I think the point I’m trying to make is you’re not VMware guys – you’re storage guys… and you’re forgetting (or not aware of) the fact that NFS vs FC with ESX does not scale linearly – due to issues with the way VMFS works, the fact most storage access on ESX is random and small, and the way the SCSI reservation queues are per LUN on an ESX host, etc.

    I guess what I was commenting on is the fact that Jesse you trashed on the design without understanding the fact that THERE IS NOTHING WRONG WITH IT from a VMware admins perspective and if properly implemented (with quality arrays, sufficient disk, and a dedicated storage LAN, etc) it could easily scale to handle 4-5x the number of VMs without a problem. A design like this is quite common, hell nearly all the NetApp/ESX implementations these days are NFS based.

    Okay not sure what else I can say to open your mind to the possibility this isn’t “wrong”… next time I’m near D.C. how about we continue over drinks or something. 😉

    • InsaneGeek on June 2, 2009 at 2:36 pm
    • Reply

    200 guests is not small potatoes (it’s not “oh my god” massive, but it’s not small either), if they are servers rather than just desktops I’ve got to imagine there is a couple of high-throughput systems on it. So IMO Jesse is pretty reasonable on being a skeptic of it being a easy “walk in the park” type engagement rather than uncovering hidden problems left and right.

    Because people seem to think of things as stove pipe buckets (san guy, vm guy); as I been managing storage for 8+ years and I also have a ESX VCP cert; maybe I can help bridge the gap between the two on this topic. Firstly, without details of the I/O load nothing can be said. They may do 100iops at 5MB/sec, or they may do hundreds of MB/sec and thousands of iops. Without out some stick to measure it’s about impossible to make an informed statement; if you are walking into a situation like this with hardly any info expect to be blind-sided by pain.

    I think this is where the divide between the storage & VM guyes come in: VM guys generally (def not all, but my experience most) have only really seen the first stage of virtualizations meaning all the low-hanging fruit that does nothing pretty much all the time, print server, AD server, DNS, NTP, etc. These entry level systems generally do little to no I/O compared to a traditional server, heck most of their load could run in a single 100mb link let alone needing a gig link. While storage guys without the whole information are going to think conservatively, and associate it with loads they normally see on big servers; because who has ever asked to hook up a DNS server to mass external storage with an expectation of running thousands of iops? So the storage guys are thinking thousands of iops and hundreds of megabytes/sec and the vm guys are thinking tens of iops and tens of mega*bits*; and they are both right and both wrong.

    Here’s my take on the whole thing, NFS gets you management benefits but there is no free lunch. Being able to take a NFS snapshot and let a non vmware host see it (i.e. backup server) is a big benefit. Being able to copy things around back and forth using traditional tools is very, very nice. The detriment is that traditionally VMware has supported new features via NFS much slower (i.e. storage vmotion), you can only have a total of 8x NFS datastores (got a cluster of 14x vmware nodes, you can only use 8x NFS datastores) vs the 256 of FC, and the big one is that you don’t really get good scaling. The first thing someone will say is that you can do some gig ethernet link aggregation and get around the issue but nobody I know does it. The problem with ethernet link aggregation is that unless you are using some stackable switches you throw away redundancy for performance (before you throw stones, this might have changed as late with vsphere). i.e. If you are uplinking to 2x different switches for redundancy, to get 4gig of bandwidth you need 8x network connections. 8x in a single channel with 4x active and 4x passive (nature of ethernet to avoid network loops). So for a 10 ESX host cluster with 4gig of bandwidth per host you will need 80x full-gig network connections. Because of this hardly anybody does aggregation and they mainly just add additional network access ports to the virtual switches. VMware will then chop up requests and split them around giving them an effective double that of a aggregated channel (8x individual ethernet ports can be active, but a max throughput of anyone is a single gig) , a VM will get put on a link and remain there until it reboots/vmotioned or a link down occurs. The max speed of any one guest is a gig (assuming no other guest is on the same link), this limit is also for the NFS/iscsi storage. VMware does not intelligently moving this around, and can have 2x high throughput guests sitting on the same ethernet uplink contending for bandwidth, while another uplink sits almost idle. The vmkernel gets a single ip address and it is used to talk to ethernet storage. The traditional problems of NFS being very, very chatty on opening/closing a file is minimized as once a guest is running it doesn’t have to do the whole negotiation game and can request the direct block offsets since the file is already open (like oracle + nfs). Another detriment that gets a good portion of lip service, but not sure it matters that much is that unless you use a full toe card ethernet uses a good portion of CPU horsepower to drive a gig card. Normally you are limited by ram rather than CPU so I’ve often found that you have extra CPU cycles hanging around.

    So coming back full circle… I have a number of guests who constantly push 60-70MB/sec which individually can fit on a gig link (alone by themselves), but when you start combining them it becomes rather difficult to try and mange them. I run FC connections to the ESX servers, I can give them lots of bandwidth and redundancy with a single dual port HBA and just not worry about it. I already had the existing FC environment so it didn’t cause me any additional mgmt overhead. I lose out on being able to take a snapshot backup using NDMP (I still can take a block level snapshot and present it to another server), and copying via traditional tools (but I can use storage vmotion) but I don’t have to micromanage anything it just simply works. NFS could work in our environment but I’d fear that some of my guests I’d have to micromanage and have some ‘fits’ with.

    If I was going into a new environment and had very clear understanding of my I/O load, I wouldn’t be opposed to using ethernet for a VMware farm, I’d be a bit skeptical though. It becomes rather interesting once people get beyond the low-utilization server and start putting bigger and bigger servers onto a virtualized environment on how you grow for future throughput needs. You don’t want to rip things out and try to redo them it’s a serious pain. One person wanting to put a big guest in the environment may push into some hefty $ choices: 10gig ethernet (does your array even support it), split the cluster into a high/low throughput farms, etc. that instead of it simply virtualizing a $5k server, it moves into a $100k+ infrastructure switchout.

    • william bishop on July 25, 2009 at 5:28 pm
    • Reply

    Andrew Storrs, you’re dead wrong. I started as a vmware guy, and still am. I run one of the largest implementations in the world, and I built it from the ground up. I tested every concievable method over the years for this environment, and others….and I have no love for NFS after all of that time.

    Hell, go find out how NFS by netapp is doing for the vmware build at the FAA….oh wait, it’s not. They ended up scrapping it in favor of FC.

    I’m VMware first, storage second. And I state again, you won’t get a lot of traction because FC is simply better.

    • Nakarti on August 18, 2011 at 3:29 pm
    • Reply

    I’m a network/server guy with some knowledge of storage, and I see nothing wrong with that setup, as long as the bandwidth out of the Celerra is 10gig or multipath.
    200 VMs on one server would be pretty insane, the 200 VM environment I work on is on 40 servers. Their design is a bunch of app servers pull from a handful of (in this case physical clustered) database servers (which have their own Symmetrix SAN,) and everything is feeding out to web servers.

    So yeah it looks scary, but its entirely doable, as long as you’re not trying to do things with the environment that the environment can’t do, 200 VMs on a Celerra is within sanity.

    1. There are *SO* many reasons not to do this… Now I would say you’re right, you can “limit” the damage but when it all comes down to it, ethernet is ethernet. It’s a connection-less protocol, designed around the concept of a flaky network.

      10G is a plus, direct-attach via crossover (though I think not supported) might make things even more stable. At the very least single-switch, single-subnet, with an enterprise switch. (I saw someone running iSCSI over a consumer-grade NetGear once, it made me CRINGE.)

      That being said, there are also issues with NFS and file locking, that with 200 VM’s I’d be afraid to try it.. Though interesting…since NFS doesn’t do the same whole filesystem locking like VMFS does there are parts of it that might be ok.

      Interesting. Now you have me thinking..

      At the very least though I’d go to iSCSI instead of NFS. It’s got at least a little more resiliency built into it, but FibreChannel will always be my first love. 😉

Leave a Reply

Your email address will not be published.