«

»

Jun 09

HPUX

It’s funny that I got my start working with HP systems, first HP-MPE, then HP-UX, because I hate HP these days. HPUX is still the only operating system that can’t tolerate something as simple as a switch migration.

HPUX is still, to this day, dependant on the /dev/dsk/cxtxdx numbering in it’s volume groups.  Unlike all *REAL* Logical Volume Managers, HPUX doesn’t write any sort of PVID to the disk, doesn’t store the disk configuration for later use.

So what happens is, if the D_ID (Destination ID) of the port changes, HPUX will generate a new cXtXdX number, and all existing volume group configuration will be lost.

So what you have to do – gather the volume group configuration using ‘vgdisplay’ and ‘syminq’ before you start.   Deactivate the volume group and export it.

Once you’ve moved the cables, and rezoned, rescan the bus and run ‘insf -e’ to build the new special files.

Run the ‘syminq’ again and use the information you get there (new symdev –> cXtXdX number)  to rebuild and reimport the volume groups.  Reactivate the volume groups and ensure everything is present.

Reboot the system and all should be good.

What’s pretty pathetic is that even Windows can handle such a simple change, Solaris doesn’t care about the port #’s, AIX cares, but the simple process of removing and rebuilding the devices in the ODM fixes that, because the AIX LVM is dependant on the PVID of the disk, not the device number.

It just seems like something that would have been so simple to fix, so long ago….

10 comments

Skip to comment form

  1. Not sure if you have ever had the joy of switching SAN cabinets in an ESX environment that you might experience something very similar.
    At least once a year I rotate in a new switch and a new SAN cabinet. All the data is copied from SAN A to B and if you connect through the old switch life is good but as soon as you swap out the switch… Things go downhill from there. I am dreading the migration this year. To mitigate it I am going to do it in a better two step process – switch the data first and then switch the switch.
    I ended up writing off 400GB worth of VMS last time due to the changeover. This time the data is in excess of 9 TB and it should be a blast!

  2. I guess I’m a bit confused… HPUX has veritas volume manager installed by default/builtin and has since at least 11.11.

  3. Jesse

    I don’t think so by default. I’m working on one engagement where in an HPUX 11.11 system the disks are mounted as raw, and there is no vxvm or vxfs on the system.

    Now I don’t know if they have removed it or if it was excluded from the install, but it’s not there.

    My primary complaint isn’t the lack of a volume manager, but the fact that unlike most civilized operating systems, their cXtXdX number is D_ID dependant, which means that if the destination ID changes, the number changes. If, like Sun does, the cXTx numbers were controller # (HBA#) and target # (WWPN mapped to a TID) then none of this would be a problem.

    As a consultant though HPUX is a great thing. The risk multiplier on a job starts at 35% when it has anything to do with HPUX, or I won’t touch it. Not worth having to deal with the potential problems.

  4. Hmmm where to start…

    First off the HP-UX Logical Volume Manager *does* write a PVID to each disk (that’s why you can do the vgimport afterwords, or use vgscan). The issue isn’t anything to do with the volume manager – it’s to do with how device special files are allocated, which in turn is down to how hardware paths are allocated. Of course these “problems” generally only occur when one of 2 things happen: i) The domain ID of the switch is changed or ii) The port the storage is plugged into changes. Not something that should be happening frequently in a well managed SAN I would have thought? And for item ii any decent switch (like the CIsco switches for instance) can be configured to tie a D_ID to a WWN. I’d also argue that there are some situations where tieing storage to WWPNs can also cause problems.

    But more importantly you seem to have completely missed the fact that HP-UX11.11 is an 8 year old OS (think Windows 2000, Solaris 8 and Linux with the 2.4 kernel!) The fact its still around is a testament to its quality and the fact that HP still supports it! The latest release of the HP-UX OS is 11.31, (out for about 18 months now) and this version certainly doesn’t suffer from the issues you describe, in fact I reckon it’s probably got the best IO stack of any of the current commercial Unices. Have a read of the intro whitepaper here:

    http://docs.hp.com/en/MassStorageStack/The_Next_Generation_Mass_Storage_Stack.pdf

    Cheers

    Duncan

  5. Jesse

    Duncan – The above reply came from within HP’s domain. (No surprise really)

    I glanced through the whitepaper – seems like HP is doing a lot to mitigate the problem. On the vgexport/vgscan, can that be done system-wide or does it require that you at least specifiy the first device in the volume group? (Like it has in every version I’ve worked with) If you could do ‘vgscan’ system wide and have it bring the volume groups in automatically – that would make life a lot easier.

    As for versions, I see much more “old” HPUX in production datacenters than I do new. Most of the systems I work on these days are K, L, or N-class servers running, at a maximum, 11.11. This is a great testimony to the reliability of the operating system but means that I can never depend on the ‘latest and greatest’ features being available. (As a result of the reliability, customers aren’t anxious to upgrade, which is a double-edged sword)

    As to storage ports/D_ID’s or WWPN’s changing I have a skewed view. When I see a system its because some component of it is being upgraded. I’ve done a lot of switch/storage upgrades (80% of my work) and above all of them, HPUX is the more difficult of the unix flavors.

    In the case I had in mind when I wrote this post – we’re migrating from a pair of Cisco MDS9216 switches to a single MDS9509. So while I can set the Switch domain per VSAN, the storage port is going to change, so no matter what I do the D_ID of the storage is going to change.

    Right now all of their volume groups are set up using the cXtXdX number of the disk as the primary path (and an alternate path, totally unrequired because the systems are running powerpath)

    Now if the volume groups were created using the /dev/emcpowerxx device ID, this wouldn’t matter. The emcpower device doesn’t change when the underlying cXtXdX number changes, so paths can be moved and reconfigured dynamically while the application is still up.

    So the migration gets put off until sometime in August because we can’t take the application down to reconfigure LVM. Add that to the fact that its a clustered OS (and I threw my MC/ServiceGuard book away years ago) increases the complexity.

  6. Yes, I’m a HP employee – but chasing round the blogosphere looking for inaccurate statements about our products isn’t in my job description (well at least not explicitly!) I wasn’t a HP employee a year ago and would have posted the same comments then… but I guess you were right to call it out, and maybe I should have mentioned it in my post.

    Just to be clear, the IO stack in 11.31 would behave pretty much like any other OS when doing the switch migrations – no need to vgexport/vgimport or vgscan etc.

    And of course there’s a lot of old systems out there (amazingly many of the K class systems introduced in 1996 will be supported right up to the end of 2013 – nearly 17 years!) – the issue was that the IO stack in these systems running 11.11 was sufficiently monlithic that fixing it would take more than a patch could acheive and would break backwards compatability. But it *is* fixed now. If I wanted to be obtuse I could spin this around and say “why doesn’t a Clariion support ALUA?” – well you’d probably point out that the Clariion has supported ALUA since the flare 26 release last September, but if my clariion is running an older release and I don’t want to upgrade…

    But I will agree that these upgrades on HP-UX 11.11 are probably harder than most other OSs (except some of the older versions of AIX – you won’t see so many of those out there though as IBM aren’t nearly so forgiving in supporting older releases). Without going into gory details, when I’ve had to do these sorts of upgrades, I’ve always managed to do them online (even within a cluster – although I may have had to fail packages over). Generally its been acheieved by doing one switch at a time using vgreduce/vgextend before/after the migration – although the ones I’ve done have always been with fully A/A disk arrays like Symmetrix and DMX – not with Clariion, so maybe there are more issues there…

    With regards to adding alternate paths to volume groups when power path is installed – you’re right there’s no need to do it, but I would say that good practice dictates you do it anyway – many admins would use the command “strings /etc/lvmtab” on HP-UX to determine which disks are “in use” – not a reliable wayof checking but a lot of folks do that. If the alternate link doesn’t show up, the admin thinks the disk is free and…

    I’m not totally sure on your question about vgscan, as I never actually do it that way myself, but are you actually renaming the /etc/lvmtab file before running vgscan? If you don’t it will try to use information in the file which I think would mean it would need to find at least one disk in each volume group already.

    Anyway – I’m glad I stumbled across this blog, as many of the other posts are very interesting too.

    Thanks,

    Duncan

  7. So, I doubt this will see a response, but why not shoot anyways.

    If you’re booting form SAN on HPUX 11.0/11.11, and you need to change the wwpn of the storage, is there any way to *easily* do this?

    I can try to beat up on it in my lab if I have time… the time thing would be the problem.

  8. Well, it worked fine 😀 Didn’t care about the wwpn since the ports didn’t change.

  9. jesse

    I’m impressed, I figured that was going to be much more trouble.

    I know 11.23 has FINALLY done away with the /dev/dsk/cxtxdx numbers, but I still haven’t seen how persistent they are.

  10. Ashish Barot

    Hi All,

    can any one tell me that, which is the bydefault package of volume manager comes with HP UX 11.31? On newly installed system?

    Thanks.
    Ashish Barot.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>