Tag Archives: lab

Issue with ESXi 5 upgrades (via VUM) and Iomega IX2-200 VMDK datastores

Over the last couple months I have been performing upgrades from ESXi 4.1, to 5.0 Beta, to 5.0 RTM, and earlier today to 5.0 GA. With the exception of the 5.0 GA release all these upgrades were handled with VMware Update Manager (VUM). I have encountered a few errors along the way I and I felt it was worthwhile to share them.

First of all to use VUM and the ESXi 5.0 GA ISO to upgrade to your hosts to 5.0 GA you must be running  ESXi 4.0.4.1 or later (per details from VUM), but not any previous release of 5.0. The odd thing is that you can do an upgrade of your pre-ESXi 5.0 GA hosts by booting with the install CD and choosing the “upgrade” option. This preserves all the settings as you would expect it to. The ESXi “depot” installer for 5.0 GA for VUM has not been released yet so I do not know if you will be able to use it to upgrade 5.0 Beta or RC to 5.0 GA (stay tuned as I have a ton on hosts running 5.0 RTM so I will test the depot install as soon as I get it!).

For further details about upgrade requirements and such visit the vSphere 5 online documentation here about that very subject. I have had nothing but success using VUM; I’ve used it for ESX 4 to ESXi 5 upgrades as well ESXi 4 toESXi 5 upgrades, even with the guest VM’s *suspended* during the upgrade (I was feeling adventurous). The onlyissue I had was with my Iomega IX-200 (Cloud Edition) that I use for iSCSI shared storage in my lab. I had no issues going from 4.X to 5.0 RTM; the datastores were available as expected after the VUM orchestrated upgrade. This morning though I went from 5.0 RTM to 5.0 GA and my datastores were not available, however the iSCSI connected devices did display.

Device View:

 Datastore View:

 Devices looks good, but where is my VMDK volume (only the local volume is shown)?

I’ve done a little work with SAN copied VMDK volumes before and as such have had to deal with VMDK volume resignaturing. VMware has a nice KB articlethat explains why resignaturing is needed:

VMFS3 metadata identifies the volumes by several properties which include the LUN number and the LUN ID (UUID or Serial Number). Because the LUNs now have new UUIDs, the resulting mismatch with the metadata leads to LVM identifying the volumes as snapshots. You must resignature the VMFS3 volumes to make them visible again.

Ignore the bit that specifies VMFS3; the KB article hasn’t been updated and it would appear that this issue applies to VMFS5 as well. In a nutshell what is happening is that ESXi sees the “new” datastore as a snapshot and as such does not mount it as a VMDK volume.
Resignaturing the drives is quick and painless although please remember that it will affect every host that connects to the datastore that may not have been upgraded yet and/or is still accessing it. I had brought down the whole lab so I wasn’t concerned about this. The steps are as follows:
  1. Power off any/all VM’s that may be running on the datastore(s) you will be resignaturing.
  2. SSH into one of the hosts that currently has access to the datastore(s) that are having problems.
  3. Execute “vmkfstools -V” from the ESXi console to rescan the volumes on the host. If this fixes the problem then you are all set. Odds are you already did this via the vSphere client so you need to move on to the next step.
  4. Remove any VM’s from the ESXi inventory that reside on the volume(s) you will resignature.
  5. Verify that the volumes are seen by the host by executing “esxcfg-mpath -l | less” from the ESXi console.
  6. From the ESXi console execute “esxcfg-advcfg -s 1 /LVM/EnableResignature“. This will resignature ALLdatastores that were detected as snapshots during the next rescan, so hopefully you remembered to take all the precautions I specified above.
  7. Repeat step three to initiate the rescan and perform the resignature operation. YOU ARE NOT DONE YET! You should however be able to see the VMDK volumes on the host now at this point (they will have new datastore names that start with “snapshot-“, if not your problem goes beyond the scope of this post.
  8. Execute “esxcfg-advcfg -s 0 /LVM/EnableResignature” to disable the resignature-during-rescan option. If you fail to do this your datastores will be resignatured during EVERY rescan, which I am fairly certain you do not want.
  9. Still not done, now execute step 3 again to make sure the volumes stay mounted after the rescan. Assuming that they appeared during step 7 they should still be present after you run another rescan. If they disappear after this step it means you did something wrong in step 8 and the drives were resignatured again. Repeat step 8 again, then this step, and verify that the volumes remain.
  10. Browse the datastore(s) and re-add all your VM’s to your inventory. You do that by browsing to the VM folder, right clicking on the vmx file within, and selecting “Add to Inventory”.
  11. Rename the datastores to match what they were before. This is an optional step but if you are like me the datastore names have meaning and they are part of your overall design.
Everything is back to normal in the lab:
I must admit that it is scary when the datastores disappear. Remain calm and remember that during a CD based (boot time) install you don’t have access to the iSCSI or NFS volumes (unlike fiber channel) so you are most likely just a resignature away from fixing your problem. The fix takes less than a couple of minutes and you will be off and running with your new ESXi 5.0 GA install.
Update (10-1-2011): I encountered this issue again after updating the firmware of my Iomega IX2-200; he same fix worked to restore access to my datastore.
- Jason

The VMware lab is (slowly) coming to life

Slowly” means that Murphy’s Law reared its ugly head; one of the Intel system boards would not post so I am awaiting an RMA replacement. Thankfully the other board was fine and one of the new systems is up and running fine.

Things I have learned so far:

1) Xeon E5310 and Core 2 Quad Q6600 processors support enough of the same features to allow vmotions between each. I am using my “old” Q6600 based host as the second node in my cluster until the new system board comes in.

2) Xeon E5310 processors and the ram they require generate a lot of heat; I needed to purchase some additional fans to keep the temperature in check. None of this is surprising but having never run this equipment outside of a prebuilt server (Dell in my case) the amount of heat generated is impressive. In my case an extra $30 worth of fans is not worth it considering I already had the processors and ram needed to build the host.

3) The crawl space of a house in North Carolina (if properly sealed) appears to maintain a very suitable temperature for storing lab servers. The only downside is that I don’t have remote power management so when I want to bring up my extra nodes to run labs (only 1 will run at all times) I have to trudge down to my makeshift datacenter. Perhaps now is the time to set up wake on lan; I’ll make that a future project.

I already had a suitable vCenter instance running so all that was required to bring the cluster up was to:

1) Transition all my existing VM’s from my legacy Q6600 host local storage to the ix2 datastore and deregister them from vCenter.

2) Rebuild all hosts to vSphere ESXi 4.1 U1.

3) Register the new hosts with vCenter and register all the (now orphaned) VM’s.

Initial impressions of the environment, in particular the Iomega ix2, are very positive. I keep the following servers running now and performance is more than adequate: Windows 2008 R2 – 5 servers in total:

  • Domain Controller
  • Virtual Center
  • VMware View Controller
  • VMware View Security Server
  • VMware View Transfer Server
  • Windows 7 x64 – VMware View Desktop (I haven’t setup VMware View Composer yet so this is a dedicated VM)
  • EMC VNX VSA (VNX simulator that can be used for VMware SRM testing or for general EMC Unisphere/NAS labs)
  • VMware vCMA appliance (vCenter management from mobile browsers or the Apple iPad vSphere client)
  • Astaro Security Gateway appliance (I use this free appliance as a VPN endpoint)

I’ve only just begun to dig into VMware View but I must admit it is a very cool product. Unfortunately outside of the work of Mike Laverick or VMware itself there isn’t a lot of information out there about it so to date most of what I am doing is trial and error. My next step is to get View Composer up and running so I can use linked clones rather than the dedicated clones I am using today. Once my new Intel board comes in I can complete the lab setup and get the VMware SRM labs up and running. I am familiar enough with the core VMware vSphere features but unfortunately none of the employers I have worked for has had to leverage FT let alone SRM, which is why I decided to put this lab together.

I must admit that I am having a lot of fun building the new lab and the VMware View environment. This isn’t a surprise mind you, but a reminder that no matter what I do I will always be an engineer at heart. To me architecting and building things is always going to be more exciting than managing them. Once I complete the lab build I will post some pictures of the my setup and go into some additional detail about what I am doing with the lab. – Jason

Time for a home VMware lab upgrade

I have developed a theory that a significant number of VMware vSphere/ESXi customers (smaller ones; a couple of hundred servers or so at most) likely don’t use much of the functionality beyond vMotion, DRS, and virtual center. This means that there are a lot of VMware admins (trained or not) who know only basic details about what it is they can do with their environment.

The VMware Install/Configure/Manage class is really an intro level class and doesn’t touch on much more than these topics either; the focus is primary on installation, basic configuration (DRS and vMotion included of course), storage integration, vSwitches (but not Nexus), and resource pools. Don’t get me wrong I enjoyed the class but the reality is that it seems to be designed exactly for the “average” VMware admin that most companies need.

As my career progresses I realize I am gravitating towards the core infrastructure and (somewhat) away from the applications that run on it. I still need to understand those applications of course, but only so I know what I need to deliver. A good example would be high availability Exchange 2010 deployments as Microsoft wants to see a certain number of cores across your virtual environment in order to be able to provide full support. While my administration days are numbered the engineering skills I am required to have are increasing by the month. No complaints here! Engineering is what keeps me up at night, and sometimes through the night. The point? I need to know more about VMware. I need to be able to do more with VMware. Deep skills in SRM, HA, resource pools, update manager, interpreting log/performance data, and so on.

My current home lab is two hosts with local storage only, which won’t suffice when you need to study the laundry list of items that I do. My short term goal is to be able to get my VCAP-DCD, and at some point my VCDX. The former is definitely possible; I don’t know yet what I will need to do to attain the latter.

Current lab details:

  • 1 x Dell PowerEdge 2950 / 16GB ram / 2 x Intel Xeon 5000 Series Dual Core
  • 1 x Generic PC / 8GB ram / 1 x Intel Q6600 Quad Core
  • ESXi 4.1
  • vCenter 4.1

New lab details:

  • “Datacenter” 1:
    • Two node primary VMware cluster (yes two of these will be built)
    • Intel S5000XVN Workstation Motherboard, 1 x Intel Xeon 5000 Series Dual Core, 16GB ram, dual onboard NIC
    • Local storage will be flash only to run ESXi 4.1U1
    • Iomega ix2-200 iSCSI capable 2TB (1TB usable) NAS device. This VMware certified device will provide iSCSI shared storage to my two node cluster.
  • “Datacenter” 2:
    • Generic PC will be updated to ESXi 4.1U1 and used for performing VMware SRM labs.This box will continue to run local storage only.

Additional details: The 2950 will be parted out to provide processors and ram for the new hosts. The server is just to loud and expensive to run for long periods of time at home. While old (going on 5 years) the components are still more than adequate for most production VMware loads and are perfect for my new builds. I will use a EMC (Uber) VNX simulator in each “datacenter” to support the SRM labs. Nick Weaver (creator of the EMC VNX and Celerra simulators) has a video explaining how to make it all work. I choose NZXT Whisper cases for my 2 builds because they get decent reviews and are reasonably quiet. Corsair CX430 power supplies are good quality, not too expensive, and should provide enough power.   The updated lab should allow me to simulate most of the things I need in order to really advance my VMware skills. While it would be nice to build a lab like this one I think I can get most of what I need out of this modest setup. I look forward to all the components coming in so I can get everything setup and start playing with some of the more advanced features of VMware.

- Jason