Tag Archives: 5.0 - Page 2

ESXi: Change vmnic / vmhba numbering

If you change your hardware configuration without reinstalling your hypervisor you might run into an issue with the nic or hba port numbering. The ESXi host writes the PCI bus to vmnic mapping to a special configuration file. If you change the pci slot from your dual port nic for example you end up with 2 new vmnics.

The configuration file is located at /etc/vmware/esx.conf

Change vmnic numbering Read more »

Storage vMotion / dvSwitch patch released (but manual fix required)

VMware vCenter 5.0 U1a has been released. This patch has a permanent fix for the Storage vMotion/dvSwitch bug which caused HA to fail in a few cases after a hardware failure. I've tested the Update in my Homelab. The fix works as expected and enables the vCenter to move the dvSwitch port information to the appropriate datastore during Storage vMotion. But there is a little limitation: The patch prevents virtual machines from getting affected, but it does not fix currently affected virtual machines. This means that you still have to fix affected virtual machines after installation.

  1. Backup vCenter Server Database
  2. Install vCenter Server 5.0.0 U1a (Build 757163)
  3. Fix affected virtual machines using one of the Scripts below

VMware also released a patch for their ESXi Hypervisor, which made 768111 the current version.

VMware ESX Version Timeline

Since 2001 VMware has release 5 major verisons from their hypervisor called ESX (Elastic Sky X). I tried to collect all relases to the market here and here. With all that information i created a timeline. This is what i got:

VMware released a Security Patch for ESXi 5

VMware has publish a security fix for their current ESX Server. There is a vulnerability which might allow an attacker to manipulate the traffic from a remote virtual device to cause the virtual machine to crash. Another vulnerability might allow an attacker with the ability to load a specially crafted checkpoint file to execute arbitrary code on the host.

The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3288 and CVE-2012-3289 to this issues. Please note that this is not the privilege escalation vulnerability which in everybody's mouth at the moment. VMware products are not affected by this issue (CVE-2012-0217).

The latest ESXi 5.0.0 Build number is now: 721882
Also affected: ESX(i) 4.1, ESX(i) 4.0, ESX(i) 3.5
Updated: ESXi Release and Build Number History

ESX(i) Build List

The first thing i check on every vSphere environment is the version or build number which is currently installed. It is always good to know which version is running and when the last update has been applied. I published my list in the VMware section a few days ago:

ESX Release and Build Number History

ESXi Release and Build Number History

I will keep this list up to date when updates are released.

How to determine the VMware ESX(i) Build Version using vSphere Client
From the vSphere Client just select the ESX server within the Host & Cluster view. The version can be seen above the configuration tabs. In this example i am running ESXi Version 5.0.0 Build 504890.

How to determine the VMware ESX(i) Build Version using SSH?
If you are connected to the ESX console just enter "vmware -v" to get the Build number. This server is running ESXi 5.0.0 Build 702118, which is the current version as i am writing (June 2012).

Windows Server 2012 Hyper-V v3 vs. VMware vSphere 5

Microsoft marketing has recently released a Competitive Advantages document which focuses against VMware. Microsoft promises to have the hypervisor which exceeds VMware capabilities. But is Microsoft really in front of VMware now? I think that it is not convincing to compare a product that is not released yet with one that has been on the market for one year now. Who knows what VMware will release in the future, maybe even before the final release of Windows Server 2012. Nevertheless, let's have a closer look at the document.

There is no doubt that Microsoft has done a great job. They have a distributed switch called extensible switch and they have migration features which allows to migrate machines without shared storage. But are there any features VMware does not provide? I could not find anything. In my opinion, Microsoft has caught up, but they are not ahead.

Trunk Mode to Virtual Machines
With Hyper-V traffic can be directed to a virtual machine. Microsoft alleges that VMware does not provide such features. But this is wrong. Using VMware you can create trunk port groups and add VLAN tags inside your virtual machines:

320 vs. 160 logical processors
Hyper-V supports twice as many logical processors as the recent version of ESXi. Sounds great from a marketing perspective, but how about the real world? Is there any server vendor which provides such a big server? HPs largest server, the ProLiant DL980 G7 for example supports 8 processors with up to 10 cores. The IBM x3950 X5 has the same limit. This is even the half of VMware configuration maximum. So the answer is, yes - Hyper-V does scale higher then ESXi. But will you encounter this limit in the real world? Not at the moment.

Cluster Size
Hyper-V allows clusters with up to 64 hosts. VMware limits to 32 hosts. But does this really limit the design? In fact, there is no disadvantage if you just create two 32 host clusters with VMware.

VDCD510 Objective 1.1 – Gather and analyze business requirements

Five-Step design process

  1. Initial design meeting
  2. Current-state analysis
  3. Stakeholder and SME training
  4. Design sessions
  5. Design deliverables

Project participants (SMEs / Key Stakeholders)
A design project manager has to determine a list of SMEs and project stakeholders who can provide information about the design requirements. This list typically includes:

  • Network administrators
  • Storage administrators
  • Server hardware administrators
  • Operating system administrators
  • Application administrators
  • Security Officer
  • HR representatives
  • C-Level executives (CEO, CTO, CIO,...)
  • Representatives

It is important to lists the people involved in this project and their contact information. This list should include:

  • Name
  • Title
  • Organisation
  • E-Mail address
  • Phone number
  • Reachability

Having contact information available will help to make progress without unnecessary interruption during the design process.

Stakeholder = A person with an interest in a project
SME = A subject-matter expert is a person who is an expert in a particular area.#

Functional / Non-functional requirements
Functional requirements are tasks or processes that must be performed by the system. For example, a functional requirement of a vSphere platform is "must allow multi tenancy" or "users must be able to create virtual machines".

Non-functional requirements are standards that the system under must have or comply with. For example, a non-functional requirements for a vSphere platform is "must be built for a total cost of $500.000". Non-Functional requirements are also called constraints.

Conceptual design
The conceptual design typically focuses on the business requirements. It is an high level overview which includes only the most important components. Concept design is the first phase of the design process.

Conceptual design from an stretched Cluster

Conceptual design from a webservices platform

Logical design
The logical design specifies the relationship between all components. It is usually stable which means that it will only change if the requierement changes. The logical design does not include physical details such as IP addresses, hostnames or hardware models.

Physical design
The physical design is usually the last and the most specific design. It shows port connections, IP addresses and model description.

Requirement tracking
It is important to document all requirements. This could be done in an excel sheet for example. Every single requirement should contain at least a unique number, desciption, timestap, priority and the originator.

VCAP5-DCD Exam Blueprint v1.1

Skills and Abilities

  • Associate a stakeholder with the information that needs to be collected.
  • Utilize customer inventory and assessment data from a current environment to define a baseline state.
  • Analyze customer interview data to explicitly define customer objectives for a conceptual design.
  • Identify the need for and apply requirements tracking.
  • Given results of a requirements gathering survey, identify requirements for a conceptual design.
  • Categorize requirements by infrastructure qualities to prepare for logical design requirements.


VCAP5-DCD - What's new?

The new VMware Certified Advanced Professional 5 Datacenter Design (VCAP5-DCD / VDCD510) Certification is now live. There is much new stuff with vSphere 5, but also the design process has changed. I was pleasantly surprised that there is a much of ITIL standardization in the new exam. As i come from the cisco world, i am already familiar with ITIL and the PPDIOO methodology. So while studying for the old VDCD410 exam i was a little bit surprised at the new design methodology VMware uses. Now with the VDCD510 exam the design process ich much more like ITIL, i think this is a good idea.

As the old exam there are no course requirements. There only requirement is a VCP5 status. Until August 17, 2012  if you are currently holding a VCAP4-DCD there are no additional requiremnts. Pass the VCAP5-DCD and achieve both, VCP5 and VCAP5-DCD.

I have worked through the VDCD510 Exam Blueprint to find the differences between the old and the new exam. As this is not a technical exam and the blueprint has been completely rewritten, it is not trivial to compare both, but i tried to extract the new stuff out of the blueprint as best as i can.

Logical Design
All design processes are now much more ITIL related. So knowledge about creating a Service Catalog is now part of the exam.

The new exam now covers Exchange Version 2010 and a completely new product: Enterprise Java. VMware now offers much more ressources for application virtualization which can be found here.

Map Service Dependencies
Timekeeping is no longer part of the exam.

Build Manageability Requirements
Building manageability requirements is now much more complex. Make yourself familiar with Operational Readiness Assessment.

Physical Storage Design
Beside the ITIL methodology the most new stuff has come with the physical storage design. There are much new features like VASA, Storage Policies, Storage DRS, Storage Tiering and the new vSphere Storage Appliance.

Configuration Maximums
One of the most important part every VMware Architect shouild know are the Configuration Maximums. Of course, there are many changes in vSphere 5, so you should memorise the new Configuration Maximums.

Additional New Features

  • Auto Deploy
  • Deploying Anti-Virus Solutions (VMware vShield Endpoint)
  • vCenter Server Virtual Appliance

How to change vCenter Server language

When you install the vCenter Server on a system in your own language (german for example) you might run into an issue that even after forcing the vSphere Client to english all error messages and e-mail notifications appers in german language. Having non-english error messages is a pain, if you try to google for solutions. If you work for an international company it also may be important that error messages are in english.

There is no official supported solution for this problem, but you can use this little workaround. This works with vSphere 4, 4.1 and 5:

  1. Connect to the Server where vCenter Server is running (RDP Client or Console)
  2. The language files are located in: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\locale\
  3. Rename the de folder (or your language) to DE.old
  4. Copy the en folder to the same directory and rename it to de
  5. Restart vCenter Server Service

All error messages and alarm emails should now appear in english language.

Filesystem consistent Linux Backups with VMware

Life is easy if you are running Windows because if you want to create image-based VMware Backups with Veeam Backup & Replication, Quest vRanger, PHD Virtual or any other VADP using competitor you can use VSS. Backing up Linux is much more complex as there is no equivalent. What you get are crash consistent copys from your virtual disks. After some research i couldn't find any established solution. This is the backup vendors answer (I am not talking about application aware backups as this is another problem):

veeam Backup & Replication v6
Veeam refers to the "Enable VMware tools quiescence" option. But is this true? Yes, there is this option and you can enable it. But the vmsync driver inside you virtual machine is disabled by default. So if you activate "Quiesce", nothing actually happens. The backup succeeds but all you get is an inconsistent state.
Source: User Guide

Quest vRanger 5.3.1
The solution Quest provides is only a small hint: Install VMware Tools, create freeze Scripts and enable Guest Quiescing. But who will support my custom script?
Source: Quest Solution SOL84967

PHD Virtual Backup
PHD Virtual does not provide any information about consistent linux backup. The only thing i could find was a note "Quiesce? Windows only!"

Possible Solution?

So, how to create a filesystem consistent linux backup with veeam, vRanger or PHD virtual? As every vendor is doing the same -triggering the VMware API-  the answer is identical. But first let's have a look at the basics. What do i have to do to get a consistent state? And how can i determine that my backup is consistent?

An inconsistent filesystem has to be recovered prior to mount. Using dmesg you can determin whether is was consistent or not:

Consistent filesystem mount:

root@ubuntu:~# dmesg |grep EXT
 [3.711991] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
 [7.685314] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro

Inconsistent filesystem mount:

 root@ubuntu:~# dmesg |grep EXT
 [3.780568] EXT4-fs (dm-0): INFO: recovery required on readonly filesystem
 [3.780855] EXT4-fs (dm-0): write access will be enabled during recovery
 [4.153234] EXT4-fs (dm-0): recovery complete
 [4.178622] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
 [8.058018] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-roThis test was made with the current Ubuntu 12.04 LTS.

Solution 1: Custom fsfreeze Script

The fsfreeze command suspends and resumes the access to an filesystem. After suspendig the access, the volume is in an consistent state and can be copied. Please note that fsfreeze is a recently published tool and not available on older systems.
1. Install VMware Tools

2. Create custom scripts:

root@ubuntu:~# touch /usr/sbin/pre-freeze-script
root@ubuntu:~# touch /usr/sbin/post-thaw-script

3. Edit both scripts and add your mountpoints. Your file should look like (Only one mount):

root@ubuntu:~# cat /usr/sbin/pre-freeze-script
 fsfreeze -f /
 root@ubuntu:~# cat /usr/sbin/post-thaw-script
 fsfreeze -u /

4. Make both files executable:

root@ubuntu:~# chmod 755 /usr/sbin/pre-freeze-script
root@ubuntu:~# chmod 755 /usr/sbin/post-thaw-script

5. Activate "Quiesce" Option in your backup client

During backup the backup client triggers the vCenter server to make a snapshot with the "quiesce" option. This involves both scripts to freeze and unfreeze the write IOs on the filesystem during the snapshot creation.

Solution 2: vmsync

As mentioned above VMware has created a sync driver that allows to create consistent backups. Unfortunately i couldn't find any information about this driver and it is disabled by default, giving a small explanation:

[EXPERIMENTAL] The VMware FileSystem Sync Driver (vmsync) is a new feature that
creates backups of virtual machines. Please refer to the VMware Knowledge Base
for more details on this capability. Do you wish to enable this feature?

This comment emphasizes that this feature is not supported at the moment. I also couldn't find this Knowledge Base details.

To enable vmsync driver you have to enable it during the installation, or run vmware-config-tools.pl later:

root@ubuntu12:~# vmware-config-tools.pl

Making sure services for VMware Tools are stopped.

vmware-tools stop/waiting

[EXPERIMENTAL] The VMware FileSystem Sync Driver (vmsync) is a new feature that
 creates backups of virtual machines. Please refer to the VMware Knowledge Base
 for more details on this capability. Do you wish to enable this feature?
 [no] yes

After activating vmsync and "quiesce" option in your backup client you can create consistent backups. I have tested both solutions in testing environments with Ubuntu 12 and RHEL6 systems and was able to create consistent backups. But please note that this is not supported by VMware or any Backup Vendor. So please test it out before you roll it out into production.