Wednesday, December 24, 2014

Extending VMware VM’s VMDK disk with command line vmkfstools

Those who have recently upgraded to vSphere 5.5 may have noticed that if you’ve also gone ahead and upgraded the virtual machine’s hardware to version 10 would result in the inability to edit the virtual machine’s settings via the vSphere Client because you are now forced to use the vSphere Web Client as the following message indicates:

You cannot use the vSphere Client to edit the settings of virtual machines of version 10 or higher.

Use the vSphere Web Client to edit the settings of this virtual machine.

image

Most administrators at this point can proceed by logging onto the vSphere Web Client to edit the settings but what if vCenter was down?  Connecting directly to the host wouldn’t allow you to edit the settings either.  The workaround to this is to actually update the client to vSphere Client 5.5 Update 2 which would result in the following message when you attempt to edit a virtual machine’s settings:

You are about to edit a virtual machine of version 10 or higher. Only version 8 features will be available for Edit using this client.

If you want to edit the advanced hardware features of this virtual machine, please use the vSphere Web Client to login to vCenter.

image

Various advanced properties won’t be available but actions such as extending a hard disk is allowed.

There has been times where I’ve found myself stuck in a datacenter without access to the internet which therefore means upgrading the vSphere Client to 5.5 Update 2 would not be possible.  The workaround to this is to use the command line via SSH or console to extend the disk and the following outline the steps to do this.

Step #1 – Check for Snapshots

The first step for this process is to ensure that there are no snapshots created on the VM that you are attempting to extend the VMDK.  There are plenty of ways to determine whether there are snapshots such as browsing into the directory of the virtual machine and looking for -delta.vmdk files as shown in the following screenshot where ls was executed before and after a snapshot was created:

image

Another method which I favour in using is to use the vim-cmd command to list snapshot information for the virtual machine.  The reason why I favour the vim-cmd command is because it is available via the ESXi’s console with no reliance on PowerCLI which requires vCenter to be up and if vCenter was up, you could just use the Web Client to increase the disk size.

Before I begin, note the following handy CLI reference site which I use quite frequently:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012964

Begin by SSH or accessing the console of the ESXi server that the VM is hosted on and execute the following command to list the VMs:

vim-cmd vmsvc/getallvms | more

The VM we’re interested in is the -FS-01 server:

image

Note the VMID # on the left which in this case is 21.

With the VMID obtained, proceed by executing the following command to list the snapshots:

vim-cmd vmsvc/snapshot.get [VMID]

For this example, we’ll execute the following:

vim-cmd vmsvc/snapshot.get 21

image

Note that there is a snapshot named Snappy for this VM.

To remove all snapshots on this VM, continue by executing the following command:

vim-cmd vmsvc/snapshot.removeall [VMID]

For this example, we’ll execute the following:

vim-cmd vmsvc/snapshot.removeall 21

image

You can verify that the snapshots are gone by executing:

vim-cmd vmsvc/snapshot.get 21

… again.

Step #2 – Expand VMDK with VMKFSTOOLS

Before I begin, note that expanding a disk with vmkfstools requires the VM be powered off and I am unable to find a way to expand the disk while the VM is powered on even though the vSphere Client allows it so when I do, I’ll update this post.  To shutdown the VM, execute the following command:

vim-cmd vmsvc/power.shutdown vmid

For this example, we’ll execute the following:

vim-cmd vmsvc/power.shutdown 21

With the VM powered down, continue by browsing to the directory of the VM and execute:

du -ah

… to list all the files and the size:

image

The disk we’re going to expand for this example is named -FS-01_3.vmdk which is currently at 1.0G.  The command we’ll use to expand the disk is:

vmkfstools -X <sizeOfVMDK>GB <diskname>.vmdk

… where we’ll need to define the size of the VMDK and the VMDK file name.  Note that the size defined here is the total size so the final size of the VMDK would be so if we use 9GB, it would expand the VMDK to the size of 9GB and not 9 + 1 = 10GB.  Also, the VMDK filename we’ll be using is the FS-01_3.vmdk and NOT the FS-01_3-flat.vmdk file even though that file has a size associated to it.

For this example, we’ll be executing:

vmkfstools -X 9GB FS-01_3.vmdk

image

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Note that you’ll receive the following error if you forget to shut down the VM prior to executing this command:

Failed to extend disk : Failed to lock the file (16392).

image

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Rerun du -ah to see the updated size:

image

Now power on VM with the following command:

vim-cmd vmsvc/power.on vmid

For this example, we’ll be executing:

vim-cmd vmsvc/power.on 21

Proceed and expand the volume in the guest VM’s operating system (i.e. Computer Management):

image

image

Sunday, December 14, 2014

Upgrading Nutanix blades from ESXi 5.1 to 5.5

I don’t usually perform upgrades of Nutanix blocks from 5.1 to 5.5 but I’ve noticed that whenever I do, I’ve always had to reference an email in my Outlook’s Draft folder with the notes and screenshots I’ve captured during a previous upgrade so just in case I needed to reference it in the future, I’m going to go ahead and get a post written on my blog.

The configuration I’ll be upgrading in this example consists of an older Nutanix 2000 series block with 2 blades and a newer Nutanix 7000 series block with one blade running ESXi 5.1.0 build 1065491:

image

image

image

This upgrade was performed a while ago so the version we’ll be upgrading to is VMware ESXi 5.5 build 1331820.

Step #1 – Download Upgrade Package

I find that one of the questions I get asked most from colleagues and clients are that they cannot find the the upgrade package by navigating through VMware’s download section.  The reason for this is usually because they are navigating to an update of version 5.5.0 with the path such as:

Home –> All Downloads –> VMware vSphere –> VMware ESXi 5.5.0 Update 1

The section above provides the following options:

  • ESXi 5.5 Update 1 Driver Rollup 2 (includes VMware Tools)
  • ESXi 5.5 Update 1 Driver Rollup 2 README
  • ESXi 5.5 Update 1 ISO image (includes VMware Tools)
  • ESXi 5.5 Update 1 Offline Bundle

imageimage

None of the above can actually be used to upgrade our blades from ESXi 5.1 to 5.5 because the ESXi 5.5 Update 1 Offline Bundle is only used for updating 5.5.0 to 5.5.0 U1.  Downloading the first package on the list would provide you with ISOs for installing ESXi 5.5U1 on a new blade.  As an example, here’s what the contents of the VMware-ESXi-5.5U1-Rollup_2ISO.iso contain:

image

image

The correct page that you’re looking for is to select the RTM version of 5.5.0 as such:

image

Here is where you will find the ESXi Offline Bundle:

image

The upgrade package should be in the form of a zip package that ends with -depot.zip:

image

image

Opening the zip package should show the following file and folders:

vib20 – Folder
vmv-ESXi-5.5.0-metadata.zip – File
index.xml – File
vendor-index.xml – File

image

Step #2 – Upload Upgrade Package to Datastore

With the upgrade package downloaded, proceed by uploading the package to a datastore accessible by all of the hosts:

imageimage

Step #3 – Connect to CVM to Execute Upgrade

With the upgrade package uploaded, proceed by SSH-ing to any one of the CVMs:

image

… then execute the following:

i in `hostips`;do echo $i && ssh root@$i "esxcli software vib install -d /vmfs/volumes/NutanixNFS/VMware-ESXi-5.5.0-1331820-depot.zip";done

Note that you will need to change the path to the upgrade package (highlighted in red) to reflect the environment you are working in.

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Note that if you receive the following error:

ssh: Could not resolve hostname hosttips: Name or service not known

image

It’s because you need to replace the characters for hostips with the symbol beside the 1 key:

for i in 'hostips';do echo $i && ssh root@$i "esxcli software vib install -d /vmfs/volumes/NutanixNFS/VMware-ESXi-5.5.0-1331820-depot.zip";done

--------------------------------------------------------------------------------------------------------------------------------------------------------------

The output of the upgrade should look something similar to the following:

image

image

The output from the SSH session should display the following for each successfully upgraded blade in the cluster:

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

Step #4 (Optional) – Upgrading individual blades

If there is any issue with upgrading a specific blade in the cluster via the command ran in the CVM or if you would like to upgrade one blade to test, it is possible to simply SSH directly to the ESXi host and use the regular esxcli command to upgrade the host as such:

esxcli software vib install -d /vmfs/volumes/NutanixNFS/VMware-ESXi-5.5.0-1331820-depot.zip

image

imageimage

Step #5 – Apply Update Packages

With the hosts upgraded to the latest version, you can proceed with repeating the same process as Step #3 or #4 to apply Update packages downloaded from the appropriate section of the site:

image

Friday, December 12, 2014

Attempting to register a package from a script file (.RSP) on a Wyse Device Manager manager fails with: “Unable to connect to the HTTP Repository”

Problem

You need to move Package Manager Images from one WDM server to another by following the instructions found at this URL:

https://support.wyse.com/OA_HTML/csksxvm.jsp?nSetId=26518&nUsePub=NO&jttst0=6_23871,23871,-1,0,&jtfm0=&etfm1=&jfn=ZGAC0F217415CC7186A83C4B85C3E8044B1347842403EDE204F64AE27372AEDC74F923EF3C8412DD9F0B17F866C966EE089B&oas=hPgVmjwPoeUVu6hGtLpgiA

image

image

image

image

image

image

… but notice that you are unable to register the image because it fails with:

Unable to connect to the HTTP Repository

image

Proceeding to use the Test Connection feature for the Software Repository as such:

image

… fails with the error:

Name: HTTP Connection
Progress: Test Testing Complete
Status: Failed

image

Clicking on the Errors tab displays the following error:

[Error:Not Available] [WebDAV error:Failed to upload buffer. HTTP Response Code: 401Remote File Path: {A33AADB7-443F-4113-A478-D5A07E5518E0}.tmp Error Code: 401]

image

Solution

While there could be various reasons why this error would be thrown, one of the users is if the rapport service account password has expired and requires a new password to be set.  To determine this, launch the Computer Management console on the Wyse Device Manager server and navigate to the Local Users and Groups –> Users node, then open the rapport service account properties:

image

Notice that the User must change password at next login is checked which means the password has expired and must be changed.  Proceed by unchecking the checkbox and check the Password never expires checkbox:

image

Run Test Connection again for the Software Repository to ensure that the test completes successfully:

image

You should now be able to register the image.

Tuesday, December 9, 2014

Notes when working with Microsoft CA (Certificate Authority) services

I recently had to work with a Microsoft support engineer to troubleshoot an Active Directory Certificate Services issue and managed to write down a few commands he used during the troubleshooting process and thought I’d write a blog post so I can refer to them in the future since I’m bound to forget.

Command to collect CA information CRL:

Before starting the troubleshooting, the engineer ran the following commands to collect base information.

The following certutil command lists the Enterprise Root or Subordinate CAs found in Active Directory:

C:\>certutil

Entry 0:

Name: `test'

Organizational Unit: `'

Organization: `'

Locality: `'

State: `'

Country/region: `'

Config: `conspschsrv.contoso\test'

Exchange Certificate: `'

Signature Certificate: `'

Description: `'

Server: `conspschsrv.contoso'

Authority: `test'

Sanitized Name: `test'

Short Name: `test'

Sanitized Short Name: `test'

Flags: `1'

Web Enrollment Servers: `'

Entry 1:

Name: `CONTOSO ROOT CA'

Organizational Unit: `'

Organization: `'

Locality: `'

State: `'

Country/region: `'

Config: `conCERTSRV.contoso\Contoso ROOT CA'

Exchange Certificate: `'

Signature Certificate: `'

Description: `'

Server: `conCERTSRV.contoso'

Authority: `CONTOSO ROOT CA'

Sanitized Name: `CONTOSO ROOT CA'

Short Name: `CONTOSO ROOT CA'

Sanitized Short Name: `CONTOSO ROOT CA'

Flags: `1'

Web Enrollment Servers: `'

Entry 2: (Local)

Name: `CONTOSO-Sub-CA'

Organizational Unit: `'

Organization: `'

Locality: `'

State: `'

Country/region: `'

Config: `conSCASRV01.contoso\CONTOSO-Sub-CA'

Exchange Certificate: `'

Signature Certificate: `conSCASRV01.contoso_CONTOSO-Sub-CA.crt'

Description: `'

Server: `conSCASRV01.contoso'

Authority: `CONTOSO-Sub-CA'

Sanitized Name: `CONTOSO-Sub-CA'

Short Name: `CONTOSO-Sub-CA'

Sanitized Short Name: `CONTOSO-Sub-CA'

Flags: `13'

Web Enrollment Servers: `'

CertUtil: -dump command completed successfully.

Note that the above output shows that there are currently 3 Enterprise CAs that is found in Active Directory.

-------------------------------------------------------------------------------------------------------------------------------------------------------

The net config rdr command shows how the redirector or workstation is currently configured on your computer.  The following output is from a command prompt on the Enterprise Subordinate CA:

C:\>net config rdr

Computer name \\conSCASRV01

Full Computer name conSCASRV01.contoso

User name TLuk

Workstation active on

NetBT_Tcpip_{4C336E8C-128B-4B8F-99EC-BFD21F1B9E6D} (005056A10F07)

Software version Windows Server 2012 Standard

Workstation domain con-CONTOSO

Workstation Domain DNS Name contoso

Logon domain con-CONTOSO

COM Open Timeout (sec) 0

COM Send Count (byte) 16

COM Send Timeout (msec) 250

The command completed successfully.

-------------------------------------------------------------------------------------------------------------------------------------------------------

The following nltest /dclist:contoso command lists the domain controllers found in Active Directory:

C:\>nltest /dclist:contoso

Get list of DCs in domain 'contoso' from '\\DC2.contoso'.

DC2.contoso [DS] Site: Default-First-Site-Name

DC3.contoso [DS] Site: Default-First-Site-Name

DC1.contoso [PDC] [DS] Site: Default-First-Site-Name

The command completed successfully

C:\>

-------------------------------------------------------------------------------------------------------------------------------------------------------

Command to dump certificate information CRL:

The support engineer suspected that there was something wrong with the published CRL so the following command was executed to check the start and end validity date of the file (I’ve highlighted this information in red:

C:\>certutil -dump Contoso-Root-CA.crl
X509 Certificate Revocation List:
Version: 2
Signature Algorithm:
    Algorithm ObjectId: 1.2.840.113549.1.1.10 RSASSA-PSS
    Algorithm Parameters:
    30 00
Issuer:
    CN=Contoso-Root-CA
  Name Hash(sha1): b8e12ef9872c9e93d65fbd132a2402fa4b0d0dde
  Name Hash(md5): 9fe720a14c52866f8ffb9819659eb114

ThisUpdate: 11/30/2013 3:12 PM
NextUpdate: 11/30/2014 3:32 AM
CRL Entries: 0
CRL Extensions: 5
    2.5.29.35: Flags = 0, Length = 18
    Authority Key Identifier
        KeyID=b2 be 9e 66 df 55 a8 33 b7 72 00 77 a5 2a e5 19 68 6a d9 90

    1.3.6.1.4.1.311.21.1: Flags = 0, Length = 3
    CA Version
        V0.0

    2.5.29.20: Flags = 0, Length = 3
    CRL Number
        CRL Number=02

    1.3.6.1.4.1.311.21.4: Flags = 0, Length = f
    Next CRL Publish
        Saturday, November 29, 2014 3:22:16 PM

    1.3.6.1.4.1.311.21.14: Flags = 0, Length = cb
    Published CRL Locations
        [1]Locations
             Distribution Point Name:
                  Full Name:
                       URL=ldap:///CN=Contoso-Root-CA,CN=BHLRCASRV01,CN=CDP,CN=Pub
lic Key Services,CN=Services,CN=Configuration,DC=contoso,DC=domain,DC=bm?certificateR
evocationList?base?objectClass=cRLDistributionPoint (ldap:///CN=Contoso-Root-CA,CN
=BHLRCASRV01,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=contoso
,DC=domain,DC=bm?certificateRevocationList?base?objectClass=cRLDistributionPoin
t)

Signature Algorithm:
    Algorithm ObjectId: 1.2.840.113549.1.1.10 RSASSA-PSS
    Algorithm Parameters:
    30 00
Signature: UnusedBits=0
    0000  cb e3 84 68 86 fc af 5c  83 6a 1d 5a ef f8 74 11
    0010  ae d8 33 c3 0f cf 99 f9  24 d7 47 d6 90 09 24 a9
    0020  c9 06 c0 4b 3e 91 bc 96  5b cc ba 87 d5 72 68 1c
    0030  60 a7 fc a3 a1 6c 00 b5  5a 76 23 a4 29 89 fe bf
    0040  0d 9c 0f 74 57 15 9a 05  39 0f a3 05 39 a5 08 33
    0050  c5 b2 31 3d 16 b2 69 dd  16 62 09 db d3 d0 37 db
    0060  65 77 bc 4e 06 84 40 0d  e9 35 7c 03 c3 a0 36 4a
    0070  57 6c c8 3f 9b 07 b2 16  f1 16 43 46 de 27 96 7e
    0080  5b 32 e9 6d 24 eb 58 8d  1f 0f e5 ea 41 e4 64 e7
    0090  66 8d 46 f0 0c 4c 3c ab  13 12 da 3a f3 ac 2f cb
    00a0  37 80 9f a5 1d 6d bf db  cc 2e 48 ae 00 b1 56 83
    00b0  e3 0e 02 9a 29 e4 55 58  22 4b a1 ba 0f 1f bd 14
    00c0  3e 0b 37 dc af b3 35 33  48 ba 1c bc b5 6f 22 47
    00d0  49 99 d5 ac e5 68 fd 4f  30 7e ba 63 25 d9 6f 3e
    00e0  0c 3e 26 84 0b 29 6c d4  76 7c 44 8f 2b 71 a0 e9
    00f0  80 71 e0 2f 11 ec e7 02  62 3e 60 05 3a 27 bb 3d
CRL Hash(md5): 40 65 78 62 9a 86 fa 9f fb 77 57 43 e1 bc 23 66
CRL Hash(sha1): 26 92 02 b6 8e ab 26 41 b3 17 18 76 3f c5 41 45 a6 75 54 5e
CRL Hash(sha256): 6da08e776a532e6b92d1ebe7ffd477606f37b7d3659361661c50b8be9919cc
c3
Signature Hash: 438535ec7850261962dff88f6b73c487ae45633c
CertUtil: -dump command completed successfully.

The CRL file this command references can be found in the following directory:

C:\Windows\System32\certsrv\CertEnroll

Note that if this is too short, the period can be extended by launching the Certificate Authority administration console:

image

… right clicking on the Revoked Certificates and selecting properties:

image

… then changing the CRL publication interval to the desired duration:

image

-------------------------------------------------------------------------------------------------------------------------------------------------------

Command to publish certificate to LDAP

Updating the CRL file for http access is simply copying the CRL file from the Root CA’s C:\Windows\System32\certsrv\CertEnroll folder to the same folder on the Subordinate CA but to update the one published through LDAP, the command certuitl -dspublish <Your-Root-CA>.crl <YourSubOrdinateCAServerName is ran on the subordinate CA:

C:\>certutil -dspublish Contoso-Root-CA.crl CONSCASRV01
ldap:///CN=Contoso-Root-CA,CN=BHLSCASRV01,CN=CDP,CN=Public Key Services,CN=Service
s,CN=Configuration,DC=contoso,DC=bhl,DC=bm?certificateRevocationList

Base CRL added to DS store.

CertUtil: -dsPublish command completed successfully.

----------------------------------------------------------------------------------------------------------------------------------------------------------

Verifying published CRL paths:

To verify the published CRL paths, simply export the subordinate CA’s certificate without the private key to a DER format file:

image

Then launch the URL Retrieval Tool by executing certuilt -url <exportedCA>.cer and use the Retrieve button with CRLs (from CDP) selected to verify the accessibility and validity of the files:

image

-------------------------------------------------------------------------------------------------------------------------------------------------------

Determining which certificate the current CA is using:

If the certificate for the CA has been renewed several times, you may end up seeing a list similar to the following:

image

While most of us would probably just look for the certificate with the latest expiry date, another way to do it is to open up the Registry Editor and navigate to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration\<YourCAName>

The CACertHash key can be used to identify which certificate is currently in use:

image

… by comparing it to the certificate’s Thumbprint as such:

image