Menu

Virtual Geek

Tales from real IT system administrators world and non-production environment

POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

July 20, 2017 08:45AM

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

In earlier chapter I shown how to migrate VM and test storage multipath using vSphere web client, here I will perform same task using commands using VMware Powercli. For this I have reverted all the setting. To setup powercli check my earlier article VMWARE VSPHERE POWERCLI INSTALLATION AND CONFIGURATION STEP BY STEP

Here I have stored specific information about esxi host, virtual machine, and datastore in the there respective powershell variable. (I can view information about all inventory by running just Get-VMhost, Get-VM, Get-Datastore)
$vmhost = Get-VMhost Esxi001.vcloud-lab.com
$vm = Get-VM winxp001
$Datastore = $VMhost | Get-Datastore Disk1_Tier3

I want to know how much is the free space left and multipath policy about selected datastore Disk1_Tier3 can be fetched using next one liner commands.
$SCSILun = $VMhost | Get-ScsiLun -LunType Disk | Where-Object {$_.CanonicalName -eq $Datastore.extensiondata.info.vmfs.extent.Diskname}
$SCSILun | Select-Object @{N='Name'; E={$Datastore.Name}}, CanonicalName, CapacityGB,@{N='FreeSpaceGB'; E={$Datastore.FreeSpaceGB}}, @{N='MountPath'; E={$Datastore.extensiondata.Info.Url}}, MultipathPolicy

vmware vcenter vsphere powercli get-vmhost, get-vm, Get-datastore, Get-scsilun multipathPolicy, CanonicalName, extent, extensiondata, freespacegb, capacityGB datastore storage, vmfs example

I am changing multipath policy of selected datastore using command.
$SCSILun | Set-ScsiLun -MultipathPolicy RoundRobin

To view multipath status of vmhost datastore use next command, and they looks good and all active.
($SCSILun | Get-ScsiLunPath).ExtensionData | Select-Object Name, PathState

vmware powercli vsphere vcenter esxi, datastore storage lun, set-scsilun multipathpolicy roundrobin rr, fixed, mru, most recently used, psp, path selection policy satp, Get-ScsilunPath, pathstate, runtime, devicename, naa

To know IP address of virtual machine using powercli and virtual harddisk location use commands, If you see all the commands from top to bottom of this articles are connected.
$vm.ExtensionData.Guest.IpAddress
$vm | Get-HardDisk

vmware vcenter esxi powercli vm extensiondata guest ipaddress, get-harddisk, filename datastore location

I have gathered all the information and changed multipathing information as well, Now for final step I am storage vmotion VM with thin provisioned disk. Once my command is executed successfully, I can verify VM location with command $vm | Get-HardDisk.
$vm | Move-VM -Datastore Disk1_Tier3 -DiskStorageFormat Thin

Side by side while storage migration I started ping (Test-connection) to VM, I didn't get any ping loss, also I simulated physical adapter connected to storage failure by removing cable from esxi server. and checked the multipathing status as expected VM is intact and 2 paths are dead, No downtime on the VM.
($SCSILun | Get-ScsiLunPath).ExtensionData | Select-Object Name, PathState

vmware vsphere esxi vcenter powercli, move-vm datastore storage vmotion, diskstorageformat thin, migrate, Get-ScsiLunPath, extensiondata, Pathstate dead, active, test-connection -count ping vm, port binding

VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING

July 18, 2017 09:02AM

Series PartsSeries PartsSeries Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

Following by my earlier chapters, My storage is setup correctly, Now I will storage vmotion, migrate virtual machines on this newly deployed datastore. Before I would like to show you connectivity and multipathing options under Datastore view >> Datastore >> Configure Tab. Here I see the mounted datastore (Device ID with NAA id), click Edit Multipathing button and change Path selection policy to Round Robin (VMware). I see paths on the datastore are in green status means OK. (Changing PSP in runtime has no issues)

By default, VMware provisions the next path selection policies. If you have a third-party PSP deployed on your host, its policy also appears on the list. (ie: Third-Party PSP example EMC Powerpath), before changing changing paths selection policy (PSP), consult your storage team as well as vendor and documentation guide for best performance and practices.

Fixed (VMware)
The host uses the chosen preferred path, if it has been configured. Otherwise, it selects the first operational path learnt at system boot period. If you want the host to use a specific preferred path, specify it manually. Fixed is the default policy for most active-active storage devices.
Note: If the host uses a default preferred path and the path's status turns to Dead, a new path is selected as preferred. However, if you explicitly designate the preferred path, it will remain preferred even when it becomes inaccessible.

MRU or Most Recently Used (VMware)
The host chooses the path that it used most recently. When the path becomes inaccessible, the host selects another path. The host does not return back to the original path when that path becomes accessible again. There is no preferred path setting with the MRU policy. MRU is the default policy for most active-passive storage devices.

Round Robin (VMware)
The host uses an automatic path selection algorithm rotating through all active paths when connecting to active-passive arrays, or through all available paths when connecting to active-active arrays. RR is the default for a number of arrays and can be used with both active-active and active-passive arrays to implement load balancing across paths for different LUNs.

vmware vsphere vcenter esxi datastore configure connectivity and multipathing details, policies,  mounted datastore, Round robin, fixed, MRU most recently used, runtime name, preferred path,nmp, native multipathing path

Once I know all storage paths are in optimal status, I will migrate VM on this new datastore, In the summary I see hard disk are located on old datastore. Also note the IP address. To view IP on summary tab, you must have installed VMware tools in Virtual Machine and IP assigned.

vmware vsphere vcenter esxi VM hardware summary cpu, hdd, memory, capacity, Harddisk datastore network change, dns, storage migration, IP addresses virtual machine

Before starting SvMotion/ storage migration, I will ping virtual machine IP continuously to see migration effect, To migrate right click virtual machine, click Migrate, Select change storage only. (Migrate the virtual machines' storage to a compatible datastore or datastore cluster), click next.

vmware vcenter vsphere VM Virtual  machine, migrate, storage vmotion, compute resource, No ping loss, IP address, summary, DNS name, compatibilty

Under select storage migration option select datastore from list (shows visible and compatible disk with host), verify  once moved VM it will have enough disk space. If selected datastore is good for migration it shows compatibility checks succeeded with green check mark. (Most of the time it might not succeed due to in sufficient space), click next. (I am changing select virtual disk format to Thin provision to save some space, hard disk will grow as data is filling inside VM. another thing I have option here in advanced to move individual vm hard disk to migrate)

VMware vsphere esxi vcenter vm virtual machine migrate storage datastore vmotion, select datastore, compatible, vmfs advanced disk format thin, thick eager zeroed, thick lazy zeroed, VM storage policies

This is last screen and ready to complete, review changes and click finish.

vmware vsphere esxi vcenter migrate, vmotion ready to complete select storage datastore, disk formation thin, virtual machine, migration type, compute resource

I see relocate virtual machine is successful in recent tasks. I reviewed my ping test, there was no ping loss (You might see 2-3 ping loss, its depending on your storage and networking latency performance). This is done without downtime.

VMware Virtual machine after vmotion storage migration no ping loss connectivity summary vm hardware status

Next I will simulate vmware esxi physical adapter failure by removing or shutting down switch. below are the before and after screenshot. In physical nic adapters, vmnic3 is connected and showing speed, and after pulling cable check the status again on vmnic3 its down and disconnected.

vmware vsphere vcenter esxi, physical network adapter cdp, lldp, down connected disconnection simulat, vmnic status down, troubleshoot failure

Lets see how connectivity and multipathing information looks like under selected datastore, My two paths are down and showing dead (degraded), but VM is still running. to know more about runtime name also called storage device naming read vmware pub document

In the vSphere Client, each storage device, or LUN, is identified by several names, including a friendly name, a UUID, and a runtime name.
Name
     This is a friendly name that the ESX host assigns to a device based on the storage type and manufacturer. You can modify the name using the vSphere Client. When you modify the name of the device on one host, the change takes affect across all hosts that have access to this device.
Identifier
     This is a universally unique identifier assigned to a device. Depending on the type of storage, different algorithms are used to create the identifier. The identifier is persistent across reboots and must be the same for all hosts sharing the device.
Runtime Name
     This is the name of the first path to the device. The runtime name is created by the host, is not a reliable identifier for the device, and is not persistent.
The runtime name has the following format: vmhba#:C#:T#:L#.

vmhba#
     The name of the storage adapter. The name refers to the physical adapter on the host, not to the SCSI controller used by the virtual machines.
C#
     The storage channel number. Software iSCSI initiators use the channel number to show multiple paths to the same target.
T#
     The target number. Target numbering is decided by the host and might change if there is a change in the mappings of targets visible to the host. Targets that are shared by different ESX hosts might not have the same target number.
 L#
     The LUN number that shows the position of the LUN within the target. The LUN number is provided by the storage system. If a target has only one LUN, the LUN number is always zero (0).

For example, vmhba1:C0:T3:L1 represents LUN1 on target 3 accessed through the storage adapter vmhba1 and channel 0.

VMware vsphere web client esxi vcenter datastore view storage, configure, connectivity and multipathing mounted NAA vmfs volumes, dead active target lun status.png

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING

POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE

July 13, 2017 08:45PM

This blog article is similar to my previous notes VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE. Here I am using VMware Powercli instead of vSphere web client to configure iScsi storage adpater and add new datastore. As this is complete powercli article I will not use GUI at all.

I am configuring my first Esxi server. Storing its information in variable called $VMhost.
$VMhost = Get-VMhost Esxi001.vcloud-lab.com

Next install and enable iSCSI software adapter.
$VMhost | Get-VMHostStorage | Set-VMHostStorage -SoftwareIScsiEnabled $True

vmware vsphere esxi powercli add enable iscsi storage adapter vmhba, host bus adapter, get-vmhost, get-vmhoststorage, Set-vmhoststorage, SoftwareIScsiEnabled

Before processing further first I will do ping and telnet test on esxi server to storage and vice versa, As this is complete powercli article, for this I will enable SSH service on Esxi server using below command.
$VMHost | Get-VMHostService | ? {$_.Key -eq 'TSM-SSH'} | Start-VMHostService -Confirm:$false

Remotely vmware vsphere esxi Get-VMHostservice TSM-SSH, Where-Object enable SSH powercli Start-VMhostService -confirm

Putty to esxi server. The telnet command is not available in any versions of ESXi and, therefore, you must use netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of the nc command is: nc -z <destination-ip> <destination-port>, I am telneting port 3260 and succeeded means good (This is also a guide how to telnet in vmware esxi)

nc -z 172.20.1.101 3260
Connection to 172.20.1.101 3260 port [tcp/*] succeeded!

To use telnet command on Microsoft Windows telnet client needs to be installed with PowerShell command Install-WindowsFeature Telnet-Client. run telnet IP port, prompt is changed means I can go ahead with further configuration.

Install-windowsFeature telnet-client, esxi telnet 3260 iscsi, nc, netcat putty esxi ssh

After enabling iScsi software adapter I need iSCSI IQN name, which I will need to add on storage server in initiator list.
$VMhost | Get-VMHostHba -Type iScsi | Select-Object Name, Status, IScsiName

vmware powercli Get-VMHostHba -type iscsi and select-object iscsiname vmhba online

Below command is ran on Microsoft iSCSI target, in the Initiator list, without this Esxi will not able to view or access LUN. Refer this article POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
Set-IscsiServerTarget -TargetName Boot-Esxi001 -InitiatorIds iqn:iqn.1998-01.com.vmware:esxi001-5abe60a5

Microsoft iSCSI Target server set-iscsiServertarget -targetName -InitiatorId for hyperv vmware esxi

Add storage target IP in the software iScsi storage adapter static discovery. (Get-VMHostHba -Type iscsi parameter shows software adapter)
$VMhost | Get-VMHostHba -Type iScsi | New-IScsiHbaTarget -Address 172.20.1.101

Next is needed devicename of iSCSI VMkernel adapter configured earlier. Ref. article POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE, with using Where-Object filtering on Get-VMHostNetworkAdapter -VMkernel I easily get the needed DeviceName. I got two results vmk1 and vmk2. note them down.
$VMhost | Get-VMHostNetworkAdapter -VMKernel | ? {$_.PortGroupName -match 'iSCSI'} | select Devicename

add target ip and information vmware vsphere powercli configure vcenter esxi get-vmhosthba -type iscsi, new-IscsiHbaTarget -address, get-VMHostNetworkAdapter -vmkernel portgroupname, deviceName vmk1.png

After getting information about VMkernel port name, I need device name of vmhba using:
$VMHost | Get-VMHostHba -Type IScsi | Select Device

Next step use EsxCli and add vmkernel ports to network port binding,
$esxcli = $VMhost | Get-EsxCli
$esxcli.iscsi.networkportal.add('vmhba65',$Null,'vmk1')
$esxcli.iscsi.networkportal.add('vmhba65',$Null,'vmk2')

Once everything is successful rescan for HBA and VMFS on esxi server.
$VMhost | Get-VMHostStorage -RescanAllHba -RescanVmfs

vmware powercli configure iscsi rescan stroage device, refresh get-vmhosthba -type iscsi, device, $esxcli vmhost, get-esxcli, $esxcli.iscsi.networkportal.add,  rescanall, rescanvmfshba

If above steps are successful, It requires NAA id of Free LUN for last step, to find free lun follow this article FIND FREE OR UNASSIGNED STORAGE LUN DISKS ON VMWARE ESXI SERVER.
Get-FreeEsxiLUNs -Esxihost Esxi001.vcloud-lab.com

vmware vsphere powercli connect-viserver get-freeEsxiLUNs, shows or find free esxi lun remote storage disk

Once NAA id is rectified use New-Datastore command to format LUN with VMFS type, I am using complete space here.
$VMhost | New-Datastore -Name Disk1_Tier3 -Path naa.60003ff44dc75adc9601165db783a4ac -Vmfs

vmware vsphere esxi vmhost, new-datastore tier -path naa find, capacitygb and freespaceGB

If you add another Esxi host adding datastore is not required, This formatted datastore will be automatically be visible once configuring VMKernel ports and iSCSI software adapter. New-Datastore need to run once only for newly presented storage disk.

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

FIND FREE OR UNASSIGNED STORAGE LUN DISKS ON VMWARE ESXI SERVER

July 13, 2017 07:35PM

This is one of my small script. And I use it to find free or unassigned unforrmatted LUNs/Disks on Esxi server. When Lun or disk from storage (most probably remote) assigned and presented to Esxi server, this LUN doesn't have any partition, and unformatted as a raw disk, This is visible when you try to add new datastore. Same list I can pull using VMware Powercli script. Script is combination of two cmdlets Get-Scsilun and Get-Datastore and processes data.

This require VMware powercli installer or module, follow this article to setup it VMWARE VSPHERE POWERCLI INSTALLATION AND CONFIGURATION STEP BY STEP. Login to esxi or vCenter with Connect-VIServer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
function Get-FreeEsxiLUNs {
    ##############################
    #.SYNOPSIS
    #Shows free or unassigned SCSI LUNs/Disks on Esxi
    #
    #.DESCRIPTION
    #The Get-FreeEsxiLUNs cmdlet finds free or unassigned SCSI LUNs/Disks on VMWare Esxi server. Free or unassigned disks are unformatted LUNs and need to format VMFS datastore or use as RDM (Raw Device Mapping)
    #
    #.PARAMETER Esxihost
    #This is VMware Esxi host name
    #
    #.EXAMPLE
    #Get-FreeEsxiLUNs -Esxihost Esxi001.vcloud-lab.com
    #
    #Shows free unassigned storage Luns disks on Esxi host name Esxi001.vcloud-lab.com
    #
    #.NOTES
    #http://vcloud-lab.com
    #Written using powershell version 5
    #Script code version 1.0
    ###############################

    [CmdletBinding()]
    param(
        [Parameter(Position=0, Mandatory=$true)]
        [System.String]$Esxihost
    )    
    Begin {
        if (-not(Get-Module vmware.vimautomation.core)) {
            Import-Module vmware.vimautomation.core
        }
        #Connect-VIServer | Out-Null
    }
    Process {
        $VMhost = Get-VMhost $EsxiHost
        $AllLUNs = $VMhost | Get-ScsiLun -LunType disk
        $Datastores = $VMhost | Get-Datastore
        foreach ($lun in $AllLUNs) {
            $Datastore = $Datastores | Where-Object {$_.extensiondata.info.vmfs.extent.Diskname -Match $lun.CanonicalName}
            if ($Datastore.Name -eq $null) {
                $lun | Select-Object CanonicalName, CapacityGB, Vendor        
            } 
        }
    }
    End {}
}

#Get-FreeEsxiLUNs -Esxihost <Esxi001.vcloud-lab.com>

After login, copy paste above script on powershell console (powercli), Run new function command with Get-FreeEsxiLUNs -Esxihost <VcenterEsxi_IP_FQDN>, It show the list of free luns and I have only one at the moment.

vmware vsphere powercli connect-viserver get-freeEsxiLUNs, shows or find free esxi lun remote storage disk

VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE

July 12, 2017 06:17PM

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

As per below diagram I have setup Microsoft Storage Target server and Esxi Networking VMKernel ports, This is last part of series and configuring ESXi software iSCSI adapter for adding storage, here I will be adding and enabling software iSCSI adapter. (It is a client and also called initiator), Same steps can be performed using VMware Powercli and it will be a subject of my next article.

VMware vsphere vcenter esxi software iscsi storage configuration diagram, multipathing 3260 port, iqn iscsi initiator configuration

Make sure you have proper connectivity between Esxi iSCSI VMKernel adapters and Storage Target, I am checking connectivity on ESXi as well as Storage server by pinging each other. If SSH is not enabled on Esxi server go to Configure tab >> System Security Profile >> scroll down and Edit Services >> Select SSH >> Services Details on the status and start service. I am using putty tool to login to Esxi server. esxcfg-vmknic -l command lists all VMKernel adapter. I have highlighted both iSCSI vmkernel adapters, I specifically need their Interface names.

vmkping -I vmk1 172.20.1.101
vmkping -I vmk2 172.20.1.102

-I is Interface, vmkping will us selected interface only to ping remote IP. It looks good from Esxi Server that connectivity is good. Next do the same ping testing from Microsoft Storage Target server and try to reach Esxi server, using powershell command Test-Connection alternative to ping.

Test-Connection -Source 172.20.1.101 -Destination 172.20.1.10
Test-Connection -Source 172.20.1.102 -Destination 172.20.1.11

-Source is the interface on storage server and -Destination is Esxi server. Pinging is successful (if ping fails you will see error on the screen with red text). I can also check connectivity using telnet port 3260, If you are facing any issue check at networking for VLAN and MTU settings (remove them if required and check), Same steps are useful for first basic troubleshooting, Most of the time I have resolved my issues with the same steps.

check connective between esxi and storage, Vmware vSphere esxi iscsi connection session vmkping, esxcfg-vmknic -l, ping get-netIPAddress, test-connection source destination, alternative ping successful

By default Software iSCSI adapter is disabled and not installed on the ESXi server, It can be added by selecting Esxi Server >> Configure Tab >> Storage >> Storage Adapters >> click + green plus sign >> Software iSCSI adapter.

It shows popup message A new software iSCSI adapter will be added to the list. after it has been added, select the adapter and use the adapter details section to complete the configuration, Make note here, only one Software iSCSI hba can be added on esxi server. also it will always have number above 35, in my scenario it is vmhba65.

vmware vcenter esxi add software iscsi storage adapter, software vmhba send targets, static, dynamic discovery

Here select vmhba65 adapter, note the type is iSCSI and status is online, I will require to add IQN identifier on Storage server for access. IQN stands for iSCSI Qualified name. It is somewhat as similar as physical address like MAC address and unique. Communication will taken care on 3260 port, by default when iSCSI software adapter is enabled or added it opens and allow port no 3260 in ESXi security profile automatically. Next I added this ESXi's IQN on Microsoft storage server. This table show how IQN name is made of.

iqn.yyyy-mm.naming-authority:unique name -- iqn.1998-01.com.vmware:esxi001-5abe60a5
IQN (iSCSI qualified name) Can be up to 255 characters long
yyyy-mm is the year and month when the naming authority was established.
naming-authority s usually reverse syntax of the Internet domain name of the naming authority. For example, the iscsi.vmware.com naming authority could have the iSCSI qualified name form of iqn.1998-01.com.vmware.iscsi. The name indicates that the vmware.com domain name was registered in January of 1998, and iscsi is a subdomain, maintained by vmware.com.
unique name is any name you want to use, for example, the name of your host. The naming authority must make sure that any names assigned following the colon are unique
EUI (extended unique identifier) Includes the eui. prefix, followed by the 16-character name. The name includes 24 bits for the company name assigned by the IEEE and 40 bits for a unique ID, such as a serial number.
 

Add iSCSI qualified name IQN initiators in the iSCSI Storage server Target

I am adding one Target (Storage Server) IP address in Dynamic Discovery (just need to add one of the IP and it will detect Targets IQN and other IPs, available paths automatically). Here it supports CHAP authentication, but for simplicity I am not touching it. Advanced options can be added or changed as per vendor recommendations and best practices documents.

vmware vcenter esxi add software iscsi storage adapter, Enable ssh and other services, Advanced configuration, Storage Properties, software vmhba send targets, static, dynamic discovery,  esxi security profiles, iscsi port 3260

Next Configure storage network port binding. Port binding is used in iSCSI when multiple VMkernel ports for iSCSI reside in the same broadcast domain and IP subnet to allow multiple paths to an iSCSI array that broadcasts a single IP address. When using port binding, you must remember that:

  • Array Target iSCSI ports must reside in the same broadcast domain and IP subnet as the VMkernel port.
  • All VMkernel ports used for iSCSI connectivity must reside in the same broadcast domain and IP subnet.
  • All VMkernel ports used for iSCSI connectivity must reside in the same vSwitch.
  • Currently, port binding does not support network routing.

Here VMware has provided very nice article on storage network port binding, It is a must read article. Select network port binding tab in the bottom on iSCSI vmhba, click + green plus button. and select iSCSI network vmkernel adapters.

vmware vsphere esxi configure storage adapter, iSCSI software adapter vmhba rescan, iqn, Network port binding, add bind vmkernel network adapter with vmhba software

You will see warning, Due to recent configuration changes, a rescan of this storage adapter is recommended. So for next step click rescan button in the red circle. It Rescans the host's storage adapter to discover newly added storage devices, detects all provided remote LUNs. Check the paths tab, If calculated Two iSCSI dedicated NICs on esxi server and two NICs on storage total 4 paths available correctly.

vmware vsphere esxi, configure storage adapters iscsi software adapter, rescans the hosts storage hba to discover storage devices, multiple paths, devices raw vmfs disk available

So far it all looks good to me. In the next step add and format assigned LUN as VMFS datastorage. To add remote storage disk/LUN  on esxi go to configure tab, from Left pane expand Storage and click datastores, click the icon (red circled) to create a new datastore. New wizard opens, Select VMFS to create it and click next.

VMware vSphere esxi vcenter Configure Storage, Datastore create a new datastore storage VMFS, NFS, VVOL from disk lun.png

In the Name and device selection give datastore name, for best practices this name should always be matching with LUN disk on the Storage. You can use Name Naa id to correctly identify LUN disk selection, by practicing this you can avoid any accident like formatting incorrect LUN disk. In production specially with EMC devices I always ask storage team Datastore name and associated NAA id to identify disks and avoid any mis-configuration or mistakes. With the name I can view information like LUN Number, Capacity, Hardware acceleration, Drive type (HDD or SSD) and new sector format.

NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique Identifier. The number is guaranteed to be unique to that LUN. The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device.

Next is VMFS Version as my all future hosts version going to be above 6.5, I am choosing VMFS 6 version, with this I get below feature. 

  • Support for 4K Native Drives in 512e mode
  • SE Sparse Default
  • Automatic Space Reclamation (UNMAP) 
  • Support for 512 devices and 2000 paths (versus 256 and 1024 in the previous versions)
  • CBRC aka View Storage Accelerator

Make decision appropriately before going for VMFS 6 as it is not compatible with earlier version of esxi and host might not be able to access the datastores. Unfortunately it is not possible to perform an in-place upgrade from VMFS 5 to VMFS 6. To upgrade, you have to perform the following steps:

  • Unmount the datastore from all ESX hosts.
  • Delete the datastore formatted with VMFS 5.
  • Create a new datastore with the VMFS 6 file system using the same LUN.

Depending on your environment and how much space you have available on your array, this can be a long and painful migration.

vmware vsphere esxi create a new VMFS datastore, Name and device selection, LUN disk, datastore name and device selection , sector format 512e, hardware acceleration, vmfs 6 version automatic space reclaimation support

Next screen is Partition configuration, I am using all available partition and whole datastore size, Block size is by default 1 MB, Specify granularity for the unmap operation. Unmap granularity equals the block size, which is 1 MB. Storage sectors of the size smaller than 1 MB are not reclaimed. There are two options for space reclamation policy. Low (default), Processes the unmap operations at a low rate. Select None option if you want to disable the space reclamation operations for the datastore, Reclamation is also possible using commands. If everything looks good review the changes in Ready to complete and click finish.

vmware esxi vsphere, storage, vmfs datastore details, partition configuration configuration and layout, datastore size, block size, space reclamation granularity, priority, free space capacity.png

Here my datastore is ready to deploy VMs. It is always a best practice to test the configured item so I will be demonstrating and removing cable from one of the physical NIC adapter to simulate switch or network card failure and will watch how storage multiple path effects to see what happens to VM on the datastore.

Datastores in esxi after adding as VMFS 6, datastore version, datastore cluster, capacity, free, status normal

If you add another Esxi host adding datastore is not required, This formatted datastore will be automatically be visible once configuring VMKernel ports and iSCSI software adapter. add datastore need to run once only for newly presented storage disk, also once LUN is presented from storage to Esxi always to detect it correctly rescan adapter as well as vmfs.

POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE

July 10, 2017 06:42PM

In earlier chapter I created new VMKernel adapter using vSphere Web client GUI, Here I am going to perform same steps but using vmware powercli, If you are new to PowerCLI, need to set it up, I am suggesting check my another blog VMWARE VSPHERE POWERCLI INSTALLATION AND CONFIGURATION STEP BY STEP. Open PowerCLI,  Login to the vCenter server or Esxi with commandlet Connect-VIServer vCenterIP_Or_FQDN. Next command is select the ESXi host and store its information in variable, so I can use it again and again for creation of vSwitch and VMKernel adapter. I have provided screenshot to show how powercli commands are mapped to vSphere web client GUI.

$VMhost = Get-VMhost Esxi001.vcloud-lab.com

vsphere vmware powercli connect-viserver, variable, get-vmhost, vcloud-lab.com, vmware web client login to esxi, and vsphere vcenter add vmkernel and standard virtual vswitch

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

As this is complete PowerCLI based article I will see how my esxi physical nic adapters are used by vSwitch and which are free. To show the complete list the of Physical VMnics, I am using $vmhost variable and keep using it with piping (combining other cmdlets) for next commands, as It has all the information related to selected host.
$vmhost | Get-VMHostNetworkAdapter -Physical |  Select-Object Name

Get-VMHostNetworkAdapter list all the physical nics as well as VMKernel adapters, As I interested in seeing Physical only I am using syntax with same name -Physical. next result is piped into Select-Object to see the VMNic names only.
$vmhost | Get-VirtualSwitch | Select-Object  Name, Nic

Once I know how many physical Nics I have, I can see which Nics are in used by virtual switches with Get-VirtualSwitch. Here make sure you are using correct Physical NIC. You can also view CDP or LLDP information to determine the correct NIC adapter, It doesn't mean if you have free NIC you can connect it anywhere, you also need to review your backend networking for perfect designing. Here I know my vmnic3 is straight away connected with same storage switch, I am selecting it.

VMware Powercli Get-VMHostNetworkAdapter -physical Select-Object -expandProperty, variable Get-VirtualSwitch vmnic information from esxi host

First oneliner cmdlet creates new standard virtual switch, Command is New-VirtualSwitch and configuration syntaxes are -Name is vSwitch2, Physical -Nic is vmnic3 and -MTU 9000.
$vmhost | New-VirtualSwitch -Name vSwitch2 -Nic vmnic3 -Mtu 9000

Once standard virtual switch is created successfully, I am creating new VMKernel adapter port with New-VMHostNetworkAdapter on the same virtual switch named vSwitch2, here Portgroup is equivalent to name of VMkernel PortGroup, Best practice is to use static IP, It also set Mtu to 9000.
$vmhost | New-VMHostNetworkAdapter -PortGroup iSCSI02 -VirtualSwitch vSwitch2 -IP 172.20.1.11 -SubnetMask 255.255.255.0 -Mtu 9000

This is the last command and setting VLAN ID on VMkernel adapter, Command querying existing VMKernel on the esxi server with Get-VirtualPortGroup and edit it using Set-VirtualPortGroup with -VLANId parameter.
$vmhost | Get-VirtualPortGroup -Name iSCSI02 | Set-VirtualPortGroup -VLanId 202

In case if you are getting error make sure Name, Portgroup and Nic are not in used (duplicate) and you have appropriate privileges to perform this task. Confirm changes under Esxi Server>>Configure tab>>Expand Networking>>virtual Switches, For me all looks good.

vmware powercli New-VirtualSwitch -Name -Nic -Mtu, New-VMHostNetworkAdapter -PortGroup -VirtualSwitch  -IP -SubnetMask -mtu, Get-VirtualPortGroup  -Name, Set-VirtualPortGroup -VLanId, Create virtual switch standard, vlan mtu

VMWARE VSPHERE POWERCLI INSTALLATION AND CONFIGURATION STEP BY STEP

July 9, 2017 07:29PM

VMWare vSphere suite (vCenter, Esxi and other products) can be managed using powercli, It is software module installed as same as about to deprecate old vSphere c# client. Prerequisite is you must have Powershell on the client desktop or server from you are managing vSphere, by default Powershell available on microsoft latest OS. There are two options available to install PowerCLI, first method is download vSphere Powercli installer from VMware website, it requires login to VMware site. Installation is very easy and not much interactive. Accept license agreement and Click next until finish.

Install vsphere VMWare Powercli Installer installshiled wizard free license agreement, vsphere vcloud air, vcd powercli, update manager, vrealize operations manager, horizon view, install

find download and launch vmware powercliYou can find VMware PowerCLI icon on desktop or in search bar, lanuch it. It opens Windows Powershell, but in the background it runs the script INITIALIZE-POWERCLIENVIRONMENT.PS1 located under location  %program files%\VMware\Infrastructure\vSphere Powercli\Scripts, and loads (Imports) all the required vmware module, These modules can be imported individually as per requirement which I will be showing later in this article when showing second installation method. Incase if you are facing issue and seeing error on the powershell console I suggest you to check below article to resolve issue POWERCLI INITIALIZE-POWERCLIENVIRONMENT.PS1 CANNOT BE LOADED BECAUSE RUNNING SCRIPTS IS DISABLED

I can see the version of VMware Powercli on the title bar. For first time it will ask for Participate in VMware Customer Experience Improvement Program CEIP, Choose your option. from here on I am ready to use it.

VMWare Powercli 6.5 release 1 console, connect-viserver, Get-VICommand, Get-PowercliHelp, Get-VM, Get-PowerCLICOmmunity, CEIP

Below are all available modules related to VMware and loaded with PowerCLI, It can be viewed imported modules with command Get-Module.

  • Initialize-VMware.VimAutomation.HorizonView
  • Initialize-VMware.VimAutomation.License
  • Initialize-VMware.VimAutomation.vROps
  • Initialize-VMware_DeployAutomation
  • Initialize-VMware_VimAutomation_Cis
  • Initialize-VMware_VumAutomation
  • VMware.DeployAutomation
  • VMware.ImageBuilder
  • VMware.VimAutomation.Cis.Core
  • VMware.VimAutomation.Cloud
  • VMware.VimAutomation.Common
  • VMware.VimAutomation.Core
  • VMware.VimAutomation.HA
  • VMware.VimAutomation.HorizonView
  • VMware.VimAutomation.License
  • VMware.VimAutomation.PCloud
  • VMware.VimAutomation.Sdk
  • VMware.VimAutomation.Storage
  • VMware.VimAutomation.Vds
  • VMware.VimAutomation.vROps
  • VMware.VumAutomation

vmware powercli get-module, vmware.vimautomation.core, vumautomation, vrops, vds, storage, sdk,pcloud, license, horizonview, ha, core, common, cloud, cis.core, deployautomation

Now I have my Powercli is installed and setup, think it is as client only, My very first command will be Connect-VIserver vCenterIp_or_FQDN. It connects to vCenter and ask for credentials - Username and Password. If it is successful I see the connection status with server name, port and username, I connected to. While connecting It shows me warning about server certificate and I can ignore it for time being until I install valid public or CA certificate, but for now I don't require it. and My second command is Get-VM, it shows the list of VMs with there status.

VMWare vSphere Powercli free class tutorial and example, Connect-VIserver, Get-VM Certificate credentials username password

Earlier installer wizard was the old method. This second new method is very straight forward, download module directly from online Microsoft PS gallary, This requires no logging to VMware site and can be automated easily in the script. It requires Internet and Administrator rights, After running Find-Module VMware* it shows me all related module in the online Microsoft PSGallary repository. Right now I require one of the main module and I am installing it using command Install-Module VMware.VimAutomation.Core -Confirm:$false, downloaded modules can be found at location C:\Program Files\WindowsPowerShell\Modules

Also make sure you are ok with Powershell question and ask for your input Yes or No:
You are installing the modules from an untrusted repository. If you trust this repository, change its Installation Policy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from 'PSGallery'?

VMware Powercli, Find-Module, Install-Module VMware.VImAuatomation, Import-Module, get-Module, VMware PSGallary Online

I can see the list of loaded modules with Get-Module. As Powershell core is available on Linux as well as Mac os there is fling available to install powercli core on Linux from https://labs.vmware.com/flings/powercli-core.

VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE

July 6, 2017 08:13PM

Series Parts
MICROSOFT WINDOWS 2012 R2 ISCSI TARGET STORAGE SERVER FOR ESXI AND HYPERV 
POWERSHELL INSTALLING AND CONFIGURING MICROSOFT ISCSI TARGET SERVER
VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
POWERCLI: VMWARE ESXI CONFIGURE (VSWITCH) VMKERNEL NETWORK PORT FOR ISCSI STORAGE
VMWARE ESXI INSTALL AND CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER FOR VMFS VERSION 6 DATASTORE
POWERCLI VMWARE: CONFIGURE SOFTWARE ISCSI STORAGE ADAPTER AND ADD VMFS DATASTORE
VMWARE VCENTER STORAGE MIGRATE/SVMOTION VM AND PORT BINDING MULTIPATHING TESTING
POWERCLI: VIRTUAL MACHINE STORAGE MIGRATE/SVMOTION AND DATASTORE PORT BINDING MULTIPATHING

Starting from my earlier article on iSCSI Target server creation on Windows server 2012 R2, Before configuring storage on Esxi, I must have networking configured.

vmware vsphere vcenter esxi, virtual switches vmkernel vms iscsi storage design architecture port binding, design.png

When using iSCSI protocol for VMFS storage configuration, ESXi server must have VMKernel port added and configured, By default there is a management port exist, It can also be used for iSCSI configuration (Not at all recommended), but for best practices always  isolate storage traffic with separate physical adapters if you are using 1 Gig network adapters. Try to keep and use separate network adapters for iSCSI Storage if possible use physical switches also separate from VM and other traffic if possible, (Now a days using 10 Gig network adapter hardware is common practice and if ESXi virtual networking designed and architected with distributed virtual switches properly only 2 network adapter in Esxi Servers are enough to perfectly handle all load of VM, vMotion, iSCSI stroage and other network flow with segregation using Distributed Virtual Switches), For this demo I am using Standard vSwitches and I have 4 one Gigabit Ethernet adapters in my Esxi server. 

As per the above visio diagram, I will install additional two standard vSwitchs, I will be concentrating only on creating and configuring iSCSI VMKernel ports and its isolated vSwitch. As you can see I have four physical adapters vmnics, three unassigned, I am using selected vmnic2 for this part, in the next part I am going to use same process using vSphere Powercli. 

Adapters list can be viewed on ESXi server >> Configuration tab >> Networking >> Physical adapters.

VMWare vSphere vCenter Esxi Server Configuration Networking Physical adapters vmnics, Used and unused

Once I know which Adapters are free and unassigned, Go to Configure tab on Esxi, choose Networking >> Virtual Switches, Here I can see there is by default vSwitch0 created, I am not touching it, click the 4th step to create new standard vSwitch >> Add host networking. New Add Networking popup box open, in the select connection type, choose VMkernel Network Adapter, (The VMkernel TCP/IP stack handles traffic for ESXi services such as vSphere vMotion, iSCSI, NFS, FCOE, Fault Tolerance, Virtual SAN and host management), Click next.

VMWare vSphere vCenter Esxi Server virtual Switches Add Host Networking Select Connection type VMKernel Network Adapter .png

Under select target device, As this is my new deployment, choose new standard switch, (select an existing standard switch option for if you want to add VMKernel port to existing vSwitch), click next.

In the next 3rd step, Assign adapter by clicking green plus sign, Select free vmnic2 (already decided above), It will be added to selected Active Adapter failover order group list, here I can see either CDP (Cisco Discovery Protocol) or LLDP (Link Layer Discovery Protocol), Physical port and switch information for the NIC also PCI location. (This is also a good way to know that I using correct vmnic). If you want to use single switch and and 2 vmkernel nics and 2 vmnics you will have to adjust the failover order group (active adpaters and unused adapter), Here in the later chapter I am configuring separate standard switch for another VMKernel and Physical adapter. 

Here is some nice whitepaper from vmware https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/iscsi_design_deploy-whitepaper.pdf

VMWare vSphere vCenter Esxi Server add networking,create new Standard switch Assign free physical adapters vmnics, active, standby, unused adapters, Add physical adapters to switch, esxi physical nic adapter properties, CDP and LLDP

Next step is important, Provide it label name, and VLAN ID, in most of the cases for storage vmkernel port other settings are not required to be configured. Here I see new option TCP/IP stack, It gives me option to provide different gateway and I can make storage network traffic routable (I am not recommending this option), also do not enable and share other available services on same VMkernel port, do not merge other traffic as other network can choke storage traffic, click next.

VMWare vSphere vCenter Esxi Server add networking configuration Port Properties, specify VMKernel port settings, Network label, VLAN ID, IP Settings, TCP IP stack, available services vmotion, provisioning, ft, management replication, vsan, virtual san, vsphere replication nfc, fault tolerance logging

Configure static IPV4.

VMWare vSphere vCenter Esxi Server add networking configuration VMKERNEL IPv4 Setting, Default gateway override, tcp ip stack, dns server addresses, static and dynamic ip

This is the last screen of Add networking, Review settings of selection before finishing the wizard, and click finish, I reviewed my settings all looks good.

It takes few seconds, and results are visible under Esxi >> Configuration >> Networking >> Virtual Switches, Select vSwitch1. It shows Nice graphical view of virtual network.

VMware vSPHERE web client vcenter esxi vswitch, virtual switches, standard switch, iSCSI, Physical adapters diagram

In the last make sure you have configured maximum (9000) MTU setting on the interface as well as Physical switch (MTU settings should be configured end to end)., If your switch don't have maximum MTU configured, below steps can be ignore, one thing to note, MTU configuration to 9000 is a best practice for iSCSI storage.

For changing MTU size on virtual switch select the vSwitch under Networking expand Virtual Switches, Select the last created vSwitch1. There is edit settings button looks like pencil. Click it. and on the properties choose 9000 amd click Ok to exit.

VMware vSPHERE web client vcenter esxi vswitch, virtual switches, standard switch, edit virtual switch properties, increase MTU, edit settings vmware

I will be configuring same setting on VMkernel adapters as well under Esxi Server>>Configure Tab>>Networking>>VMkernel adapters>> Select vmkernel adapter>> edit settings>>NIC settings>>9000>>Ok.

VMware vSPHERE web client vcenter esxi vswitch, virtual switches, standard switch, edit virtual switch properties, increase change MTU settings on VMKernel Port, NIC settings , edit settings vmware

I have configured only one VMKernel interface in this article, As I am planning for Port binding for redundant storage path,  and need another VMKernel Port which will be creating and configure using Powercli in next series.

POWERSHELL CONVERT EXCEL TO DSC (DESIRED STATE CONFIGURATION) CONFIGURATION HASHTABLE FORMAT

July 2, 2017 10:09PM

I use Powershell DSC (Desired State Configuration) a lot for windows servers configuration, And I make use of DSC hashtable (configurationdata) frequently. Here is Microsoft fantastic article guide for writing DSC hashtable. But while adding multiple node I find sometimes it takes time to adding extra configuration, specifically comma "s, parentheses, braces, equal sign,  (Tabs for in better readable format) need to have sometime special care and one has to spend more time typing them. I wanted a best automated way to write the hashtable format, so it can be easily exported in below dsc data schema, and it will have few or less errors in it.

$MyData = 
@{
    AllNodes = @()
    NonNodeData = ""   
}

Useful Blog related to DSC
POWERSHELL PS REMOTING BETWEEN STANDALONE WORKGROUP COMPUTERS
DSC (DESIRED STATE CONFIGURATION) : DEPLOYING ACTIVE DIRECTORY
PART 1 : POWERSHELL - CREATE LOCAL IIS WEB REPOSITORY FOR DSC MODULE
POWERSHELL: USE PARAMETERS AND CONFIGURATION FROM INI FILE, USE AS SPLATTING
POWERSHELL CREATE HASHTABLE FROM MICROSOFT EXCEL

Most of the time I gather information in Microsoft Excel or my colleagues provide information in same format to me, so I decided to utilize Excel for this task, writing configuration to multiple tabs is easy, I can easily use them and convert to AllNodes automatic hashtable DSC variable, also NonNodedata can also can be mentioned as tab. Below is the strict Excel file format need to follow, Create AllNodes (Name should be same) tab names must contain AllNodes suffixed with . (dot) and numberings, NonNodeData must be as a tab with same name even though it is blank.

Powershell Convert Microsoft Excel to Dsc desired State Configuration Excel file format with tabs

Below script I am running as .ps1 file, make sure you can execute ps1 scripts on your computer, Set the execution policy with Set-ExecutionPolicy RemoteSigned.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
<#  
  .Synopsis  
   Converts Excel to DSC Configuration Data HashTable format text file.
  .Description  
   This script parses Microsoft Excel file data and coverts to ready to use Powershell DSC (Desired State Configuration) configuration Hashtable format text file. It requires Microsoft Excel to be installed from where you are running this script. Once Excel file is proecess it opens notepad with result, Notpad data can be copied and modified easily.
  .Example  
   ConvertTo-DSCHashTable -ExcelFile C:\temp\Test.xlsx
     
   It parses Microsoft Excel file and create DSC configuration text file, Notpad will automatically open.
  .Example
   Microsoft Excel file format and example, AllNodes data must 
   ----------------------                -----------------------------------   ---------------------------------
   | NodeName | LogPath |                | NodeName | Role      | SiteName |   | ConfigFileContents            | 
   |--------------------|                |----------------------------------   |-------------------------------|
   | *        | C:\Logs |                | VM1      | WebServer | Website1 |   |(Get-Content C:\Temp\Conf.xml) |

   ------------------------------------  ------------------------------------  ------------------------------------
   |*AllNodes.0|AllNodes.1|NonNodeData|  |AllNodes.0|*AllNodes.1|NonNodeData|  |AllNodes.0|AllNodes.1|*NonNodeData| <------ Tabs
   ------------------------------------  ------------------------------------  ------------------------------------
  .OutPuts  
    ConvertTo-DSCHashTable -ExcelFile C:\temp\Test.xlsx
    
    $ConfigurationData = @{
        AllNodes = @(
            @{
                LogPath='C:\Logs'; 
                NodeName='*'
            },

            @{
                Nodename='VM-1'; 
                Role='WebServer'; 
                SiteName='Website1'
            },
        );
        
        NonNodeData = @{		
		    ConfigFileContents=(Get-Content C:\Temp\conf.xml)
	    }
    }

  .Notes  
   NAME: ConvertTo-DSCHashTable
   AUTHOR: Kunal Udapi
   CREATIONDATE: 12 February 2017
   LASTEDIT: 2 July 2017  
   KEYWORDS: Convert excel tab to DSC configuration hashtable  
  .Link  
   #Check Online version: http://kunaludapi.blogspot.com
   #Check Online version: http://vcloud-lab.com
   #Referances: https://msdn.microsoft.com/en-us/powershell/dsc/configdata
   #Requires -Version 3.0  
  #>
[CmdletBinding(SupportsShouldProcess=$True,ConfirmImpact='Medium')]
param(
	[Parameter(
        Position=0, 
        Mandatory=$true,
        ValueFromPipeline=$true,
        HelpMessage='Type the full path of Excel file'
    )]
    [alias('ExcelFile')]
    [ValidateScript({
        If (Test-Path $_) {
            $true
        }
        else{
            "Invalid path given: $_"
        }
    })]
    [System.String]$File
)
$ConfigurationData = @()
#Region  (Parse Excel)
#$File = 'C:\temp\test.xlsx'
$TempFolder = [System.IO.Path]::GetTempPath()
$Guid = [System.Guid]::NewGuid()
$TempDir = Join-Path -Path $TempFolder -ChildPath $Guid.Guid
New-Item -Path $TempDir -ItemType Directory | Out-Null
$excelObj =  New-Object -ComObject Excel.Application
$excelObj.Visible = $false
$excelObj.DisplayAlerts = $false
$WorkBook = $excelObj.Workbooks.Open($File)
$WorkSheets = $WorkBook.Worksheets
Foreach ($WorkSheet in $WorkSheets) {
    $CSVFileName = $WorkSheet.Name
    $CSVFile = Join-Path -Path $TempDir -ChildPath $CSVFileName
    $WorkSheet.SaveAs($CSVFile, 6)
}
$excelObj.Quit()
#Endregion

#Region  (AllNodes)
$AllNodesCSV = Get-ChildItem -Path $TempDir -Filter AllNodes.* | Select-Object -ExpandProperty FullName
$CompleteAllNode =@()
Foreach ($AllNodeRow in $AllNodesCSV) {
    $RowHash = @()
    $ALLNodeCSVRow = Import-Csv $AllNodeRow | Where-Object { ($_.PSObject.Properties | ForEach-Object {$_.Value}) -ne $null}
    $Header = $ALLNodeCSVRow | Get-Member -MemberType NoteProperty | Select-Object -ExpandProperty Name
    foreach ($head in $header) {
        $HashValue = ($ALLNodeCSVRow.$Head | Out-String).Trim()
        $RowHash += "`t`t`t{0}=`"{1}`"" -f $Head, $HashValue
        #"`"$Head`"=`"$($ALLNodeCSVRow.$Head)`""
    }
    $CompleteAllNode += "`t`t@{`n$($RowHash -Join "; `n")`n`t`t}"
}
$AllNodeInfo = $CompleteAllNode -join ",`n`n"
#Endregion

#Region  (NonNodeData)
$NonNodeDataCSV = Get-ChildItem -Path $TempDir -Filter NonNodeData.csv | Select-Object -ExpandProperty FullName
$NonNodeDataCSVRow = Import-Csv $NonNodeDataCSV | Where-Object { ($_.PSObject.Properties | ForEach-Object {$_.Value}) -ne $null}
$CompleteNonNode = @()
if ($NonNodeDataCSVRow -eq $null) {
    $CompleteNonNode = "`t`t`tNonNodeData = `"`""
}
else {
    $NonHeader = $NonNodeDataCSVRow | Get-Member -MemberType NoteProperty | Select-Object -ExpandProperty Name
    $NonNodeRowHeadHash = @()
    foreach ($Nonhead in $NonHeader) {
        $NonNodeHashValue = ($NonNodeDataCSVRow.$Nonhead | Out-String).Trim()
        $NonNodeRowHeadHash += "`t`t{0}=`"{1}`"" -f $NonHead, $NonNodeHashValue
    }
    $CompleteNonNode += "`t`t`n$($NonNodeRowHeadHash -Join "; `n")`n`t"
}
#Endregion
$TmpFileName = [System.IO.Path]::GetTempFileName()
$ConfigurationData = "`$ConfigurationData = @{`n`tAllNodes = @(`n$AllNodeInfo`n`t); `n`n`tNonNodeData = @{$CompleteNonNode}`n}"
Set-Content -Path $TmpFileName -Value $($ConfigurationData -replace "`n", "`r`n")
Notepad $TmpFileName
#$ConfigurationData | clip.exe
$ConfigurationData

There is only one parameter named File or ExcelFile, is the path to Excel and it is mandetory. Once it is executed it opens notepad with the required formatted data, here on notepad  (Powershell ISE, Visual Studio Code Copy past data) I can again verify the data if there is any mistake, and for minor changes information can be modified straight in the readymade text formatted notepad or choice of editor, instead running script again, Also same information is shown on the console as well, for future use save notepad as it is opened as temporary file.

Powershell Convert Microsoft Excel to Dsc desired State Configuration Excel file format with tabs auto format hashtable format nodename, Role, NonNodeData, AllNodes, Automatic Variable, confdata, configurationdata

Incase you require this hashtable in the variable first load the data in variable, and run Invoke-Expression. 

$ConfigurationData = C:\Temp\ConvertTo-DSCHashTable.ps1 -ExcelFile C:\Temp\Test.xlsx
Invoke-Expression $ConfigurationData

All the name and value information within hashtable is stored into variable $ConfigurationData and ready to use in Desired State Configuration

Powershell Convert Microsoft Excel to Dsc desired State Configuration ConvertTo-DSCHashTable, Invoke-Expression, hashtable format nodename, Role, NonNodeData, AllNodes, Automatic Variable, confdata, configurationdata

POWERSHELL CREATE HASHTABLE FROM MICROSOFT EXCEL

July 1, 2017 08:09PM

After writing my last blog on using splatting from INI file, where it has format of Key and Value concatenated with equal = sign. It can be executed easily but sometimes writing INI file can be little messy if there is slight mistake, Still I find INI is best for writing parameter and syntax due to its simplicity, If you want run same configuration against multiple entity like users or computers, Need to write extra coding or add another blocks to INI. For Normal basic use cases INI is always best.

POWERSHELL: USE PARAMETERS AND CONFIGURATION FROM INI FILE, USE AS SPLATTING

This is why I wrote my  another script for parsing Excel file, Same Key and Value can be added to Excel file, and it can have multiple rows. Only requirement is Excel should be installed on the machine where this script will run. (There are modules and scripts available where Excel can be manipulated or extract information without installing it using PowerShell). 

Below is the my Excel format looks like, in the bottom I have Tab with configuration names.

 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
<#  
  .Synopsis  
   Excel to Powershell Hashtable.
  .Description  
   This script converts Selected Excel data to Powershell Hashtable. It requires Microsoft Excel to be installed from where you are running this script.
  .Example  
   Get-ExcelHashtable -ExcelFile c:\temp\Test.xlsx -Tab Service
     
   It parses Excel file and for selected tab create hashtable with same name.
  .Example
   Excel file format and example
   --------------------------------
   | Name          | ComputerName |
   | ------------------------------
   | LanManServer  | Localhost    |
   | WorkStation   | 127.0.0.1    |
   --------------------------------
   |Service| <---------------------------Excel Tabs
   ---------
  .OutPuts  
    Get-ExcelHashtable -ExcelFile c:\temp\Test.xlsx -Tab Service
    $service | Foreach-Object {$syntex = $_; Get-Service @syntex}

    Status   Name               DisplayName                           
    ------   ----               -----------                           
    Stopped  LanManServer       Server                                
    Running  LanmanWorkstation  Workstation
  .Notes  
   NAME: Get-ExcelHashtable
   AUTHOR: Kunal Udapi
   CREATIONDATE: 01 July 2017
   LASTEDIT: 1 July 2017  
   KEYWORDS: Export excel tab to hashtable  
  .Link  
   #Check Online version: http://kunaludapi.blogspot.com
   #Check Online version: http://vcloud-lab.com
   #Requires -Version 3.0  
  #>
[CmdletBinding(SupportsShouldProcess=$True,ConfirmImpact='Medium')]
param(
	[Parameter(
        Position=0, 
        Mandatory=$true,
        ValueFromPipeline=$true,
        HelpMessage='Type the full path of Excel file'
    )]
    [alias('ExcelFile')]
    [ValidateScript({
        If (Test-Path $_) {
            $true
        }
        else{
            "Invalid path given: $_"
        }
    })]
    [System.String]$File,
    [Parameter(Position=1, Mandatory=$true)]
    [alias('Configuration')]
	[System.String]$Conf 
)
#$Path = 'C:\temp\test.xlsx'
#$Conf = 'Service'
$TabName = [System.IO.Path]::GetTempFileName()
$excelObj =  New-Object -ComObject Excel.Application
$excelObj.Visible = $false
$excelObj.DisplayAlerts = $false
$WorkBook = $excelObj.Workbooks.Open($File)
$WorkSheet = $WorkBook.Worksheets | Where-Object {$_.Name -eq $Conf}
$WorkSheet.SaveAs($TabName, 6)
$excelObj.Quit()
$TabCSV = Import-Csv $TabName
$Header = $TabCSV | Get-Member -MemberType NoteProperty | Select-Object -ExpandProperty Name
$CompleteHash =@()
$Information = @()
foreach ($row in $TabCSV) {
    $RowHash = @()
    foreach ($head in $header) {
        $RowHash += "`"$Head`"=`"$($row.$Head)`""
    }
    $CompleteHash += "@{$($RowHash -Join '; ')}"
}
$Information = $CompleteHash -join ', '
New-Variable -Scope Global -Name $Conf -Value (Invoke-Expression $Information) -Force

Copy Paste above script in PS1, Make sure you can execute PS1 script, it can be modified using Set-Executionpolicy remotesigned. Next once below command run it will be asking for excel file path and Configuration tab name. When running information it doesn't require much coding. It creates same configuration or Tab named variable, In my case $service, I am using foreach block with $psitem as splatting.

Powershell excel to hashtable, service restart from hashtable

View older posts »