iwan.hoogendoorn
NSX · · 19 min read

The NSX PowerOPS Tool unleashed

A few months back one of my colleagues and good friend Dominic did a presentation about NSX PowerOPS. Due to a busy schedule, I was not able to try it and test it up until last week.

Business Problem

NSX has transformed customer’s approach to Networking & Security and has achieved a significant installed base. Operations is challenged with Day-2 understanding, documentation & health of their NSX deployments.

The main business problem that arises here is that one has all the tools suggested to operationalize NSX but the team is still at a loss when they have to use them to troubleshoot NSX”

For who is this

The NSX Power Operations (Tool) is for people who are responsible for the installation or administrators of the NSX environment.

PowerOPS - What is it and what does it do

The NSX Power Operations (Tool) is a tool that is Day-2 focused tool to extend product usefulness & adoption, Automated documentation, focusing on daily operational activities.

The installation

The installation steps are outlined below.

Check version of PowerShell

Make sure version 5 or above of PowerShell is installed, and if it is installed already you can verify this with the following command:

Terminal window
PS C:\Users\Administrator> $PSVersionTable
Name Value
---- -----
PSVersion 5.1.14393.0
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.14393.0
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
PS C:\Users\Administrator>

Here you see that I already had version 5.x installed of PowerShell.

Change Execution Policy

Terminal window
PS C:\Users\Administrator> Set-ExecutionPolicy RemoteSigned
Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
you to the security risks described in the about_Execution_Policies help topic at
http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the execution policy?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): A
PS C:\Users\Administrator>

Install PowerCLI

Terminal window
PS C:\Users\Administrator> Find-Module VMware.PowerCLI

A quick verification tells us that PowerCLI in installed correctly.

Terminal window
PS C:\Users\Administrator> Find-Module VMware.PowerCLI
Version Name Repository Description
---- ---- ---------- -----------
10.2.0.... VMware.PowerCLI PSGallery This Windows PowerShell module contains VMware.Po...

We can also verify what VMware PowerCLI modules are now available.

Terminal window
PS C:\Users\Administrator> Get-Module VMware* -ListAvailable
Directory: C:\Users\Administrator\Documents\WindowsPowerShell\Modules
ModuleType Version Name ExportedCommands
---- ------- ---- ----------------
Script 6.7.0.8... VMware.DeployAutomation {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-De...
Script 6.7.0.8... VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-EsxI...
Manifest 10.2.0.... VMware.PowerCLI
Script 6.7.0.9... VMware.Vim
Script 10.1.0.... VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Script 10.0.0.... VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Ge...
Script 10.1.0.... VMware.VimAutomation.Common
Script 10.1.0.... VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAda...
Script 6.5.4.7... VMware.VimAutomation.HA Get-DrmInfo
Script 7.5.0.8... VMware.VimAutomation.HorizonView {Connect-HVServer, Disconnect-HVServer}
Script 10.0.0.... VMware.VimAutomation.License Get-LicenseDataManager
Script 10.2.0.... VMware.VimAutomation.Nsxt {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtService}
Script 10.0.0.... VMware.VimAutomation.PCloud {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstan...
Script 10.1.0.... VMware.VimAutomation.Sdk
Script 10.0.0.... VMware.VimAutomation.Srm {Connect-SrmServer, Disconnect-SrmServer}
Script 10.1.0.... VMware.VimAutomation.Storage {Add-KeyManagementServer, Copy-VDisk, Export-SpbmStoragePol...
Script 1.2.0.0 VMware.VimAutomation.StorageUtility Update-VmfsDatastore
Script 10.1.0.... VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Ex...
Script 10.0.0.... VMware.VimAutomation.Vmc {Connect-Vmc, Disconnect-Vmc, Get-VmcService, Connect-VmcSe...
Script 10.0.0.... VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OM...
Script 6.5.1.7... VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Complian...

Download GIT and Download PowerOPS with using a command windows not powershell

Download GIT from this website Download PowerOPS with using a regular command window (NOT PowerShell).

Terminal window
C:\Users\Administrator>git clone https://github.com/vmware/nsx-powerops.git
Cloning into 'nsx-powerops'...
remote: Counting objects: 849, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 849 (delta 23), reused 28 (delta 13), pack-reused 800R
Receiving objects: 100% (849/849), 1.14 MiB

Verify if the git repository is cloned properly.

Terminal window
C:\Users\Administrator>dir
Volume in drive C has no label.
Volume Serial Number is BA5F-76FD
Directory of C:\Users\Administrator
09/18/2018 04:19 AM <DIR> .
09/18/2018 04:19 AM <DIR> ..
09/18/2018 02:19 AM <DIR> Contacts
09/18/2018 02:19 AM <DIR> Desktop
09/18/2018 02:34 AM <DIR> Documents
09/18/2018 04:12 AM <DIR> Downloads
09/18/2018 02:19 AM <DIR> Favorites
09/18/2018 02:19 AM <DIR> Links
09/18/2018 02:19 AM <DIR> Music
09/18/2018 04:19 AM <DIR> nsx-powerops
09/18/2018 02:19 AM <DIR> Pictures
09/18/2018 02:19 AM <DIR> Saved Games
09/18/2018 02:19 AM <DIR> Searches
09/18/2018 02:19 AM <DIR> Videos
0 File(s) 0 bytes
14 Dir(s) 89,933,549,568 bytes free
C:\Users\Administrator>

Run the PowerOPS script after the installation

Terminal window
PS C:\Users\Administrator\nsx-powerops> .\nsx-PowerOps.ps1

When you run PowerOPS for the first time you can set up “Connection Profiles” with hostnames and log in credentials for different componenenets. Once these are initially set these can be changed later on in the menu if required.

Initial startup menu
Terminal window
NSX-PowerOps Version: 2.5.
*************************************************************************************************************************
NSX-PowerOps Main Menu
1 - Install NSX-PowerOps Dependancies [ Installed ]
2 - Connection Profiles [ Default: 10.11.11.122 ]
3 - Configure Scheduled Tasks [ ENABLED ]
4 - NSX-PowerOps Documentation Tools [ ENABLED ]
5 - NSX-PowerOps HealthChecks [ ENABLED ]
*************************************************************************************************************************
Default Connection Profile: 10.11.11.122
Current Connection Profile: 10.11.11.122
Connected: True
Output Directory: C:\Users\Administrator\nsx-powerops\Report\2018-09-18_23-56
*************************************************************************************************************************
Enter the number of your choice, h to get item specific help, x to go back or q to quit.
Select 4 for NSX - PowerOps Documentation Tools
Terminal window
NSX-PowerOps Version: 2.5.
*************************************************************************************************************************
NSX-PowerOps Main Menu > NSX-PowerOps Documentation Tools
Auto Run All
1 - Run All Following Documentation Menu Items [ ENABLED ]
Environment Documentation
2 - Document All NSX Components [ ENABLED ]
3 - Document ESXi Host(s) Info [ ENABLED ]
Networking Documentation
4 - Document NSX Environment via Visio Diagramming Tool [ ENABLED ]
5 - Document Routing information [ ENABLED ]
6 - Document Load Balancing Information [ ENABLED ]
Security Documentation
7 - Document NSX DFW info to Excel via DFW2Excel (small environments) [ ENABLED ]
8 - Document NSX DFW info to Excel via DFW2Excel (large environments) [ ENABLED ]
*************************************************************************************************************************
Default Connection Profile: 10.11.11.122
Current Connection Profile: 10.11.11.122
Connected: True
Output Directory: C:\Users\Administrator\nsx-powerops\Report\2018-09-18_23-56
*************************************************************************************************************************
Enter the number of your choice, h to get item specific help, x to go back or q to quit.
Select 2 - Document All NSX Components

Generated File by PowerOPS 2._NSX-Components-2018-09-19_00-06.xlsx

Select 3 - Document ESXi Host Info

Output errors after selecting the option

Terminal window
out-event : A specified parameter was not correct: argument[0]
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:426 char:17
+ out-event -entrytype error $ErrorMessage
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,out-event
Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary: 'VXLAN Installed' Key being added: 'VXLAN Installed'"
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:427 char:17
+ ... $sshCommandOutputDataLogicalSwitch.Add("VXLAN Installed", ...
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ArgumentException
out-event : A specified parameter was not correct: argument[0]
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:426 char:17
+ out-event -entrytype error $ErrorMessage
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,out-event
Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary: 'VXLAN Installed' Key being added: 'VXLAN Installed'"
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:427 char:17
+ ... $sshCommandOutputDataLogicalSwitch.Add("VXLAN Installed", ...
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ArgumentException
out-event : A specified parameter was not correct: argument[0]
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:426 char:17
+ out-event -entrytype error $ErrorMessage
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,out-event
Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary: 'VXLAN Installed' Key being added: 'VXLAN Installed'"
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:427 char:17
+ ... $sshCommandOutputDataLogicalSwitch.Add("VXLAN Installed", ...
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ArgumentException

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS 3._ESXi-Hosts-2018-09-19_00-07.xlsx

Select 4 - Document NSX Environment via Visio Diagramming Tool

Output after selecting the option

Terminal window
PowerNSX Object Capture Script - Version 2.0.1
Getting NSX Objects
Getting LogicalSwitches
Getting DV PortGroups
Getting VSS PortGroups
Getting Logical Routers
Getting Edges
Getting NSX Controllers
Getting VMs
Getting IP and MAC details from Spoofguard
Getting configured IP Set objects
Getting configured Services
Getting configured Service groups
Getting configured Security groups
Getting configured Security tags
Getting configured DFW Rules
Getting Security Policies
Creating Object Export Bundle
Capture Bundle created at C:\Users\Administrator\nsx-powerops\Report\2018-09-18_23-56\NSX-CaptureBundle-2018-09-19_00-10.zip
PowerNSX Object Diagram Script
Launching Microsoft Visio.
Building Diagram
Adding DLR-01 to diagram with stencil Logical Router
Adding LS-TRANSIT to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch with text: uplink: 192.168.10.2
Adding LS-PRODUCTION-01 to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.4 with text: internal: 172.16.10.1
Adding LS-DHCPTEST to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.6 with text: internal: 172.16.11.1
Adding LS-MARCEL to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.8 with text: internal: 10.11.30.1
Adding RoutedNetwork-92c01f5a-e699-4568-8726-e1161fdf0963 to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.10 with text: internal: 172.16.30.129
Adding RoutedNetwork-5213d324-af25-4fa1-a7c1-a19771fc08af to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.12 with text: internal: 172.16.30.161
Adding LS-PRODUCTION-02 to diagram with stencil Logical Switch
Connecting Logical Router with Logical Switch.14 with text: internal: 172.16.12.1
Adding ESG-01 to diagram with stencil Edge
Adding VLAN16-vDS-NSX to diagram with stencil PortGroup
Connecting Edge with PortGroup with text: uplink: 10.11.16.1
Connecting Edge with Logical Switch with text: internal: 192.168.10.1
Adding Edge-DemoApp-99f68d14-4bb1-4a81-9594-62f73e6529fb to diagram with stencil Edge
Connecting Edge.20 with Logical Switch.12 with text: uplink: 172.16.30.162 172.16.30.163
Saved diagram at C:\Users\Administrator\nsx-powerops\Report\2018-09-18_23-56\NSX-CaptureBundle-2018-09-19_00-10.vsdx
Done. Hit any key to continue.

Generated Files by PowerOPS 4._NSX-CaptureBundle-2018-09-19_00-10.vsdx

Select 5 - Document Routing information

Generated File by PowerOPS 5._NSX-Routing-2018-09-19_00-12.xlsx

Select 6 - Document Load Balancing Information

Output errors after selecting the option

Terminal window
You typed an invalid name for a sheet or chart. Make sure that:
• The name that you type does not exceed 31 characters.
• The name does not contain any of the following characters: : \ / ? * [ or ]
• You did not leave the name blank.
At C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1:972 char:9
+ $wsStats.Name = $edge.name
+ 16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)16:34, 24 September 2018 (BST)~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS 6._LBToExcel-2018-09-19_00-13.xlsx

Select 7 - Document NSX DFW info to Excel via DFW2Excel small environments

Output after selecting the option

Terminal window
Collection of IP Addresses Enabled
Collection of Security Tag VM Membership Enabled
Collection of Security Group VM Membership Enabled
Retrieving IP Addresses for ALL Virtual Machines in vCenter environment.
*** This may take a while ***.
Retrieving Services configured in NSX-v.
Retrieving Service Groups configured in NSX-v.
Retrieving MACSETS configured in NSX-v.
Retrieving IPSETS configured in NSX-v.
Retrieving Security Groups configured in NSX-v.
Retrieving Security Groups Effective Membership in NSX-v.
Collection of VM Sec Membership Enabled
Retrieving Security Tags configured in NSX-v.
Retrieving VMs in DFW Exclusion List
Retrieving DFW Redirect FW Rules
Retrieving DFW Layer 2 FW Rules
Retrieving DFW Layer 3 FW Rules
Retrieving Environment Summary

Generated File by PowerOPS 7. DfwToExcel-2018-09-19_00-15.xlsx

Select 8 - Document NSX DFW info to Excel via DFW2Excel large environments

This output would be the same as option 7.

Select x to go back and select 5 for the health checks

Select 5 for NSX - PowerOps HealthChecks
Terminal window
NSX-PowerOps Version: 2.5.
*************************************************************************************************************************
NSX-PowerOps Main Menu > NSX-PowerOps HealthChecks
Non-Interactive Healthchecks Menu
1 - NSX Connectivity Test [ ENABLED ]
2 - NSX Manager Test [ ENABLED ]
3 - NSX Controllers Appliance Test [ ENABLED ]
4 - NSX Logical Switch Test [ ENABLED ]
5 - NSX DFW Memory Test (requires SSH to ESXi) [ ENABLED ]
6 - Check VIB Version [ ENABLED ]
Interactive Healthchecks Menu
7 - Check DLR Instance [ ENABLED ]
8 - Check vTEP to vTEP connectivity [ ENABLED ]
Beta Healthchecks Menu
9 - Compare DLR and Hosts Routing Tables [BETA] [ ENABLED ]
*************************************************************************************************************************
Default Connection Profile: 10.11.11.122
Current Connection Profile: 10.11.11.122
Connected: True
Output Directory: C:\Users\Administrator\nsx-powerops\Report\2018-09-18_23-56
*************************************************************************************************************************
Enter the number of your choice, h to get item specific help, x to go back or q to quit.
Select 1 - NSX Connectivity Test

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing Basic System Connectivity and Time Tests
NSX Manager: 10.11.11.122
[+] is connected to vCenter 585ms
[-] is within 5 minutes of the local system time (Note: Local system and NSX Manager must be in same TimeZone) 165ms
Expected {419.986275675} to be less than {5}
49: $totalAbsMinsDrift

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXConnections-2018-09-19_00-24.xml

Select 2 - NSX Manager Test

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing NSX Manager
[+] performed an Inventory less than 30 minutes ago 648ms
[+] has the NSX Manager Service in a running state 5.59s
[+] has the vPostgres Service in a running state 81ms
[+] has the RabbitMQ Service in a running state 22ms
[-] has the NSX Replicator Service in a stopped state 150ms
Expected: {} to match the expression {STOPPED}
75: $_.name -match 'NSX Replicator'}).status

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXManager-2018-09-19_00-26.xml

Select 3 - NSX Controllers Appliance Test

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing NSX Controllers
The NSX controller cluster
[-] has the supported number of controller nodes 4.14s
Expected: {3}
But was: {1}
54: It "has the supported number of controller nodes" {$NsxControllers.Count

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXControllers-2018-09-19_00-33.xml

Select 4 - NSX Logical Switch Test

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing Logical Switches
the Logical Switch LS-TRANSIT
Logical Switch control plane tests
[-] is found on all controllers 2.56s
String lengths are both 26. Strings differ at index 13.
Expected: {VNI found on 1 controllers}
But was: {VNI found on 0 controllers}
------------------------^
115: "VNI found on $($CurrLsVniResults.count) controllers"

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXLogicalSwitch-2018-09-19_00-28.xml

Select 5 - NSX DFW Memory Test 〈requires SSH to ESXi〉

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing Distributed Firewall Memory heaps
[+] esx-02.home.local is reachable via ssh 1.55s
[+] esx-02.home.local returns system heaps 158ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-attr-0x4322ab5a3000/ for esx-02.home.local 86ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-flow-0x4310336f6000/ for esx-02.home.local 115ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-ipdiscovery-0x4310136f3000/ for esx-02.home.local 76ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-rules-0x4323f77a0000/ for esx-02.home.local 70ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-state-0x430f736f0000/ for esx-02.home.local 86ms
[+] esx-02.home.local has not exceeded the 80% memory threshold on memory heap vsip-module-0x430ef36ed000/ for esx-02.home.local 71ms
[+] esx-01.home.local is reachable via ssh 199ms
[+] esx-01.home.local returns system heaps 139ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-attr-0x43251f631000/ for esx-01.home.local 73ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-flow-0x4310ec78d000/ for esx-01.home.local 85ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-ipdiscovery-0x430f1fae6000/ for esx-01.home.local 73ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-rules-0x43245f82e000/ for esx-01.home.local 84ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-state-0x43104c78a000/ for esx-01.home.local 69ms
[+] esx-01.home.local has not exceeded the 80% memory threshold on memory heap vsip-module-0x430fcc787000/ for esx-01.home.local 78ms
[+] esx-04.home.local is reachable via ssh 226ms
[+] esx-04.home.local returns system heaps 137ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-attr-0x430f0df72000/ for esx-04.home.local 88ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-flow-0x430c1a068000/ for esx-04.home.local 77ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-ipdiscovery-0x430979eeb000/ for esx-04.home.local 78ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-rules-0x430eae06f000/ for esx-04.home.local 69ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-state-0x430e6e06c000/ for esx-04.home.local 87ms
[+] esx-04.home.local has not exceeded the 80% memory threshold on memory heap vsip-module-0x430dee069000/ for esx-04.home.local 70ms
Tests completed in 3.85s
Passed: 24 Failed: 0 Skipped: 0 Pending: 0 Inconclusive: 0

Generated File by PowerOPS TestNSXDistributedFirewallHeap-2018-09-19_00-36.xml

Select 6 - Check VIB Version

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
[-] Error occurred in test script 'C:\Users\Administrator\nsx-powerops/HealthCheck/testNSXVIBVersion.Tests.ps1' 640ms
NotSupportedException: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer's first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\HealthCheck\testNSXVIBVersion.Tests.ps1: line 72
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 297
at Invoke-Pester, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 310
at runNSXTest, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1322
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1715
at Show-MenuV2, C:\Users\Administrator\nsx-powerops\util.ps1: line 328
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1667
at Show-MenuV2, C:\Users\Administrator\nsx-powerops\util.ps1: line 328
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1915
at <ScriptBlock>, <No file>: line 1
Tests completed in 640ms
Passed: 0 Failed: 1 Skipped: 0 Pending: 0 Inconclusive: 0

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXVIBVersion-2018-09-19_00-37.xml

Select 7 - Check DLR Instance

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Please provide the Compute Cluster name [example: Compute Cluster A1]: Compute
Please provide Cluster's VDS name [example: ComputeA_VDS]: vDS-Compute
Please provide VDN ID [example: 10000]: 5001
Describing NSX VDR Port Tests
Cluster Compute
WARNING:
The Get-EsxCli cmdlet provides a new interface to the ESXCLI functionality.
Use the -V2 parameter to switch to the new cmdlet interface.
Check the cmdlet help for more information.
Scripts that use the old cmdlet interface may not be compatible across two different versions of the ESX server.
The old cmdlet interface is deprecated and will be removed in a future version.
Checking Host: esx-02.home.local
Checking Host: esx-01.home.local
Checking Host: esx-04.home.local
[+] got VDR deployed 322.1s
Tests completed in 322.1s
Passed: 1 Failed: 0 Skipped: 0 Pending: 0 Inconclusive: 0

Generated File by PowerOPS TestNSXVDR-2018-09-19_00-38.xml

Select 8 - Check vTEP to vTEP connectivity

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Describing NSX Manager
>> Please provide the MTU size to test [Default: 1572]:
>> Run this test from 'one' host or 'all' [Default: all]:
******************************
Pinging From Host: esx-01.home.local
Pinging From VMKnic: vmk3
Pinging From VMKnic IP: 10.11.15.7
Ping MTU Size is: 1572
******************************
Pinging To Host: esx-01.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.7
[+] Ping test result 17.25s
[+] round-trip min/avg/max=0.041/0.044/0.049ms
Pinging To Host: esx-02.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.6
[+] Ping test result 2.9s
[+] round-trip min/avg/max=0.25/0.274/0.299ms
Pinging To Host: esx-04.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.8
[+] Ping test result 2.7s
[+] round-trip min/avg/max=0.367/0.426/0.49ms
******************************
Pinging From Host: esx-02.home.local
Pinging From VMKnic: vmk3
Pinging From VMKnic IP: 10.11.15.6
Ping MTU Size is: 1572
******************************
Pinging To Host: esx-01.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.7
[+] Ping test result 2.79s
[+] round-trip min/avg/max=0.189/0.234/0.268ms
Pinging To Host: esx-02.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.6
[+] Ping test result 2.65s
[+] round-trip min/avg/max=0.045/0.062/0.072ms
Pinging To Host: esx-04.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.8
[+] Ping test result 2.73s
[+] round-trip min/avg/max=0.414/0.436/0.451ms
******************************
Pinging From Host: esx-04.home.local
Pinging From VMKnic: vmk3
Pinging From VMKnic IP: 10.11.15.8
Ping MTU Size is: 1572
******************************
Pinging To Host: esx-01.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.7
[+] Ping test result 2.81s
[+] round-trip min/avg/max=0.26/0.304/0.37ms
Pinging To Host: esx-02.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.6
[+] Ping test result 3.74s
[+] round-trip min/avg/max=0.414/0.448/0.494ms
Pinging To Host: esx-04.home.local
Pinging To its VMKnic: vmk3
Pinging To its IP: 10.11.15.8
[+] Ping test result 2.93s
[+] round-trip min/avg/max=0.109/0.115/0.12ms
Tests completed in 40.49s
Passed: 9 Failed: 0 Skipped: 0 Pending: 0 Inconclusive: 0

Generated Files by PowerOPS VMKnicPingTestOutput-2018-09-19_00-46.xlsx TestNSXMTUUnderlay-2018-09-19_00-45.xml

Select 9 - Compare DLR and Hosts Routing Tables BETA

Output after selecting the option

Terminal window
Using ESXi credentials from connection profile 10.11.11.122
Using NSX Controller credentials from connection profile 10.11.11.122
Using NSX Manager credentials from connection profile 10.11.11.122
Using existing PowerCLI connection to 10.11.11.52
Found single logical router DLR-01
Collecting routing table from DLR-01
Collecting routing table from host esx-02.home.local
Collecting routing table from host esx-01.home.local
Collecting routing table from host esx-04.home.local
Validating routing table of ESXi server esx-02.home.local
[-] Error occurred in test script 'C:\Users\Administrator\nsx-powerops/HealthCheck/testNSXDlrHostRoutingTable.Tests.ps1' 4.38s
RuntimeException: The It command may only be used inside a Describe block.
at Assert-DescribeInProgress, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Functions\Describe.ps1: line 125
at ItImpl, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Functions\It.ps1: line 140
at It, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Functions\It.ps1: line 117
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\HealthCheck\testNSXDlrHostRoutingTable.Tests.ps1: line 287
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 297
at Invoke-Pester, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 310
at runNSXTest, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1322
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1736
at Show-MenuV2, C:\Users\Administrator\nsx-powerops\util.ps1: line 328
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1667
at Show-MenuV2, C:\Users\Administrator\nsx-powerops\util.ps1: line 328
at <ScriptBlock>, C:\Users\Administrator\nsx-powerops\nsx-PowerOps.ps1: line 1915
at <ScriptBlock>, <No file>: line 1
Tests completed in 4.38s
Passed: 0 Failed: 1 Skipped: 0 Pending: 0 Inconclusive: 0

The errors displayed above are probably due to a compatibility problem of the combination of the version PowerNSX and the version NSX that I am using. I did not try to troubleshoot these as the goal of this exercise only was to demonstrate how the PowerOPS script worked.

Generated File by PowerOPS TestNSXDlrHostRoutingTable-2018-09-19_00-47.xml

Originally published on the old wiki at /index.php/The_NSX_PowerOPS_Tool_unleashed.