Now that we prepared the environment to start our Terraform coding it is time to actually start some Terraform coding. For our automated nested lab we need some network segments that the infra that we are building will consume.
The Segments we need to create with the subnets are listed below:
- LX-APP-MGMT-11 | 192.168.11.0/24
- Subnet for Management applications, like vCenter Server, NSX-T Manager, etc.
- LX-ESXI-MGMT-UT | 192.168.12.0/24
- Subnet for all vmk0 interfaces for ESXi management traffic.
- LX-VMOTION-13 | 192.168.13.0/24
- Subnet for all vmk1 interfaces for ESXi vMotion traffic.
- LX-VSAN-14 | 192.168.14.0/24
- Subnet for all vmk2 interfaces for ESXi VSAN traffic.
- LX-REPLICATION-15 | 192.168.15.0/24
- Subnet for all vmk3 interfaces for ESXi Replication traffic.
- LX-OVERLAY-HOST-16 | 192.168.16.0/24
- Subnet for all Host (ESXi) Transport Node TEP interfaces for GENEVE traffic.
- LX-OVERLAY-EDGE-17 | 192.168.17.0/24
- Subnet for all Edge (VM) Transport Node TEP interfaces for GENEVE traffic.
- LX-BGP-UPLINK-01-18 | 192.168.18.0/24
- Subnet for BGP uplink 01 for N/S Connectivity.
- LX-BGP-UPLINK-02-19 | 192.168.19.0/24
- Subnet for BGP uplink 02 for N/S Connectivity.
- LX-TRANSIT-LEAF-SPINE-20 | 192.168.20.0/24
- Subnet to connect the virtual ToR switches/leafs (VX01 and VX02) with the core/spine (VX00).
The terraform script in order to deploy these segments on my NSX-T infrastructure NSX-T Manager can be found below:
\❯ tree├── main.tfmain.tf
Output: main.tf code
# use Terraform 0.12 to execute script
# NSX-T Manager Credentialsprovider "nsxt" { host = “<your NSX-T manager IP/Hostname>” username = “admin” password = “<your NSX-T manager password>” allow_unverified_ssl = true max_retries = 10 retry_min_delay = 500 retry_max_delay = 5000 retry_on_status_codes = [429]}
data “nsxt_policy_transport_zone” “overlay_tz” { display_name = “TZ-OVERLAY”}
resource “nsxt_policy_segment” “tf_app_mgmt” { display_name = “LX-APP-MGMT-11” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_esxi_mgmt” { display_name = “LX-ESXI-MGMT-UT” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_vmotion” { display_name = “LX-VMOTION-13” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_vsan” { display_name = “LX-VSAN-14” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource "nsxt_policy_segment” “tf_replication” { display_name = “LX-REPLICATION-15” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_overlay_host” { display_name = “LX-OVERLAY-HOST-16” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_overlay_edge” { display_name = “LX-OVERLAY-EDGE-17” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_uplink_01” { display_name = “LX-BGP-UPLINK-01-18” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_uplink_02” { display_name = “LX-BGP-UPLINK-02-19” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}
resource “nsxt_policy_segment” “tf_transit_leaf_spine” { display_name = “LX-TRANSIT-LEAF-SPINE-20” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path}Validate your code:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24[INFO] Switching to v0.12.24[INFO] Switching completedihoogendoor-a01:Test iwanhoogendoorn$ terraform validatePlan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform planExecute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform applyWhen the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroyBefore the above script can be executed (as the time of this writing: March 2020) we need to compile the latest NSX-T Terraform Provider from source. The NSX-T Terraform Provider that is downloaded automatically is a version that does not contain all features and the policy management features are not fully available and in beta at the moment.
In order to create the segments with terraform in NSX-T in the Simple UI the default nsx-t terraform provider is not supported. With the default you can only create segments in the advanced UI (that is not visible in the Simple UI). In order to get the latest terraform NSX-T provider you need to compile this yourself.
Using the latest version of the NSX–T provider
By default when using the NSX-T provider in your Terraform script the “Default” provider will be downloaded and used. This “default” version is the latest and stable version of the provider, but not the one with the latest features. There are a lot more features and commands under development and these can only be used with the latest version of the NSX-T provider. In order to get this new NSX-T provider with the latest and greatest features we need to clone the GitHub repo of the provider and compile it from source using Go Lang.
So the first step is to install Go Lang on my Mac. We do this using brew. How brew works can be found here.
ihoogendoor-a01:~ iwanhoogendoorn$ # Go development
ihoogendoor-a01:~ iwanhoogendoorn$ export GOPATH=“${HOME}/.go”
ihoogendoor-a01:~ iwanhoogendoorn$ export GOROOT=“$(brew —prefix golang)/libexec”
ihoogendoor-a01:~ iwanhoogendoorn$ export PATH=“$PATH:${GOPATH}/bin:${GOROOT}/bin”
ihoogendoor-a01:~ iwanhoogendoorn$ test -d “${GOPATH}”ihoogendoor-a01:~ iwanhoogendoorn$ brew install goUpdating Homebrew...==> Auto-updated Homebrew!Updated 2 taps (homebrew/core and homebrew/cask).==> New Formulaeazcopy container-structure-test dhall-yaml mtoc publish swift-format==> Updated Formulaeappium dvc gsoap libvirt run tiny-fuguearp-sk ettercap hugo minio selenium-server-standalone tomee-plumearping exploitdb inspircd minio-mc skaffold tomee-plusarpoison fop legit miniserve spades tomee-webprofilebalena-cli frege libarchive nanopb-generator sslsplit travisbash-completion@2 gitlab-gem libcbor nativefier suricata vicecalicoctl gitversion libfido2 poppler tccutil wpscandocbook glooctl libnet ripgrep tcptraceroute wtfutildpkg golang-migrate libnids root tflint zeek==> Updated Casks1password-cli fontforge lazarus qownnotes unity-android-support-for-editorangry-ip-scanner fontgoggles lrtimelapse rectangle unity-download-assistantanydo frhelper maccy refined-github-safari unity-ios-support-for-editorbeaker-browser gifox macgamestore ripcord unity-lumin-support-for-editorchromium glyphs metaz supercollider unity-webgl-support-for-editorclashx hbuilderx meteorologist texworks unity-windows-support-for-editorelectron-fiddle icq mochi thonny webcatalogengine-prime jami multitouch tinderboxespresso jprofiler odrive trilium-notesexodus kapitainsky-rclone-browser pd-l2ork unity==> Deleted Casksgoogle-nik-collection
==> Downloading https{{fqm}}//homebrew.bintray.com/bottles/go-1.14.catalina.bottle.tar.gz==> Downloading from https{{fqm}}//akamai.bintray.com/a4/a4e81b1025b77af039a34888545ce0f4a3a332b1ed6358ca8107e878d25e46aa?__gda__=exp=1584369109~hmac=e7bb7a5cce083ba4b6c323ef9######################################################################## 100.0%==> Pouring go-1.14.catalina.bottle.tar.gz🍺 /usr/local/Cellar/go/1.14: 9,431 files, 424MBihoogendoor-a01:~ iwanhoogendoorn$Now that Go Lang is installed we can clone the new NSX-Tprovider code:
ihoogendoor-a01:~ iwanhoogendoorn$ cd Coding/nestedlabs/Terraform/
ihoogendoor-a01:Terraform iwanhoogendoorn$ mkdir terratemp
ihoogendoor-a01:Terraform iwanhoogendoorn$ cd terratemp/
ihoogendoor-a01:terratemp iwanhoogendoorn$ pwd/Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/terratemp
ihoogendoor-a01:terratemp iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.gitCloning into ‘terraform-provider-nsxt’…remote: Enumerating objects: 86, done.remote: Counting objects: 100% (86/86), done.remote: Compressing objects: 100% (51/51), done.remote: Total 15107 (delta 46), reused 51 (delta 34), pack-reused 15021Receiving objects: 100% (15107/15107), 16.23 MiBNow that the code is downloaded we need to compile it from source:
Output: compile provider from source output
ihoogendoor-a01:terratemp iwanhoogendoorn$ ls -ltotal 0drwxr-xr-x 20 iwanhoogendoorn staff 640 Mar 16 15:04 terraform-provider-nsxtihoogendoor-a01:terratemp iwanhoogendoorn$ cd terraform-provider-nsxt/ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ lsCHANGELOG.md GNUmakefile NOTICE.txt go.mod main.go scripts vendorCONTRIBUTING.md LICENSE.txt README.md go.sum nsxt tools website
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go buildgo: downloading github.com/hashicorp/terraform-plugin-sdk v1.1.0go: downloading github.com/google/uuid v1.1.1go: downloading github.com/vmware/go-vmware-nsxt v0.0.0-20191219213550-f4221331f638go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.2.0go: downloading github.com/vmware/vsphere-automation-sdk-go/runtime v0.1.1go: downloading github.com/vmware/vsphere-automation-sdk-go/lib v0.1.1go: downloading github.com/hashicorp/go-multierror v1.0.0go: downloading github.com/zclconf/go-cty v1.1.0go: downloading github.com/beevik/etree v1.1.0go: downloading github.com/hashicorp/go-hclog v0.9.2go: downloading github.com/gorilla/mux v1.7.3go: downloading golang.org/x/text v0.3.2go: downloading github.com/mitchellh/mapstructure v1.1.2go: downloading github.com/agext/levenshtein v1.2.2go: downloading google.golang.org/grpc v1.23.0go: downloading github.com/vmihailenco/msgpack v3.3.3+incompatiblego: downloading github.com/mitchellh/reflectwalk v1.0.1go: downloading github.com/hashicorp/go-version v1.2.0go: downloading github.com/spf13/afero v1.2.2go: downloading github.com/hashicorp/hcl2 v0.0.0-20190821123243-0c888d1241f6go: downloading github.com/hashicorp/logutils v1.0.0go: downloading github.com/mitchellh/copystructure v1.0.0go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45go: downloading github.com/hashicorp/go-getter v1.4.0go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500dbgo: downloading golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586go: downloading github.com/davecgh/go-spew v1.1.1go: downloading github.com/apparentlymart/go-textseg v1.0.0go: downloading github.com/hashicorp/go-plugin v1.0.1go: downloading google.golang.org/api v0.9.0go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3dgo: downloading github.com/aws/aws-sdk-go v1.19.39go: downloading golang.org/x/net v0.0.0-20200114155413-6afb5195e5aago: downloading github.com/mitchellh/go-wordwrap v1.0.0go: downloading cloud.google.com/go v0.45.1go: downloading github.com/google/go-cmp v0.3.0go: downloading github.com/hashicorp/hil v0.0.0-20190212112733-ab17b08d6590go: downloading github.com/hashicorp/errwrap v1.0.0go: downloading github.com/mitchellh/cli v1.0.0go: downloading github.com/gibson042/canonicaljson-go v1.0.3go: downloading github.com/oklog/run v1.0.0go: downloading github.com/armon/go-radix v1.0.0go: downloading github.com/mitchellh/go-homedir v1.1.0go: downloading github.com/ulikunitz/xz v0.5.5go: downloading github.com/hashicorp/go-cleanhttp v0.5.1go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55go: downloading github.com/hashicorp/go-uuid v1.0.1go: downloading github.com/golang/protobuf v1.3.2go: downloading github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75dgo: downloading github.com/hashicorp/terraform-config-inspect v0.0.0-20190821133035-82a99dc22ef4go: downloading github.com/posener/complete v1.2.1go: downloading github.com/hashicorp/go-safetemp v1.0.0go: downloading github.com/bgentry/speakeasy v0.1.0go: downloading github.com/hashicorp/hcl v0.0.0-20170504190234-a4b07c25de5fgo: downloading github.com/mattn/go-isatty v0.0.4go: downloading github.com/googleapis/gax-go/v2 v2.0.5go: downloading github.com/zclconf/go-cty-yaml v1.0.1go: downloading github.com/fatih/color v1.7.0go: downloading github.com/mitchellh/go-testing-interface v1.0.0go: downloading github.com/mattn/go-colorable v0.0.9go: downloading go.opencensus.io v0.22.0go: downloading github.com/apparentlymart/go-cidr v1.0.1go: downloading github.com/hashicorp/golang-lru v0.5.1go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439afWhen we look in the directory we see the following file “terraform-provider-nsxt”:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls -ltotal 88824-rw-r--r-- 1 iwanhoogendoorn staff 8857 Mar 16 15:04 CHANGELOG.md-rw-r—r— 1 iwanhoogendoorn staff 2422 Mar 16 15:04 CONTRIBUTING.md-rw-r—r— 1 iwanhoogendoorn staff 1839 Mar 16 15:04 GNUmakefile-rw-r—r— 1 iwanhoogendoorn staff 15248 Mar 16 15:04 LICENSE.txt-rw-r—r— 1 iwanhoogendoorn staff 485 Mar 16 15:04 NOTICE.txt-rw-r—r— 1 iwanhoogendoorn staff 7322 Mar 16 15:04 README.md-rw-r—r— 1 iwanhoogendoorn staff 456 Mar 16 15:23 go.mod-rw-r—r— 1 iwanhoogendoorn staff 28565 Mar 16 15:04 go.sum-rw-r—r— 1 iwanhoogendoorn staff 427 Mar 16 15:04 main.godrwxr-xr-x 248 iwanhoogendoorn staff 7936 Mar 16 15:04 nsxtdrwxr-xr-x 6 iwanhoogendoorn staff 192 Mar 16 15:04 scripts-rwxr-xr-x 1 iwanhoogendoorn staff 45391620 Mar 16 15:23 terraform-provider-nsxt <-- new filedrwxr-xr-x 10 iwanhoogendoorn staff 320 Mar 16 15:04 toolsdrwxr-xr-x 8 iwanhoogendoorn staff 256 Mar 16 15:04 vendordrwxr-xr-x 4 iwanhoogendoorn staff 128 Mar 16 15:04 websiteihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$We need to copy this file in our working directory where our Terraform script is in that we want to execute for deployment.
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test/terraform-provider-nsxt
ihoogendoor-a01:Test iwanhoogendoorn$ cd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/TestI needed to remove the cached provided, because I executed the “terraform init” before where it used the downloaded “default” NSX-T provider. I want to use the”local” and “just compiled” provider so I need to remove the .terraform directory.
ihoogendoor-a01:Test iwanhoogendoorn$ ls -lahtotal 88664drwxr-xr-x 5 iwanhoogendoorn staff 160B Mar 16 15:36 .drwxr-xr-x 15 iwanhoogendoorn staff 480B Mar 16 15:43 ..drwxr-xr-x 3 iwanhoogendoorn staff 96B Mar 14 22:59 .terraform-rw-r—r— 1 iwanhoogendoorn staff 838B Mar 16 15:28 networks.tf-rwxr-xr-x 1 iwanhoogendoorn staff 43M Mar 16 15:26 terraform-provider-nsxtihoogendoor-a01:Test iwanhoogendoorn$ rm -R .terraform/Now lets run the “init” command:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24[INFO] Switching to v0.12.24[INFO] Switching completedihoogendoor-a01:Test iwanhoogendoorn$ terraform init
Initializing the backend...
Initializing provider plugins…
Terraform has been successfully initialized!You may now begin working with Terraform. Try running “terraform plan” to see any changes that are required for your infrastructure. All Terraform commands should now work.
If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialise your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
And when we check the version we see the “unversioned” remark, and this tells us we are using our NSX-T provider we have just compiled.
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24[INFO] Switching to v0.12.24[INFO] Switching completedihoogendoor-a01:Test iwanhoogendoorn$ terraform versionTerraform v0.12.24+ provider.nsxt (unversioned)So we are ready to use the “newest” version of the NSX-T provider that is required to deploy our NSX-T segments.
Validate your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform validatePlan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform planExecute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform applyWhen the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroyThe terraform provider is currently under development. If you download and compile the latest version (that is still under development) it can be that old functionality that was working first, is now broken. To go back to a specific version with specific commits you can do this by following the below procedure (Thank you Rutger Blom for this):
Browse to a new fresh directory:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cd terraform-provider-nsxt-v0/The directory is really empty:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ lsClone the full repository:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.gitCloning into 'terraform-provider-nsxt'...remote: Enumerating objects: 142, done.remote: Counting objects: 100% (142/142), done.remote: Compressing objects: 100% (87/87), done.remote: Total 15163 (delta 84), reused 78 (delta 54), pack-reused 15021Receiving objects: 100% (15163/15163), 16.25 MiB 8.33 MiB/s, done.Resolving deltas: 100% (8351/8351), done.Checking out files: 100% (4088/4088), done.Now lets browse into the directory:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ lsterraform-provider-nsxtihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ cd terraform-provider-nsxtihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ lsCHANGELOG.md GNUmakefile NOTICE.txt go.mod main.go scripts vendorCONTRIBUTING.md LICENSE.txt README.md go.sum nsxt tools websiteFetch a specific version + commits:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git fetch origin c01d221ceb98e114aadd84126707633d25fcb43bFrom https://github.com/terraform-providers/terraform-provider-nsxt * branch c01d221ceb98e114aadd84126707633d25fcb43b ->ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git reset --hard is now at c01d221 Policy block subnet - wait for delete realizationCompile the specific version :
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go buildgo: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.1.1ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$‘Copy the new provider to the repository directory of your NSX-T Terraform code:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/#create-T1Sources
Originally published on the old wiki at /index.php/Terraform:_NSX-T_infra_segments.