- Introduction
- Prerequisites
- Important Configuration Notes
- BYOL Licensing
- BIG-IQ License Manager
- Installation Example
- Configuration Example
This solution uses a Terraform template to launch a 3-NIC deployment of a cloud-focused BIG-IP VE cluster (Active/Standby) in Google GCP. Traffic flows to the BIG-IP VE which then processes the traffic to application servers. The BIG-IP VE instance is running with multiple interfaces: management, external, internal. NIC0 is associated with the external network.
The BIG-IP VEs have the Local Traffic Manager (LTM) module enabled to provide advanced traffic management functionality. In addition, the Application Security Module (ASM) can be enabled to provide F5's L4/L7 security features for web application firewall (WAF) and bot protection.
Terraform is beneficial as it allows composing resources a bit differently to account for dependencies into Immutable/Mutable elements. For example, mutable includes items you would typically frequently change/mutate, such as traditional configs on the BIG-IP. Once the template is deployed, there are certain resources (network infrastructure) that are fixed while others (BIG-IP VMs and configurations) can be changed.
Example...
-> Run once
- Deploy the entire infrastructure with all the neccessary resources, then use Declarative Onboarding (DO) to configure the BIG-IP cluster, Application Services (AS3) to create a sample app proxy, then lastly use Service Discovery to automatically add the DVWA container app to the BIG-IP pool.
-> Run many X
Networking Stack Type: This solution deploys into an EXISTING networking stack. You are required to have existing VPC networks, firewall rules, and proper routing. Refer to the Prerequisites. Visit DevCentral to read Service Discovery in Google Cloud with F5 BIG-IP where I show you my basic VPC setup (networks, subnets) along with firewall rules.
- Important: When you configure the admin password for the BIG-IP VE in the template, you cannot use the character #. Additionally, there are a number of other special characters that you should avoid using for F5 product user accounts. See K2873 for details.
- This template requires one or more service accounts for the BIG-IP instance to perform various tasks:
- See Google's Understanding service accounts
- Google Secret Manager secrets access - requires "Secrets Manager Secret Accessor"
- Performed by VM instance during onboarding to retrieve passwords and private keys
- Backend pool service discovery - requires "Compute Viewer"
- Performed by F5 Application Services AS3
- Google Cloud Monitoring (aka StackDriver) - requires "Monitoring Editor"
- Performed by F5 Telemetry Streaming
- Cloud failover via API - requires R/W access to compute and storage (see F5 CloudDocs Create and assign an IAM role)
- This template requires a service account to deploy with the Terraform Google provider and build out all the neccessary Google objects
- See the Terraform Google Provider "Adding Credentials" for details. Also, review the available Google GCP permission scopes too.
- Permissions will depend on the objects you are creating
- My service account for Terraform deployments in GCP uses the following roles:
- Compute Admin
- Storage Admin
- Service Account User
- Service Account Admin
- Project IAM Admin
- Note: Make sure to practice least privilege
- Shared Service Accounts: For lab purposes, you can create one service account and use it for everything. Alternatively, you can create a more secure environment with separate service accounts for various functions. Example...
- Service Account #1 - the svc-acct used for Terraform to deploy cloud objects
- Service Account #2 - the svc-acct assigned to BIG-IP instance during creation (ex. service discovery, query Pub/Sub, storage, failover)
- Service Account #3 - the svc-acct used in F5 Telemetry Streaming referenced in ts.json (ex. analytics)
- Passwords and secrets are located in Google Cloud Secret Manager. Make sure you have an existing Google Cloud "secret" with the data containing the clear text passwords for each relevant item: BIG-IP password, service account credentials, BIG-IQ password, etc.
- 'usecret' contains the value of the adminstrator password (ex. "Default12345!")
- 'ksecret' Contains the value of the 'svc_acct' private key. Currently used for BIG-IP telemetry streaming to Google Cloud Monitoring (aka StackDriver). If you are not using this feature, you do not need this secret in Secret Manager.
- Refer to Template Parameters
- This template deploys into an existing network
- You must have a VPC for management and a VPC for data traffic (client/server). The management VPC will have one subnet for management traffic. The External VPC will have one subnet for data traffic. The Internal VPC will have one subnet as well.
- Firewall rules are required to pass traffic to the application
- BIG-IP will require tcp/22 and tcp/443 on the mgmt network
- Application access will require tcp/80 and tcp/443 on the external network
- Storage bucket is used for F5 Cloud Failover. See F5 Cloud Failover GCP Setup.
- If you require a new network first, see the Infrastructure Only folder to get started.
- The parameter 'dns_suffix' must match the DNS suffix assigned by the GCP project. You can retrieve this value by logging into an existing VM in the same project and running 'uname -a' or reviewing the /etc/resolv.conf file. Failure to properly set 'dns_suffix' will result in failed hostname lookup during HA setup.
- Variables are configured in variables.tf
- Sensitive variables like Google SSH keys are configured in terraform.tfvars
- Note: Other items like BIG-IP password are stored in Google Cloud Secret Manager. Refer to the Prerequisites.
- The BIG-IP instance will query Google Metadata API to retrieve the service account's token for authentication.
- The BIG-IP instance will then use the secret name and the service account's token to query Google Metadata API and dynamically retrieve the password for device onboarding.
- This template uses Declarative Onboarding (DO), Application Services 3 (AS3), and Cloud Failover Extension packages for the initial configuration. As part of the onboarding script, it will download the RPMs automatically. See the AS3 documentation and DO documentation for details on how to use AS3 and Declarative Onboarding on your BIG-IP VE(s). The Telemetry Streaming extension is also downloaded and can be configured to point to F5 Beacon, Google Cloud Monitoring (old name StackDriver), or many other consumers. The Cloud Failover Extension documentation is also available.
- Files
- bigip.tf - resources for BIG-IP, NICs, public IPs
- main.tf - resources for provider, versions
- onboard.tpl - onboarding script which is run by startup-script (user data). It will be copied to startup-script=path-to-file upon bootup. This script is responsible for downloading the neccessary F5 Automation Toolchain RPM files, installing them, and then executing the onboarding REST calls.
- do.json - contains the L1-L3 BIG-IP configurations used by DO for items like VLANs, IPs, and routes
- as3.json - contains the L4-L7 BIG-IP configurations used by AS3 for items like pool members, virtual server listeners, security policies, and more
- ts.json - contains the BIG-IP configurations used by TS for items like telemetry streaming, CPU, memory, application statistics, and more
- cfe.json - contains the BIG-IP configurations used for failover operations of cloud objects like IPs and routes
This template uses PayGo BIG-IP image for the deployment (as default). If you would like to use BYOL licenses, then these following steps are needed:
- Find available images/versions with "byol" in the name using Google gcloud:
gcloud compute images list --project=f5-7626-networks-public | grep f5
# example output...
--snippet--
f5-bigip-13-1-3-2-0-0-4-payg-best-1gbps-20191105210022
f5-bigip-13-1-3-2-0-0-4-payg-best-200mbps-20191105210022
f5-bigip-13-1-3-2-0-0-4-byol-all-modules-2slot-20191105200157
...and some more
f5-bigip-14-1-2-3-0-0-5-byol-ltm-1boot-loc-191218142225
f5-bigip-15-1-2-1-0-0-10-payg-best-1gbps-210115161130
f5-bigip-15-1-2-1-0-0-10-byol-ltm-2boot-loc-210115160742
...and more...
- In the "variables.tf", modify image_name with the image name from gcloud CLI results
# BIGIP Image
variable image_name { default = "projects/f5-7626-networks-public/global/images/f5-bigip-15-1-2-1-0-0-10-byol-ltm-2boot-loc-210115160742" }
- In the "variables.tf", modify license1 with a valid regkey
# BIGIP Setup
variable license1 { default = "" }
- In the "do.json", add the "myLicense" block under the "Common" declaration (full declaration example here)
"myLicense": {
"class": "License",
"licenseType": "regKey",
"regKey": "${regKey}"
},
This template uses PayGo BIG-IP image for the deployment (as default). If you would like to use BYOL/ELA/Subscription licenses from BIG-IQ License Manager (LM), then these following steps are needed:
- Find BYOL image. Reference BYOL Licensing step #1.
- Replace BIG-IP image_name in "variables.tf". Reference BYOL Licensing step #2.
- In the "variables.tf", modify the BIG-IQ license section to match your environment
- In the "do.json", add the "myLicense" block under the "Common" declaration (full declaration example here)
"myLicense": {
"class": "License",
"licenseType": "${bigIqLicenseType}",
"bigIqHost": "${bigIqHost}",
"bigIqUsername": "${bigIqUsername}",
"bigIqPassword": "$${bigIqPassword}",
"licensePool": "${bigIqLicensePool}",
"skuKeyword1": "${bigIqSkuKeyword1}",
"skuKeyword2": "${bigIqSkuKeyword2}",
"unitOfMeasure": "${bigIqUnitOfMeasure}",
"reachable": false,
"hypervisor": "${bigIqHypervisor}",
"overwrite": true
},
Note: The onboard.tpl startup script will use the same 'usecret' payload value (aka password) for BIG-IP password AND the BIG-IQ password. In the onboard.tpl file, this happens in the 'passwd' variable. You can use a separate password for BIG-IQ by creating a new Google Secret Manager secret for the BIG-IQ password, then add a new variable for the secret in variables.tf, modify bigip.tf to include the secret in the local templatefile section similar to 'usecret', then update onboard.tpl to query Secret Manager for the BIG-IQ secret name. Reference code example usecret='${usecret}'.
| Name | Version |
|---|---|
| terraform | ~> 0.14 |
| ~> 3 |
| Name | Version |
|---|---|
| 3.86.0 |
No modules.
| Name | Type |
|---|---|
| google_compute_address.vip1 | resource |
| google_compute_forwarding_rule.vip1 | resource |
| google_compute_instance.f5vm01 | resource |
| google_compute_instance.f5vm02 | resource |
| google_compute_target_instance.f5vm01 | resource |
| google_compute_target_instance.f5vm02 | resource |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| AS3_URL | URL to download the BIG-IP Application Service Extension 3 (AS3) module | string |
"https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.30.0/f5-appsvcs-3.30.0-5.noarch.rpm" |
no |
| CFE_URL | URL to download the BIG-IP Cloud Failover Extension module | string |
"https://github.com/F5Networks/f5-cloud-failover-extension/releases/download/v1.9.0/f5-cloud-failover-1.9.0-0.noarch.rpm" |
no |
| DO_URL | URL to download the BIG-IP Declarative Onboarding module | string |
"https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.23.0/f5-declarative-onboarding-1.23.0-4.noarch.rpm" |
no |
| TS_URL | URL to download the BIG-IP Telemetry Streaming module | string |
"https://github.com/F5Networks/f5-telemetry-streaming/releases/download/v1.22.0/f5-telemetry-1.22.0-1.noarch.rpm" |
no |
| adminSrcAddr | Trusted source network for admin access | string |
"0.0.0.0/0" |
no |
| alias_ip_range | An array of alias IP ranges for the BIG-IP network interface (used for VIP traffic, SNAT IPs, etc) | string |
"10.1.10.100/32" |
no |
| bigIqHost | This is the BIG-IQ License Manager host name or IP address | string |
"" |
no |
| bigIqHypervisor | BIG-IQ hypervisor | string |
"gce" |
no |
| bigIqLicensePool | BIG-IQ license pool name | string |
"" |
no |
| bigIqLicenseType | BIG-IQ license type | string |
"licensePool" |
no |
| bigIqSkuKeyword1 | BIG-IQ license SKU keyword 1 | string |
"key1" |
no |
| bigIqSkuKeyword2 | BIG-IQ license SKU keyword 2 | string |
"key2" |
no |
| bigIqUnitOfMeasure | BIG-IQ license unit of measure | string |
"hourly" |
no |
| bigIqUsername | Admin name for BIG-IQ | string |
"admin" |
no |
| bigipMachineType | Google machine type to be used for the BIG-IP VE | string |
"n1-standard-8" |
no |
| customImage | A custom SKU (image) to deploy that you provide. This is useful if you created your own BIG-IP image with the F5 image creator tool. | string |
"" |
no |
| customUserData | The custom user data to deploy when using the 'customImage' paramater too. | string |
"" |
no |
| dns_server | Leave the default DNS server the BIG-IP uses, or replace the default DNS server with the one you want to use | string |
"8.8.8.8" |
no |
| dns_suffix | DNS suffix for your domain in the GCP project | string |
"example.com" |
no |
| extSubnet | External subnet | string |
null |
no |
| extVpc | External VPC network | string |
null |
no |
| f5_cloud_failover_label | This is a tag used for F5 Cloud Failover Extension to identity which cloud objects to move during a failover event. | string |
"mydeployment" |
no |
| gceSshPubKey | SSH public key for admin authentation | string |
null |
no |
| gcp_project_id | GCP Project ID for provider | string |
null |
no |
| gcp_region | GCP Region for provider | string |
"us-west1" |
no |
| gcp_zone | GCP Zone for provider | string |
"us-west1-b" |
no |
| host1_name | Hostname for the first BIG-IP | string |
"f5vm01" |
no |
| host2_name | Hostname for the second BIG-IP | string |
"f5vm02" |
no |
| image_name | F5 SKU (image) to deploy. Note: The disk size of the VM will be determined based on the option you select. Important: If intending to provision multiple modules, ensure the appropriate value is selected, such as AllTwoBootLocations or AllOneBootLocation. | string |
"projects/f5-7626-networks-public/global/images/f5-bigip-15-1-2-1-0-0-10-payg-best-1gbps-210115161130" |
no |
| intSubnet | Internal subnet | string |
null |
no |
| intVpc | Internal VPC network | string |
null |
no |
| ksecret | Contains the value of the 'svc_acct' private key. Currently used for BIG-IP telemetry streaming to Google Cloud Monitoring (aka StackDriver). If you are not using this feature, you do not need this secret in Secret Manager. | string |
"" |
no |
| license1 | The license token for the first F5 BIG-IP VE (BYOL) | string |
"" |
no |
| license2 | The license token for the second F5 BIG-IP VE (BYOL) | string |
"" |
no |
| managed_route1 | A UDR route can used for testing managed-route failover. Enter address prefix like x.x.x.x/x. | string |
"192.0.2.0/24" |
no |
| mgmtSubnet | Management subnet | string |
null |
no |
| mgmtVpc | Management VPC network | string |
null |
no |
| ntp_server | Leave the default NTP server the BIG-IP uses, or replace the default NTP server with the one you want to use | string |
"0.us.pool.ntp.org" |
no |
| onboard_log | This is where the onboarding script logs all the events | string |
"/var/log/cloud/onboard.log" |
no |
| owner | This is a tag used for object creation. Example is last name. | string |
null |
no |
| prefix | This value is inserted at the beginning of each Google object (alpha-numeric, no special character) | string |
"demo" |
no |
| privateKeyId | ID of private key for the 'svc_acct' used in Telemetry Streaming to Google Cloud Monitoring. If you are not using this feature, you do not need this secret in Secret Manager. | string |
"" |
no |
| svc_acct | Service Account for VM instance | string |
null |
no |
| timezone | If you would like to change the time zone the BIG-IP uses, enter the time zone you want to use. This is based on the tz database found in /usr/share/zoneinfo (see the full list here). Example values: UTC, US/Pacific, US/Eastern, Europe/London or Asia/Singapore. | string |
"UTC" |
no |
| uname | User name for the Virtual Machine | string |
"admin" |
no |
| usecret | Used during onboarding to query the Google Cloud Secret Manager API and retrieve the admin password (use the secret name, not the secret value/password) | string |
null |
no |
| Name | Description |
|---|---|
| f5vm01_ext_selfip | f5vm01 external self IP private address |
| f5vm01_ext_selfip_pip | f5vm01 external self IP public address |
| f5vm01_mgmt_ip | f5vm01 management private IP address |
| f5vm01_mgmt_name | f5vm01 management device name |
| f5vm01_mgmt_pip | f5vm01 management public IP address |
| f5vm01_mgmt_pip_url | f5vm01 management public URL |
| f5vm02_ext_selfip | f5vm02 external self IP private address |
| f5vm02_ext_selfip_pip | f5vm02 external self IP public address |
| f5vm02_mgmt_ip | f5vm02 management private IP address |
| f5vm02_mgmt_name | f5vm02 management device name |
| f5vm02_mgmt_pip | f5vm02 management public IP address |
| f5vm02_mgmt_pip_url | f5vm02 management public URL |
| public_vip | public IP address for application |
| public_vip_url | public URL for application |
To run this Terraform template, perform the following steps:
- Clone the repo to your favorite location
- Modify terraform.tfvars with the required information
# BIG-IP Environment
uname = "admin"
usecret = "my-secret"
gceSshPubKey = "ssh-rsa xxxxx
prefix = "mydemo123"
adminSrcAddr = "0.0.0.0/0"
mgmtVpc = "xxxxx-net-mgmt"
extVpc = "xxxxx-net-ext"
intVpc = "xxxxx-net-int"
mgmtSubnet = "xxxxx-subnet-mgmt"
extSubnet = "xxxxx-subnet-ext"
intSubnet = "xxxxx-subnet-int"
dns_suffix = "c.xxxxx.xxxxx.internal"
# BIG-IQ Environment
bigIqUsername = "admin"
# Google Environment
gcp_project_id = "xxxxx"
gcp_region = "us-west1"
gcp_zone = "us-west1-b"
svc_acct = "[email protected]"
- Initialize the directory
terraform init
- Test the plan and validate errors
terraform plan
- Finally, apply and deploy
terraform apply
- When done with everything, don't forget to clean up!
terraform destroy
The following is an example configuration diagram for this solution deployment. In this scenario, all access to the BIG-IP VE cluster (Active/Standby) is direct to each BIG-IP via the management interface. The IP addresses in this example may be different in your implementation.
For more information on F5 solutions for Google, including manual configuration procedures for some deployment scenarios, see the Google GCP section of F5 CloudDocs. Also check out the Using Cloud Templates for BIG-IP in Google on DevCentral. This particular HA example is based on the BIG-IP Cluster "HA via API" F5 GDM Cloud Template on GitHub.
In order to pass traffic from your clients to the servers through the BIG-IP system, you must create a virtual server on the BIG-IP VE. In this template, the AS3 declaration creates 2 VIPs: one for public internet facing, and one for private internal usage. It is preconfigured as an example.
In this template, the Google public IP address is associated with the active BIG-IP device NIC0. The address is created with a Google Forwarding Rule, and this IP address will be the same IP you see as a virtual server on the BIG-IP.
Note: These next steps illustrate the manual way in the GUI to create a virtual server
- Open the BIG-IP VE Configuration utility
- Click Local Traffic > Virtual Servers
- Click the Create button
- Type a name in the Name field
- Type an address (ex. x.x.x.x/x) in the Destination/Mask field
- Type a port (ex. 443) in the Service Port
- Configure the rest of the virtual server as appropriate
- Select a pool name from the Default Pool list
- Click the Finished button
- Repeat as necessary for other applications
This example illustrates how to replace or upgrade the BIG-IP VE.
- Change the image_name variable to the desired release
- Revoke the problematic BIG-IP VE's license (if BYOL)
- Run command
terraform taint google_compute_instance.f5vm01
terraform taint google_compute_instance.f5vm02
terraform taint google_compute_target_instance.f5vm01
terraform taint google_compute_target_instance.f5vm02
terraform taint google_compute_forwarding_rule.vip1
- Run command
terraform apply