Part 9: VCF Day 2 Operations

Day 2 Operations in VCF

Getting VCF deployed is Day 1. Day 2 is everything that comes after: patching, certificate renewals, password rotations, capacity expansions, backup and recovery, health monitoring, and troubleshooting. VCF’s biggest operational advantage over standalone VMware is that SDDC Manager centralizes and automates almost all of these tasks. This article is a deep dive into what you’ll actually do after the bring-up is complete.

SDDC Manager Dashboard

SDDC Manager’s web UI (FQDN configured during bring-up, e.g., sddc-manager.vcf.local) is your operational command center. Key dashboard sections:

• Workload Domains — Lists all domains with health status, host count, and resource summary.
• Hosts — Shows all commissioned hosts, their domain assignment, and validation status.
• Lifecycle Management (LCM) — The patching and upgrade hub. Shows current component versions, available updates, and update history.
• Certificate Management — View and rotate TLS certificates for all managed components.
• Password Management — View, rotate, and schedule password rotations for all service accounts.
• Developer Center — API explorer for SDDC Manager’s REST API, with Swagger UI and downloadable OpenAPI spec.

Lifecycle Management (LCM) Deep Dive

LCM is the most critical Day 2 function in VCF. It ensures your stack stays on a validated BOM and handles the complexity of updating interdependent components in the correct sequence.

How LCM Works

1. Bundle sync: SDDC Manager connects to Broadcom’s depot (depot.vmware.com) and downloads update bundle metadata. If your SDDC Manager is air-gapped, you use the Offline Bundle Transfer Utility (OBTU) to manually upload bundles.
2. Bundle download: When a bundle is available (e.g., VCF 9.1.1), you initiate download from the UI. Bundles include pre-checks, rollback mechanisms, and all component update payloads (ESXi VIBs, vCenter ISO, NSX upgrade package).
3. Pre-check validation: Before applying any update, LCM runs pre-checks: sufficient disk space on datastores, all hosts connected, no vSAN resync in progress, DRS enabled, HA enabled. Pre-checks fail if any condition is violated.
4. Coordinated upgrade sequence: LCM updates components in a validated order to maintain interoperability at every step. Example sequence for a VCF minor update: NSX upgrade → ESXi rolling update (vSAN maintenance mode, one host at a time) → vCenter update → SDDC Manager update.
5. Rollback: If an update fails mid-process, LCM supports rollback for most component updates. Always take a snapshot of SDDC Manager before major upgrades.

Update Bundles for Air-Gapped Environments

Many enterprises run VCF in environments without internet access. For these environments:

• Use the Offline Bundle Transfer Utility (OBTU) — a standalone tool that downloads bundles on a connected system and transfers them via USB/file share to the air-gapped SDDC Manager
• Configure SDDC Manager’s depot settings to point to an internal web server hosting the bundles
• Alternative: use the SDDC Manager API to stage bundles via authenticated upload

Certificate Management

VCF manages TLS certificates for all components: vCenter Server, NSX Manager, SDDC Manager itself, and ESXi hosts. Certificates are issued by one of three supported Certificate Authorities:

• VMware Certificate Authority (VMCA) — The built-in CA integrated with vCenter. Simple to use but certificates are self-signed and may not be trusted by enterprise browsers/clients by default.
• Microsoft CA (MSCA) — Enterprise Microsoft Active Directory Certificate Services. SDDC Manager integrates via MSCA API — certificates are signed by your enterprise CA and are automatically trusted by domain-joined systems.
• HashiCorp Vault — Supported in VCF 9.x as an enterprise secrets management integration. Best for organizations already using Vault as their PKI.

Certificate Rotation via SDDC Manager

Certificates approaching expiration are flagged in the SDDC Manager Certificate Management dashboard. Rotation is a 2-click operation — SDDC Manager issues new certificates from the configured CA, pushes them to each component, and restarts services in sequence with no manual intervention.

Password Management

SDDC Manager tracks credentials for all service accounts in the VCF stack: vCenter admin, NSX admin, SDDC Manager admin, ESXi root, vSAN health service, and more. Key capabilities:

• View passwords: SDDC Manager stores all credentials in an encrypted internal vault. Admins with sufficient privileges can view current credentials via UI or API.
• Rotate passwords: On-demand or scheduled rotation. SDDC Manager rotates the password on the component AND updates all dependent services that use that credential.
• Rotation schedule: Configure automatic rotation policies (e.g., every 90 days) per credential type for compliance requirements.
• Audit log: All credential views and rotations are logged for compliance audit trails.

Backup and Recovery

VCF does not provide built-in backup for workload VMs — that’s handled by separate backup solutions (Veeam, Commvault, etc. integrated with vSphere). However, SDDC Manager manages backups for platform management components:

SDDC Manager Backup

• SDDC Manager supports scheduled backups of its configuration database to an external SFTP server
• Configure in SDDC Manager UI → Administration → Backup
• Recovery: deploy a fresh SDDC Manager OVA and restore from backup — SDDC Manager reconnects to all managed components

vCenter Server Backup

• vCenter’s built-in File-Based Backup (FBB) backs up vCenter configuration (not VMs) to SFTP, FTP, HTTP, HTTPS, or NFS
• Configure in vCenter VAMI (https://vcenter:5480) → Backup
• Recovery via vCenter Recovery tool from VAMI — restores full vCenter configuration including host inventory, cluster settings, distributed switches

NSX Manager Backup

• NSX Manager maintains its own configuration backup to SFTP/SCP
• Configure in NSX Manager UI → System → Backup & Restore
• NSX backup includes all policies, segments, gateways, edge configurations, and DFW rules

Health Monitoring

vSAN Health Service

Accessible via vCenter → vSAN → Skyline Health. Runs 200+ health checks across the vSAN cluster. Pay special attention to:

• Disk capacity: Alert at 70%, critical at 80% — vSAN performance degrades significantly above 80% capacity
• Resync operations: Ongoing resyncs reduce cluster resilience. Monitor after any host maintenance or disk failure
• Controller firmware: vSAN validates NVMe/SAS controller firmware against VCG — outdated firmware causes health warnings

NSX Alert Monitoring

NSX Manager → Monitor → Alarms provides visibility into:

• Transport node connectivity (TEP reachability between hosts)
• Edge node health (CPU, memory, BGP session status)
• Certificate expiration warnings for NSX components

SDDC Manager Integrated Alerts

SDDC Manager surfaces critical alerts from all managed components in a unified view. Integrate SDDC Manager alerts with your enterprise monitoring platform via its REST API or configure email alerts in Administration → Email Notifications.

Troubleshooting Common VCF Issues

• LCM bundle download fails: Check SDDC Manager connectivity to depot.vmware.com (port 443). Verify proxy settings in SDDC Manager → Administration → Network Settings.
• Host commissioning fails: Run SDDC Manager’s Host Validation tool — it checks ESXi version, NIC configuration, vSAN disk availability, and network pool IP availability.
• vSAN resync stuck: Check for host connectivity issues (vmkping across vSAN VMkernel) and controller firmware compatibility. Check vSAN Health for component health details.
• NSX TEP connectivity issues: Verify MTU 9000 is configured on the TEP VLAN end-to-end. Use esxcli network ip interface ipv4 get to verify TEP IPs are assigned correctly.

Broadcom Documentation

• VCF Lifecycle Management — TechDocs
• VCF Certificate Management
• VCF Backup and Restore
• SDDC Manager Password Management