VCF 9.1 Lab Part 11: Certificates

Why certificates, and why now

Everything in a fresh VCF 9.1 fleet wears a certificate signed by the VMware Certificate Authority built into vCenter. It works, and every browser complains about it. I run a Windows certificate authority in the lab for everything else, so I wanted VCF to use it too. One root, one trust store, no warnings.

This part covers the management domain: vCenter, both NSX Manager entries, SDDC Manager and the log management service from Part 10. The ESX hosts and the rest of the fleet components come in the next post.

Two places to set the CA

9.1 splits the job. VCF Operations looks after the fleet components, which means log management, the management services and the rest of that family. SDDC Manager looks after vCenter, NSX Manager, the ESX hosts and itself. You point each one at your CA separately, and each has its own certificate screen.

One detail decides whether a certificate is accepted: 9.1 wants both the FQDN and the IP address in the Subject Alternative Name. The SDDC Manager dialog only lists the FQDN, so you type the IP yourself.

My CA was not ready

VCF does not talk to the Certificate Authority service directly. It talks to the web enrollment pages on the CA server, over HTTPS, with basic authentication. My CA had the authority role and nothing else. No web enrollment, no IIS bindings, no web server certificate. The first sign was a connection timeout on the enrollment page.

So the first job was fixing the CA: add the web enrollment role, give IIS a proper certificate, turn on basic authentication, create a service account with no admin rights, and publish a certificate template for it to use.

Certificate Templates list on the CA with the new VCF template selected

Two things cost me time. Templates are created in the template console, not the main CA console, which only lets you publish or delete them. I nearly deleted the Web Server template looking for a Duplicate button. And a new template does not appear in the publish list until Active Directory has replicated it.

Replace, do not renew

In VCF Operations the certificate screen has a Renew button at the top. I almost clicked it. The dialog says VMCA and 730 days, so it issues a new certificate from the same VMware CA that signed the old one. Nothing changes except the date.

Renew Certificates dialog showing the type is VMCA

To use your own CA you generate a signing request first, then pick Replace With Configured CA Certificate from the three-dots menu. It is hidden one level down.

Three-dots menu with Replace With Configured CA Certificate highlighted

The first certificate

I started with log management, because I could confirm it worked by watching logs arrive. The replacement took ten minutes. The certificate now shows my root as the issuer, and Operate, Logs carried on receiving events without a gap.

Certificate details for log management issued by the lab root CA

SDDC Manager would not save the CA

Same details, same account, different console. SDDC Manager answered with “Unable to create CA” and a reference token.

Configure Certificate Authority dialog in SDDC Manager showing Unable to create CA

The log said “Failed to authenticate with CA server”, so I tested the login straight from the SDDC Manager shell with curl. It returned 401. On the CA server, the Security log (event 4625) gave the reason for each attempt. With the domain in front of the name, the failure code meant the account existed and the password was wrong. With a bare username, the code meant no such user. IIS basic authentication treats a bare name as a local account on the server, so that form can never work.

Two fixes. I reset the service account password and typed it carefully, and I entered the username as DOMAIN\name. The curl test returned 200 and SDDC Manager saved the CA.

The lesson: VCF Operations had accepted the same settings without testing them, and it keeps its own copy of the password. After the reset I updated it there too.

vCenter, NSX and SDDC Manager

The workflow in SDDC Manager is the same for each component. Open the domain, go to the Certificates tab, tick the row, generate the signing request, generate the signed certificate from the Microsoft CA, then install it. I did vCenter on its own first, both NSX Manager rows together, and SDDC Manager last because its own interface restarts during the install.

Certificates tab with the vCenter certificate issued by the lab root CA

Each install restarted services and took a few minutes. Remember to add the IP to the Subject Alternative Name before you generate the request, or the certificate comes back without it.

vCenter and SDDC Manager rows issued by the lab root CA

Result

vCenter, both NSX Manager entries and SDDC Manager are now signed by my own root, valid for two years, as set by the template. I expected the VCF Operations connections to complain after the rotation. They did not. Every account in Integrations still shows Collecting.

VCF Operations Integrations page with all accounts collecting

Lesson

Check the CA side before you touch VCF. Web enrollment, HTTPS, basic authentication and a working service account are all needed, and the errors from VCF will not tell you which one is missing. When a login fails, ask the CA server’s Security log what it saw. It answers in one line.

Next

The ESX hosts and the remaining fleet components: VCF Operations itself, the identity broker, the services runtime and VCF Automation. I want the management services backup working before I touch the identity broker, so that comes first.