Replace ESX host certificates with CA signed wildcard certificate using PowerShell!

Leave a reply

Recently we were requested to install CA signed certificates on our ESX hosts to pass a security audit.

The thought of doing this manually bored me! so I wrote the following script – which recursively puts each host into maintenance, installs new certificate, then reboots the host, takes it out of maintenance and tests the certificate! The script also produces a detailed log file.

This script requires plink and pscp EXE files to be present in the C:\Windows folder.
Download @ https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

You need access to vCenter over port 443 from the machine you are running this script from.

You need access to your hosts over port 22 from the machine you are running this script from.

You need to generate a CA signed cert and Private key prior to running this script – rename cert as rui.crt, key as rui.key, and place both in a local directory on the machine you are running this script from – there are plenty of blog posts out there with instructions on how to setup a local MS CA, generate a cert and key. Or check out my blog post about replacing machine SSL certs on PSC servers using MS CA – it covers the required steps.

We decided to use a WILDCARD cert – so COMMON NAME and SUBJECT ALTERNATIVE NAME parameters for cert was defined as *.domain.com

Make sure to add your root and intermediate certs into the trusted certs store on vCenter – you can use the UI for this now.

Recommend disabling vSphere HA for the duration process, it does get confused after the cert is changed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
#############################################################################
#       Author: Cengiz Ulusahin
#       Version: 1.3
#       Date: 29/03/2021
#       Description: Replace ESX self-signed certs with a CA signed wildcard
#       certificate
#      
#       !IMPORTANT NOTES!
#
#       This script requires plink and pscp EXE files to be present in the
#       C:\Windows folder. 
#       Download @ https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
#       You need access to vCenter over port 443 from the machine you are running
#       this script from.
#       You need access to your hosts over port 22 from the machine you are running
#       this script from.
#       You need to generate a CA signed cert and Private key prior to running
#       this script - rename cert as rui.crt, key as rui.key, and place both in
#       a local directory on the machine you are running this script from.
#       Make sure to add your root and intermediate certs into the trusted certs
#       store on vCenter.
#      
#       !WARNING!
#
#       This script will recursively restart your ESX hosts after putting them
#       in maintenance.
#
#       Recommend disabling vSphere HA for the duration process,
#       it does get confused with the cert change.
#
#############################################################################
 
#Function to generate wait time progress bar
function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while ($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping" -Status "Sleeping..." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Sleeping" -Status "Sleeping..." -SecondsRemaining 0 -Completed
}
 
#Function for Logging
Function Write-Log {
    Param ([string]$logstring)
    $stamp = (Get-Date).toString("yyyy-MM-dd HH:mm:ss")
    $line = "$stamp $logstring"
    Add-content $logfile -value $line
}
 
#Function to check SSL cert
#Put this together using existing functions detailed in links below
#https://docs.microsoft.com/en-us/archive/blogs/parallel_universe_-_ms_tech_blog/reading-a-certificate-off-a-remote-ssl-server-for-troubleshooting-with-powershell
#https://dscottraynsford.wordpress.com/2016/12/24/test-website-ssl-certificates-continuously-with-powershell-and-pester/
#https://www.powershellgallery.com/packages/Remoting/0.2.1.4/Content/functions%5CGet-RemoteCert.ps1
function check_ssl {
    #Create a TCP Socket to the host and a port number
    $tcpsocket = New-Object System.Net.Sockets.Socket( `
            [System.Net.Sockets.SocketType]::Stream,
        [System.Net.Sockets.ProtocolType]::Tcp)
    $tcpsocket.connect($esx, $port)
 
    #Test if the socket got connected
    if (!$tcpsocket) {
        Write-Host 'Error Opening Connection: 443 on' $esx 'Host unreachable, cannot test cert, make sure host is functioning, terminating script now!'
        Write-Log ("Error Opening Connection: 443 on $($esx) Host unreachable, cannot test cert, make sure host is functioning, terminating script now!")
        Exit
    }
    else {
        #Socket got connected get the TCP stream ready to read the certificate
        Write-Host 'Successfully connected to' $esx 'on port' $port
        $tcpstream = New-Object System.Net.Sockets.NetworkStream($tcpsocket, $true)
        Write-Host 'Reading SSL certificate...'
        #Create an SSL Connection
        $sslstream = New-Object System.Net.Security.SslStream($tcpstream, $true)
        #Force the SSL Connection to send us the certificate
        $sslstream.AuthenticateAsClient($esx, $null, $protocolname, $false)
        #Read the certificate
        $certinfo = [System.Security.Cryptography.X509Certificates.X509Certificate2]$sslstream.remotecertificate
        if ($certinfo.SerialNumber -eq $wildcardserial) {
            Write-Host 'Certificate on' $esx 'OK!'
            Write-Log ("Certificate on $($esx) OK!")
        }
        else {
            Write-Host 'Certificate test on' $esx 'failed, check cert manually, moving on to next host!'
            Write-Log ("Certificate test on $($esx) failed, check cert manually, moving on to next host!")
        }
    }  
}
 
#Variables
$vcserver = Read-Host -Prompt 'Enter vCenter name'
$cluster = Read-Host -Prompt 'Enter Host Cluster name'
$vccreds = Get-Credential -Message "Please enter VC creds"
$esxcreds = Get-Credential -Message "Please enter ESX SSH creds"
$esxnewsslpath = Read-Host -Prompt 'Enter path to CA signed cert and key e.g c:\temp\new-cert'
$logfilepath = Read-Host -Prompt 'Enter path to log file e.g c:\temp\new-cert'
$date = (Get-Date).toString("yyyy-MM-dd_HH-mm-ss")
$logfile = New-Item $logfilepath\log-file-$date.txt -Force
$wildcardserial = Read-Host -Prompt 'Enter wildcard cert serial'
 
connect-viserver $vcserver -username $vcusername -credential $vccreds
 
Write-Host 'Connected to' $vcserver
Write-Log ("Connected to $($vcserver)")
 
$esxhosts = Get-Cluster $cluster | Get-VMHost
#Use below to test script by importing multiple host names from CSV
#$csv = (Import-Csv $esxnewsslpath\esx-hosts.csv).Name
#$esxhosts = Get-VMHost $csv
#Use below to test script on a single host
#$esxhosts = Get-VMHost HOST NAME
 
foreach ($esx in $esxhosts) {
 
    Write-Host 'Putting' $esx 'in Maintenance Mode'
    Write-Log ("Putting $($esx) in Maintenance Mode")
     
    # Place the selected host into Maintenance Mode.
    $esx | Set-vmhost -State Maintenance
 
    while ((Get-VMHost $esx).ConnectionState -ne 'Maintenance') {
           
        Start-Sleep -Seconds 10
    }
 
    Write-Host $esx 'is in Maintenance Mode'
    Write-Log ("$($esx) is in Maintenance Mode")
 
    Write-Host 'Checking' $esx 'Lockdown Mode status'
    Write-Log ("Checking $($esx) Lockdown Mode status")
 
    If ((($esx | Get-View).config.admindisabled) -eq $true) {
 
        Write-Host 'Disabling Lockdown Mode on' $esx
        Write-Log ("Disabling Lockdown Mode on $($esx)")
 
        #Disable Lockdown Mode
        ($esx | Get-View).ExitLockdownMode()
    }
    else {
 
        Write-Host 'Lockdown Mode on' $esx 'is already disabled'
        Write-Log ("Lockdown Mode on $($esx) is already disabled")   
    }
 
    #More variables
    $esxusername = $esxcreds.GetNetworkCredential().username   
    $esxpassword = $esxcreds.GetNetworkCredential().password
    $esxsslpath = "/etc/vmware/ssl"
 
    Write-Host 'Checking SSH service status on' $esx
    Write-Log ("Checking SSH service status on $($esx)")
 
    #Start SSH service on Host
    $sshservice = (Get-VMHostService -VMHost $esx -Server $vcserver | Where { $_.Key -eq "TSM-SSH" })
 
    if ($sshservice.Running -eq $false) {
         
        Write-Host 'Starting SSH service on' $esx
        Write-Log ("Starting SSH service on $($esx)")
 
        Start-VMHostService -HostService $sshservice -Confirm:$false
    }
    else {
 
        Write-Host 'SSH service is already running on' $esx
        Write-Log ("SSH service is already running on $($esx)")  
    }
 
    Write-Host 'Checking SSH credentials for' $esx
    Write-Log ("Checking SSH credentials for $($esx)")
 
    #Check ESX authentication
    #Run once with no -batch to accept ssh key
    Echo "y" | pscp.exe -scp -pw $esxpassword -ls $esxusername@"$esx":$esxsslpath
    #Run again with -batch to populate variable
    $auth = (Echo "y" | pscp.exe -scp -batch -pw $esxpassword -ls $esxusername@"$esx":$esxsslpath)
 
    if ($auth -eq $null) {
     
        Write-Host 'Authentication check for' $esx 'was unsuccessful, check your ESX root creds and start again, terminating script now!'
        Write-Log ("Authentication check for $($esx) was unsuccessful, check your ESX root creds and start again, terminating script now")
        Stop-VMHostService -HostService $sshservice -Confirm:$false
        ($esx | Get-View).EnterLockdownMode()
        Exit  
    }
    else {
 
        Write-Host 'Authentication check for' $esx 'was successful!'
        Write-Log ("Authentication check for $($esx) was successful!")
    }
 
    Write-Host 'Creating ESX cert backup path on' $esx
    Write-Log ("Creating ESX cert backup path on $($esx)")
 
    #Get ESX scratch partition
    $esxscratch = $esx | Get-AdvancedSetting -Name "ScratchConfig.CurrentScratchLocation"
    $esxsslbkppath = $esxscratch.value
 
    #Create folder on ESX scratch partition to backup certs
    echo "y" | plink.exe -batch -ssh -pw $esxpassword $esxusername@$esx "mkdir -p $esxsslbkppath/certs"
 
    Write-Host 'Backing up ESX cert and key to backup path'
    Write-Log ("Backing up ESX cert and key to backup path $($esxsslbkppath)")
     
    #Backup current certs
    echo "y" | plink.exe  -batch -ssh -pw $esxpassword $esxusername@"$esx" "cp -f $esxsslpath/rui.crt $esxsslbkppath/certs"
    echo "y" | plink.exe  -batch -ssh -pw $esxpassword $esxusername@"$esx" "cp -f $esxsslpath/rui.key $esxsslbkppath/certs"
 
    Write-Host 'Uploading new ESX cert and key'
    Write-Log ("Uploading new ESX cert and key to $($esx)")
 
    #Upload new certs
    echo "y" | pscp.exe -scp -pw $esxpassword "$esxnewsslpath\rui.crt" $esxusername@"$esx":"$esxsslpath" 
    echo "y" | pscp.exe -scp -pw $esxpassword "$esxnewsslpath\rui.key" $esxusername@"$esx":"$esxsslpath"
 
    Write-Host 'Stopping SSH service on' $esx
    Write-Log ("Stopping SSH service on $($esx)")
 
    #Stop SSH service on Host
    Stop-VMHostService -HostService $sshservice -Confirm:$false
 
    Write-Host 'Enabling Lockdown Mode on' $esx
    Write-Log ("Enabling Lockdown Mode on $($esx)")
 
    #Enable Lockdown mode
    ($esx | Get-View).EnterLockdownMode()
 
    Write-Host 'Rebooting' $esx
    Write-Log ("Rebooting $($esx)")
     
    #Reboot host
    Restart-VMHost $esx -confirm:$false -force
 
    while ((Get-VMHost $esx).ConnectionState -ne 'NotResponding') {
 
        Start-Sleep -Seconds 10
    }
 
    Write-Host 'Still rebooting' $esx 
    Write-Log ("Still rebooting $($esx)")
 
    while ((Get-VMHost $esx).ConnectionState -ne 'Maintenance') {
 
        #In my environment hosts take about 5-10 minutes to reboot
        Start-Sleep -Seconds 600
        #This if/else statement is for when the host after successful reboot doesn't automatically connect back to vCenter
        #If the connection is forced and host is still not connected, the sleep cycle will start again - won't stop until host is connected and is in maintenance
        #Make sure host is healthy and is in a rebooting state if the sleep cycle starts second time around
        #Set-VMHost cmd within if will throw an error, that's OK!
        if ((Get-VMHost $esx).ConnectionState -ne 'Maintenance') {
            Set-VMHost -VMHost $esx -State Maintenance
            Start-Sleep -Seconds 10
            Write-Host $esx 'was forced to connect'
            Write-Log ("$($esx) was forced to connect")
        }
        else {
            Write-Host $esx 'connected back automatically'
            Write-Log ("$($esx) connected back automatically")
        }
    }
 
    #Dont know why but vCenter needs this Disconnect / Connect step to accept the new cert, reboot is not enough, otherwise I've experienced issues with vSphere HA
    Write-Host 'Disconnecting' $esx 'from vCenter'
    Write-Log ("Disconnecting $($esx) from vCenter")
 
    Get-VMHost -Name $esx | set-vmhost -State Disconnected
 
    Start-Sleep -Seconds 10
 
    Write-Host 'Connecting' $esx 'to vCenter'
    Write-Log ("Connecting $($esx) to vCenter")
 
    #Set-VMHost cmd will throw an error, that's OK!
    Get-VMHost -Name $esx | set-vmhost -State Maintenance
 
    Start-Sleep -Seconds 10
 
    Write-Host $esx 'reboot process complete, taking host out of Maintenance Mode'
    Write-Log ("$($esx) reboot process complete, taking host out of Maintenance Mode")
 
    Get-VMHost -Name $esx | set-vmhost -State Connected
 
    while ((Get-VMHost $esx).ConnectionState -ne 'Connected') {
 
        Start-Sleep -Seconds 10
    }
 
    Write-Host $esx 'is out of Maintenance Mode'
    Write-Log ("$($esx) is out of Maintenance Mode")
 
    #Test certificate
    #More variables
    $protocolname = "tls12"
    $port = "443"
    check_ssl
}
 
Write-Host 'Disconnecting' $vcserver
Write-Log ("Disconnecting $($vcserver)")
 
#Disconnect vCenter server
Disconnect-VIServer $vcserver -Confirm:$false
Post Views: 7,081

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.