Connectivity issues reported by some users to the cloud
Postmortem

Since applying this month's Windows Patches, we have had 3 unexplained issues where our servers have become unresponsive/unreliable, and the resolution has been to restart the server.

This has only started since the patches were applied - we are in contact with Microsoft (again) and DELL, our hardware provider and looking for their guidance as we are not seeing any failures in any hardware that is causing this.

Depending on the response from Microsoft we may decide to roll back these updates which will require a maintenance window on the weekend to allow it to be done.

We will as always, continue to communicate transparently with you on these issues as we try to resolve them and I apologise for the multiple interruptions that have occurred this month since applying these patches.

UPDATE: 29/08/2023 19:22 AEST

We have been advised to apply upgrades to network drivers on all our primary Dell servers.

This may result in a temporary loss of connectivity during the driver upgrades so we have planned to schedule this later this evening to minimise impact and connected users.

You may experience a disconnection of your session for up to 5 minutes as the drivers are upgraded.

This is treated as an emergency update and therefore is scheduled during normal operations to try to address the issues caused due to server instability after applying August 2023 Updates

Posted Aug 29, 2023 - 18:16 AEST

Resolved
This incident has been resolved.
Posted Aug 29, 2023 - 16:08 AEST
Update
We are continuing to monitor for any further issues.
Posted Aug 29, 2023 - 16:08 AEST
Update
A number of servers are taking longer than expected to come back online but we believe the primary issue has been resolved as a number of users have reported that they are back on line and working successfully
Posted Aug 29, 2023 - 15:39 AEST
Monitoring
The server has rebooting and servers are coming back on line
We are monitoring the overall situation to determine if the issue has been resolved
Posted Aug 29, 2023 - 15:18 AEST
Update
The server has required a hard reset and we are now waiting for it to come back on line
Posted Aug 29, 2023 - 14:58 AEST
Update
We have rebooted a core server to try to resolve the issue which will believe is still related to the last Windows update..we expect that the system will be operational again in the next 10 minutes once the server reboot is completed
Posted Aug 29, 2023 - 14:42 AEST
Update
We are seeing an issue that is causing connectivity to drop and restart. We are investigating as a matter of urgency
Posted Aug 29, 2023 - 14:19 AEST
Investigating
We are currently investigating the issue reported by some users regarding a connectivity issue to the cloud servers
Posted Aug 29, 2023 - 14:06 AEST
This incident affected: Remote Desktop Services.