Microsoft services under investigation
Incident
Report for IT
Postmortem
Microsoft identified the following root cause for this issue:
“A configuration change in a portion of our Azure backend workloads, caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections.”
Resolved
All systems are operating normally. This issue is resolved.
Update
As of 7:31AM central time, the following Microsoft services may still be impacted:
- PowerBI: Users may notice that their service is in read-only mode while we address impact.
- Microsoft 365 admin center: Admins may be intermittently unable to access the Microsoft 365 admin center and any action may be delayed if accessible.
Microsoft's internal telemetry and customer signals indicate that the following services are recovered:
- Microsoft Defender
- Microsoft Defender for Endpoint
- Microsoft Defender Experts
- Microsoft Intune
- Microsoft OneNote
- OneDrive for Business
- SharePoint Online
- Windows 365
- Viva Engage
- Microsoft Purview
- Microsoft Teams
- Microsoft Fabric
- PowerBI: Users may notice that their service is in read-only mode while we address impact.
- Microsoft 365 admin center: Admins may be intermittently unable to access the Microsoft 365 admin center and any action may be delayed if accessible.
Microsoft's internal telemetry and customer signals indicate that the following services are recovered:
- Microsoft Defender
- Microsoft Defender for Endpoint
- Microsoft Defender Experts
- Microsoft Intune
- Microsoft OneNote
- OneDrive for Business
- SharePoint Online
- Windows 365
- Viva Engage
- Microsoft Purview
- Microsoft Teams
- Microsoft Fabric
Update
Some Microsoft services and applications may still be operating in a degraded state.
Microsoft reports that they are "still observing a positive trend of service availability as our mitigation efforts progress. In parallel, we are working on a mitigation to address the suspected root cause of this event. We remain committed to treating this incident with the highest priority to ensure it resolves in a timely manner."
Microsoft reports that they are "still observing a positive trend of service availability as our mitigation efforts progress. In parallel, we are working on a mitigation to address the suspected root cause of this event. We remain committed to treating this incident with the highest priority to ensure it resolves in a timely manner."
Monitoring
Microsoft reports that their services are showing signs of recovery. The university's online payment processing site is up and functioning normally at this time. We're continuing to monitor for any reoccurring problems.
Update
Microsoft provided the following update:
Users may be unable to access various Microsoft 365 apps and services. Impact includes but is not limited to the following apps and services:
SharePoint: Users may be unable to access SharePoint Online sites.
OneDrive: Users may be unable to access OneDrive for Business content.
Teams: Users may be unable to leverage Microsoft Teams functions including chat, messaging, and meeting management.
Power BI - Users may be unable to access the Power BI service and receive an error message. (Something went wrong. Please try again later), or will show as read-only.
Microsoft Admin Center: Admins may be intermittently unable to access the Microsoft 365 admin center and any action may be delayed if accessible.
Microsoft Intune: Users may experience failures with device check-in, syncing and device enrollment.
Viva Engage: Users may be unable to access Viva Engage.
Additional information is available on their website at https://status.cloud.microsoft/
Users may be unable to access various Microsoft 365 apps and services. Impact includes but is not limited to the following apps and services:
SharePoint: Users may be unable to access SharePoint Online sites.
OneDrive: Users may be unable to access OneDrive for Business content.
Teams: Users may be unable to leverage Microsoft Teams functions including chat, messaging, and meeting management.
Power BI - Users may be unable to access the Power BI service and receive an error message. (Something went wrong. Please try again later), or will show as read-only.
Microsoft Admin Center: Admins may be intermittently unable to access the Microsoft 365 admin center and any action may be delayed if accessible.
Microsoft Intune: Users may experience failures with device check-in, syncing and device enrollment.
Viva Engage: Users may be unable to access Viva Engage.
Additional information is available on their website at https://status.cloud.microsoft/
Identified
Microsoft has posted the following update regarding issues in the Central US region:
Impact Statement: Starting at approximately 4:56PM central time on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services.
Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.
This message was last updated at 6:17 central time on 18 July 2024.
Impact Statement: Starting at approximately 4:56PM central time on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services.
Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.
This message was last updated at 6:17 central time on 18 July 2024.
Investigating
A problem with Microsoft hosted services is under investigation. Online payment processing for various university websites is currently impacted as a result of an Azure outage. More information is available at https://azure.status.microsoft/en-us/status
Other services may be affected. Technicians are investigating the issue.
Questions 402-472-3970 or support@nebraska.edu
Other services may be affected. Technicians are investigating the issue.
Questions 402-472-3970 or support@nebraska.edu
This incident affected: Infrastructure (Data Center Services) and Communication & Collaboration (Office 365 (Email & Apps)).