Login Unavailability (4 minutes)
Incident Report for Pivotal Web Services
Resolved
Summary

On Wednesday May 23 from 19:00-19:04 UTC, users were unable to login using the CF CLI. Users attempting to login were shown an error response from UAA:

{
"error": "unauthorized",
"error_description": "Unable to sign token, misconfigured JWT signing keys"
}

Root Cause

The addition of new properties in v59 that were unable to be deserialized by the v58.1 instances caused `cf login` failures until the deployment of the new v59 instances was complete.

During the time of the outage a deploy was underway to upgrade from UAA v58.1 to v59 (to rollout GDPR compliance changes). In v59, new branding properties where the GDPR consent link and text are specified were introduced. When the first UAA VMs got updated these new properties were stored it in the database where they were visible to VMs still running 58.1 code. When a VM still on v58.1 received a token request, it looked in the database and found a saved configuration it was unable to parse and use. Without being able to use the saved configuration, which contained the JWT key used for token signing, the UAA was unable to issue tokens in response to the `cf login` requests.

The issue resolved once all UAA VMs updated to the UAA v59 code.

Impact

During the four minutes of the outage, no new oauth tokens could be authored by UAA vms. Customers experienced this as the failure of their `cf login` commands. Any token requests from other sources (such as any components within CF that are clients of the UAA) would also have failed during this time. However, any tokens that were previously issued should have continued to work, limiting the impact to only those customers trying to login.

Resolution

PWS returned to normal operation as soon as all UAA VMs were updated to the v59 release.

Take-aways

UAA team should make changes to ensure the addition of new Identity Zone configuration properties will not cause any outage during future.
Posted 26 days ago. May 23, 2018 - 12:00 PDT