Service Broker Authentication Errors (~2 hours)
Incident Report for Pivotal Web Services
Postmortem

Summary

Beginning Thursday August 2nd 2018, 21:22 UTC through 23:00 UTC, a service broker on Pivotal Web Services (PWS) was misconfigured, resulting in errors provisioning, updating, deleting, and binding service instances managed by this broker.

Root Cause

On Thursday August 2nd, 2018, while rotating a set of service broker credentials, we inadvertently changed the service broker password a second time.

Impact

During this time, users attempting to provision, update, delete, or bind to services managed by the service broker would have seen an authentication error. We observed approximately 200 errors that would have been similar to:

❯ cf create-service <service> <plan> <service-name>
Creating service instance <service-name> in org <organization> / space <space> as <user>...
FAILED
Server error, status code: 502, error code: 10001, message: Authentication with the service broker failed. Double-check that the username and password are correct: <https://<broker-uri>/api/custom/cloudfoundry/v2/service_instances/08c0364a-b1c6-4fa4-abd8-bf6de667514c?accepts_incomplete=true>

Resolution

Upon saving the configuration with the correct service broker password, requests to the service broker resumed working. We plan on taking the following corrective actions to help mitigate this problem in the future:

  1. Add an automated test that exercises this service broker on a regular cadence.
Posted 4 months ago. Aug 07, 2018 - 09:55 PDT

Resolved
Beginning Thursday August 2nd 2018, 21:22 UTC through 23:00 UTC, a service broker on Pivotal Web Services (PWS) was misconfigured, resulting in errors provisioning, updating, deleting, and binding service instances managed by this broker.
Posted 4 months ago. Aug 02, 2018 - 21:15 PDT