Beginning Thursday August 2nd 2018, 21:22 UTC through 23:00 UTC, a service broker on Pivotal Web Services (PWS) was misconfigured, resulting in errors provisioning, updating, deleting, and binding service instances managed by this broker.
On Thursday August 2nd, 2018, while rotating a set of service broker credentials, we inadvertently changed the service broker password a second time.
During this time, users attempting to provision, update, delete, or bind to services managed by the service broker would have seen an authentication error. We observed approximately 200 errors that would have been similar to:
❯ cf create-service <service> <plan> <service-name> Creating service instance <service-name> in org <organization> / space <space> as <user>... FAILED Server error, status code: 502, error code: 10001, message: Authentication with the service broker failed. Double-check that the username and password are correct: <https://<broker-uri>/api/custom/cloudfoundry/v2/service_instances/08c0364a-b1c6-4fa4-abd8-bf6de667514c?accepts_incomplete=true>
Upon saving the configuration with the correct service broker password, requests to the service broker resumed working. We plan on taking the following corrective actions to help mitigate this problem in the future: