Errors creating and binding service instances
Incident Report for Pivotal Web Services
Postmortem

Summary

On Thursday 5/31 from 17:15 to 20:18 UTC, attempts to create, bind, and delete to many marketplace services failed.

Root Cause

Many marketplace services are managed by a single third-party service broker. The credentials to the service broker were rotated at 17:15 UTC. After the credentials were rotated, some marketplace service broker operations succeeded (e.g. get-catalog), but some failed.

Ex. Attempts to create or delete service instances failed with:

[{"code":"Internal Server Error","message":"class org.hibernate.exception.ConstraintViolationException: could not execute statement"}]

Impact

At least three customer opened support tickets reporting problems creating, binding, and unbinding service instances. Existing service bindings were unaffected.

service_outage

Resolution

PWS reverted back to the previous credentials which has mitigated the issue.

Take-aways

  • PWS does not have alerting for binding/unbinding of marketplace services
  • We limited the testing of the new creds to get-catalog, which succeeded
  • We did not alert internal teams that credentials to the third-party broker were being rotated
Posted 6 months ago. Jun 01, 2018 - 07:28 PDT

Resolved
This incident has been resolved.
Posted 6 months ago. May 31, 2018 - 16:05 PDT
Update
We are continuing to monitor for any further issues.
Posted 6 months ago. May 31, 2018 - 12:45 PDT
Monitoring
We have resolved the issue and are monitoring the situation.
Posted 6 months ago. May 31, 2018 - 12:44 PDT
Investigating
We are currently investigating an issue that is causing errors creating, binding and unbinding service instances for some marketplace services.
Posted 6 months ago. May 31, 2018 - 12:07 PDT
This incident affected: Marketplace Services API.