Summary

Beginning around 16:24 UTC on July 16, 2018 latency on requests to the API nodes that back the Apps Manager developer console jumped significantly. Investigation narrowed the problem to an issue with DNS resolution on one API instance. The issue was resolved by restarting bosh-dns on the affected instance.

Root Cause

The issue was a bug in bosh-dns that manifested itself on a single API instance. From that instance uaa.service.cf.internal could not be resolved.

curl --cacert uaa_ca.crt https://uaa.service.cf.internal:8443/token_keys
curl: (6) Could not resolve host: uaa.service.cf.internal

Before restarting bosh-dns on the instance, the bosh team extracted logs from the instance for further analysis.

Impact

Customers may have noticed occasional long page load times in Apps Manager.

Resolution

After the bosh team had analyzed the affected instance, the bosh-dns job was restarted, which has mitigated the issue for now. The fix was deployed on 2018-07-26.