Beginning around 16:24 UTC on July 16, 2018 latency on requests to the API nodes that back the Apps Manager developer console jumped significantly. Investigation narrowed the problem to an issue with DNS resolution on one API instance. The issue was resolved by restarting bosh-dns
on the affected instance.
The issue was a bug in bosh-dns
that manifested itself on a single API instance. From that instance uaa.service.cf.internal
could not be resolved.
curl --cacert uaa_ca.crt https://uaa.service.cf.internal:8443/token_keys
curl: (6) Could not resolve host: uaa.service.cf.internal
Before restarting bosh-dns
on the instance, the bosh team extracted logs from the instance for further analysis.
Customers may have noticed occasional long page load times in Apps Manager.
After the bosh team had analyzed the affected instance, the bosh-dns
job was restarted, which has mitigated the issue for now. The fix was deployed on 2018-07-26.