February 2017

Some audit events temporarily not recording
Crash events have been restored. Should you have any questions, please contact support@run.pivotal.io
Feb 15, 17:18 - Feb 16, 12:17 PST
Pivotal Web Services API errors
From approx 18:45 to 19:05 UTC, the API experienced delays for all requests that would generate audit events, which is most creates, updates, and deletes of API resources. Additionally, some asynchronous tasks may have taken longer to complete until approx 19:12 UTC. Users may have seen asynchronous tasks (like deleting a space) take longer to complete or timeout. Although users may have seen timeouts, the asynchronous tasks all eventually completed.
Feb 16, 11:25-12:12 PST
PWS Cloud Controller API Unavailable
On February 14, 2017 from 10:30 - 10:33 AM Pacific the PWS API (Cloud Controller API on the endpoint api.run.pivotal.io) was unavailable. This was due to an underlying system update. When the update completed, service to the API endpoint returned. If you have any questions, please contact support@run.pivotal.io
Feb 14
Issues concerning new signups for PWS and Pivotal Support
The fix to enable creating new accounts for Pivotal Web Services (PWS) and Pivotal Support has been deployed. Users are now able to create new accounts. If you are trying to create a new account for PWS or Pivotal Support and encountering issues, please contact support@run.pivotal.io.
Feb 13, 10:29-14:47 PST
AppsManager is experiencing some disruption among low-priority background jobs
A patch has been deployed to address the background jobs. Billing processing and other background tasks are resuming. If you continue to have issues with billing please contact support@run.pivotal.io
Feb 9, 17:16 - Feb 10, 11:42 PST

January 2017

SSL certificate errors
This incident has been resolved.
Jan 27, 05:51-06:57 PST
PWS Web Console (Apps Manager) not available. CLI functioning normally.
AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io
Jan 26, 10:50-11:19 PST
[Scheduled] PWS Key Rotation Effects on Spring Cloud Services
The scheduled maintenance has been completed.
Jan 26, 10:00-11:00 PST
PCF Metrics Upgrade
PCF Metrics upgraded from 1.2 to 1.3. It was down during this time.
Jan 23, 16:11 PST
[Scheduled] PCF Metrics Upgrade
The scheduled maintenance has been completed.
Jan 23, 13:45-15:22 PST
AppsManager is experiencing some disruption among low-priority background jobs
AppsManager's low-priority background jobs are operational. If you continue to have issues, please contact support@run.pivotal.io
Jan 19, 17:23-17:25 PST
Apps Manager unable to be accessed
The resolution to the apps manager has been successfully deployed. Apps Manager access has returned. Should you continue to have issues, please contact support@run.pivotal.io
Jan 19, 16:44-16:48 PST
AppsManager is unavailable
AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io
Jan 19, 16:35-16:39 PST
[Scheduled] Scheduled Maintenance for PWS
The scheduled maintenance has been completed.
Jan 19, 13:30-14:30 PST
High response times from the API
Around 15:57-16:09 UTC we noticed unusually high response times of approximately 700-800 ms from the API. The issue has now been resolved and we are actively monitoring the situation.
Jan 17
Cloud Controller functionality degraded
From approximately 10:10 AM to 10:50 AM UTC on 2017-01-04, the Cloud Controller was running with degraded functionality. Any application-related tasks (start, stop, push, restart, instances) would fail, but existing application instances continued to run and to be routable.
Jan 4

December 2016

Cloud Controller functionality degraded
From approximately 2:55 AM to 4:15 AM UTC on 2016-12-28, the Cloud Controller was running with degraded functionality. Any application-related tasks (start, stop, push, restart) would fail, but existing application instances continued to run and to be routable.
Dec 27
Cloud controller downgraded functionality.
Earlier today, around 1 PM UTC the cloud controller was running with degraded functionality. Any application related tasks (start, stop, push, restart) would fail. This impacted users for about 30 minutes. No applications were affected during this time.
Dec 27
Maintenance for PCF Metrics
This incident has been resolved.
Dec 22, 14:52-15:16 PST
Maintenance for PCF Metrics
This incident has been resolved.
Dec 21, 09:10-09:23 PST
PCF Metrics not loading logs
This incident has been resolved.
Dec 19, 19:38 PST
PCF Metrics is currently not operational
PCF Metrics functionality has been restored. No data was lost.
Dec 19, 10:02-14:47 PST
Access to PCF Metrics via Apps Manager unavailable
AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io
Dec 13, 13:39-17:03 PST
AppsManager is unavailable
AppsManager functionality has returned. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io
Dec 13, 14:28-14:31 PST
Monitoring Systems detecting issues with UAA (Authentication) Service
Our automated monitoring systems now report that ]UAA (User Authentication and Authorization) service performance is back to normal and the issue is considered resolved. If you continue to experience issues with logging in or with UAA API calls, please contact support@run.pivotal.io.
Dec 10, 00:23-00:59 PST
[Scheduled] Scheduled Maintenance on the p-redis service on PWS
The scheduled maintenance has been completed.
Dec 9, 11:15-13:15 PST
PCF Metrics Slowdown
Metrics capture and storage performance has returned to normal. There will be a gap in container and network metrics visible in PCF Metrics during the time of this incident.
Dec 8, 10:15-11:35 PST
Monitoring Systems detecting issues with Auto Scaler Service
The Autoscaler service was intermittently unavailable between 13:22 PST and 15:22 PST on Dec 02, 2016. The issue was due to a failed deployment of new features. The deployment was reverted and services restored.
Dec 2, 13:22 - Dec 5, 08:24 PST

November 2016

Issues with pushing apps.
The system has returned to a state of consistent successful application pushes and the incident is resolved. If you see issues with application pushes please contact support@run.pivotal.io.
Nov 29, 13:52-14:08 PST
Potential issues with logging
We found the issue to be related to https://www.pivotaltracker.com/story/show/131556799 and are monitoring this further.
Nov 25, 07:40-08:14 PST
PCF Metrics is encountering issues
Latency and container metric functionality has been restored. Metrics were not recorded between 18:39 and 00:00.
Nov 22, 13:43-16:15 PST
Loggerator is encountering issues
This incident has been resolved.
Nov 21, 11:11-14:27 PST
Issues with UAA and api.run.pivotal.io are causing issues with authentication for user of Apps Manager and CF CLI
We have been monitoring the corrective measures that we put in place to address this issue and at this time believe that all systems have returned to normal. All control, management, app logging and application execution are working as normal. This issue has been resolved. If you see issues, please contact support@run.pivotal.io
Nov 17, 17:41 - Nov 18, 15:21 PST
Customers will be unable to CF push
All systems on PWS should have returned to normal as we have rolled back an upgrade. The ability to push has returned and applications should be stable. We are performing root cause analysis to determine the cause of the issue. We apologize for the inconvenience. If you continue to have issues please contact support@run.pivotal.io
Nov 16, 12:03-14:42 PST
PWS Incident - Increased API Response Times
API response time has normalized. This incident is over. Thank you for your patience.
Nov 8, 07:04-19:23 PST
Deploying Python Apps requires explicit start command declaration.
We deployed a fix to the platform and Python applications no longer require a procfile. It is sufficient to specify a start command using the -C flag of the CLI. Should you have continue to have issues deploying python applications please contact support@run.pivotal.io
Oct 12, 13:40 - Nov 8, 08:58 PST
Routing tier experiencing Errors impacting pushing of applications and use of Apps Manager
We have re-established the complete operation of the etcd component of PWS and the routing tier has returned to normal operations. At this time all operations are normal including pushing applications, logging and use of the App Manager. Should you continue to see issues please contact support@run.pivotal.io
Nov 7, 16:29-17:49 PST
Issues with Pivotal SSL for certain Apps
This incident has been resolved.
Nov 2, 17:56-21:15 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on November 1 , 2016
The scheduled maintenance has been completed.
Nov 1, 10:01 - Nov 2, 09:55 PST

October 2016

Maintenance for PCF Metrics
This incident has been resolved.
Oct 31, 10:15-12:41 PST
Maintenance for PCF Metrics
This incident has been resolved.
Oct 31, 10:37-12:41 PST
Maintenance for PCF Metrics
This incident has been resolved.
Oct 31, 11:00-11:04 PST
Autoscaler service intermittently disables itself
We have deployed a fix that addresses the issue of the autoscaler self deactivating. While the issue only applies to apps bound to autoscaler after September 22, 2016, we recommend all autoscaler users check on the current status of their autoscaler settings and if incorrectly set to “disabled”, re-enable the service and the intended behavior should be in effect. Thank you for your patience. Should you continue to encounter issues with the autoscaler, please contact support@run.pivotal.io
Oct 25, 09:30 - Oct 28, 14:26 PST
Intermittent connection failures by the router
This incident has been resolved based on our monitoring and information that the .IO ccTLD team has restored service to the .io TLD. If you continue to have issues accessing your application and/or the CC API please contact support@run.pivotal.io
Oct 28, 02:06-10:45 PST
Currently the UAA (login.run.pivotal.io) is down, all services requiring authentication are unable to complete.
We have made changes and authentication activities should be correctly functioning. If you continue to have issues, please refresh the browser and/or authentication token. If issues persist, please contact support@run.pivotal.io
Oct 25, 16:40-17:43 PST
Issue with HTTP Headers with Zipkin enabled and interoperability with Spring Cloud Sleuth
Monday October 17th, 1:15pm PST time, PWS enabled zipkin HTTP headers for inbound http requests that passed through the PWS routers. This change impacted applications using the spring-cloud-sleuth library, which is not resilient to header values to invalid hex characters. There was one report of application downtime as well as several applications that reported additional errors in their logs. After diagnosing the issue, we started to turn off the zipkin headers at 11:11pm PST, and configuration change was finished by Tuesday, October 18th, 12:09am PST. We are sorry this change impacted users and we are adding tests to prevent a change like this from impacting applications again.
Oct 18, 15:49 PST
Issues accessing services management pages when Managing Marketplace Services
The issue has been resolved, we are working to understand the root cause and implement a fix. Thank you for your patience. If you are still encountering issues with this matter, please contact support@run.pivotal.io
Oct 17, 08:38 - Oct 18, 13:19 PST
Maintenance for PCF Metrics
This incident has been resolved.
Oct 11, 11:23-12:44 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on October 11, 2016
The scheduled maintenance has been completed.
Oct 11, 09:31-12:31 PST
[Scheduled] Scheduled Maintenance for PWS
The scheduled maintenance has been completed.
Oct 9, 22:00-22:25 PST
Loggerator is encountering issues
Both the latest version of the cf CLI and Apps Manager are able to process logs with the lastest deploy. Should you continue to find issues, please contact support@run.pivotal.io
Oct 3, 14:46 - Oct 5, 16:18 PST

September 2016

[Informational] Potential Breaking Change PWS static Outbound IPs are Eliminated after 9/29/2016
The deprecation of the Static IPs has completed. If you have issues and need assistance, please contact support@run.pivotal.io.
Sep 22, 12:03 - Sep 30, 14:11 PST
Ongoing Outbound Networking Issue resulting in Latency for some Apps
After some investigation, it appeared there was an issue that resulted in stale routes which vacating a container. This issue result in some misdirected requests which would lead to issues in performance. The issue has been resolved. If you continue to have issues, please contact support@run.pivotal.io.
Sep 20, 10:51 - Sep 22, 11:01 PST
PWS p-mysql service issue
The issue has been resolved. There was no data loss and service is now working as intended.
Sep 8, 15:28-15:44 PST

August 2016

cloudfoundry.org DNS down, impacting pushing of Applications on PWS
The issue has been resolved. If you continue to encounter issues, please contact support@run.pivotal.io.
Aug 31, 10:56-11:31 PST
PWS p-mysql service issue
The PWS p-MySQL cluster has been at restarted at 2016-08-27 15:20:12 UTC / 2016-08-27 11:20:12 EDT / 2016-08-27 08:20:12 PDT All apps can access their bound p-MySQL service. Please note, not all apps/ORMs are resilient to database restarts. If your app is still failing, please restart.
Aug 27, 07:48-08:37 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on August 25, 2016
The scheduled maintenance has been completed.
Aug 25, 11:00-12:01 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on August 23, 2016
The scheduled maintenance has been completed.
Aug 23, 15:58-16:12 PST

July 2016

[Scheduled] Scheduled Maintenance for PWS Apps Manager on July 26, 2016
The scheduled maintenance has been completed.
Jul 26, 11:00-11:30 PST
Networking change may require application restart
We have identified an issue with our networking infrastructure. A machine running our outbound network address translation service crashed and was recreated. What this means is that some apps relying on long-lasting outbound connections may no longer have access to those established connections. Some apps have long timeouts or no timeouts, and may not reestablish those connections. We recommend that PWS customers log into their CF and validate that their apps are healthy. If not, please restart your app so that it can reestablish contact with its Internet-based service.
Jul 15, 11:34 PST
Network change may require application restart
This incident has been resolved.
Jul 13, 17:56-18:07 PST
[Scheduled] Software maintenance on the p-mysql service on PWS on July 12 , 2016
The scheduled maintenance has been completed.
Jul 12, 10:46-15:30 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on July 5th, 2016
The scheduled maintenance has been completed.
Jul 5, 14:00-15:00 PST

June 2016

Slowness with UAA impacting PWS Functions including deploy and web console
This issue has been resolved. Apps Manager and cf CLI operations have returned to normal.
Jun 29, 14:15-22:49 PST
[Scheduled] Scheduled Maintenance on the p-mysql service on PWS on June 9, 2016
The scheduled maintenance has been completed.
Jun 9, 15:00-17:00 PST
[Scheduled] Scheduled Maintenance on the p-mysql service on PWS on June 8, 2016
The scheduled maintenance has been completed.
Jun 8, 11:00-11:30 PST

May 2016

[Scheduled] Scheduled Maintenance for PWS Apps Manager on May 17, 2016
The scheduled maintenance has been completed.
May 17, 14:00-14:39 PST
Slow deploy of applications
Due to a network capacity issue we encountered a backlog of requests that slowed down the buildpack download step impacting deploys. We have mitigated those network issues and application deploy is now performing normally. Thank you for your patiencein this matter. If you continue to have issues, please contact support@run.pivotal.io
May 3, 12:24-15:19 PST

April 2016

Apps Manager Outage
We found an issue that impacted the ability of Apps Manager to communicate with the authentication service. We have fixed the problem and Apps Manager has returned to normal operation. Should you encounter further issues, please contact support@run.pivotal.io.
Apr 25, 11:21-11:54 PST
Unscheduled Maintenance for PWS Apps Manager on April 22, 2016
This incident has been resolved.
Apr 22, 16:24-17:44 PST
[Monitoring] Apps Manager partial outage
We found a performance issue that resulted from an unexpected use case. We have mitigated that pattern and are working on a longer term fix to address the larger issue. These can be monitored in our public Pivotal Trackers for Cloud Foundry. Should you encounter further issues, please contact support@run.pivotal.io
Apr 21, 23:31 - Apr 22, 14:03 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on April 20, 2016
The scheduled maintenance has been completed.
Apr 20, 14:00-14:30 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on April 15, 2016
The scheduled maintenance has been completed.
Apr 15, 11:00-11:56 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on April 13, 2016
The scheduled maintenance has been completed.
Apr 13, 14:00-14:35 PST

March 2016

Emails to support@run.pivotal.io experiencing delivery issues
This incident has been resolved.
Mar 30, 11:39 - Mar 31, 16:36 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on March 28, 2016
The scheduled maintenance has been completed.
Mar 28, 14:00-14:30 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on March 22, 2016
The scheduled maintenance has been completed.
Mar 22, 14:00-14:54 PST
Investigating issues with PWS Apps Manager
This incident has been resolved.
Mar 10, 06:46-13:30 PST

February 2016

[Scheduled] Scheduled Maintenance for PWS Apps Manager on February 23, 2016
The scheduled maintenance has been completed.
Feb 23, 14:00-14:15 PST
Application Start Up issues on PWS
On February 10, 2016, PWS Cloud Operations deployed the new Cloud Foundry Stemcell 3192 that resolved the issues impacting the start up of applications and the ongoing refreshing of cells to deal with resource capacity on the application runners. If you wish to reduce the timeout interval permitted for application deployment, this can be done by re-pushing your application with the '-t' interval set to 60 s which is the default. If you continue to see issues with slow application starts please contact support@run.pivotal.io with details. Thank you for your patience on this matter.
Feb 5, 13:46 - Feb 11, 10:42 PST
[Informational] Pivotal Web Services Notification: Recent Cloud Foundry CVE announcements and remediation steps.
The actions in this incident notice are still recommended. The notification window has completed. If you need more information about the actions described in the notice, please contact support@run.pivotal.io
Jan 21, 15:30 - Feb 8, 15:48 PST

January 2016

Inability to provision certain paid services on the Marketplace
Working with our partners we have resolved the issue with the marketplace. All services and plans are able to be provisioned at this time. Should you encounter continued issues with provisioning please contact support@run.pivotal.io
Jan 22, 10:45-13:35 PST
[Informational and Ongoing] Degraded Performance when Deploying/Pushing Applications
Improvements to the buildpack caching on the platform has resolved this issue. Note that in some cases there may be a prolonged deploy but subsequent deploys should be performant.
Nov 5, 16:40 - Jan 22, 10:34 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager on January 21, 2016
The scheduled maintenance has been completed.
Jan 21, 10:00-10:15 PST
Applications experiencing degraded performance in routing, access and deploy.
On Monday January 11, 2016, PWS support received reports that a narrow set of applications hosted on the platform were experiencing degraded performance. Initial analysis was unable to uncover platform issues as deploy and new software causes were considered as possible causes. During the week, more reports were received that provided engineering with additional data via customer provided logs to investigate the cause of the substandard performance for the impacted applications. Additional analysis identified that the subset of applications impacted were having issues with outbound connections primarily impacting DB access for MySQL instances across many providers. Further investigation revealed that only instances in one of the deployed Availability Zones (AZ) were impacted. We deployed a set of ping-out apps that were able to provide latency metrics to nearby test instances. On the evening of January 15, 2016, we were able to positively identify indicators of high-latency on one of our NAT servers. According to performance tools, the NAT instance did not indicate saturation of RAM, disk, CPU, or network. However, the NAT server consistently added tens of milliseconds to test destinations. This indicated congestion somewhere in the network path. We coordinated with AWS support but were unable to identify what was causing the congestion. We replaced the problematic NAT server. On Monday, January 18, 2016, all of our NAT servers were upgraded to larger capacity machines in the hope that this would both refresh the remaining NAT servers, and obtain a relatively larger share of the underlying virtualization host. This action remedied the issue for all impacted applications and we have been monitoring to verify. We have added additional monitoring around this and do not expect this issue to occur again without being alerted. After an internal incident retrospective is concluded, a post mortem will be published. If you have any questions or concerns, please contact support@run.pivotal.io
Jan 14, 11:05 - Jan 20, 13:31 PST
Experiencing Application Slow Downs in Deploys, Routing and Response
The load resulting in degraded performance in deploys, routing and response has subsided and all services appear to be functioning as normal. In the event of continued impact, please contact support@run.pivotal.io
Jan 13, 14:35-21:15 PST
Some app pushes may fail to stage.
This incident has been resolved.
Jan 12, 15:24-16:03 PST

December 2015

[Scheduled] Routing Layer upgrade impacting outbound connection from PWS
The scheduled maintenance has been completed.
Dec 14, 12:25 PST
SSL Service for PWS dashboard unavailable
This issue has been resolved.
Nov 30, 19:32 - Dec 1, 11:23 PST

November 2015

[informational] Application Execution Pool upgrade and change over from Oct 15 - Nov 2.
This incident has been resolved.
Oct 15, 17:48 - Nov 5, 14:38 PST

October 2015

Degraded Performance on Pushing Apps to PWS
The incident affecting PWS has been resolved.
Oct 28, 10:41-12:50 PST
[Scheduled] Planned Maintenance on 26 Oct 2015 Impacting Application Management and Developer Console
Scheduled maintenance is complete. Thank you for your patience. Should you experience any issues, please contact support@run.pivotal.io.
Oct 26, 11:00-13:51 PST

September 2015

No incidents reported for this month.

August 2015

Failure to push apps due to expired cert on PWS
Some users were experiencing failures when pushing apps to PWS this morning. The affected users were: * users of the "online" a.k.a. "uncached" java-buildpack * users of the "online/uncached" ruby-buildpack 1.5.0 and later who haved opted-in to JRuby. The failure was due to expired SSL certificate of download.run.pivotal.io. The issue started happening around 5:00am PDT and was fixed at 8:50am PDT. We apologize for the inconvenience. If you are still experiencing issues pushing apps, please contact support@run.pivotal.io.
Aug 31, 09:30 PST
Issues with Apps Manager
We have rolled back to an earlier stable release.
Aug 27, 11:37-18:55 PST
[Scheduled] Deprecating the 'consoleapi.run.pivotal.io' endpoint for 'cf' CLI tool on July 29, 2015 12:00 PM PDT
The 'cf' CLI API endpoint 'consoleapi.run.pivotal.io' has been removed as planned. Please use endpoint 'api.run.pivotal.io' (instructions in previous update).
Aug 10, 07:29 PST
Application Errors due to Amazon S3 Issues
The incident is resolved after Amazon fixed the S3 issues.
Aug 10, 02:06-05:20 PST

July 2015

Application Availability Issues
This incident has been resolved.
Jul 27, 22:56 - Jul 28, 12:19 PST
Issues with creating the Blazemeter Service
All actions on the Blazemeter service are now available. Thank you for your patience. If you continue to have issues, please contact support@run.pivotal.io.
Jul 15, 17:25 - Jul 21, 12:09 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager
The Scheduled Maintenance for PWS has completed. Thank you for your patience. Should you experience any issues, please contact support@run.pivotal.io.
Jul 15, 16:16-16:38 PST
Degraded Performance on Pushing Apps to PWS
The issue impacting pushing applications to PWS has been identified and resolved. Thank you for your patience. If you continue to have issues deploying apps, please contact support@run.pivotal.io.
Jul 15, 14:45-16:22 PST
Issues involving service actions with ClearDB MySQL database
Working with our partners, we were able to identify and resolve the issue impacting the management of ClearDB database instances. Existing instances were not impacted. All management actions are now available. Thank you for your patience on this matter. If you continue to have issues, please contact support@run.pivotal.io
Jul 8, 16:35 - Jul 10, 11:46 PST
Slow and intermittent performance on App Manager
This incident has been resolved.
Jul 8, 16:31-20:13 PST

June 2015

App Manager undergoing maintenance.
The unplanned maintenance for the Web Console has been completed. Thank you for your patience.
Jun 23, 15:11-15:29 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager
The scheduled maintenance has been completed.
Jun 15, 10:00-10:37 PST
[Scheduled] Scheduled Maintenance for Application Execution Pool
The scheduled maintenance has been completed.
Jun 3, 09:30-14:55 PST
[Scheduled] Continuation: Auto-Migration of Apps from lucid to cflinuxfs2 stack
All applications have been migrated and lucid64 stack has been removed.
May 29, 15:30 - Jun 1, 16:15 PST

May 2015

[Scheduled] Continuation: Auto-Migration of Apps from lucid to cflinuxfs2 stack
The migration of applications to the new cflinuxfs2 stack has completed. If you have any issues, please contact support@run.pivotal.io
May 29, 11:45-14:28 PST
[Scheduled] Auto-Migration of Apps from lucid to cflinuxfs2 stack
The scheduled maintenance has been completed.
May 28, 14:15-23:45 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager
The scheduled maintenance has been completed.
May 19, 10:00-11:55 PST

April 2015

[Scheduled] Scheduled Maintenance Impacting Pivotal Web Services API
The scheduled maintenance has been completed.
Apr 22, 14:00-14:51 PST

March 2015

CI Smoketests failures
Confirmed: CI tests failing due to Github issue. Cloud Controllers Apdex has been > 95% for last 24 hours. Investigation resolved.
Mar 29, 11:03-11:43 PST
Degraded performance in PWS API
This incident has been resolved.
Mar 27, 14:23 - Mar 28, 23:06 PST
Issues with Binding and Unbinding Third Party Services
We have resolved this issue with the latest v204 push. With the exception of service instances that were incorrectly managed as a result of this issue, users should be able to manage services instances without issues at this time. If you have an affected service instance, please contact support@run.pivotal.io for assistance.
Mar 17, 13:43 - Mar 20, 11:00 PST
Email notification issues
This incident has been resolved.
Mar 16, 12:43-13:58 PST
Increased PWS API response time
This incident has been resolved.
Mar 11, 15:04 - Mar 12, 09:51 PST
Degraded Apps Manager Performance
This incident has been resolved.
Mar 4, 17:59 - Mar 9, 13:52 PST
[Scheduled] PWS Apps Manager maintenance
The scheduled maintenance has been completed.
Mar 4, 15:00-17:19 PST
PWS Incident - intermittent failures when pushing and staging applications
We've experienced an issue of bad DEA instances causing intermittent failures when users pushed or staged applications on PWS. The issue happened between 1:00pm and 4:00pm PST. The issue since been resolved.
Mar 3, 17:45 PST
[Scheduled] Ongoing PWS Apps Manager Maintenance
The scheduled maintenance has been completed.
Mar 3, 11:45-12:02 PST

February 2015

PWS Incident - Apps Manager Database Failover
This incident has been resolved.
Feb 26, 20:50-21:29 PST
Users unable to login to PWS via Apps Manager or CLI
On February 3, 2015 from 0804 - 0824 PST, users of PWS were unable to login to their accounts. The issue resolved itself and we are investigating the cause. We apologize for any inconvenience this may have caused. If your continue to have issues or questions please contact support@run.pivotal.io
Feb 3, 11:11 PST

January 2015

GHOST Vulnerability Response
On January 27, 2015, A vulnerability in the glibc package’s gethostbyname() and gethostbyname2() function calls was identified. This vulnerability could lead to a remote attacker executing arbitrary code using local user’s permission. This vulnerability is being referred to by the name “GHOST”. More details are available in CVE-2015-0235 [1]. The current release of Cloud Foundry v197 deployed on PWS has been patched with the glibc security fix. Since glibc could be statically linked to your applications running on PWS, we recommend you to take the following actions: If you are using a Cloud Foundry system buildpack All CF system buildpacks have been evaluated and were not found to statically link against glibc. If you are concerned that your application may statically link against glibc, we recommend restaging your application: $ cf restage If you are using a custom buildpack You need to evaluate whether your custom buildpack uses or creates binaries statically linked against a vulnerable glibc. If it is, update the buildpack with the security fix. Then, push your app again with the updated buildpack: $ cf push -b If you have any questions, please contact support@run.pivotal.io [1] http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-0235
Jan 29, 16:31 PST
Cannot manage service instances
This incident has been resolved.
Jan 27, 11:50-21:18 PST
[Scheduled] Scheduled Maintenance for PWS Apps Manager
The scheduled maintenance has been completed.
Jan 19, 10:00-10:51 PST

December 2014

[Scheduled] Scheduled Maintenance for PWS Developer Console
The scheduled maintenance has been completed.
Dec 19, 10:00-10:47 PST
Cannot View Recent Application Logs
We will continue to monitor for any further issues regarding this incident.
Dec 14, 15:33-17:36 PST
[Scheduled] Scheduled
The scheduled maintenance has been completed.
Dec 10, 18:01-22:00 PST
[Scheduled] Scheduled Maintenance for PWS Developer Console
The scheduled maintenance has been completed.
Dec 8, 13:45-13:58 PST

November 2014

502 Bad Gateway errors / resource capacity issues
All routing issues should be resolved.
Nov 17, 14:19-14:27 PST
cf-java-client (used by the Maven plugin, Gradle plugin, and Eclipse plugin) encounters 500 errors with cf push
A new version of Cloud Foundry was deployed to PWS. Applications utilizing cf-java-client for deploying apps should work without issue. If you are experiencing continued issues with app deployment via these tools using cf-java-client please contact support@run.pivotal.io.
Oct 22, 13:11 - Nov 7, 16:33 PST
[Scheduled] Scheduled Maintenance for PWS Developer Console
The scheduled maintenance has been completed.
Nov 4, 13:45-14:22 PST

October 2014

[Scheduled] Planned Maintenance impacting Application Management and Developer Console
The scheduled maintenance has been completed.
Oct 28, 11:45-13:35 PST
[Scheduled] Planned Maintenance impacting Application Management and Developer Console
The scheduled maintenance has been completed.
Oct 28, 10:45-11:30 PST
[Scheduled] Schedule Maintenance for Developer Console
The scheduled maintenance has been completed.
Oct 20, 15:15-15:45 PST
[Scheduled] Schedule Maintenance for PWS Developer Console
The scheduled maintenance has been completed.
Oct 20, 13:45-14:15 PST
[Scheduled] Planned Maintenance for PWS Logging Systems
The scheduled maintenance has been completed.
Oct 15, 16:00-16:22 PST
POODLE SSL Vulnerability Response
On October 14, 2014, A vulnerability in SSL Version 3 was publicly disclosed by Google researchers which is being referred to as the “POODLE” vulnerability. Details on this vulnerability are available in CVE-2014-3566 [1]. We have responded to this issue by disabling SSL V3 on our load balancers which is the point of termination for SSL in our system. Customer applications running in their corresponding Warden containers are not exposed to this vulnerability. If you have any issues, please do not hesitate to contact us at support@run.pivotal.io [1] http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3566
Oct 15, 12:18 PST
Issues pushing apps results in "Instance (index x) failed to start accepting connections" errors
Our investigation into this issue indicates this log message "Instance (index x) failed to start accepting connections" was being emitted incorrectly, and is not indicative of applications failing to start. We have resolved this incorrect error message. If you continue to receive this message, please contact support@run.pivotal.io with the results of your "CF_TRACE=true cf push your-app", as well as any logging traces.
Oct 1, 18:16 - Oct 9, 10:43 PST
[Scheduled] Planned Maintenance
The scheduled maintenance has been completed.
Oct 8, 10:07-10:14 PST
Access to Services Management Pages not working
We have resolved access to the management consoles of third party services. if you continue to have issues, please contact us at support@run.pivotal.io
Sep 30, 17:49 - Oct 1, 09:39 PST

September 2014

Outage: Customers apps were unreachable
Customers apps were unreachable from 15:40 to 15:55 Pacific Time. Note that all customer app instances continued to run in their app containers during the outage. However, customer apps were unreachable during this time because of an issue with the routing infrastructure. We have rolled a change to production to restore customer routes. Also during 15:40 to 16:40 logging subsystems were affected.
Sep 30, 16:36 PST
Issues managing and creating marketplace service instances
This incident has been resolved.
Sep 30, 11:12-15:40 PST
Adding/updating billing info may fail.
This incident has been resolved.
Sep 29, 14:54-17:19 PST
[Scheduled] Scheduled maintenance
The scheduled maintenance has been completed.
Sep 27, 06:00 - Sep 29, 12:00 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Sep 29, 11:15-11:45 PST
Bash Security Advisory
At this time, we have taken the following actions to protect our customers and their applications. We have updated our Warden container rootfs and restarted our customer applications to get the patched Bash. We have patched all our perimeter hosts. As always, please contact us at support@run.pivotal.io with any questions.
Sep 26, 17:49-17:49 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Sep 23, 16:15-16:45 PST
[Scheduled] Scheduled Maintenance to Application Runners
Maintenance has completed, all systems functioning normally.
Sep 15, 16:30 - Sep 22, 17:44 PST
PWS Web Console Unable to create spaces
This incident has been resolved.
Sep 5, 11:53-14:20 PST
Failures in pushing applications to PWS
From 11:49AM PDT - 12:23PM PDT Cloud Foundry had no stagers available, which prevented apps from pushing. This had no effect on running apps. The issue was caused by an update process being run out of order. The stagers and restored and system is stable.
Sep 4, 12:14-13:40 PST
Intermittent Logging issue to Third Party Providers
With the system release on 2013 Sep 2, the logging issue should be resolved. If you continue to have issues, please contact support@run.pivotal.io. We apologize for the inconvenience.
Aug 26, 14:19 - Sep 2, 14:20 PST

August 2014

API / Developer Console Degraded Performance
We've identified an issue with the router dropping requests to the API and believe the issue is fixed. We do not foresee users being affected, please contact support@run.pivotal.io for any further related issues.
Aug 28, 15:23 - Aug 29, 17:42 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Aug 20, 11:30-11:45 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Aug 19, 15:45-16:00 PST
[Scheduled] Updating Developer Console
The scheduled maintenance has been completed.
Aug 5, 10:45-11:15 PST

July 2014

[Scheduled] Updating Developer Console
The scheduled maintenance has been completed.
Jul 21, 09:30-10:50 PST
Problem creating CloudAMQP Instances
The issue has been resolved by the service provider.
Jul 3, 06:59-07:11 PST

June 2014

Problems managing SendGrid instances via the developer console.
This issue has been resolved. Consumers of a SendGrid instance should be able to load the management page.
Jun 30, 08:56-14:41 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Jun 24, 11:47-12:30 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Jun 6, 10:14 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Jun 2, 10:16 PST

May 2014

New CloudAMQP service instances reported having problems connecting to service
This issue has been resolved with the service provider. Connections to services should be working now.
May 21, 12:23 - May 22, 14:10 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
May 13, 10:16 PST
Users are currently unable to provision Rediscloud services
This issue has been resolved.
May 6, 14:20 - May 9, 10:15 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
May 6, 11:28 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
May 5, 13:30-14:32 PST

April 2014

Mongolab "sandbox" plan currently unavailable.
Issue has been resolved. Users can now create MongoLab instance without issue.
Apr 22, 12:21 - Apr 25, 10:01 PST
SSL "Heartbleed" bug status
We have renewed the final certificate for "*.cfapps.io" and it has been installed. All SSL certs have now been reissued after all vulnerable OpenSSL instances were upgraded. Pivotal Web Services is no longer vulnerable to the "Heartbleed" bug.
Apr 9, 17:58 - Apr 23, 10:21 PST
A small subset of users received erroneous emails from Pivotal Web Services
This incident has been resolved.
Apr 17, 13:50 - Apr 21, 14:36 PST
"Manage" link to SendGrid service instance not working
This issue should now be resolved.
Apr 10, 17:42 - Apr 18, 10:30 PST
Users are currently unable to access their organization on the Developer Console
Functionality restored to all but a few accounts.
Apr 16, 14:07-20:58 PST
Production stemcell update to 2366
The nats instance has restarted. The rest of the deployment continues.
Apr 16, 14:02-14:39 PST
console.run.pivotal.io is down
Console has been restored.
Apr 16, 12:09-12:58 PST
Mongolab "sandbox" plan fails to provision new service instances
This error was identified to only affect a specific user space. We're resolving the issue for that user. Other users should be able to provision a mongolab service instance.
Apr 14, 10:08-23:17 PST
Unable to provision services
A vendor was updating ssl certificates and was temporarily unavailable. The cert is updated and they are back online.
Apr 14, 18:12-18:50 PST
Error on first attempt to provision a service in a new space
Resolved the issue with an update to our marketplace. All service creation attempts expected to succeed now.
Apr 10, 15:31 - Apr 11, 20:37 PST
Problems provisioning some marketplace service instances
All accounts should be able to provision all services again.
Apr 8, 10:50-17:59 PST

March 2014

Existing service instances cannot be deleted
This incident has been resolved.
Mar 28, 14:56-16:02 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Mar 27, 14:51 PST
All node.js apps are experiencing failure to start due a breaking change introduced by NPM
This incident has been resolved.
Feb 28, 15:18 - Mar 26, 22:09 PST
Trial users unable to provision some free tier service plans.
This has been resolved. Customers with a free trial account should now be able to create instances of free tier services.
Mar 26, 12:20-16:18 PST
Staging apps is degraded
12:00 PM PDT Staging apps is back to fully available.
Mar 26, 11:59-12:07 PST
[Scheduled] Service Marketplace Maintenance
The scheduled maintenance has been completed.
Mar 25, 19:00-22:25 PST
cf-162 push is causing an issue with 'available stagers'
Issue is resolved. The cause was a new release that caused a minority of DEAs to consume all available CPU. We reverted the release.
Mar 13, 14:50-21:23 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Mar 13, 12:22 PST
Degraded Availability
We have multiple Network Address Transtaltion boxes, one of which went down. The NAT box was replaced and service was restored to normal.
Mar 8, 12:49-14:06 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Mar 5, 11:39 PST
Developer Console background jobs delayed
Our job scheduler had lost its connection to our database. Restarting the scheduler resolved the issue. Deeper investigation to follow.
Mar 4, 19:10-19:18 PST

February 2014

[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Feb 28, 13:58 PST
Loss of Application logs (loggregator)
Loggregator update was rolled back. All application logs should be delivered to syslog drains. During this issue some application log lines were not delivered to their corresponding drains.
Feb 26, 17:34 - Feb 27, 10:42 PST
[Scheduled] Developer Console Upgrade
The scheduled maintenance has been completed.
Feb 25, 11:15 PST
Running app instances in zone 1 experienced a brief loss of connectivity
Between 21:17 and 21:45 PST app instances running in zone 1 of our Application Execution Pool lost connectivity and were unable to respond to requests.
Feb 18, 22:20 PST
API Outage
Incident resolved. All services available again.
Feb 6, 15:31-17:40 PST

January 2014

No incidents reported for this month.

December 2013

No incidents reported for this month.