DocsBot AI - Notice history

After performing a DB upgrade and restart, training data ingestion seems to be performing well now. We are now re-enabling training in our Dashboard, and will continue to monitor performance.

We are continuing to work on a fix for this incident. Writes are working again but running to slow to enable for customers. A backup and upgrade of the DB is being performed right now to see if that can improve the issue with batch imports.

From provider:

About 6 hours ago the pod for the primary docsbot production cluster was moved to a new kubernetes node. due to this, it is going through a startup process still where it is cleaning up old/stale data and in read-only mode. Unfortunately we do not have a good estimate for how long this will take.

Unfortunately we received no notice of this maintenance or that the DB would be unwritable for an extended period of time. We have disabled all training actions on the site temporarily until the DB is ready to avoid further confusion or accidental deleting of sources.

A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string).

Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully.

Cluster has been running smoothly for many hours now. Our provider reports that the root cause was a pod wasn't scheduling as it got stuck when the node it was on was upgrading machine. They have fixed the process internally.

We are working on getting assurances that better monitoring is in place to minimize downtime if something similar happens again. We are also researching if moving to a HA setup would prevent similar issues in the future.

Our cloud provider has implemented a fix which appears to have brought the cluster back online and are currently monitoring the result. Analysis to follow.

Our cloud provider has confirmed they can see where the issue lies and are right now liaising with the SRE team for right approach to repairing the cluster.