DocsBot AI - Notice history

We've confirmed the incident has not recurred, and implementing monitoring improvements to automatically deal with this failure mode in the future.

After performing a DB upgrade and restart, training data ingestion seems to be performing well now. We are now re-enabling training in our Dashboard, and will continue to monitor performance.

We are continuing to work on a fix for this incident. Writes are working again but running to slow to enable for customers. A backup and upgrade of the DB is being performed right now to see if that can improve the issue with batch imports.

From provider:

About 6 hours ago the pod for the primary docsbot production cluster was moved to a new kubernetes node. due to this, it is going through a startup process still where it is cleaning up old/stale data and in read-only mode. Unfortunately we do not have a good estimate for how long this will take.

Unfortunately we received no notice of this maintenance or that the DB would be unwritable for an extended period of time. We have disabled all training actions on the site temporarily until the DB is ready to avoid further confusion or accidental deleting of sources.