DocsBot AI - Notice history

This incident has been resolved. We apologize for the short outage that was due to an incomplete weekend automatic software update. We are putting in place additional automated checks to detect this kind of issue more quickly in the future.

The reboot seems to have solved the DB issues. A small number of new bots created during the outage may be broken, we are recreating those now.

Our cloud provider is having trouble finding the root cause, but they are rebooting the DB for now as a temporary fix (this worked last week for the same issue).

The incident seems to have recurred, impacting training sources. We are communicating with our cloud provider.

Note, if any sources were added or refreshed during the outage, they may be temporarily stuck in a failed or queued state. You can simply click Retry if available, wait a bit for stuck sources to timeout, or simply add the same source again and delete the old one.

We triggered a restart and minor update of the DB cluster, and that seems to have fixed the write issues. We are currently monitoring the result, and awaiting a full postmortem from our cloud provider.

Our internal monitoring notified us immediately to the increased error rate and we began investigating.

It appears that the internal OpenAI key we use by default for chatbots that have not added their own key was deleted from our OpenAI account. We have verified account security and that this deletion was inadvertent.

After reissuing a new key and deploying all services should be back online. We apologize that this affected our customers, and will further restrict production key delete capabilities to make sure this won't happen again.