DocsBot AI - Intermittent Database search errors for subset of bots – Incident details

It seems that overnight the issue resurfaced with a small handful of bots. The cloud team is still looking into a permanent fix for the root issue that caused that to reoccur.

This incident has been resolved. Repair script seems to have been completed.

Affected bots count is dropping as they are fixed one by one. It's projected that all bots will be repaired within the hour.

They implemented a fix and are currently monitoring the results as it gradually repairs affected bots across all database shards.

We realize that part of the confusion with this outage is that, previously, when there would be database issues, it would return a user-friendly error message in the chat interface. With our architecture recently launched a few weeks ago, this was no longer a fatal error, and the AI instead was passed the error message. It would still return an answer to the user, just letting them know that it couldn't find the sources.

We have pushed a hotfix to our API so that when searching for sources fails, it again returns an actual error instead of responding fully to the user with the potential for hallucination.

Waiting on the status from our provider on how those fixes are going.

Our cloud provider has identified a performance configuration on the cluster that was preventing entrypoint repairs from completing. They are now rolling out an update to remove this setting and restore normal repair operations.

The cloud team is continuing to work on a fix for this incident. They attempted a full cluster restart but that did not solve the issue.

For a subset of bots error rates remain higher than 75% when searching docs. Our cloud provider is still working on clearing this issue that seems to be affecting a subset of our high availability nodes.

Error rates have dropped nearly completely, but our cloud provider is still working on fixing the underlying root cause with configuration tweaks.