Duration: 1 day, 2 hours and 43 minutes

Affected Components: Website

Feb 27, 17:00:00 GMT+0
Identified - We plan to begin migrating bots to our new Vector DB for better performance and stability. Migrations will happen bot by bot, and are expected to cause no downtime in chatbots. During the migration there may be a short period of a few minutes where managing sources will be disabled on the bot being migrated..

Feb 28, 19:43:26 GMT+0
Completed - Maintenance has completed successfully..

Duration: 2 days, 21 hours and 41 minutes

Affected Components: Chatbots

Feb 23, 07:30:00 GMT+0
Identified - We've seen increasing error rates with our training database that show up as a database search error or "I couldn't access the documentation" when chatting with a bot. The problem is intermittent. At its peak, we are seeing as much as 5% of requests triggering the error. We are in communication with our vector database cloud provider, who is working on fixing this issue on their side..

Feb 23, 16:30:00 GMT+0
Monitoring - Error rates have dropped nearly completely, but our cloud provider is still working on fixing the underlying root cause with configuration tweaks..

Feb 23, 18:43:19 GMT+0
Monitoring - For a subset of bots error rates remain higher than 75% when searching docs. Our cloud provider is still working on clearing this issue that seems to be affecting a subset of our high availability nodes..

Feb 25, 23:28:32 GMT+0
Monitoring - Only one bot remains in the repair queue. The beta release of our DB software that contains the permanent fix is in RCA..

Feb 23, 22:50:29 GMT+0
Identified - The cloud team is continuing to work on a fix for this incident. They attempted a full cluster restart but that did not solve the issue..

Feb 23, 23:49:09 GMT+0
Identified - Our cloud provider has identified a performance configuration on the cluster that was preventing entrypoint repairs from completing. They are now rolling out an update to remove this setting and restore normal repair operations..

Feb 24, 00:15:28 GMT+0
Identified - We realize that part of the confusion with this outage is that, previously, when there would be database issues, it would return a user-friendly error message in the chat interface. With our architecture recently launched a few weeks ago, this was no longer a fatal error, and the AI instead was passed the error message. It would still return an answer to the user, just letting them know that it couldn't find the sources. We have pushed a hotfix to our API so that when searching for sources fails, it again returns an actual error instead of responding fully to the user with the potential for hallucination. Waiting on the status from our provider on how those fixes are going. .

Feb 24, 00:38:00 GMT+0
Monitoring - They implemented a fix and are currently monitoring the results as it gradually repairs affected bots across all database shards..

Feb 24, 01:26:40 GMT+0
Monitoring - Affected bots count is dropping as they are fixed one by one. It's projected that all bots will be repaired within the hour. .

Feb 25, 15:29:57 GMT+0
Monitoring - We’ve been working closely with our cloud provider who have now confirmed the root causes of the recent intermittent vector index errors. What happened \- When compression was enabled, some tenants were in a lazy-loading state. \- In certain cases, compression could begin before the cache was fully loaded, leading to intermittent shard-level errors. \- Because each tenant runs across multiple shards, responses can come from different shards, which made the issue appear inconsistent. \- This has only affected a small subset of bots at any given time. Impact \- Of 592 tenants with compressed data (those potentially impacted), only 4 required requantization due to errors. \- Errors have been limited and intermittent rather than systemic. Actions taken by our cloud provider \- Cluster upgraded to patch version 1.35.10, which includes a fix for the RQ compression behavior. \- Async indexing enabled to immediately compress remaining shards. \- Repair tasks are running to restore any missing vector index elements or entry points. \- A full repair script is in progress to ensure all tenants are fully remediated. \- The fix has been identified and is moving through their QA pipeline. Current status \- Repair tasks are approximately 50% complete and progressing on the impacted bots. \- A separate issue impacting new tenant creation is under investigation. \- We expect remaining cleanup and stabilization to be completed by mid day. We will continue monitoring closely and provide further updates as remediation completes..

Feb 24, 03:34:59 GMT+0
Resolved - This incident has been resolved. Repair script seems to have been completed..

Feb 24, 16:47:35 GMT+0
Monitoring - It seems that overnight the issue resurfaced with a small handful of bots. The cloud team is still looking into a permanent fix for the root issue that caused that to reoccur. .

Feb 26, 05:11:28 GMT+0
Resolved - This incident has been resolved..

Duration: 4 hours and 5 minutes

Affected Components: Chatbots, Website

Jan 24, 20:00:00 GMT+0
Identified - Our cloud providers will be working on upgrading our Vector databases to a new more powerful cluster with High Availability. This will provide faster and more reliable service. This may lead to temporary periods of read only access that would show error messages on bot creation or during source training or refreshing. Deployed bots should remain functional for users to chat with..

Jan 24, 20:00:00 GMT+0
Identified - Reminder about DB maintenance starting tomorrow. All sources will be read only and bot creation will be disabled during the migration..

Jan 24, 21:10:19 GMT+0
Identified - During the migration we are also experiencing intermittent DB connection errors that may cause chatbot responses to fail..

Jan 25, 00:05:10 GMT+0
Completed - Maintenance has completed successfully..

Duration: 15 hours and 33 minutes

Affected Components: Chatbots, Website

Oct 20, 07:30:00 GMT+0
Investigating - Is having an intermittent problems with our website and chat widgets on which our hosting provider has a dependency on AWS (though designed for global failover)..

Oct 20, 11:03:00 GMT+0
Monitoring - We're only seeing intermittent impacts to customers, such as chat widgets disappearing for some bots, and inability to start webcrawls..

Oct 20, 17:03:00 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 35 minutes

Affected Components: Chatbots, Website

Sep 9, 05:40:06 GMT+0
Monitoring - We implemented a temporary workaround for DNS resolution while our cloud provider is still experiencing a global outage and are currently monitoring the result. Bots seem to be working, though it may take a bit of time for server load and response times to decrease..

Sep 9, 03:45:00 GMT+0
Investigating - We are currently investigating this incident. It appears to be DNS related with our API cloud provider..

Sep 9, 05:12:15 GMT+0
Identified - Our API cloud provider has confirmed the DNS resolution incident affecting many of their regional datacenters: "We are currently experiencing a partial outage in our Chicago, Dallas, Frankfurt, London, Los Angeles, Seoul, Singapore, Sydney, Tokyo, Toronto location. Our network engineers are actively working with our upstream providers to restore connectivity as quickly as possible. This has been effecting DNS resolution and overall network connectivity at this time." <https://status.vultr.com/> We are awaiting their resolution. When back online our chat APIs should also be restored..

Sep 9, 06:19:36 GMT+0
Resolved - This incident has been resolved thanks to our workaround. We will continue to monitor our cloud provider incident to learn the root cause..

Duration: 1 hour and 34 minutes

Affected Components: Chatbots, Website

Aug 14, 19:30:00 GMT+0
Investigating - Our vectorDB provider had an outage in their cloud, leading to failures in training and querying bots..

Aug 14, 21:04:00 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 41 minutes

Affected Components: Chatbots

Jul 9, 13:11:20 GMT+0
Resolved - This incident has been resolved..

Jul 9, 05:30:00 GMT+0
Identified - We made a minor configuration change to our security headers to improve site security, but it appears to have overridden our custom rules for the iframe embed urls, causing iframe embeds of our widget to break. We have reverted this change and iframe embeds should be working again with no changes needed on your side. Thank you for your patience and understanding!.

Duration: 3 hours and 29 minutes

Affected Components: Chatbots

May 30, 15:50:00 GMT+0
Investigating - Our chat API is currently being attacked with a DDoS. We are working on mitigating this traffic. It is causing slow or timeout responses primarily for our Agent API..

May 30, 19:18:36 GMT+0
Resolved - This incident has been resolved..

May 30, 18:16:10 GMT+0
Monitoring - We implemented blocks and are currently monitoring the result..

Duration: 8 minutes

Affected Components: Chatbots

Apr 26, 22:10:00 GMT+0
Investigating - We are currently investigating this incident affecting chatting with your bots..

Apr 26, 23:07:12 GMT+0
Resolved - This incident has been resolved. We apologize for the short outage that was due to an incomplete weekend automatic software update. We are putting in place additional automated checks to detect this kind of issue more quickly in the future..

Duration: 2 days, 17 hours and 26 minutes

Affected Components: Website

Apr 23, 18:34:26 GMT+0
Resolved - This incident has been resolved..

Apr 21, 04:26:23 GMT+0
Resolved - We triggered a restart and minor update of the DB cluster, and that seems to have fixed the write issues. We are currently monitoring the result, and awaiting a full postmortem from our cloud provider..

Apr 21, 01:08:00 GMT+0
Investigating - We have reports and our monitoring is showing problems with our vector database provider when creating or updating sources. This may show as timeouts, or strange errors like: class TenantDocument has multi-tenancy enabled, but request was without tenant We are currently investigating this incident and are in contact with our cloud provider..

Apr 21, 04:29:40 GMT+0
Resolved - Note, if any sources were added or refreshed during the outage, they may be temporarily stuck in a failed or queued state. You can simply click Retry if available, wait a bit for stuck sources to timeout, or simply add the same source again and delete the old one..

Apr 23, 15:47:59 GMT+0
Identified - The incident seems to have recurred, impacting training sources. We are communicating with our cloud provider. .

Apr 23, 17:08:10 GMT+0
Identified - Our cloud provider is having trouble finding the root cause, but they are rebooting the DB for now as a temporary fix (this worked last week for the same issue)..

Apr 23, 17:27:02 GMT+0
Monitoring - The reboot seems to have solved the DB issues. A small number of new bots created during the outage may be broken, we are recreating those now..

Duration: 50 minutes

Affected Components: Website, Chatbots

Mar 27, 16:27:53 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 27, 16:00:00 GMT+0
Investigating - Bots using our default OpenAI credentials are returning an API key error message. We are investigating..

Mar 27, 16:49:44 GMT+0
Resolved - Our internal monitoring notified us immediately to the increased error rate and we began investigating. It appears that the internal OpenAI key we use by default for chatbots that have not added their own key was deleted from our OpenAI account. We have verified account security and that this deletion was inadvertent. After reissuing a new key and deploying all services should be back online. We apologize that this affected our customers, and will further restrict production key delete capabilities to make sure this won't happen again..

Duration: 32 minutes

Affected Components: Website

Dec 13, 16:22:26 GMT+0
Identified - It looks like there is some kind of Github outage with a package server we depend on for processing Word, PPT, and txt document files. We are looking at workarounds. .

Dec 13, 16:54:50 GMT+0
Resolved - This incident has been resolved by adding a workaround from the package server..

Duration: 4 hours

Affected Components: Website, Chatbots

Nov 13, 10:00:00 GMT+0
Identified - We're rolling out some key improvements to our VectorDB to keep your experience top-notch and to reduce any potential issues..

Nov 13, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 13, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour and 58 minutes

Affected Components: Chatbots

Sep 18, 14:36:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Sep 18, 15:37:36 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 18, 16:34:26 GMT+0
Resolved - We've confirmed the incident has not recurred, and implementing monitoring improvements to automatically deal with this failure mode in the future..

Duration: 11 hours and 9 minutes

Affected Components: Website

Sep 5, 14:56:56 GMT+0
Identified - From provider: About 6 hours ago the pod for the primary docsbot production cluster was moved to a new kubernetes node. due to this, it is going through a startup process still where it is cleaning up old/stale data and in read-only mode. Unfortunately we do not have a good estimate for how long this will take. Unfortunately we received no notice of this maintenance or that the DB would be unwritable for an extended period of time. We have disabled all training actions on the site temporarily until the DB is ready to avoid further confusion or accidental deleting of sources..

Sep 5, 09:12:00 GMT+0
Investigating - We are currently investigating this incident. It appears that while bots are functional, training bots is triggering a timeout error message 'properties'. We are in contact with our cloud provider to look into the status of our cluster..

Sep 5, 17:39:53 GMT+0
Identified - We are continuing to work on a fix for this incident. Writes are working again but running to slow to enable for customers. A backup and upgrade of the DB is being performed right now to see if that can improve the issue with batch imports..

Sep 5, 18:22:35 GMT+0
Monitoring - After performing a DB upgrade and restart, training data ingestion seems to be performing well now. We are now re-enabling training in our Dashboard, and will continue to monitor performance..

Sep 5, 20:20:43 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour

Affected Components: Chatbots

Aug 7, 22:02:26 GMT+0
Completed - Maintenance has completed successfully.

Aug 7, 21:02:27 GMT+0
Identified - Maintenance is now in progress.

Duration: 8 hours and 3 minutes

Affected Components: Chatbots

Jul 30, 06:57:00 GMT+0
Investigating - A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string). Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully..

Jul 30, 15:00:08 GMT+0
Resolved - A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string). Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully..

Duration: 12 hours and 38 minutes

Affected Components: Website, Chatbots

Jul 19, 02:40:00 GMT+0
Identified - We have been unable to access our primary production VectorDB managed by our cloud provider. Their status shows it undergoing unplanned maintenance, and we are waiting to hear back from them a status report on progress of restoring access. This affects bot creation, source creation/updates, and bot usage. We will update as soon as we have any news..

Jul 19, 07:42:20 GMT+0
Identified - Our cloud provider has confirmed they can see where the issue lies and are right now liaising with the SRE team for right approach to repairing the cluster. .

Jul 19, 08:58:00 GMT+0
Monitoring - Our cloud provider has implemented a fix which appears to have brought the cluster back online and are currently monitoring the result. Analysis to follow. .

Jul 19, 09:17:00 GMT+0
Resolved - Cluster has been running smoothly for many hours now. Our provider reports that the root cause was a pod wasn't scheduling as it got stuck when the node it was on was upgrading machine. They have fixed the process internally. We are working on getting assurances that better monitoring is in place to minimize downtime if something similar happens again. We are also researching if moving to a HA setup would prevent similar issues in the future. .

Duration: 5 hours and 30 minutes

Affected Components: Website, Chatbots

Jul 16, 06:00:00 GMT+0
Investigating - DNS propagation + SSL renewal causing an infinite redirect on the website..

Jul 16, 11:30:00 GMT+0
Resolved - This incident has been resolved. Last night we switched our DNS provider to add more sophisticated DDoS protection to the [DocsBot.ai](http://DocsBot.ai) website and API. Unfortunately for some global regions the DNS propagation caused an SSL redirection loop that made the website, admin API, and widget inaccessible. We apologize profusely for the outage and want to assure our users that this was a one-time maintenance operation that will not need to happen again..

Duration: 16 hours and 53 minutes

Affected Components: Chatbots

Mar 29, 23:50:38 GMT+0
Identified - Migration is proceeding, no outages so far..

Mar 29, 16:43:56 GMT+0
Identified - We are planning for a scheduled maintenance during this time. It may lead to temporary slowness or outages for some bots during the migration process..

Mar 30, 16:43:56 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours and 48 minutes

Affected Components: Chatbots

Mar 26, 23:26:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 26, 20:26:00 GMT+0
Identified - We are doing DB maintenance and a migration. We do not expect much if any downtime as we migrate bots one-by-one..

Mar 26, 20:38:22 GMT+0
Identified - Maintenance is now in progress..

Duration: 5 hours and 20 minutes

Affected Components: Website, Chatbots

Feb 12, 22:04:17 GMT+0
Identified - An update from the WCS DB team: * Cluster was in crashloop with a fairly repeating pattern. We thought it might have been hitting liveness limits * However, then we saw it crash live after only \~7min, so we know it had to be something else Our engineers are still working on this and we hope to have this resolved as soon as possible. Thank you for your patience. .

Feb 12, 22:24:53 GMT+0
Resolved - This incident has been resolved. We will followup with root cause when we have one..

Feb 12, 17:04:53 GMT+0
Investigating - We are currently investigating this incident with our managed database provider..

Feb 12, 19:26:50 GMT+0
Identified - They have engaged our WCS engineering team to investigate further into this issue. They have now attempted to move to a new node and were waiting for startup. I have escalated this issue now and have raised a higher level incident for this ticket. Unfortunately, I don't have an exact timeframe for when this will be resolved. Rest assured we are prioritizing this with our WCS engineering team and I am monitoring activities actively..

Feb 12, 17:18:39 GMT+0
Identified - Our Database provider is currently investigating..

Duration: 1 day and 16 hours

Affected Components: Chatbots

Oct 25, 08:05:00 GMT+0
Investigating - There was a very high error rate from the OpenAI API around that time, but our VectorDB cluster also began having connection issues. It may be that the OpenAI outage was the root cause of the DB issues. I'm having our DB provider review the health of our cluster again today just to be sure..

Oct 25, 08:45:00 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 23 minutes

Affected Components: Website

Oct 24, 17:38:35 GMT+0
Identified - We are currently investigating this incident. It appears that our cloud provider needs to increase disk space for our DB cluster. We are working with them now to do this (it should be automated)..

Oct 24, 23:36:07 GMT+0
Identified - Our cloud provider has increased the disks and is continuing to work on a fix for this incident..

Oct 25, 00:01:07 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 15 minutes

Affected Components: Website, Chatbots

Oct 12, 22:45:00 GMT+0
Identified - Our database provider has recently updated our credentials to include multifactor authentication, which unfortunately caused a disruption in the authentication process to the database on our API. Rest assured, your data is safe we just can't access it for the moment, and we are actively trying to get in contact with them to revert back to the previous authentication method. We deeply regret the inconvenience caused, and should hopefully be back online shortly! .

Oct 13, 06:00:00 GMT+0
Resolved - This incident has been resolved..

Duration: 4 hours

Affected Components: Chatbots, OpenAI → API

Oct 11, 20:50:00 GMT+0
Investigating - OpenAI introduced a breaking bug in their steaming API. This is resulting in about 30% of chat widget responses to return an error message and not save the question to logs. We are waiting for them to resolve this bug..

Oct 12, 00:50:00 GMT+0
Resolved - This incident has been resolved..