DocsBot AI - Code regression impacting <2% of Bots – Incident details

Code regression impacting <2% of Bots

Resolved
Partial outage
Started about 2 months agoLasted about 8 hours

Affected

Chatbots

Partial outage from 6:57 AM to 3:00 PM

Updates
  • Resolved
    Resolved

    A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string).

    Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully.

  • Investigating
    Investigating

    A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string).

    Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully.