DocsBot AI - Code regression impacting <2% of Bots – Incident details

Code regression impacting <2% of Bots

Resolved
Partial outage
Started about 2 years agoLasted about 8 hours
Updates
  • Resolved
    UTC
    Resolved

    A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string).

    Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully.

  • Investigating
    UTC
    Investigating

    A code regression was deployed that was affecting a small subset of bots that had a specific unexpected metadata format in their training data (a url field saved with empty string).

    Unfortunately our automated testing and monitoring scripts did not detect this because it did not affect our test bots and only impacted <2% of customer bots. We are working on adding an additional monitoring solution that could detect these kind of edge-case regressions in the future hopefully.