PYTHON-5388 - AI/ML Testing Pipeline Automatic Retries (#106)

Author: NoahStapp

.evergreen/execute-tests.sh

@@ -13,4 +13,25 @@ set +a
 
 cd ${REPO_NAME}
 
-bash ${ROOT_DIR}/${DIR}/run.sh
+MAX_ATTEMPTS=3
+ATTEMPT=1
+EXIT_CODE=0
+set +e
+
+while [ $ATTEMPT -le $MAX_ATTEMPTS ]; do
+  bash ${ROOT_DIR}/${DIR}/run.sh
+
+  EXIT_CODE=$?
+
+  if [ $EXIT_CODE -eq 0 ]; then
+    break
+  else
+    echo "Tests failed on attempt $ATTEMPT of $MAX_ATTEMPTS..."
+    ((ATTEMPT++))
+  fi
+done
+
+if [ $EXIT_CODE -ne 0 ]; then
+  echo "Tests failed after $MAX_ATTEMPTS attempts."
+fi
+exit $EXIT_CODE

README.md

@@ -187,7 +187,7 @@ evergreen patch -p ai-ml-pipeline-testing --param REPO_ORG=caseyclements --param
 
 ### Handling Failing Tests
 
-Tests are run periodically (nightly) and any failures will propagate into both the `dbx-ai-ml-testing-pipline-notifications` and `dbx-ai-ml-testing-pipeline-notifications-{language}` channel. Repo owners of this `ai-ml-testing-pipeline` library are required to join the `dbx-ai-ml-testing-pipeline-notifications`. Pipeline specific implementers must **at least** join `dbx-ai-ml-testing-pipline-notifications-{language}` (e.g. whomever implemented `langchain-js` must at least be a member of `dbx-ai-ml-testing-pipeline-notifications-js`).
+Tests are run periodically (nightly). All failing test suites are automatically retried up to two times. Any failures will propagate into both the `dbx-ai-ml-testing-pipline-notifications` and `dbx-ai-ml-testing-pipeline-notifications-{language}` channel. Repo owners of this `ai-ml-testing-pipeline` library are required to join the `dbx-ai-ml-testing-pipeline-notifications`. Pipeline specific implementers must **at least** join `dbx-ai-ml-testing-pipline-notifications-{language}` (e.g. whomever implemented `langchain-js` must at least be a member of `dbx-ai-ml-testing-pipeline-notifications-js`).
 
 If tests are found to be failing, and cannot be addressed quickly, the responsible team MUST create a JIRA ticket within their team's project (e.g. a python failure should generate an `INTPYTHON` ticket), and disable the relevant tests
 in the `config.yml` file, with a comment about the JIRA ticket that will address it.