custom-models/bedrock-fine-tuning/claude-haiku/DataValidation/Claude-3 Haiku Fine-Tuning Training and Validation Data Validator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "998b60bf",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This Jupyter Notebook provides an interactive interface for validating your training data (and optionally validation data) in `JSONL` format. It ensures that your data meets the requirements for Haiku Fine-Tuning in terms of file size, line count, token count, and data structure.\n",
    "\n",
    "## Data Validation Checks\n",
    "\n",
    "The notebook uses a custom data validation script to perform the following checks:\n",
    "\n",
    "1. **File Structure**: Ensures each line in the JSONL file is valid JSON.\n",
    "2. **File Size**:\n",
    "   - Training data: Maximum of 10GB\n",
    "   - Validation data: Maximum of 1GB\n",
    "3. **Line Count**:\n",
    "   - Training data: Between 32 and 10,000 lines\n",
    "   - Validation data: Between 32 and 1,000 lines\n",
    "   - Total (training + validation): Must not exceed 10,000 lines\n",
    "4. **Data Structure**: Validates the structure of each entry in the JSONL file.\n",
    "5. **Message Structure**: Checks the order and roles of messages in each entry.\n",
    "6. **Token Count**: Ensures each entry has fewer than 32,000 tokens.\n",
    "7. **Reserved Keywords**: Checks for the absence of Anthropic's reserved keywords in prompts.\n",
    "\n",
    "#### Reserved Keywords\n",
    "\n",
    "The validation process now includes a check for Anthropic's reserved keywords. The following keywords must not appear in any prompt (system message or user/assistant messages):\n",
    "\n",
    "- \"\\nHuman:\"\n",
    "- \"\\nAssistant:\"\n",
    "\n",
    "Note that variations of these keywords without the colon (e.g., \"\\nHuman\" or \"\\nAssistant\") are allowed.\n",
    "\n",
    "## Using the Notebook\n",
    "\n",
    "### Data Location\n",
    "\n",
    "This script requires your data to be locally available. If your provided training and validation datasets are stored in S3, the notebook will run a function to download the datasets locally.\n",
    "\n",
    "**Note**: If your datasets are stored in S3, ensure that the notebook has sufficient permissions to access S3.\n",
    "\n",
    "### Interpreting the Results\n",
    "\n",
    "After running the validation, you will see output indicating whether the validation was successful or not:\n",
    "\n",
    "- If the validation is `successful`, you will see the message \"All data passed validation!\"\n",
    "- If there are any `errors`, they will be listed with specific details about the issue and its location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "023ae649",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries and data validator\n",
    "\n",
    "import os\n",
    "import boto3\n",
    "from data_validation import validate_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aab3b70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to download files from S3 if necessary\n",
    "\n",
    "def get_local_path(file_path):\n",
    "    if file_path.startswith('s3://'):\n",
    "        s3 = boto3.client('s3')\n",
    "        local_path = os.path.basename(file_path)\n",
    "        bucket, key = file_path[5:].split('/', 1)\n",
    "        # download file to local\n",
    "        s3.download_file(bucket, key, local_path)\n",
    "        return local_path\n",
    "    return file_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c06c01c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the paths to your data files, provide either S3 URI or local file path\n",
    "\n",
    "training_file = \"path to your training JSONL file\"\n",
    "validation_file = \"path to your validation JSONL file\" # optional"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d1b6d0a-e270-4af8-9efd-340b7d651064",
   "metadata": {},
   "source": [
    "This script requires your data to be locally available. If your provided training and validation datasets are stored in S3, the notebook will run a function to download the datasets locally.\n",
    "\n",
    "**Note**: If your datasets are stored in S3, ensure that the notebook has sufficient permissions to access S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51b2029e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get local path\n",
    "\n",
    "local_training_path = get_local_path(training_file)\n",
    "local_validation_path = get_local_path(validation_file) if validation_file else None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9353d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run validation\n",
    "validate_data(local_training_path, local_validation_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "925f0c36",
   "metadata": {},
   "source": [
    "## Next Step\n",
    "\n",
    "After the validation is successful, your data is ready to use for Claude-3 Haiku fine-tuning job!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}