Building an LLM Evaluation Tool with Synthetic Data Generation in 20 mins

LLM Evaluation Tool: Bringing Order to the Hype

One Thursday afternoon, I was scrolling through Twitter and saw the buzz about yet another new LLM release. Claims of "best-in-class performance" and "revolutionary capabilities" were flying left and right. But as a skeptical engineer, I couldn't help but wonder: How good is it really?

Instead of joining the hype train or dismissing it outright, I decided to build my own LLM evaluation tool. The goal? Create a flexible system that could generate task-specific datasets, evaluate LLM performance, and provide meaningful metrics. Oh, and make it work with any model. No pressure, right?

Framing the Problem

Before diving into the code, I needed to clearly define what I wanted to achieve. The key requirements for my LLM evals tool were:

  1. Dynamic task definition
  2. Synthetic data generation
  3. Flexible model evaluation
  4. Detailed performance metrics

With these goals in mind, I broke down the project into three main components:

  1. Schema Generation
  2. Synthetic Data Creation
  3. Model Evaluation

Let's dive into each of these components and see how they came together.

Schema Generation: The Foundation of Flexible LLM Evals

The first challenge was creating a system that could adapt to any task. I didn't want to hard-code schemas for specific tasks like sentiment analysis or named entity recognition. Instead, I wanted the tool to generate appropriate schemas based on a task description.

The solution? Use an LLM to create Pydantic schemas dynamically. Here's a snippet of the magic:

def generate_schema(task_description: str, pair_generation_model: str) -> Dict[str, Any]:
    prompt = f"""
    Task: Create a Pydantic schema for input and output based on the following task description:
 
    "{task_description}"
 
    Instructions:
    1. Analyze the task description carefully.
    2. Determine appropriate input and output fields based on the task.
    3. Create a JSON object with two keys: "input_schema" and "output_schema".
    4. For each schema, specify field names and their corresponding Python type hints.
    5. Use appropriate Python type hints: str, int, float, bool, List[str], List[int], List[float], Dict[str, Any], etc.
    6. The schema should be flexible enough to capture the essence of the task.
    7. Provide your response as a valid JSON object, nothing else.
 
    Example of a valid response for a text classification task:
    {
      "input_schema": {
        "text": "str"
      },
      "output_schema": {
        "category": "str",
        "confidence": "float"
      }
    }
    """
 
    # ... [code to call the LLM and process the response] ...
 
    return schema

This approach allows the tool to create appropriate schemas for a wide range of tasks, from simple classification to more complex scenarios like translation or summarization.

Synthetic Data Generation: Powering LLM Evals with Diverse Datasets

With our schema in place, the next step was generating diverse and relevant datasets for evaluation. Again, I turned to the power of LLMs to create synthetic data. The key innovation here was allowing users to provide sample data to guide the generation process.

Here's a glimpse of how it works:

def generate_input_output_pairs(task_description: str, pair_generation_model: str, InputModel: BaseModel, OutputModel: BaseModel, num_pairs: int = 5, data_samples: Optional[List[Dict[str, Any]]] = None) -> List[Dict[str, Any]]:
    sample_prompt = ""
    if data_samples:
        sample_prompt = f"Use these data samples as inspiration: {json.dumps(data_samples)}\n"
 
    prompt = f"""
    Task description: {task_description}
 
    {sample_prompt}
    Generate {num_pairs} input-output pairs for the above task.
 
    Input schema: {InputModel.schema_json()}
    Output schema: {OutputModel.schema_json()}
 
    Respond with a JSON array of objects, each containing 'input' and 'output' keys.
    Ensure that the types match the schema exactly.
    """
 
    # ... [code to call the LLM and process the response] ...
 
    return pairs

This approach allows for the creation of tailored datasets that match the specific requirements of each evaluation task. By leveraging LLMs for synthetic data generation, we can quickly produce large, diverse datasets that would be time-consuming and expensive to create manually.

Model Evaluation: Putting LLMs to the Test

With our schemas defined and synthetic data generated, we're ready for the main event: evaluating LLM performance. The challenge here was creating an evaluation system flexible enough to handle various output structures while still providing meaningful comparisons.

Here's a snippet of the evaluation logic:

def evaluate_model(model_name: str, task_description: str, pairs: List[Dict[str, Any]], InputModel: BaseModel, OutputModel: BaseModel) -> Tuple[Dict[str, float], List[Dict[str, Any]]]:
    # ... [setup code] ...
 
    for i, pair in enumerate(pairs):
        input_data = InputModel(**pair['input'])
        expected_output = OutputModel(**pair['output'])
 
        prompt = f"""Task description: {task_description}
 
        Given the input: {input_data.json()}, perform the task described above and provide the output.
 
        The output should be a valid JSON object matching this schema: {OutputModel.schema_json()}
        """
 
        # ... [code to call the LLM and process the response] ...
 
        try:
            actual_output = OutputModel(**json_response)
 
            # Flexible comparison of expected and actual output
            output_match = True
            differences = []
            for field, expected_value in expected_output.dict().items():
                actual_value = getattr(actual_output, field)
                if isinstance(expected_value, (int, float)):
                    if not isclose(expected_value, actual_value, rel_tol=0.1):
                        output_match = False
                        differences.append(f"{field}: expected {expected_value}, got {actual_value}")
                elif expected_value != actual_value:
                    output_match = False
                    differences.append(f"{field}: expected {expected_value}, got {actual_value}")
 
            # ... [code to record results] ...
 
        except Exception as e:
            logger.error(f"Error parsing model output: {str(e)}")
 
    # ... [code to calculate and return metrics] ...

This evaluation approach allows for nuanced comparisons, taking into account the specific requirements of each task while still providing quantitative metrics for overall performance.

Overcoming Challenges and Technical Hurdles

Of course, it wasn't all smooth sailing. I hit some roadblocks with JSON parsing and Pydantic warnings. But hey, what's a coding session without a few facepalm moments, right?

One particularly tricky issue was handling the variability in LLM outputs. Sometimes the model would return perfectly formatted JSON, other times it would include additional text or formatting. To address this, I had to implement robust error handling and parsing logic.

Another challenge was ensuring the tool could handle a wide range of task types without becoming overly complex. This required careful thought about the balance between flexibility and simplicity in the schema generation and evaluation processes.

Results and Reflections

After 20 mins of coding, debugging, the result was a tool that can:

  • Generate task-specific schemas
  • Create custom synthetic datasets
  • Evaluate any LLM on any task
  • Provide detailed performance metrics

All while being flexible enough to handle whatever you throw at it!

The experience reinforced a few key learnings:

  1. The power of LLMs for meta-tasks: Using LLMs to generate schemas and synthetic data opened up new possibilities for flexible, adaptable evaluation systems.

  2. The importance of robust error handling: When working with LLMs, expect the unexpected. Comprehensive error handling and logging are crucial for debugging and improving the system.

  3. The value of flexible evaluation metrics: Different tasks require different evaluation approaches. Building in flexibility from the start allows for more nuanced and meaningful comparisons.

Future Prospects and Improvements

While the current version of the tool is functional and flexible, there's always room for improvement. Some areas I'm considering for future development:

  1. Expanding synthetic data generation: Incorporating more advanced techniques for creating diverse and challenging datasets.

  2. Implementing additional evaluation metrics: Going beyond simple accuracy to include task-specific metrics and more nuanced performance indicators.

  3. Improving the user interface: Creating a more user-friendly interface for defining tasks and viewing results.

  4. Integrating with existing benchmarks: Allowing users to easily compare results with established LLM benchmarks.

Conclusion

What started as a quick project to objectively evaluate a new LLM release has turned into a flexible tool for LLM evals and synthetic data generation. It's a reminder of the power of curiosity and the rapid pace of innovation in the world of AI and machine learning.

Whether you're a researcher looking to benchmark the latest models, a developer fine-tuning LLMs for specific tasks, or just a curious tinkerer like me, I hope this tool can be useful in your explorations of the fascinating world of large language models.

The code for this project is available on GitHub . Feel free to clone, fork, and let me know how you're using it!