SDK + Guides
The Prompt Learning SDK is fully open source. View the SDK here. Check out Guides for some example use cases of prompt optimization through Prompt Learning!- Optimizing Coding Agent Prompts for Execution - Optimizing Coding Agents
- Optimizing your LLM-as-Judge evaluators - Optimizing your LLM-as-Judge evaluators
- Optimizing LLMs for Structured Output Generation - Optimizing LLMs for Structured Output Generation
SDK Components
The Prompt Learning SDK consists of several key components:Core Classes
PromptLearningOptimizer
The main class that orchestrates the prompt optimization process.
MetaPrompt
Handles the construction of meta-prompts used for optimization.
TiktokenSplitter
Manages token counting and batching for large datasets.
Annotator
Generates additional annotations to guide the optimization process.
Key Features
- Automatic batching based on token limits
- Template variable detection and preservation
- Multiple evaluation methods support
- Flexible input formats (strings, message lists, PromptVersion objects)
- OpenAI model integration for optimization
Setup
First clone the Prompt Learning repository and install the packages the optimizer imports. Run the examples below from the repository root, because the SDK imports its siblingcore, config and interfaces packages.
Basic Usage
1. Initialize the Optimizer
2. Prepare Your Dataset
Your dataset should contain:- Input columns: The data your prompt will use (e.g.,
question) - Output column: The LLM’s response (e.g.,
answer) - Feedback columns: Evaluation results (e.g.,
correctness,explanation)
3. Run Evaluators (Optional)
If you don’t have pre-existing feedback, you can run evaluators:4. Optimize the Prompt
Advanced Usage
Batch Processing and Context Management
The SDK automatically handles large datasets by splitting them into batches that fit within your specified context window.context_size_k Parameter
- Purpose: Controls the maximum token limit for each optimization batch
- Default: 128,000 tokens
- Impact: Larger values allow more examples per batch but may increase memory usage
- Recommendation: Start with 128k and adjust based on your model’s context window
Template Variable Handling
The SDK automatically detects and preserves template variables in your prompts:Multiple Evaluation Criteria
You can use multiple evaluators and feedback columns for comprehensive optimization:Custom Annotations
Use the annotator to generate additional guidance for optimization. This allows you to pass in all your outputs and evals into another LLM call for a final, comprehensive eval. Each annotation prompt must include the{baseline prompt} and {examples} placeholders. The annotator replaces them with your prompt and with the dataset rows, outputs and feedback.
Complete Example
Here’s a complete example showing the full workflow:Configuration Options
Model Selection
Setmodel_choice to any OpenAI chat model ID. The optimizer model rewrites your prompt, so a full-size model gives the best results:
Input Format Flexibility
The SDK accepts multiple prompt formats:Best Practices
1. Dataset Quality
- Ensure your dataset is representative of real-world usage
- Include diverse examples that cover edge cases
- Balance positive and negative feedback
2. Evaluation Criteria
- Define clear, measurable evaluation criteria
- Use multiple evaluators for comprehensive feedback
- Consider both objective (accuracy) and subjective (style) metrics
3. Context Window Management
- Start with smaller context windows for faster iteration
- Increase context size for more comprehensive optimization
- Monitor token usage to optimize costs
4. Iterative Improvement
- Run multiple optimization loops
- Monitor performance metrics across iterations
- Stop when performance plateaus or meets your criteria
5. Template Variable Preservation
- Always verify that template variables are preserved
- Test optimized prompts with new data
- Ensure backward compatibility
Conclusion
The Prompt Learning SDK provides a powerful, automated approach to optimizing LLM prompts. By leveraging evaluation feedback and meta-prompt optimization, you can systematically improve prompt performance across various use cases. Key benefits:- Automated optimization reduces manual prompt engineering
- Data-driven improvements based on actual performance metrics
- Scalable approach for production systems
- Flexible integration with existing evaluation frameworks