Alternatives to DeepEval
Compare DeepEval alternatives for your business or organization using the curated list below. SourceForge ranks the best alternatives to DeepEval in 2026. Compare features, ratings, user reviews, pricing, and more from DeepEval competitors and alternatives in order to make an informed decision for your business.
-
1
Vertex AI
Google
Build, deploy, and scale machine learning (ML) models faster, with fully managed ML tools for any use case. Through Vertex AI Workbench, Vertex AI is natively integrated with BigQuery, Dataproc, and Spark. You can use BigQuery ML to create and execute machine learning models in BigQuery using standard SQL queries on existing business intelligence tools and spreadsheets, or you can export datasets from BigQuery directly into Vertex AI Workbench and run your models from there. Use Vertex Data Labeling to generate highly accurate labels for your data collection. Vertex AI Agent Builder enables developers to create and deploy enterprise-grade generative AI applications. It offers both no-code and code-first approaches, allowing users to build AI agents using natural language instructions or by leveraging frameworks like LangChain and LlamaIndex. -
2
Literal AI
Literal AI
Literal AI is a collaborative platform designed to assist engineering and product teams in developing production-grade Large Language Model (LLM) applications. It offers a suite of tools for observability, evaluation, and analytics, enabling efficient tracking, optimization, and integration of prompt versions. Key features include multimodal logging, encompassing vision, audio, and video, prompt management with versioning and AB testing capabilities, and a prompt playground for testing multiple LLM providers and configurations. Literal AI integrates seamlessly with various LLM providers and AI frameworks, such as OpenAI, LangChain, and LlamaIndex, and provides SDKs in Python and TypeScript for easy instrumentation of code. The platform also supports the creation of experiments against datasets, facilitating continuous improvement and preventing regressions in LLM applications. -
3
Maxim
Maxim
Maxim is an agent simulation, evaluation, and observability platform that empowers modern AI teams to deploy agents with quality, reliability, and speed. Maxim's end-to-end evaluation and data management stack covers every stage of the AI lifecycle, from prompt engineering to pre & post release testing and observability, data-set creation & management, and fine-tuning. Use Maxim to simulate and test your multi-turn workflows on a wide variety of scenarios and across different user personas before taking your application to production. Features: Agent Simulation Agent Evaluation Prompt Playground Logging/Tracing Workflows Custom Evaluators- AI, Programmatic and Statistical Dataset Curation Human-in-the-loop Use Case: Simulate and test AI agents Evals for agentic workflows: pre and post-release Tracing and debugging multi-agent workflows Real-time alerts on performance and quality Creating robust datasets for evals and fine-tuning Human-in-the-loop workflowsStarting Price: $29/seat/month -
4
Confident AI
Confident AI
Confident AI offers an open-source package called DeepEval that enables engineers to evaluate or "unit test" their LLM applications' outputs. Confident AI is our commercial offering and it allows you to log and share evaluation results within your org, centralize your datasets used for evaluation, debug unsatisfactory evaluation results, and run evaluations in production throughout the lifetime of your LLM application. We offer 10+ default metrics for engineers to plug and use.Starting Price: $39/month -
5
Arize Phoenix
Arize AI
Phoenix is an open-source observability library designed for experimentation, evaluation, and troubleshooting. It allows AI engineers and data scientists to quickly visualize their data, evaluate performance, track down issues, and export data to improve. Phoenix is built by Arize AI, the company behind the industry-leading AI observability platform, and a set of core contributors. Phoenix works with OpenTelemetry and OpenInference instrumentation. The main Phoenix package is arize-phoenix. We offer several helper packages for specific use cases. Our semantic layer is to add LLM telemetry to OpenTelemetry. Automatically instrumenting popular packages. Phoenix's open-source library supports tracing for AI applications, via manual instrumentation or through integrations with LlamaIndex, Langchain, OpenAI, and others. LLM tracing records the paths taken by requests as they propagate through multiple steps or components of an LLM application.Starting Price: Free -
6
Langfuse
Langfuse
Langfuse is an open source LLM engineering platform to help teams collaboratively debug, analyze and iterate on their LLM Applications. Observability: Instrument your app and start ingesting traces to Langfuse Langfuse UI: Inspect and debug complex logs and user sessions Prompts: Manage, version and deploy prompts from within Langfuse Analytics: Track metrics (LLM cost, latency, quality) and gain insights from dashboards & data exports Evals: Collect and calculate scores for your LLM completions Experiments: Track and test app behavior before deploying a new version Why Langfuse? - Open source - Model and framework agnostic - Built for production - Incrementally adoptable - start with a single LLM call or integration, then expand to full tracing of complex chains/agents - Use GET API to build downstream use cases and export dataStarting Price: $29/month -
7
OpenPipe
OpenPipe
OpenPipe provides fine-tuning for developers. Keep your datasets, models, and evaluations all in one place. Train new models with the click of a button. Automatically record LLM requests and responses. Create datasets from your captured data. Train multiple base models on the same dataset. We serve your model on our managed endpoints that scale to millions of requests. Write evaluations and compare model outputs side by side. Change a couple of lines of code, and you're good to go. Simply replace your Python or Javascript OpenAI SDK and add an OpenPipe API key. Make your data searchable with custom tags. Small specialized models cost much less to run than large multipurpose LLMs. Replace prompts with models in minutes, not weeks. Fine-tuned Mistral and Llama 2 models consistently outperform GPT-4-1106-Turbo, at a fraction of the cost. We're open-source, and so are many of the base models we use. Own your own weights when you fine-tune Mistral and Llama 2, and download them at any time.Starting Price: $1.20 per 1M tokens -
8
Orbit Eval
Turning Point HR Solutions Ltd
Orbit Eval is part of the Orbit Software Suite and is analytical job evaluation software. Job evaluation is a consistent & systematic process for defining the relative size or ranking of jobs within an organisation, by applying a consistent set of criteria to job roles. Analytical schemes offer a higher degree of rigour and objectivity. They enable a systematic approach to be applied providing a rationale as to why jobs are ranked differently. Application of the same method throughout the evaluation ensures consistency while minimising subjectivity and gender bias Orbit Eval is easy to use, very transparent and ensures consistency. The tool has been designed to be ‘owned’ by the organisation & requires minimal amounts of training. . It is hosted in the cloud with access permission levels. You can also input your current paper based scheme into the web-based data storage facility in Orbit Eval© to accommodate various systems including: NJC, GLPC & others. -
9
ChainForge
ChainForge
ChainForge is an open-source visual programming environment designed for prompt engineering and large language model evaluation. It enables users to assess the robustness of prompts and text-generation models beyond anecdotal evidence. Simultaneously test prompt ideas and variations across multiple LLMs to identify the most effective combinations. Evaluate response quality across different prompts, models, and settings to select the optimal configuration for specific use cases. Set up evaluation metrics and visualize results across prompts, parameters, models, and settings, facilitating data-driven decision-making. Manage multiple conversations simultaneously, template follow-up messages, and inspect outputs at each turn to refine interactions. ChainForge supports various model providers, including OpenAI, HuggingFace, Anthropic, Google PaLM2, Azure OpenAI endpoints, and locally hosted models like Alpaca and Llama. Users can adjust model settings and utilize visualization nodes. -
10
Opik
Comet
Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle. Log traces and spans, define and compute evaluation metrics, score LLM outputs, compare performance across app versions, and more. Record, sort, search, and understand each step your LLM app takes to generate a response. Manually annotate, view, and compare LLM responses in a user-friendly table. Log traces during development and in production. Run experiments with different prompts and evaluate against a test set. Choose and run pre-configured evaluation metrics or define your own with our convenient SDK library. Consult built-in LLM judges for complex issues like hallucination detection, factuality, and moderation. Establish reliable performance baselines with Opik's LLM unit tests, built on PyTest. Build comprehensive test suites to evaluate your entire LLM pipeline on every deployment.Starting Price: $39 per month -
11
EvalsOne
EvalsOne
An intuitive yet comprehensive evaluation platform to iteratively optimize your AI-driven products. Streamline LLMOps workflow, build confidence, and gain a competitive edge. EvalsOne is your all-in-one toolbox for optimizing your application evaluation process. Imagine a Swiss Army knife for AI, equipped to tackle any evaluation scenario you throw its way. Suitable for crafting LLM prompts, fine-tuning RAG processes, and evaluating AI agents. Choose from rule-based or LLM-based approaches to automate the evaluation process. Integrate human evaluation seamlessly, leveraging the power of expert judgment. Applicable to all LLMOps stages from development to production environments. EvalsOne provides an intuitive process and interface, that empowers teams across the AI lifecycle, from developers to researchers and domain experts. Easily create evaluation runs and organize them in levels. Quickly iterate and perform in-depth analysis through forked runs. -
12
Cognee
Cognee
Cognee is an open source AI memory engine that transforms raw data into structured knowledge graphs, enhancing the accuracy and contextual understanding of AI agents. It supports various data types, including unstructured text, media files, PDFs, and tables, and integrates seamlessly with several data sources. Cognee employs modular ECL pipelines to process and organize data, enabling AI agents to retrieve relevant information efficiently. It is compatible with vector and graph databases and supports LLM frameworks like OpenAI, LlamaIndex, and LangChain. Key features include customizable storage options, RDF-based ontologies for smart data structuring, and the ability to run on-premises, ensuring data privacy and compliance. Cognee's distributed system is scalable, capable of handling large volumes of data, and is designed to reduce AI hallucinations by providing AI agents with a coherent and interconnected data landscape.Starting Price: $25 per month -
13
Trusys AI
Trusys
Trusys.ai is a unified AI assurance platform that helps organizations evaluate, secure, monitor, and govern artificial intelligence systems across their full lifecycle, from early testing to production deployment. It offers a suite of tools: TRU SCOUT for automated security and compliance scanning against global standards and adversarial vulnerabilities, TRU EVAL for comprehensive functional evaluation of AI applications (text, voice, image, and agent) assessing accuracy, bias, and safety, and TRU PULSE for real-time production monitoring with alerts for drift, performance degradation, policy violations, and anomalies. It provides end-to-end observability and performance tracking, enabling teams to catch unreliable output, compliance gaps, and production issues early. Trusys supports model-agnostic evaluation with a no-code, intuitive interface and integrates human-in-the-loop reviews and custom scoring metrics to blend expert judgment with automated metrics.Starting Price: Free -
14
BiG EVAL
BiG EVAL
The BiG EVAL solution platform provides powerful software tools needed to assure and improve data quality during the whole lifecycle of information. BiG EVAL's data quality management and data testing software tools are based on the BiG EVAL platform - a comprehensive code base aimed for high performance and high flexibility data validation. All features provided were built by practical experience based on the cooperation with our customers. Assuring a high data quality during the whole life cycle of your data is a crucial part of your data governance and is very important to get the most business value out of your data. This is where the automation solution BiG EVAL DQM comes in and supports you in all tasks regarding data quality management. Ongoing quality checks validate your enterprise data continuously, provide a quality metric and supports you in solving the quality issues. BiG EVAL DTA lets you automate testing tasks in your data oriented project. -
15
EvalExpert
AlgoDriven
EvalExpert empowers dealerships by giving them the vehicle appraisal tools to make data-driven decisions about used cars. We offer a fully automated, single platform for vehicle appraisal, price guidance and analysis. Our industry leading data, partnered with proprietary algorithms; help reduce paperwork, eliminate mistakes of manual entry, improve productivity & provide great service to your customers. Using our propriety algorithms and industry leading data, EvalExpert streamlines the appraisal process with our easy to use, 3 step appraisal process - scan the vehicles registration or VIN, take photos, enter current information & condition details - done! EvalExpert’s Web Dashboard instantly syncs all your dealerships evaluations from any device. It provides overview statistics for the dealership and sales team with the most advanced reporting tools available in the market. -
16
Revolution FTO
Wayne Enterprises
Documenting the training of new officers is serious business. Liability is generally determined by training or the lack of it. Our police and sheriff FTO evaluation software was created by sworn officers having over 23 years of experience in managing FTOs and training new officers. This software is web-based and allows your training officers to document all daily and monthly activities of your newer officers. Through an annual contract with your agency, we can provide 24/7 phone, web, and onsite technical support. You will get direct assistance from a developer of the software. Create evaluations in half the time. FTO's can only change the evals they create. Finalization prevents changes in evaluations. Use from any computer inside the department. Use dailies to create monthlies, trainees can log on and sign evals without FTO. Chronological one-button approval of evaluations. Create statistical reports and track the effectiveness of police academies. -
17
BenchLLM
BenchLLM
Use BenchLLM to evaluate your code on the fly. Build test suites for your models and generate quality reports. Choose between automated, interactive or custom evaluation strategies. We are a team of engineers who love building AI products. We don't want to compromise between the power and flexibility of AI and predictable results. We have built the open and flexible LLM evaluation tool that we have always wished we had. Run and evaluate models with simple and elegant CLI commands. Use the CLI as a testing tool for your CI/CD pipeline. Monitor models performance and detect regressions in production. Test your code on the fly. BenchLLM supports OpenAI, Langchain, and any other API out of the box. Use multiple evaluation strategies and visualize insightful reports. -
18
HumanLayer
HumanLayer
HumanLayer is an API and SDK that enables AI agents to contact humans for feedback, input, and approvals. It guarantees human oversight of high-stakes function calls with approval workflows across Slack, email, and more. By integrating with your preferred Large Language Model (LLM) and framework, HumanLayer empowers AI agents with safe access to the world. The platform supports various frameworks and LLMs, including LangChain, CrewAI, ControlFlow, LlamaIndex, Haystack, OpenAI, Claude, Llama3.1, Mistral, Gemini, and Cohere. HumanLayer offers features such as approval workflows, human-as-tool integration, and custom responses with escalations. Pre-fill response prompts for seamless human-agent interactions. Route to specific individuals or teams, and control which users can approve or respond to LLM requests. Invert the flow of control, from human-initiated to agent-initiated. Add a variety of human contact channels to your agent toolchain.Starting Price: $500 per month -
19
Chainlit
Chainlit
Chainlit is an open-source Python package designed to expedite the development of production-ready conversational AI applications. With Chainlit, developers can build and deploy chat-based interfaces in minutes, not weeks. The platform offers seamless integration with popular AI tools and frameworks, including OpenAI, LangChain, and LlamaIndex, allowing for versatile application development. Key features of Chainlit include multimodal capabilities, enabling the processing of images, PDFs, and other media types to enhance productivity. It also provides robust authentication options, supporting integration with providers like Okta, Azure AD, and Google. The Prompt Playground feature allows developers to iterate on prompts in context, adjusting templates, variables, and LLM settings for optimal results. For observability, Chainlit offers real-time visualization of prompts, completions, and usage metrics, ensuring efficient and trustworthy LLM operations. -
20
Agency
Agency
Agency helps enterprises build, evaluate, and monitor AI agents. From the team at AgentOps.ai. Agen.cy (Agency AI) develops cutting edge AI agents using CrewAI, AutoGen, CamelAI, LLamaIndex, Langchain, Cohere, MultiOn + many more. -
21
Ragas
Ragas
Ragas is an open-source framework designed to test and evaluate Large Language Model (LLM) applications. It offers automatic metrics to assess performance and robustness, synthetic test data generation tailored to specific requirements, and workflows to ensure quality during development and production monitoring. Ragas integrates seamlessly with existing stacks, providing insights to enhance LLM applications. The platform is maintained by a team of passionate individuals leveraging cutting-edge research and pragmatic engineering practices to empower visionaries redefining LLM possibilities. Synthetically generate high-quality and diverse evaluation data customized for your requirements. Evaluate and ensure the quality of your LLM application in production. Use insights to improve your application. Automatic metrics that helps you understand the performance and robustness of your LLM application.Starting Price: Free -
22
ProdEval
Texas Computer Works
There is no such thing as a typical user of this system. Users include; independent reservoir engineers doing reserve reports, production engineers working up AFE’s and monitoring daily production, bank engineers tracking petroleum loan packages, CFOs tracking their borrowing base, property tax professionals assessing ad-valorem value, plus investors buying and selling producing properties. TCW’s ProdEval software is a quick and comprehensive Economic Evaluation system for both reserve reporting and prospect analysis. ProdEval has a very easy-to-use and straightforward approach to economic analysis and this methodology serves the user well. For example, the projecting of future production using sophisticated curve fitting techniques that allow the user to simply adjust the curves is one of the big factors that new users find attractive. The system is a rather open-ended system in that it accepts data from many sources; excel worksheets, commercial data sources. -
23
Martian
Martian
By using the best-performing model for each request, we can achieve higher performance than any single model. Martian outperforms GPT-4 across OpenAI's evals (open/evals). We turn opaque black boxes into interpretable representations. Our router is the first tool built on top of our model mapping method. We are developing many other applications of model mapping including turning transformers from indecipherable matrices into human-readable programs. If a company experiences an outage or high latency period, automatically reroute to other providers so your customers never experience any issues. Determine how much you could save by using the Martian Model Router with our interactive cost calculator. Input your number of users, tokens per session, and sessions per month, and specify your cost/quality tradeoff. -
24
Valid Eval
Valid Eval
Complex group deliberations don't have to be painful. Whether you're tasked with ranking hundreds of competing proposals, judging a dozen live pitches, or managing a multi-phase innovation program, there's an easier way. A better way. Valid Eval is an online evaluation system for organizations that make and defend tough decisions. It's a secure SaaS platform that works efficiently at virtually any scale so you can involve as many applicants, subjects, domain experts, and judges as it takes to do the job right. Combining best practices from the learning sciences and systems engineering, Valid Eval delivers defensible, data driven results and provides robust reporting tools that help you measure and monitor performance and demonstrate mission alignment. Best of all, it provides an unprecedented degree of transparency that promotes accountability and builds trust in the process. -
25
Llama 3
Meta
We’ve integrated Llama 3 into Meta AI, our intelligent assistant, that expands the ways people can get things done, create and connect with Meta AI. You can see first-hand the performance of Llama 3 by using Meta AI for coding tasks and problem solving. Whether you're developing agents, or other AI-powered applications, Llama 3 in both 8B and 70B will offer the capabilities and flexibility you need to develop your ideas. With the release of Llama 3, we’ve updated the Responsible Use Guide (RUG) to provide the most comprehensive information on responsible development with LLMs. Our system-centric approach includes updates to our trust and safety tools with Llama Guard 2, optimized to support the newly announced taxonomy published by MLCommons expanding its coverage to a more comprehensive set of safety categories, code shield, and Cybersec Eval 2.Starting Price: Free -
26
NVIDIA NeMo Guardrails
NVIDIA
NVIDIA NeMo Guardrails is an open-source toolkit designed to enhance the safety, security, and compliance of large language model-based conversational applications. It enables developers to define, orchestrate, and enforce multiple AI guardrails, ensuring that generative AI interactions remain accurate, appropriate, and on-topic. The toolkit leverages Colang, a specialized language for designing flexible dialogue flows, and integrates seamlessly with popular AI development frameworks like LangChain and LlamaIndex. NeMo Guardrails offers features such as content safety, topic control, personal identifiable information detection, retrieval-augmented generation enforcement, and jailbreak prevention. Additionally, the recently introduced NeMo Guardrails microservice simplifies rail orchestration with API-based interaction and tools for enhanced guardrail management and maintenance. -
27
EvalFlow
EvalFlow
EvalFlow — Performance Management Software for Distributed SMB Teams. EvalFlow is an AI-native performance management platform designed for small and mid-sized businesses with distributed, field-based, and operations-driven workforces — the segment that enterprise tools like Lattice and 15Five systematically price out and underserve. EvalFlow brings together the full performance management cycle in a single platform: structured review cycles, continuous feedback, OKR and goal tracking with hierarchy and ownership, peer recognition, pulse surveys, 1:1 meeting management, and project and task tracking. The platform supports multi-entity team structures and is accessible in English, French, and Spanish — making it one of the only performance management tools with native Spanish-language support for US Hispanic SMB teams.Starting Price: $6/month/user -
28
RagMetrics
RagMetrics
RagMetrics is a production-grade evaluation and trust platform for conversational GenAI, designed to assess AI chatbots, agents, and RAG systems before and after they go live. The platform continuously evaluates AI responses for accuracy, groundedness, hallucinations, reasoning quality, and tool-calling behavior across real conversations. RagMetrics integrates directly with existing AI stacks and monitors live interactions without disrupting user experience. It provides automated scoring, configurable metrics, and detailed diagnostics that explain when an AI response fails, why it failed, and how to fix it. Teams can run offline evaluations, A/B tests, and regression tests, as well as track performance trends in production through dashboards and alerts. The platform is model-agnostic and deployment-agnostic, supporting multiple LLMs, retrieval systems, and agent frameworks.Starting Price: $20/month -
29
eVal
eVal
eVal's free data and peer company analysis tools include historic valuation multiples, historical share price data, company financial information, and Valuation Multiples by Industry sector reports, for use in investment and business valuations. In addition to the provision of financial data and peer company analysis tools, eVal provides investment and company valuations. eVal offers expert business, investment, and company valuations based on our proprietary data-driven valuation software and platform. Our investment and business valuation service is tailored for valuation professionals, business owners, investors, and investment advisors. If you're a business owner and require a business valuation; or if you're an investor and require a private company valuation for your portfolio, please contact us directly regarding our business valuation service. Our outlier detection tool provides an overview of the peer group valuation multiples.Starting Price: Free -
30
LlamaIndex
LlamaIndex
LlamaIndex is a “data framework” to help you build LLM apps. Connect semi-structured data from API's like Slack, Salesforce, Notion, etc. LlamaIndex is a simple, flexible data framework for connecting custom data sources to large language models. LlamaIndex provides the key tools to augment your LLM applications with data. Connect your existing data sources and data formats (API's, PDF's, documents, SQL, etc.) to use with a large language model application. Store and index your data for different use cases. Integrate with downstream vector store and database providers. LlamaIndex provides a query interface that accepts any input prompt over your data and returns a knowledge-augmented response. Connect unstructured sources such as documents, raw text files, PDF's, videos, images, etc. Easily integrate structured data sources from Excel, SQL, etc. Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs. -
31
Selene 1
atla
Atla's Selene 1 API offers state-of-the-art AI evaluation models, enabling developers to define custom evaluation criteria and obtain precise judgments on their AI applications' performance. Selene outperforms frontier models on commonly used evaluation benchmarks, ensuring accurate and reliable assessments. Users can customize evaluations to their specific use cases through the Alignment Platform, allowing for fine-grained analysis and tailored scoring formats. The API provides actionable critiques alongside accurate evaluation scores, facilitating seamless integration into existing workflows. Pre-built metrics, such as relevance, correctness, helpfulness, faithfulness, logical coherence, and conciseness, are available to address common evaluation scenarios, including detecting hallucinations in retrieval-augmented generation applications or comparing outputs to ground truth data. -
32
20 Dollar Eval
SVI
With its user-friendly interface, 20 Dollar Eval provides easy-to-follow prompts and automated features, requiring no technical expertise to operate. 20 Dollar Eval is powered by SVI, an organizational development company that focuses on creating irresistible companies and extraordinary people. Over the years, SVI has launched thousands of performance reviews within some of the world’s largest and most complex organizations. You can rest comfortably knowing that, while the price is low, the system and industry expertise supporting it are proven to be best-in-class.Starting Price: $20 per review -
33
Tapt Health
Tapt Health
Tapt Health completes your documentation while you treat. Leverage AI to better engage patients, expedite evals, and minimize after-hours documentation.Starting Price: $91/month/user -
34
Latitude
Latitude
Latitude is an open-source prompt engineering platform designed to help product teams build, evaluate, and deploy AI models efficiently. It allows users to import and manage prompts at scale, refine them with real or synthetic data, and track the performance of AI models using LLM-as-judge or human-in-the-loop evaluations. With powerful tools for dataset management and automatic logging, Latitude simplifies the process of fine-tuning models and improving AI performance, making it an essential platform for businesses focused on deploying high-quality AI applications.Starting Price: $0 -
35
Weavel
Weavel
Meet Ape, the first AI prompt engineer. Equipped with tracing, dataset curation, batch testing, and evals. Ape achieves an impressive 93% on the GSM8K benchmark, surpassing both DSPy (86%) and base LLMs (70%). Continuously optimize prompts using real-world data. Prevent performance regression with CI/CD integration. Human-in-the-loop with scoring and feedback. Ape works with the Weavel SDK to automatically log and add LLM generations to your dataset as you use your application. This enables seamless integration and continuous improvement specific to your use case. Ape auto-generates evaluation code and uses LLMs as impartial judges for complex tasks, streamlining your assessment process and ensuring accurate, nuanced performance metrics. Ape is reliable, as it works with your guidance and feedback. Feed in scores and tips to help Ape improve. Equipped with logging, testing, and evaluation for LLM applications.Starting Price: Free -
36
PromptLayer
PromptLayer
The first platform built for prompt engineers. Log OpenAI requests, search usage history, track performance, and visually manage prompt templates. manage Never forget that one good prompt. GPT in prod, done right. Trusted by over 1,000 engineers to version prompts and monitor API usage. Start using your prompts in production. To get started, create an account by clicking “log in” on PromptLayer. Once logged in, click the button to create an API key and save this in a secure location. After making your first few requests, you should be able to see them in the PromptLayer dashboard! You can use PromptLayer with LangChain. LangChain is a popular Python library aimed at assisting in the development of LLM applications. It provides a lot of helpful features like chains, agents, and memory. Right now, the primary way to access PromptLayer is through our Python wrapper library that can be installed with pip.Starting Price: Free -
37
Giskard
Giskard
Giskard provides interfaces for AI & Business teams to evaluate and test ML models through automated tests and collaborative feedback from all stakeholders. Giskard speeds up teamwork to validate ML models and gives you peace of mind to eliminate risks of regression, drift, and bias before deploying ML models to production.Starting Price: $0 -
38
HoneyHive
HoneyHive
AI engineering doesn't have to be a black box. Get full visibility with tools for tracing, evaluation, prompt management, and more. HoneyHive is an AI observability and evaluation platform designed to assist teams in building reliable generative AI applications. It offers tools for evaluating, testing, and monitoring AI models, enabling engineers, product managers, and domain experts to collaborate effectively. Measure quality over large test suites to identify improvements and regressions with each iteration. Track usage, feedback, and quality at scale, facilitating the identification of issues and driving continuous improvements. HoneyHive supports integration with various model providers and frameworks, offering flexibility and scalability to meet diverse organizational needs. It is suitable for teams aiming to ensure the quality and performance of their AI agents, providing a unified platform for evaluation, monitoring, and prompt management. -
39
EXAONE Deep
LG
EXAONE Deep is a series of reasoning-enhanced language models developed by LG AI Research, featuring parameter sizes of 2.4 billion, 7.8 billion, and 32 billion. These models demonstrate superior capabilities in various reasoning tasks, including math and coding benchmarks. Notably, EXAONE Deep 2.4B outperforms other models of comparable size, EXAONE Deep 7.8B surpasses both open-weight models of similar scale and the proprietary reasoning model OpenAI o1-mini, and EXAONE Deep 32B shows competitive performance against leading open-weight models. The repository provides comprehensive documentation covering performance evaluations, quickstart guides for using EXAONE Deep models with Transformers, explanations of quantized EXAONE Deep weights in AWQ and GGUF formats, and instructions for running EXAONE Deep models locally using frameworks like llama.cpp and Ollama.Starting Price: Free -
40
Klu
Klu
Klu.ai is a Generative AI platform that simplifies the process of designing, deploying, and optimizing AI applications. Klu integrates with your preferred Large Language Models, incorporating data from varied sources, giving your applications unique context. Klu accelerates building applications using language models like Anthropic Claude, Azure OpenAI, GPT-4, and over 15 other models, allowing rapid prompt/model experimentation, data gathering and user feedback, and model fine-tuning while cost-effectively optimizing performance. Ship prompt generations, chat experiences, workflows, and autonomous workers in minutes. Klu provides SDKs and an API-first approach for all capabilities to enable developer productivity. Klu automatically provides abstractions for common LLM/GenAI use cases, including: LLM connectors, vector storage and retrieval, prompt templates, observability, and evaluation/testing tooling.Starting Price: $97 -
41
EVALS
EVALS
EVALS is the most dynamic mobile skills assessment and tracking solution for public safety, providing students and instructors with powerful tools to enhance learning and performance. Record, stream, upload and review videos to reinforce the knowledge, skills, attitudes and beliefs associated with the proper process. Design realistic scenarios and situational evaluations that help students develop the specialized skills needed to be effective in the real world. Track on-the-job training hours and performance requirements using our unique Digital Taskbook and Time Tracking modules. Select the components you need to streamline and simplify your training evaluations, including Digital Taskbook, an embedded events calendar, attendance, and time tracking, private message boards, academic testing, and more. Access the platform from anywhere via a web-enabled device and use the iOS app to perform field and video assessments without an internet connection. -
42
Vizcab Eval
Vizcab
Vizcab Eval is the solution to allow you to produce reliable, robust building ACV studies and percussive in one minimum time. Import your DPGF-type measurements and your RSET in a few clicks. Complete your entry using our research panel by keyword. Automatically associate your components and make simple corrections with our alert system. View results globally or in batches in real-time in the form of tables and graphs and validate compliance with thresholds. Identify at a glance the most impactful cards of your project, and bring efficient optimizations. Choose the most virtuous products with our scoring system of FDES. Work together and exchange easily with our fashion collaborative. Export your results in the form of graphs, and study reports according to your needs. Recover one RSEE export from your study to Excel format. You import your data directly into Vizcab Eval, and your components are automatically associated with plugs. -
43
Mistral Forge
Mistral AI
Mistral AI’s Forge platform enables enterprises to build customized AI models tailored to their internal data, workflows, and domain expertise. It provides end-to-end model development capabilities, covering everything from pre-training and synthetic data generation to reinforcement learning and evaluation. Organizations can integrate proprietary datasets and decision frameworks to create models that align closely with their business needs. Forge supports flexible deployment options, allowing companies to run models on-premises, in private cloud environments, or through Mistral infrastructure. The platform emphasizes security and governance, ensuring strict data isolation and compliance with enterprise policies. It also includes advanced evaluation tools that measure performance based on business-specific KPIs rather than generic benchmarks. By managing the full AI lifecycle in one system, Forge helps companies transform institutional knowledge into high-performing AI. -
44
DeepCover
DeepCover
Deep Cover aims to be the best coverage tool for Ruby code. More accurate line coverage, and branch coverage. It can be used as a drop-in replacement for the built-in Coverage library. It reports a more accurate picture of your code usage. In particular, a line is considered covered if and only if it is entirely executed. Optionally, branch coverage will detect if some branches are never taken. MRI considers every method defined, including methods defined on objects or via define_method, class_eval, etc. For Istanbul output, DeepCover has a different approach and covers all def and all blocks. DeepCover doesn't consider loops to be branches, but it's easy to support them if needed. Even after DeepCover is required and configured, only a very minimal amount of code is actually loaded and coverage is not started. To make it easier to transition for projects already using the builtin Coverage library deep-cover can inject itself into those tools.Starting Price: Free -
45
viEval
viGlobal
Evaluate every professional's performance with ease, efficiency & precision. Your annual review process doesn't have to be time-consuming. With our help, simplify any number of evaluations into one easy annual workflow. We understand the results your professional services firm needs to capture, including performance on projects and client work. viEval is the best-in-class tool for performance evaluation of professional work. All client work and hours are automatically pulled in from billing systems, so evaluations can be completed quickly and easily. We build high-performance cultures with 360-degree annual evaluation and integration with real-time feedback for continuous improvement. Our system can be easily customized for any role, department, or practice area. Create a performance management process of any complexity with our intelligent process builder. Use our pre-built templates for professional services firms or design your own process to capture precise feedback. -
46
Weights & Biases
Weights & Biases
Experiment tracking, hyperparameter optimization, model and dataset versioning with Weights & Biases (WandB). Track, compare, and visualize ML experiments with 5 lines of code. Add a few lines to your script, and each time you train a new version of your model, you'll see a new experiment stream live to your dashboard. Optimize models with our massively scalable hyperparameter search tool. Sweeps are lightweight, fast to set up, and plug in to your existing infrastructure for running models. Save every detail of your end-to-end machine learning pipeline — data preparation, data versioning, training, and evaluation. It's never been easier to share project updates. Quickly and easily implement experiment logging by adding just a few lines to your script and start logging results. Our lightweight integration works with any Python script. W&B Weave is here to help developers build and iterate on their AI applications with confidence. -
47
Phi-4-mini-reasoning
Microsoft
Phi-4-mini-reasoning is a 3.8-billion parameter transformer-based language model optimized for mathematical reasoning and step-by-step problem solving in environments with constrained computing or latency. Fine-tuned with synthetic data generated by the DeepSeek-R1 model, it balances efficiency with advanced reasoning ability. Trained on over one million diverse math problems spanning multiple levels of difficulty from middle school to Ph.D. level, Phi-4-mini-reasoning outperforms its base model on long sentence generation across various evaluations and surpasses larger models like OpenThinker-7B, Llama-3.2-3B-instruct, and DeepSeek-R1. It features a 128K-token context window and supports function calling, enabling integration with external tools and APIs. Phi-4-mini-reasoning can be quantized using Microsoft Olive or Apple MLX Framework for deployment on edge devices such as IoT, laptops, and mobile devices. -
48
TruLens
TruLens
TruLens is an open-source Python library designed to systematically evaluate and track Large Language Model (LLM) applications. It provides fine-grained instrumentation, feedback functions, and a user interface to compare and iterate on app versions, facilitating rapid development and improvement of LLM-based applications. Programmatic tools that assess the quality of inputs, outputs, and intermediate results from LLM applications, enabling scalable evaluation. Fine-grained, stack-agnostic instrumentation and comprehensive evaluations help identify failure modes and systematically iterate to improve applications. An easy-to-use interface that allows developers to compare different versions of their applications, facilitating informed decision-making and optimization. TruLens supports various use cases, including question-answering, summarization, retrieval-augmented generation, and agent-based applications.Starting Price: Free -
49
AgentSea
AgentSea
AgentSea is an open source platform designed to build, deploy, and share AI agents with ease. It delivers a collection of libraries and tools for building AI agent apps, favoring the UNIX philosophy of doing one thing well. Tools can be used individually or stacked together into a single agent app, and are compatible with frameworks like LlamaIndex and LangChain. Key components include SurfKit, a Kubernetes-style orchestrator for agents; DeviceBay, offering pluggable devices like file systems and desktops; ToolFuse, a library that wraps scripts, third-party apps, and APIs as Tool implementations; AgentD, a daemon making a Linux desktop OS accessible to bots; AgentDesk, a library for running AgentD-powered VMs; Taskara, for task management; ThreadMem, for building multi-role persistent threads; and MLLM, simplifying communication with multiple LLMs and multimodal LLMs. AgentSea also offers alpha agents like SurfPizza and SurfSlicer, which navigate GUIs using multimodal approaches.Starting Price: Free -
50
Entry Point AI
Entry Point AI
Entry Point AI is the modern AI optimization platform for proprietary and open source language models. Manage prompts, fine-tunes, and evals all in one place. When you reach the limits of prompt engineering, it’s time to fine-tune a model, and we make it easy. Fine-tuning is showing a model how to behave, not telling. It works together with prompt engineering and retrieval-augmented generation (RAG) to leverage the full potential of AI models. Fine-tuning can help you to get better quality from your prompts. Think of it like an upgrade to few-shot learning that bakes the examples into the model itself. For simpler tasks, you can train a lighter model to perform at or above the level of a higher-quality model, greatly reducing latency and cost. Train your model not to respond in certain ways to users, for safety, to protect your brand, and to get the formatting right. Cover edge cases and steer model behavior by adding examples to your dataset.Starting Price: $49 per month
