---
title: "Compare LLM Models"
slug: "compare-llm-models"
date: "2024-11-13"
updated: "2024-11-13"
status: "published"
featured: false
access: "free"
difficulty: "beginner"
one_liner: "Learn how to use Chatbot Arena to test the leading LLM models."
tools: ["Chatbot Arena"]
topics: ["General", "Prompt engineering"]
roles: ["Content Creator"]
categories: ["AI Fundamentals"]
---

<!-- migrated_from_webflow_csv -->

<!-- UNRESTRICTED -->

<div class="unrestricted">

<p id="">In this tutorial, you'll learn how to use <a href="https://lmarena.ai/" target="_blank" rel="noopener noreferrer">Chatbot Arena</a> to test the leading LLM models to get experimental and community data on what’s the best model for your various use cases. </p><p id="">Chatbot Arena is an open-source platform for evaluating AI through human preference. It was developed by researchers at UC Berkeley <a href="https://sky.cs.berkeley.edu/" target="_blank" rel="noopener noreferrer" id="">SkyLab</a> and <a href="https://lmsys.org/" target="_blank" rel="noopener noreferrer" id="">LMSYS</a>. It’s a powerful, free tool that allows you to:</p><ul id=""><li id="">Compare different models side-by-side</li></ul><ul id=""><li id="">View a categorical leaderboard of models by use case</li></ul><ul id=""><li id="">And direct message over 70 of the leading models to help you find the best one for your needs</li></ul><p id="">Chatbot Arena offers the ability to test proprietary models like OpenAI’s GPT-4o and Anthropic’s Claude Sonnet 3.5, as well as open-source models like Meta’s Llama-3.1 and Mistral-Large. </p><p id="">Whether you're building an AI-powered product or looking for the ideal chatbot for your workflow, Chatbot Arena can help you streamline your model decision-making process and allow you to give back to the open-source AI community at the same time.</p><p id=""><strong id="">Steps we’ll follow in this tutorial:</strong></p><ul id=""><li id="">Review leading LLM models by category</li></ul><ul id=""><li id="">Chat with any leading LLM model</li></ul><ul id=""><li id="">Compare two models side-by-side</li></ul><ul id=""><li id="">Blind-test models</li></ul>

</div>

<!-- RESTRICTED -->

<div class="restricted" data-access="pro">

<h3 id="">Review leading LLM models by category</h3><p id="">Navigate to the <a href="https://lmarena.ai/" id="" rel="noopener noreferrer" target="_blank">Chatbot Arena website</a> to get started. We’ll begin with reviewing the LLM model leaderboard to get a sense of which models are the best at certain categories of tasks. To do this, click on the “Leaderboard” tab in the top navigation.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1be2_2628d01e650948c_6704eca6c80b838256d98958_image.webp" width="auto"/></div></figure><p id="">To filter the leaderboard by a specific use case, click the “Category” dropdown filter. In this example, we’ll first filter by “Longer Query,” to see which models have the best performance for longer queries.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1b8b_650947c_6704eca6c80b838256d98955_image_201.webp" width="auto"/></div></figure><p id="">Unsurprisingly, OpenAI’s latest models are at the top of the list. But more surprisingly, Google’s models dominate the rest of the top 10 spots for this specific category of task. Anthropic’s most performant model, Claude 3.5 Sonnet, isn’t even in the top 10 — ranking below xAI’s Grok-2 for this use case.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1bdc_6509480_6704eca6c80b838256d98996_image_202.webp" width="auto"/></div></figure><p id="">Next, we’ll update the Category filter to “Math” now to see the top models for this task type.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1bd9_65094be_6704eca7c80b838256d9899a_image_203.webp" width="auto"/></div></figure><div data-rt-embed-type="true"><div style="padding: 20px; margin: 10px 0; background-color: #ffe8ed; border-left: 5px solid #f78ae0;"><strong>💡 Tip:</strong> Clicking on any of the model names will take you to their release notes or most recent documentation — helpful if you want to dive deeper into the details of any of the models.</div></div><h3 id="">Chat with any leading LLM model</h3><p id="">Now that we have a sense of the model leaders for specific tasks, we can directly chat with them in Chatbot Arena to get a better sense of their performance with specific prompts. To do this, click the “Direct Chat” tab in the top navigation.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637eab30ea50c1bb1b70_650949e_6704eca6c80b838256d98951_image_204.webp" width="auto"/></div></figure><p id="">In the dropdown menu above the response display, you can search and select the model you want to test.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637eab30ea50c1bb1b79_6509483_6704eca6c80b838256d9895b_image_205.webp" width="auto"/></div></figure><p id="">We’re going to go with Meta’s Llama-3.1 405b parameter model.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b6b_6509486_6704eca6c80b838256d98969_image_206.webp" width="auto"/></div></figure><div data-rt-embed-type="true"><div style="padding: 20px; margin: 10px 0; background-color: #ffe8ed; border-left: 5px solid #f78ae0;"><strong>💡 Tip:</strong> You can expand the toggle list below the model selector to see all available model descriptions.</div></div><p id="">We’re going to prompt Llama-3.1 with a generic summarization and analyses prompt.</p><p id=""><strong id="">Sample Prompt:</strong></p><blockquote id="">Analyze the primary themes present in Jane Austen's <em id="">Pride and Prejudice</em>.</blockquote><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b5a_6509474_6704eca6c80b838256d9893a_image_207.webp" width="auto"/></div></figure><p id="">The model will respond in the text response window.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1bd6_65094bb_6704eca6c80b838256d98993_image_208.webp" width="auto"/></div></figure><h3 id="">Compare two models side-by-side</h3><p id="">Now that you’ve got a handle on how to direct message models 1-on-1, we can start comparing models side-by-side, which is one of the most helpful and powerful features of Chatbot Arena. To do this, click on the “Arena (side-by-side)” tab in the top navigation.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1b88_6509489_6704eca6c80b838256d98979_image_209.webp" width="auto"/></div></figure><p id="">On the resulting page, we can now select two models to compare. We’ll start with GPT-4o vs. Claude Sonnet 3.5. To select the models, click on the dropdown list and type/select the first model you want to compare. Then, do the same thing on the other chat window dropdown.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637eab30ea50c1bb1b7c_509498_6704eca6c80b838256d98962_image_2010.webp" width="auto"/></div></figure><p id="">When both models are selected, we can start our comparison with a single prompt.</p><p id=""><strong id="">Sample Prompt:</strong></p><blockquote id="">Write an email declining a job offer.</blockquote><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b60_5094d2_6704eca6c80b838256d9896f_image_2011.webp" width="auto"/></div></figure><p id="">The models will generate their responses side-by-side.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1bdf_509492_6704eca7c80b838256d9899d_image_2012.webp" width="auto"/></div></figure><p id="">Chatbot Arena lets you provide feedback on which response is better.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1bd3_50948f_6704eca6c80b838256d9898d_image_2013.webp" width="auto"/></div></figure><div data-rt-embed-type="true"><div style="padding: 20px; margin: 10px 0; background-color: #ffe8ed; border-left: 5px solid #f78ae0;"><strong>💡 Tip:</strong> This feedback does not go towards the leaderboard rankings. Only the blind testing does, which we’ll get to in the next section.</div></div><p id="">You can also use random prompt examples provided by Chatbot Arena to test the models. To do so, click the “Random Example” button at the bottom of the page.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b56_50949b_6704eca6c80b838256d98966_image_2014.webp" width="auto"/></div></figure><p id="">Chatbot Arena will provide an image that you can pair with a prompt. In this example, it’s a bar chart image — we’ll ask the two models to analyze the data.</p><p id=""><strong id="">Sample Prompt:</strong></p><blockquote id="">Analyze the attached data.</blockquote><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b5d_5094cf_6704eca6c80b838256d98972_image_2015.webp" width="auto"/></div></figure><p id="">Again, they’ll output a side-by-side response to the prompt.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1b85_5094c1_6704eca6c80b838256d98990_image_2016.webp" width="auto"/></div></figure><p id="">This side-by-side comparison is incredibly helpful if you're building a product with a specific type of prompt built in, or if you're looking for the best AI product chatbot for a certain workflow. It allows you to see the nuances in how different models interpret and respond to the same prompt.</p><h3 id="">Blind-test models</h3><p id="">Finally, we can blind-test models in the “Arena (battle)” tab. This feedback <em id="">does</em> get included in the Chatbot Arena LLM Leaderboard — a fun way you can give back to the open-source AI community!</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1b91_5094c4_6704eca6c80b838256d9896c_image_2017.webp" width="auto"/></div></figure><p id="">To blind-test the models, enter a prompt into the input field on the page.</p><p id=""><strong id="">Sample Prompt:</strong></p><blockquote id="">Translate the following English sentence into French, Spanish, and Mandarin Chinese:<br/>‍<em id="">"The advancement of technology shapes the future of humanity."</em></blockquote><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637dab30ea50c1bb1b53_5094c7_6704eca6c80b838256d9895f_image_2018.webp" width="auto"/></div></figure><p id="">The models will respond, only this time, you will not know what models are on either side of the response window. You can vote for the best response below the generations.</p><figure class="w-richtext-figure-type-image w-richtext-align-fullwidth" data-rt-align="fullwidth" data-rt-max-width="3830px" data-rt-type="image" id="" style="max-width:3830px"><div id=""><img alt="__wf_reserved_inherit" height="auto" id="" loading="lazy" src="https://cdn.prod.website-files.com/66eeba0c7a01b10910c883da/6726637fab30ea50c1bb1b8e_509495_6704eca7c80b838256d989b2_image_2019.webp" width="auto"/></div></figure><p id="">And that’s it! You now know how to use Chatbot Arena to test the leading LLM models. Have fun experimenting, voting, and learning with this wonderful tool.</p><p id="">‍</p><p id="">‍</p><p id="">‍</p><p id="">This tutorial was created by <a href="https://garretthoughton.com/" id="" rel="noopener noreferrer" target="_blank">Garrett</a>.</p>

</div>
