Today, we’re going to to create a playground to using . If you want to see this in prod, check out our . evaluate multiple LLM Providers in less than 10 minutes LiteLLM website What will it look like? : We'll build and connect it to our template frontend, ending up with a working playground UI by the end! How will we do this? the server Before you start, make sure you have followed the guide. Please note that this tutorial relies on you having API keys from at least 1 model provider (E.g. OpenAI). environment-setup Quick start Let's make sure our keys are working. Run this script in any environment of your choice (e.g. ). Google Colab 🚨 Don't forget to replace the placeholder key values with your keys! pip install litellm from litellm import completion ## set ENV variables os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS messages = [{ "content": "Hello, how are you?","role": "user"}] # openai call response = completion(model="gpt-3.5-turbo", messages=messages) # cohere call response = completion("command-nightly", messages) # ai21 call response = completion("j2-mid", messages) Set-up Server Let's build a basic Flask app as our backend server. We'll give it a specific route for our completion calls. : Notes 🚨 Don't forget to replace the placeholder key values with your keys! : LLM API calls can fail in production. This function wraps the normal litellm completion() call with to retry the call in case it fails. completion_with_retries tenacity LiteLLM specific snippet: import os from litellm import completion_with_retries ## set ENV variables os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS @app.route('/chat/completions', methods=["POST"]) def api_completion(): data = request.json data["max_tokens"] = 256 # By default let's set max_tokens to 256 try: # COMPLETION CALL response = completion_with_retries(**data) except Exception as e: # print the error print(e) return response Complete code: import os from flask import Flask, jsonify, request from litellm import completion_with_retries ## set ENV variables os.environ["OPENAI_API_KEY"] = "openai key" ## REPLACE THIS os.environ["COHERE_API_KEY"] = "cohere key" ## REPLACE THIS os.environ["AI21_API_KEY"] = "ai21 key" ## REPLACE THIS app = Flask(__name__) # Example route @app.route('/', methods=['GET']) def hello(): return jsonify(message="Hello, Flask!") @app.route('/chat/completions', methods=["POST"]) def api_completion(): data = request.json data["max_tokens"] = 256 # By default let's set max_tokens to 256 try: # COMPLETION CALL response = completion_with_retries(**data) except Exception as e: # print the error print(e) return response if __name__ == '__main__': from waitress import serve serve(app, host="0.0.0.0", port=4000, threads=500) Let’s test it Start the server: python main.py Run this curl command in your terminal to test it: curl -X POST localhost:4000/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "gpt-3.5-turbo", "messages": [{ "content": "Hello, how are you?", "role": "user" }] }' This is what you should see Connect to our UI template Download template For our frontend, we'll use - this enables us to build a simple python web-app. Streamlit Let's download the playground template we (LiteLLM) have created: git clone https://github.com/BerriAI/litellm_playground_fe_template.git Run it Make sure our server from step 2 is still running on port 4000 If you used another port, no worries - just make sure you change in your playground template's app.py Note: this line Now let's run our app: cd litellm_playground_fe_template && streamlit run app.py If you're missing Streamlit - just pip install it (or check out their ) installation guidelines pip install streamlit This is what you should see Congratulations 🚀 You've created your first LLM Playground - with the ability to call 50+ LLM APIs. Next Steps: Check out the full list of LLM Providers you can now add Deploy your server using Render Deploy your playground using Streamlit
