← Back to Cookbook
cookbook langfuse mistral sdk integration
Details
File: third_party/Langfuse/cookbook_langfuse_mistral_sdk_integration.ipynb
Type: Jupyter Notebook
Use Cases:
Integrations: Langfuse
Content
Notebook content (JSON format):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "8_OORulNGD8a"
},
"source": [
"<center>\n",
" <p style=\"text-align:center\">\n",
" <img alt=\"langfuse logo\" src=\"https://langfuse.com/langfuse_logo.png\" width=\"200\"/>\n",
" <br>\n",
" <a href=\"https://langfuse.com/docs\">Docs</a>\n",
" |\n",
" <a href=\"https://github.com/langfuse/langfuse\">GitHub</a>\n",
" |\n",
" <a href=\"https://discord.langfuse.com/\">Discord</a>\n",
" </p>\n",
"</center>\n",
"<h1 align=\"center\">Mistral AI SDK Integration with Langfuse</h1>\n",
"\n",
"This cookbook provides step-by-step examples of integrating Langfuse with the Mistral AI SDK (v1) in Python. By following these examples, you'll learn how to seamlessly log and trace interactions with Mistral's language models, enhancing the transparency, debuggability, and performance monitoring of your AI-driven applications.\n",
"\n",
"---\n",
"\n",
"Note: Langfuse is also natively integrated with [LangChain](https://langfuse.com/docs/integrations/langchain/tracing), [LlamaIndex](https://langfuse.com/docs/integrations/llama-index/get-started), [LiteLLM](https://langfuse.com/docs/integrations/litellm/tracing), and [other frameworks](https://langfuse.com/docs/integrations/overview). If you use one of them, any use of Mistral models is instrumented right away.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1uZk2TfjjbW7"
},
"source": [
"## Overview\n",
"\n",
"In this notebook, we will explore various use cases where Langfuse can be integrated with Mistral AI SDK, including:\n",
"\n",
"- **Basic LLM Calls:** Learn how to wrap standard Mistral model interactions with Langfuse's @observe decorator for comprehensive logging.\n",
"- **Chained Function Calls:** See how to manage and observe complex workflows where multiple model interactions are linked together to produce a final result.\n",
"- **Async and Streaming Support:** Discover how to use Langfuse with asynchronous and streaming responses from Mistral models, ensuring that real-time and concurrent interactions are fully traceable.\n",
"- **Function Calling:** Understand how to implement and observe external tool integrations with Mistral, allowing the model to interact with custom functions and APIs.\n",
"\n",
"For more detailed guidance on the Mistral SDK or the **@observe** decorator from Langfuse, please refer to the [Mistral SDK repo](https://github.com/mistralai/client-python) and the [Langfuse Documentation](https://langfuse.com/docs/sdk/python/decorators#log-any-llm-call).\n",
"\n",
"## What is Langfuse?\n",
"\n",
"[Langfuse](https://langfuse.com/) is an open-source LLM engineering platform. It includes features such as [traces](https://langfuse.com/docs/tracing), [evals](https://langfuse.com/docs/scores/overview), and [prompt management](https://langfuse.com/docs/prompts/get-started) to help you debug and improve your LLM app."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VTQfiFQnEsto"
},
"source": [
"## Setup\n",
"\n",
"[Sign up](https://cloud.langfuse.com/auth/sign-up) for Langfuse if you haven't already. Copy your [API keys](https://langfuse.com/faq/all/where-are-langfuse-api-keys) from your project settings and add them to your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install mistralai langfuse"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "Dqlx_v410_1r"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# get keys for your project from https://cloud.langfuse.com\n",
"os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-lf-xxx\"\n",
"os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-lf-xxx\"\n",
"os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
"# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region\n",
"\n",
"# Your Mistral key\n",
"os.environ[\"MISTRAL_API_KEY\"] = \"xxx\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Set your Mistral API key as an environment variable. If you haven't already, [sign up for a Mistral acccount](https://console.mistral.ai/). Then [subscribe](https://console.mistral.ai/billing/) to a free trial or billing plan, after which you'll be able to [generate an API key](https://console.mistral.ai/api-keys/)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "GfynEpsKX2ra"
},
"outputs": [],
"source": [
"from mistralai import Mistral\n",
"\n",
"# Initialize Mistral client\n",
"mistral_client = Mistral(api_key=os.environ[\"MISTRAL_API_KEY\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YPD8vDyYFvYb"
},
"source": [
"## Examples"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K-9KuO4cTOz1"
},
"source": [
"### 1. Completions\n",
"\n",
"We are integrating the Mistral AI SDK with Langfuse using the [@observe decorator](https://langfuse.com/docs/sdk/python/decorators), which is crucial for logging and tracing interactions with large language models (LLMs). The @observe(as_type=\"generation\") decorator specifically logs LLM interactions, capturing inputs, outputs, and model parameters. The resulting `mistral_completion` method can then be used across your project."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "I-ypnTvk1TSf"
},
"outputs": [],
"source": [
"from langfuse.decorators import langfuse_context, observe\n",
"\n",
"# Function to handle Mistral completion calls, wrapped with @observe to log the LLM interaction\n",
"@observe(as_type=\"generation\")\n",
"def mistral_completion(**kwargs):\n",
" # Clone kwargs to avoid modifying the original input\n",
" kwargs_clone = kwargs.copy()\n",
"\n",
" # Extract relevant parameters from kwargs\n",
" input = kwargs_clone.pop('messages', None)\n",
" model = kwargs_clone.pop('model', None)\n",
" min_tokens = kwargs_clone.pop('min_tokens', None)\n",
" max_tokens = kwargs_clone.pop('max_tokens', None)\n",
" temperature = kwargs_clone.pop('temperature', None)\n",
" top_p = kwargs_clone.pop('top_p', None)\n",
"\n",
" # Filter and prepare model parameters for logging\n",
" model_parameters = {\n",
" \"maxTokens\": max_tokens,\n",
" \"minTokens\": min_tokens,\n",
" \"temperature\": temperature,\n",
" \"top_p\": top_p\n",
" }\n",
" model_parameters = {k: v for k, v in model_parameters.items() if v is not None}\n",
"\n",
" # Log the input and model parameters before calling the LLM\n",
" langfuse_context.update_current_observation(\n",
" input=input,\n",
" model=model,\n",
" model_parameters=model_parameters,\n",
" metadata=kwargs_clone,\n",
"\n",
" )\n",
"\n",
" # Call the Mistral model to generate a response\n",
" res = mistral_client.chat.complete(**kwargs)\n",
"\n",
" # Log the usage details and output content after the LLM call\n",
" langfuse_context.update_current_observation(\n",
" usage={\n",
" \"input\": res.usage.prompt_tokens,\n",
" \"output\": res.usage.completion_tokens\n",
" },\n",
" output=res.choices[0].message.content\n",
" )\n",
"\n",
" # Return the model's response object\n",
" return res"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UaoL0rQwRNK9"
},
"source": [
"Optionally, other functions (api handlers, retrieval functions, ...) can be also decorated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "75C-g5IeFpNm"
},
"source": [
"#### 1.1 Simple Example\n",
"\n",
"In the following example, we also added the decorator to the top-level function `find_best_painter_from`. This function calls the mistral_completion function, which is decorated with @observe(as_type=\"generation\"). This hierarchical setup hels to trace more complex applications which involve multiple LLM calls and other non-llm methods which are decorated with @observe.\n",
"\n",
"You can use langfuse_context.update_current_observation or langfuse_context.update_current_trace to add additional details such as input, output, and model parameters to the trace."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 53
},
"id": "jsuNohCxFnm0",
"outputId": "3ce61721-c23b-4336-9e5d-bad790b63396"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Claude Monet, renowned for his role as a founder of French Impressionist painting, is often considered one of the best painters from France.'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@observe()\n",
"def find_best_painter_from(country=\"France\"):\n",
" response = mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" temperature=0.4,\n",
" messages=[\n",
" {\n",
" \"content\": \"Who is the best painter from {country}? Answer in one short sentence.\".format(country=country),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
" return response.choices[0].message.content\n",
"\n",
"find_best_painter_from()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tTcf5MsAHX3B"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/836a9585-cfcc-47f7-881f-85ebdd9f601b\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JoGaSQDXFjr-"
},
"source": [
"#### 1.2 Chained Completions"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MCh8Cf-tYt8T"
},
"source": [
"\n",
"This example demonstrates chaining multiple LLM calls using the @observe decorator. The first call identifies the best painter from a specified country, and the second call uses that painter's name to find their most famous painting. Both interactions are logged by Langfuse as we use the wrapped `mistral_completion` method created above, ensuring full traceability across the chained requests."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Dd0a7m8dCRuT",
"outputId": "c379e557-e847-4406-df5a-17a24afeb49d"
},
"outputs": [
{
"data": {
"text/plain": [
"ChatCompletionResponse(id='8bb8512749fd4ddf88720aec0021378c', object='chat.completion', model='mistral-small-latest', usage=UsageInfo(prompt_tokens=23, completion_tokens=23, total_tokens=46), created=1726597735, choices=[ChatCompletionChoice(index=0, message=AssistantMessage(content='Albrecht Dürer\\'s most famous painting is \"Self-Portrait at Twenty-Eight.\"', tool_calls=None, prefix=False, role='assistant'), finish_reason='stop')])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@observe()\n",
"def find_best_painting_from(country=\"France\"):\n",
" response = mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" temperature=0.1,\n",
" messages=[\n",
" {\n",
" \"content\": \"Who is the best painter from {country}? Only provide the name.\".format(country=country),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
" painter_name = response.choices[0].message.content\n",
" return mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" messages=[\n",
" {\n",
" \"content\": \"What is the most famous painting of {painter_name}? Answer in one short sentence.\".format(painter_name=painter_name),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
"\n",
"find_best_painting_from(\"Germany\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "86xDBe6EIEp-"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/a3360c6f-24ad-455c-aae7-eb9d5c6f5dac\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lofuuUMrIAqr"
},
"source": [
"### 2. Streaming Completions"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pqHdueUsXbBw"
},
"source": [
"The following example demonstrates how to handle streaming responses from the Mistral model using the @observe(as_type=\"generation\") decorator. The process is similar to the *Completion* example but includes handling streamed data in real-time.\n",
"\n",
"Just like in the previous example, we wrap the streaming function with the @observe decorator to capture the input, model parameters, and usage details. Additionally, the function processes the streamed output incrementally, updating the Langfuse context as each chunk is received."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 643
},
"id": "z6q4JsrjIdtm",
"outputId": "b242f793-e4cf-4770-ef0a-4b6344c859e0"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The\n",
" best\n",
" five\n",
" pain\n",
"ters\n",
" from\n",
" Spain\n",
" are\n",
" Diego\n",
" Vel\n",
"áz\n",
"que\n",
"z\n",
",\n",
" Francisco\n",
" G\n",
"oya\n",
",\n",
" P\n",
"ablo\n",
" Pic\n",
"asso\n",
",\n",
" Salvador\n",
" Dal\n",
"í\n",
",\n",
" and\n",
" Joan\n",
" Mir\n",
"ó\n",
".\n",
"\n"
]
},
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'The best five painters from Spain are Diego Velázquez, Francisco Goya, Pablo Picasso, Salvador Dalí, and Joan Miró.'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Wrap streaming function with decorator\n",
"@observe(as_type=\"generation\")\n",
"def stream_mistral_completion(**kwargs):\n",
" kwargs_clone = kwargs.copy()\n",
" input = kwargs_clone.pop('messages', None)\n",
" model = kwargs_clone.pop('model', None)\n",
" min_tokens = kwargs_clone.pop('min_tokens', None)\n",
" max_tokens = kwargs_clone.pop('max_tokens', None)\n",
" temperature = kwargs_clone.pop('temperature', None)\n",
" top_p = kwargs_clone.pop('top_p', None)\n",
"\n",
" model_parameters = {\n",
" \"maxTokens\": max_tokens,\n",
" \"minTokens\": min_tokens,\n",
" \"temperature\": temperature,\n",
" \"top_p\": top_p\n",
" }\n",
" model_parameters = {k: v for k, v in model_parameters.items() if v is not None}\n",
"\n",
" langfuse_context.update_current_observation(\n",
" input=input,\n",
" model=model,\n",
" model_parameters=model_parameters,\n",
" metadata=kwargs_clone,\n",
" )\n",
"\n",
" res = mistral_client.chat.stream(**kwargs)\n",
" final_response = \"\"\n",
" for chunk in res:\n",
" content = chunk.data.choices[0].delta.content\n",
" final_response += content\n",
" yield content\n",
"\n",
" if chunk.data.choices[0].finish_reason == \"stop\":\n",
" langfuse_context.update_current_observation(\n",
" usage={\n",
" \"input\": chunk.data.usage.prompt_tokens,\n",
" \"output\": chunk.data.usage.completion_tokens\n",
" },\n",
" output=final_response\n",
" )\n",
" break\n",
"\n",
"# Use stream_mistral_completion as you'd usually use the SDK\n",
"@observe()\n",
"def stream_find_best_five_painter_from(country=\"France\"):\n",
" response_chunks = stream_mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" messages=[\n",
" {\n",
" \"content\": \"Who are the best five painter from {country}? Answer in one short sentence.\".format(country=country),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
" final_response = \"\"\n",
" for chunk in response_chunks:\n",
" final_response += chunk\n",
" # You can also do something with each chunk here if needed\n",
" print(chunk)\n",
"\n",
" return final_response\n",
"\n",
"stream_find_best_five_painter_from(\"Spain\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KTqfPlV-KJWx"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/75a2a4fe-088d-4134-9797-ba9c21be01b2\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HGuqF2dK8r3A"
},
"source": [
"### 3. Async Completion\n",
"\n",
"\n",
"This example showcases the use of the @observe decorator in an asynchronous context. It wraps an async function that interacts with the Mistral model, ensuring that both the request and the response are logged by Langfuse. The async function allows for non-blocking LLM calls, making it suitable for applications that require concurrency while maintaining full observability of the interactions."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Xueu1ypx8zG9",
"outputId": "8822b913-8650-4727-d679-da593d2375ce"
},
"outputs": [
{
"data": {
"text/plain": [
"ChatCompletionResponse(id='589fa6216c5346cc984586209c693a41', object='chat.completion', model='mistral-small-latest', usage=UsageInfo(prompt_tokens=17, completion_tokens=33, total_tokens=50), created=1726597737, choices=[ChatCompletionChoice(index=0, message=AssistantMessage(content=\"One of the most renowned musicians from Spain is Andrés Segovia, a classical guitarist who significantly impacted the instrument's modern repertoire.\", tool_calls=None, prefix=False, role='assistant'), finish_reason='stop')])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Wrap async function with decorator\n",
"@observe(as_type=\"generation\")\n",
"async def async_mistral_completion(**kwargs):\n",
" kwargs_clone = kwargs.copy()\n",
" input = kwargs_clone.pop('messages', None)\n",
" model = kwargs_clone.pop('model', None)\n",
" min_tokens = kwargs_clone.pop('min_tokens', None)\n",
" max_tokens = kwargs_clone.pop('max_tokens', None)\n",
" temperature = kwargs_clone.pop('temperature', None)\n",
" top_p = kwargs_clone.pop('top_p', None)\n",
"\n",
" model_parameters = {\n",
" \"maxTokens\": max_tokens,\n",
" \"minTokens\": min_tokens,\n",
" \"temperature\": temperature,\n",
" \"top_p\": top_p\n",
" }\n",
" model_parameters = {k: v for k, v in model_parameters.items() if v is not None}\n",
"\n",
" langfuse_context.update_current_observation(\n",
" input=input,\n",
" model=model,\n",
" model_parameters=model_parameters,\n",
" metadata=kwargs_clone,\n",
"\n",
" )\n",
"\n",
" res = await mistral_client.chat.complete_async(**kwargs)\n",
"\n",
" langfuse_context.update_current_observation(\n",
" usage={\n",
" \"input\": res.usage.prompt_tokens,\n",
" \"output\": res.usage.completion_tokens\n",
" },\n",
" output=res.choices[0].message.content\n",
" )\n",
"\n",
" return res\n",
"\n",
"@observe()\n",
"async def async_find_best_musician_from(country=\"France\"):\n",
" response = await async_mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" messages=[\n",
" {\n",
" \"content\": \"Who is the best musician from {country}? Answer in one short sentence.\".format(country=country),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
" return response\n",
"\n",
"await async_find_best_musician_from(\"Spain\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BjxwFNd2KWVI"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/1f7d91ce-45dd-41bf-8e6f-1875086ed32f\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pZrkiNk5DjIU"
},
"source": [
"### 4. Async Streaming"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HJ0WyKckZY_-"
},
"source": [
"This example demonstrates the use of the @observe decorator in an asynchronous streaming context. It wraps an async function that streams responses from the Mistral model, logging each chunk of data in real-time."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 785
},
"id": "XKs-IEA1KvoE",
"outputId": "728de600-efff-48e2-c37a-a0a932383f7a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The\n",
" five\n",
" most\n",
" renown\n",
"ed\n",
" musicians\n",
" from\n",
" Spain\n",
" include\n",
":\n",
" Andr\n",
"és\n",
" Seg\n",
"ov\n",
"ia\n",
",\n",
" Pac\n",
"o\n",
" de\n",
" Luc\n",
"ía\n",
",\n",
" En\n",
"rique\n",
" I\n",
"gles\n",
"ias\n",
",\n",
" Ale\n",
"j\n",
"andro\n",
" San\n",
"z\n",
",\n",
" and\n",
" Ros\n",
"al\n",
"ía\n",
".\n",
"\n"
]
},
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'The five most renowned musicians from Spain include: Andrés Segovia, Paco de Lucía, Enrique Iglesias, Alejandro Sanz, and Rosalía.'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import asyncio\n",
"\n",
"# Wrap async streaming function with decorator\n",
"@observe(as_type=\"generation\")\n",
"async def async_stream_mistral_completion(**kwargs):\n",
" kwargs_clone = kwargs.copy()\n",
" input = kwargs_clone.pop('messages', None)\n",
" model = kwargs_clone.pop('model', None)\n",
" min_tokens = kwargs_clone.pop('min_tokens', None)\n",
" max_tokens = kwargs_clone.pop('max_tokens', None)\n",
" temperature = kwargs_clone.pop('temperature', None)\n",
" top_p = kwargs_clone.pop('top_p', None)\n",
"\n",
" model_parameters = {\n",
" \"maxTokens\": max_tokens,\n",
" \"minTokens\": min_tokens,\n",
" \"temperature\": temperature,\n",
" \"top_p\": top_p\n",
" }\n",
" model_parameters = {k: v for k, v in model_parameters.items() if v is not None}\n",
"\n",
" langfuse_context.update_current_observation(\n",
" input=input,\n",
" model=model,\n",
" model_parameters=model_parameters,\n",
" metadata=kwargs_clone,\n",
" )\n",
"\n",
" res = await mistral_client.chat.stream_async(**kwargs)\n",
" final_response = \"\"\n",
" async for chunk in res:\n",
" content = chunk.data.choices[0].delta.content\n",
" final_response += content\n",
" yield content\n",
"\n",
" if chunk.data.choices[0].finish_reason == \"stop\":\n",
" langfuse_context.update_current_observation(\n",
" usage={\n",
" \"input\": chunk.data.usage.prompt_tokens,\n",
" \"output\": chunk.data.usage.completion_tokens\n",
" },\n",
" output=final_response\n",
" )\n",
" break\n",
"\n",
"@observe()\n",
"async def async_stream_find_best_five_musician_from(country=\"France\"):\n",
" response_chunks = async_stream_mistral_completion(\n",
" model=\"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" messages=[\n",
" {\n",
" \"content\": \"Who are the best five musician from {country}? Answer in one short sentence.\".format(country=country),\n",
" \"role\": \"user\",\n",
" },\n",
" ]\n",
" )\n",
" final_response = \"\"\n",
" async for chunk in response_chunks:\n",
" final_response += chunk\n",
" # You can also do something with each chunk here if needed\n",
" print(chunk)\n",
"\n",
" return final_response\n",
"\n",
"# Run the async function\n",
"await async_stream_find_best_five_musician_from(\"Spain\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_fCkQkl0K6LU"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/36608110-f6cf-4566-a080-7c18777e2dbf\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dObEp9bZbOV6"
},
"source": [
"### 5. Tool Calling"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qXT1bi_xeQyt"
},
"source": [
"This snippet introduces Mistral's function-calling capability, where you can define custom functions to retrieve specific data, like payment status and date, based on a transaction ID. These functions are then registered with the Mistral model, allowing it to call them when processing queries. For a deeper dive into function calling with Mistral, refer to the official [Mistral documentation](https://docs.mistral.ai/capabilities/function_calling/)."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "05Cqr3dvbQ-e"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import json\n",
"import functools\n",
"\n",
"\n",
"# Sample payment transaction data\n",
"data = {\n",
" 'transaction_id': ['T1001', 'T1002', 'T1003', 'T1004', 'T1005'],\n",
" 'customer_id': ['C001', 'C002', 'C003', 'C002', 'C001'],\n",
" 'payment_amount': [125.50, 89.99, 120.00, 54.30, 210.20],\n",
" 'payment_date': ['2021-10-05', '2021-10-06', '2021-10-07', '2021-10-05', '2021-10-08'],\n",
" 'payment_status': ['Paid', 'Unpaid', 'Paid', 'Paid', 'Pending']\n",
"}\n",
"\n",
"# Create a DataFrame from the data\n",
"df = pd.DataFrame(data)\n",
"\n",
"# Function to retrieve payment status given a transaction ID\n",
"def retrieve_payment_status(df: data, transaction_id: str) -> str:\n",
" if transaction_id in df.transaction_id.values:\n",
" # Return the payment status as a JSON string\n",
" return json.dumps({'status': df[df.transaction_id == transaction_id].payment_status.item()})\n",
" return json.dumps({'error': 'transaction id not found.'})\n",
"\n",
"# Function to retrieve payment date given a transaction ID\n",
"def retrieve_payment_date(df: data, transaction_id: str) -> str:\n",
" if transaction_id in df.transaction_id.values:\n",
" # Return the payment date as a JSON string\n",
" return json.dumps({'date': df[df.transaction_id == transaction_id].payment_date.item()})\n",
" return json.dumps({'error': 'transaction id not found.'})\n",
"\n",
"# Define tools for the Mistral model with JSON schemas\n",
"tools = [\n",
" {\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"retrieve_payment_status\",\n",
" \"description\": \"Get payment status of a transaction\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"transaction_id\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The transaction id.\",\n",
" }\n",
" },\n",
" \"required\": [\"transaction_id\"],\n",
" },\n",
" },\n",
" },\n",
" {\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"retrieve_payment_date\",\n",
" \"description\": \"Get payment date of a transaction\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"transaction_id\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The transaction id.\",\n",
" }\n",
" },\n",
" \"required\": [\"transaction_id\"],\n",
" },\n",
" },\n",
" }\n",
"]\n",
"\n",
"# Define tools for the Mistral model with JSON schemas\n",
"names_to_functions = {\n",
" 'retrieve_payment_status': functools.partial(retrieve_payment_status, df=df),\n",
" 'retrieve_payment_date': functools.partial(retrieve_payment_date, df=df)\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O9RcDqrJckOw"
},
"source": [
"The *check_transaction_status* function demonstrates the use of Mistral's function-calling capabilities. The function's result is then incorporated into the LLM's response, which is logged and traced in Langfuse. This example illustrates how external function calls can be seamlessly integrated into a Langfuse by using the the wrapped *mistral_completion* function, ensuring that every step — from tool selection to final output - is captured for thorough observability."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "bYLfckOjbvJb",
"outputId": "baa36a67-3ebb-4115-ed30-1447eca9d9d4"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Your transaction T1005 is currently pending.'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@observe()\n",
"def tool_calling_check_transaction_status(id=\"T1001\"):\n",
"\n",
" # Construct the initial user query message\n",
" messages = [{\"role\": \"user\", \"content\": \"What's the status of my transaction {id}?\".format(id=id)}]\n",
"\n",
" # Use the Langfuse-decorated Mistral completion function to generate a tool-assisted response\n",
" response = mistral_completion(\n",
" model = \"mistral-small-latest\",\n",
" messages = messages,\n",
" max_tokens=512,\n",
" temperature=0.1,\n",
" tools = tools,\n",
" tool_choice = \"any\",\n",
" )\n",
"\n",
"\n",
" messages.append(response.choices[0].message)\n",
"\n",
" # Extract the tool call details from the model's response\n",
" tool_call = response.choices[0].message.tool_calls[0]\n",
" function_name = tool_call.function.name\n",
" function_params = json.loads(tool_call.function.arguments)\n",
"\n",
" # Execute the selected function with the extracted parameters\n",
" function_result = names_to_functions[function_name](**function_params)\n",
"\n",
" messages.append({\"role\":\"tool\", \"name\":function_name, \"content\":function_result, \"tool_call_id\":tool_call.id})\n",
"\n",
" # Call the Langfuse-wrapped Mistral completion function again to generate a final response using the tool's result\n",
" response = mistral_completion(\n",
" model = \"mistral-small-latest\",\n",
" max_tokens=1024,\n",
" temperature=0.5,\n",
" messages = messages\n",
" )\n",
"\n",
" return response.choices[0].message.content\n",
"\n",
"tool_calling_check_transaction_status(\"T1005\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1DljCfwGLSqz"
},
"source": [
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/e986408a-f96b-40dc-8278-5d0eb0286f82\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oAQYR1MsLVRM"
},
"source": [
"# Feedback\n",
"---\n",
"\n",
"If you have any feedback or requests, please create a GitHub [Issue](https://langfuse.com/issue) or share your idea with the community on [Discord](https://discord.langfuse.com/)."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}