← Back to Cookbook
cookbook mistral opentelemetry
Details
File: third_party/openlit/cookbook_mistral_opentelemetry.ipynb
Type: Jupyter Notebook
Use Cases:
Integrations: Openlit, Opentelemetry
Content
Notebook content (JSON format):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "IBxQQ9lPW2nr"
},
"source": [
"<center>\n",
" <p style=\"text-align:center\">\n",
" <img alt=\"langfuse logo\" src=\"https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true\" width=\"200\"/>\n",
" <br>\n",
" <a href=\"https://docs.openlit.io/\">Docs</a>\n",
" |\n",
" <a href=\"https://github.com/openlit/openlit\">GitHub</a>\n",
" |\n",
" <a href=\"https://discord.langfuse.com/\">Slack</a>\n",
" </p>\n",
"</center>\n",
"<h1 align=\"center\">Monitoring Mistral AI with OpenTelemetry</h1>\n",
"\n",
"This cookbook will cover the process of integrating OpenLIT with the Mistral SDK. A straightforward guide demonstrates how adding a single line of code can seamlessly enable OpenLIT to track various metrics, including cost, tokens, prompts, responses, and all chat/completion activities from the Mistral SDK using OpenTelemetry.\n",
"\n",
"## About OpenLIT\n",
"\n",
"**OpenLIT** is an open-source AI Engineering tool that help you to simplify your AI development workflow, especially for Generative AI and LLMs. It streamlines essential tasks like experimenting with LLMs, organizing and versioning prompts, and securely handling API keys. With just one line of code, you can enable **OpenTelemetry-native** observability, offering full-stack monitoring that includes LLMs, vector databases, and GPUs. This enables developers to confidently build AI features and applications, transitioning smoothly from testing to production.\n",
"\n",
"This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0dFNPPanZCGf",
"outputId": "00257742-afa2-43a4-8308-74a9ffa39ed2",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: mistralai in /opt/conda/lib/python3.10/site-packages (1.2.3)\n",
"Requirement already satisfied: openlit in /opt/conda/lib/python3.10/site-packages (1.31.1)\n",
"Requirement already satisfied: langchain_community in /opt/conda/lib/python3.10/site-packages (0.3.8)\n",
"Requirement already satisfied: eval-type-backport<0.3.0,>=0.2.0 in /opt/conda/lib/python3.10/site-packages (from mistralai) (0.2.0)\n",
"Requirement already satisfied: httpx<0.28.0,>=0.27.0 in /opt/conda/lib/python3.10/site-packages (from mistralai) (0.27.2)\n",
"Requirement already satisfied: jsonpath-python<2.0.0,>=1.0.6 in /opt/conda/lib/python3.10/site-packages (from mistralai) (1.0.6)\n",
"Requirement already satisfied: pydantic<3.0.0,>=2.9.0 in /opt/conda/lib/python3.10/site-packages (from mistralai) (2.9.2)\n",
"Requirement already satisfied: python-dateutil==2.8.2 in /opt/conda/lib/python3.10/site-packages (from mistralai) (2.8.2)\n",
"Requirement already satisfied: typing-inspect<0.10.0,>=0.9.0 in /opt/conda/lib/python3.10/site-packages (from mistralai) (0.9.0)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil==2.8.2->mistralai) (1.16.0)\n",
"Requirement already satisfied: anthropic<0.22.0,>=0.21.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (0.21.3)\n",
"Requirement already satisfied: boto3<2.0.0,>=1.34.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.35.67)\n",
"Requirement already satisfied: botocore<2.0.0,>=1.34.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.35.67)\n",
"Requirement already satisfied: openai<2.0.0,>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.55.0)\n",
"Requirement already satisfied: opentelemetry-api<2.0.0,>=1.27.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.27.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp<2.0.0,>=1.27.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.27.0)\n",
"Requirement already satisfied: opentelemetry-instrumentation<0.49,>=0.48b0 in /opt/conda/lib/python3.10/site-packages (from openlit) (0.48b0)\n",
"Requirement already satisfied: opentelemetry-sdk<2.0.0,>=1.27.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.27.0)\n",
"Requirement already satisfied: requests<3.0.0,>=2.26.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (2.32.3)\n",
"Requirement already satisfied: schedule<2.0.0,>=1.2.2 in /opt/conda/lib/python3.10/site-packages (from openlit) (1.2.2)\n",
"Requirement already satisfied: tiktoken<0.8.0,>=0.7.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (0.7.0)\n",
"Requirement already satisfied: xmltodict<0.14.0,>=0.13.0 in /opt/conda/lib/python3.10/site-packages (from openlit) (0.13.0)\n",
"Requirement already satisfied: PyYAML>=5.3 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (6.0.2)\n",
"Requirement already satisfied: SQLAlchemy<2.0.36,>=1.4 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (2.0.35)\n",
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (3.10.10)\n",
"Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (0.6.7)\n",
"Requirement already satisfied: httpx-sse<0.5.0,>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (0.4.0)\n",
"Requirement already satisfied: langchain<0.4.0,>=0.3.8 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (0.3.8)\n",
"Requirement already satisfied: langchain-core<0.4.0,>=0.3.21 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (0.3.21)\n",
"Requirement already satisfied: langsmith<0.2.0,>=0.1.125 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (0.1.146)\n",
"Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (1.26.4)\n",
"Requirement already satisfied: pydantic-settings<3.0.0,>=2.4.0 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (2.6.1)\n",
"Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in /opt/conda/lib/python3.10/site-packages (from langchain_community) (9.0.0)\n",
"Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (2.4.3)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (24.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.4.1)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (6.1.0)\n",
"Requirement already satisfied: yarl<2.0,>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.14.0)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (4.0.3)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /opt/conda/lib/python3.10/site-packages (from anthropic<0.22.0,>=0.21.0->openlit) (3.7.1)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /opt/conda/lib/python3.10/site-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.9.0)\n",
"Requirement already satisfied: sniffio in /opt/conda/lib/python3.10/site-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.3.1)\n",
"Requirement already satisfied: tokenizers>=0.13.0 in /opt/conda/lib/python3.10/site-packages (from anthropic<0.22.0,>=0.21.0->openlit) (0.20.3)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /opt/conda/lib/python3.10/site-packages (from anthropic<0.22.0,>=0.21.0->openlit) (4.12.2)\n",
"Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0.0,>=1.34.0->openlit) (1.0.1)\n",
"Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0.0,>=1.34.0->openlit) (0.10.4)\n",
"Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /opt/conda/lib/python3.10/site-packages (from botocore<2.0.0,>=1.34.0->openlit) (1.26.20)\n",
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /opt/conda/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain_community) (3.23.1)\n",
"Requirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from httpx<0.28.0,>=0.27.0->mistralai) (2024.8.30)\n",
"Requirement already satisfied: httpcore==1.* in /opt/conda/lib/python3.10/site-packages (from httpx<0.28.0,>=0.27.0->mistralai) (1.0.7)\n",
"Requirement already satisfied: idna in /opt/conda/lib/python3.10/site-packages (from httpx<0.28.0,>=0.27.0->mistralai) (3.10)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /opt/conda/lib/python3.10/site-packages (from httpcore==1.*->httpx<0.28.0,>=0.27.0->mistralai) (0.14.0)\n",
"Requirement already satisfied: langchain-text-splitters<0.4.0,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from langchain<0.4.0,>=0.3.8->langchain_community) (0.3.2)\n",
"Requirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.4.0,>=0.3.21->langchain_community) (1.33)\n",
"Requirement already satisfied: packaging<25,>=23.2 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.4.0,>=0.3.21->langchain_community) (24.1)\n",
"Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /opt/conda/lib/python3.10/site-packages (from langsmith<0.2.0,>=0.1.125->langchain_community) (3.10.12)\n",
"Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from langsmith<0.2.0,>=0.1.125->langchain_community) (1.0.0)\n",
"Requirement already satisfied: jiter<1,>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from openai<2.0.0,>=1.1.1->openlit) (0.7.1)\n",
"Requirement already satisfied: tqdm>4 in /opt/conda/lib/python3.10/site-packages (from openai<2.0.0,>=1.1.1->openlit) (4.66.5)\n",
"Requirement already satisfied: deprecated>=1.2.6 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (1.2.14)\n",
"Requirement already satisfied: importlib-metadata<=8.4.0,>=6.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (8.4.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc==1.27.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.27.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-http==1.27.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.27.0)\n",
"Requirement already satisfied: googleapis-common-protos~=1.52 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.65.0)\n",
"Requirement already satisfied: grpcio<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.66.2)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.27.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.27.0)\n",
"Requirement already satisfied: opentelemetry-proto==1.27.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.27.0)\n",
"Requirement already satisfied: protobuf<5.0,>=3.19 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-proto==1.27.0->opentelemetry-exporter-otlp-proto-grpc==1.27.0->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (3.20.3)\n",
"Requirement already satisfied: setuptools>=16.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (74.1.2)\n",
"Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (1.16.0)\n",
"Requirement already satisfied: opentelemetry-semantic-conventions==0.48b0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-sdk<2.0.0,>=1.27.0->openlit) (0.48b0)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from pydantic<3.0.0,>=2.9.0->mistralai) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.23.4 in /opt/conda/lib/python3.10/site-packages (from pydantic<3.0.0,>=2.9.0->mistralai) (2.23.4)\n",
"Requirement already satisfied: python-dotenv>=0.21.0 in /opt/conda/lib/python3.10/site-packages (from pydantic-settings<3.0.0,>=2.4.0->langchain_community) (1.0.1)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0,>=2.26.0->openlit) (3.3.2)\n",
"Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.10/site-packages (from SQLAlchemy<2.0.36,>=1.4->langchain_community) (3.1.1)\n",
"Requirement already satisfied: regex>=2022.1.18 in /opt/conda/lib/python3.10/site-packages (from tiktoken<0.8.0,>=0.7.0->openlit) (2024.11.6)\n",
"Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from typing-inspect<0.10.0,>=0.9.0->mistralai) (1.0.0)\n",
"Requirement already satisfied: exceptiongroup in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.5.0->anthropic<0.22.0,>=0.21.0->openlit) (1.2.2)\n",
"Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<=8.4.0,>=6.0->opentelemetry-api<2.0.0,>=1.27.0->openlit) (3.20.2)\n",
"Requirement already satisfied: jsonpointer>=1.9 in /opt/conda/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.21->langchain_community) (3.0.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /opt/conda/lib/python3.10/site-packages (from tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (0.26.2)\n",
"Requirement already satisfied: propcache>=0.2.0 in /opt/conda/lib/python3.10/site-packages (from yarl<2.0,>=1.12.0->aiohttp<4.0.0,>=3.8.3->langchain_community) (0.2.0)\n",
"Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (3.16.1)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (2024.9.0)\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install mistralai openlit langchain_community"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lY3vpFlYZwTy"
},
"source": [
"Set your Mistral API key as an environment variable. If you haven't already, [sign up for a Mistral acccount](https://console.mistral.ai/). Then [subscribe](https://console.mistral.ai/billing/) to a free trial or billing plan, after which you'll be able to [generate an API key](https://console.mistral.ai/api-keys/)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "fX4NzuqeZZjn"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# Your Mistral key\n",
"os.environ[\"MISTRAL_API_KEY\"] = \"YOUR_MISTRAL_AI_API_KEY\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "KBFmwn8FihbI"
},
"outputs": [],
"source": [
"# Synchronous Example\n",
"from mistralai import Mistral\n",
"import os\n",
"\n",
"client = Mistral(\n",
" api_key=os.getenv(\"MISTRAL_API_KEY\", \"\"),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "eYjMf04Oio3g"
},
"outputs": [],
"source": [
"import openlit\n",
"\n",
"openlit.init(disable_metrics=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v7AYQi2cZ9vR"
},
"source": [
"## Chat Completions\n",
"\n",
"Once OpenLIT is initialized in the application, It auto-instruments all Mistral Chat function usage from the SDK. This helps track LLM interactions, capturing inputs, outputs, model parameters along with cost."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "T1UuOmEkehcf",
"outputId": "e1c99f8b-8486-45f8-f3db-6a8591fc13a2",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"name\": \"mistral.chat\",\n",
" \"context\": {\n",
" \"trace_id\": \"0x7139cc77345ae1ab316a4a564b0a7997\",\n",
" \"span_id\": \"0xe16faa004e5e45ef\",\n",
" \"trace_state\": \"[]\"\n",
" },\n",
" \"kind\": \"SpanKind.CLIENT\",\n",
" \"parent_id\": null,\n",
" \"start_time\": \"2024-11-26T13:42:22.531723Z\",\n",
" \"end_time\": \"2024-11-26T13:42:23.049932Z\",\n",
" \"status\": {\n",
" \"status_code\": \"OK\"\n",
" },\n",
" \"attributes\": {\n",
" \"telemetry.sdk.name\": \"openlit\",\n",
" \"gen_ai.system\": \"mistral\",\n",
" \"gen_ai.operation.name\": \"chat\",\n",
" \"gen_ai.endpoint\": \"mistral.chat\",\n",
" \"gen_ai.response.id\": \"5cf4cabea8d84623a1f34a5f1ff35972\",\n",
" \"gen_ai.environment\": \"default\",\n",
" \"gen_ai.application_name\": \"default\",\n",
" \"gen_ai.request.model\": \"mistral-small-latest\",\n",
" \"gen_ai.request.temperature\": 0.7,\n",
" \"gen_ai.request.top_p\": 1.0,\n",
" \"gen_ai.request.max_tokens\": -1,\n",
" \"gen_ai.request.seed\": \"\",\n",
" \"gen_ai.request.is_stream\": false,\n",
" \"gen_ai.response.finish_reasons\": [\n",
" \"stop\"\n",
" ],\n",
" \"gen_ai.usage.input_tokens\": 16,\n",
" \"gen_ai.usage.output_tokens\": 11,\n",
" \"gen_ai.usage.total_tokens\": 27,\n",
" \"gen_ai.usage.cost\": 9.8e-05\n",
" },\n",
" \"events\": [\n",
" {\n",
" \"name\": \"gen_ai.content.prompt\",\n",
" \"timestamp\": \"2024-11-26T13:42:23.049891Z\",\n",
" \"attributes\": {\n",
" \"gen_ai.prompt\": \"user: Who is the best French painter? Answer in one short sentence.\"\n",
" }\n",
" },\n",
" {\n",
" \"name\": \"gen_ai.content.completion\",\n",
" \"timestamp\": \"2024-11-26T13:42:23.049908Z\",\n",
" \"attributes\": {\n",
" \"gen_ai.completion\": \"Claude Monet is often considered the best French painter.\"\n",
" }\n",
" }\n",
" ],\n",
" \"links\": [],\n",
" \"resource\": {\n",
" \"attributes\": {\n",
" \"telemetry.sdk.language\": \"python\",\n",
" \"telemetry.sdk.name\": \"openlit\",\n",
" \"telemetry.sdk.version\": \"1.27.0\",\n",
" \"service.name\": \"default\",\n",
" \"deployment.environment\": \"default\"\n",
" },\n",
" \"schema_url\": \"\"\n",
" }\n",
"}\n"
]
}
],
"source": [
"res = client.chat.complete(model=\"mistral-small-latest\", messages=[\n",
" {\n",
" \"content\": \"Who is the best French painter? Answer in one short sentence.\",\n",
" \"role\": \"user\",\n",
" },\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yKK1gXg2oVLL"
},
"source": [
"## Embeddings\n",
"\n",
"Once OpenLIT is initialized in the application, It auto-instruments all Mistral embedding function usage from the SDK. This helps track embedding inputs, outputs, model parameters along with cost."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "CkTlT5IWoe1b",
"outputId": "64f420f1-2b1c-4acc-f3a3-104c162236f2",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"name\": \"mistral.embeddings\",\n",
" \"context\": {\n",
" \"trace_id\": \"0xc4887fcaf1b7a46b4a4744539a1b5ccf\",\n",
" \"span_id\": \"0xe510abcd3d63a20e\",\n",
" \"trace_state\": \"[]\"\n",
" },\n",
" \"kind\": \"SpanKind.CLIENT\",\n",
" \"parent_id\": null,\n",
" \"start_time\": \"2024-11-26T13:42:26.831681Z\",\n",
" \"end_time\": \"2024-11-26T13:42:27.184374Z\",\n",
" \"status\": {\n",
" \"status_code\": \"OK\"\n",
" },\n",
" \"attributes\": {\n",
" \"telemetry.sdk.name\": \"openlit\",\n",
" \"gen_ai.system\": \"mistral\",\n",
" \"gen_ai.operation.name\": \"embedding\",\n",
" \"gen_ai.endpoint\": \"mistral.embeddings\",\n",
" \"gen_ai.environment\": \"default\",\n",
" \"gen_ai.application_name\": \"default\",\n",
" \"gen_ai.request.model\": \"mistral-embed\",\n",
" \"gen_ai.request.embedding_format\": \"float\",\n",
" \"gen_ai.response.id\": \"beb457f7913148198008a1ccd50073bc\",\n",
" \"gen_ai.usage.input_tokens\": 15,\n",
" \"gen_ai.usage.total_tokens\": 15,\n",
" \"gen_ai.usage.cost\": 1.5e-06\n",
" },\n",
" \"events\": [\n",
" {\n",
" \"name\": \"gen_ai.content.prompt\",\n",
" \"timestamp\": \"2024-11-26T13:42:27.184344Z\",\n",
" \"attributes\": {\n",
" \"gen_ai.prompt\": \"Embed this sentence., As well as this one.\"\n",
" }\n",
" }\n",
" ],\n",
" \"links\": [],\n",
" \"resource\": {\n",
" \"attributes\": {\n",
" \"telemetry.sdk.language\": \"python\",\n",
" \"telemetry.sdk.name\": \"openlit\",\n",
" \"telemetry.sdk.version\": \"1.27.0\",\n",
" \"service.name\": \"default\",\n",
" \"deployment.environment\": \"default\"\n",
" },\n",
" \"schema_url\": \"\"\n",
" }\n",
"}\n"
]
}
],
"source": [
"res = client.embeddings.create(inputs=[\n",
" \"Embed this sentence.\",\n",
" \"As well as this one.\",\n",
"], model=\"mistral-embed\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jkZz6xYpkGal"
},
"source": [
"# Sending Traces and metrics to OpenLIT\n",
"\n",
"By default, OpenLIT generates OpenTelemetry traces and metrics that are logged to your console. To set up a detailed monitoring environment, this guide outlines how to deploy OpenLIT and direct all traces and metrics there. You also have the flexibility to send the telemetry data to any OpenTelemetry-compatible endpoint, such as Grafana, Jaeger, or DataDog.\n",
"\n",
"## Deploy OpenLIT Stack\n",
"\n",
"1. Clone the OpenLIT Repository\n",
"\n",
" Open your terminal or command line and execute:\n",
"\n",
" ```shell\n",
" git clone git@github.com:openlit/openlit.git\n",
" ```\n",
"\n",
"2. Host it Yourself with Docker\n",
"\n",
" Deploy and start OpenLIT using the command:\n",
"\n",
" ```shell\n",
" docker compose up -d\n",
" ```\n",
"\n",
"> For instructions on installing in Kubernetes using Helm, refer to the [Kubernetes Helm installation guide](https://docs.openlit.io/latest/installation#kubernetes).\n",
"\n",
"Configure the telemetry data destination as follows:\n",
"\n",
"| Purpose | Parameter/Environment Variable | For Sending to OpenLIT |\n",
"|-------------------------------------------|--------------------------------------------------|--------------------------------|\n",
"| Send data to an HTTP OTLP endpoint | `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `\"http://127.0.0.1:4318\"` |\n",
"| Authenticate telemetry backends | `otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |\n",
"\n",
"> 💡 Info: If the `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.\n",
"\n",
"## Visualize and Optimize!\n",
"\n",
"With the Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your AI application's performance, behavior, and identify areas of improvement.\n",
"\n",
"Just head over to OpenLIT at `127.0.0.1:3000` on your browser to start exploring. You can login using the default credentials\n",
" - **Email**: `user@openlit.io`\n",
" - **Password**: `openlituser`\n",
"\n",
"\n",
""
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"environment": {
"kernel": "conda-base-py",
"name": "workbench-notebooks.m125",
"type": "gcloud",
"uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m125"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel) (Local)",
"language": "python",
"name": "conda-base-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.15"
}
},
"nbformat": 4,
"nbformat_minor": 4
}