← Back to Cookbook
cookbook langfuse posthog llama langfuse
Details
File: third_party/Langfuse/cookbook_langfuse_posthog_llama_langfuse.ipynb
Type: Jupyter Notebook
Use Cases:
Integrations: Langfuse
Content
Notebook content (JSON format):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "kSWOGK6XlOaT"
},
"source": [
"<center>\n",
" <p style=\"text-align:center\">\n",
" <img alt=\"langfuse logo\" src=\"https://langfuse.com/langfuse_logo.png\" width=\"200\"/>\n",
" <br>\n",
" <a href=\"https://langfuse.com/docs\">Docs</a>\n",
" |\n",
" <a href=\"https://github.com/langfuse/langfuse\">GitHub</a>\n",
" |\n",
" <a href=\"https://discord.langfuse.com/\">Discord</a>\n",
" </p>\n",
"</center>\n",
"<h1 align=\"center\">Monitoring LlamaIndex + Mistral applications with PostHog and Langfuse</h1>\n",
"\n",
"In this cookbook, we show you how to build a RAG application with [LlamaIndex](https://www.llamaindex.ai/), observe the steps with [Langfuse](https://langfuse.com/), and analyze the data in [PostHog](https://posthog.com/).\n",
"\n",
"## What is Langfuse?\n",
"\n",
"[Langfuse](https://langfuse.com/) is an open-source LLM engineering platform. It includes features such as [traces](https://langfuse.com/docs/tracing), [evals](https://langfuse.com/docs/scores/overview), and [prompt management](https://langfuse.com/docs/prompts/get-started) to help you debug and improve your LLM app.\n",
"\n",
"## What is LlamaIndex?\n",
"\n",
"LlamaIndex [(GitHub)](https://github.com/run-llama/llama_index) is a data framework designed to connect LLMs with external data sources. It helps structure, index, and query data effectively. This makes it easier for developers to build advanced LLM applications.\n",
"\n",
"## What is PostHog?\n",
"\n",
"[PostHog](https://posthog.com/) is a popular choice for product analytics. Combining Langfuse's LLM analytics with PostHog's product analytics makes it easy to answer questions about your app like:\n",
"\n",
"* \"What are my LLM costs by customer, model, and in total?\"\n",
"* \"Does interacting with LLM features correlate with other metrics (retention, usage, revenue, etc.)?\"\n",
"* \"How does the user feedback that I capture in Langfuse correlate with user behavior in PostHog?\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PW9tSXUsU2Np"
},
"source": [
"## **How to build a Simple RAG app with LlamaIndex and Mistral**\n",
"\n",
"In this example, we create a chat app that answers questions about how to care for hedgehogs. LlamaIndex vectorizes a [hedgehog care guide](https://www.pro-igel.de/downloads/merkblaetter_engl/wildtier_engl.pdf) using the [Mistral 8x22B model](https://docs.mistral.ai/getting-started/models/). Then, all model generations are traced using Langfuse's [LLamaIndex integration](https://langfuse.com/docs/integrations/llama-index/get-started).\n",
"\n",
"Lastly, the [PostHog integration](https://langfuse.com/docs/analytics/posthog) enables you to view detailed analytics about your hedgehog app directly in PostHog.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CwDWQEXdyifj"
},
"source": [
"### Step 1: Set up LlamaIndex and Mistral\n",
"\n",
"First, we set our Mistral API key as an environment variable. If you haven't already, [sign up for a Mistral acccount](https://console.mistral.ai/). Then [subscribe](https://console.mistral.ai/billing/) to a free trial or billing plan, after which you'll be able to [generate an API key](https://console.mistral.ai/api-keys/) (💡 You can use any other model supported by LlamaIndex; we just use Mistral in this cookbook).\n",
"\n",
"Then, we use LlamaIndex to initialize both a Mistral language model and an embedding model. We then set these models in the LlamaIndex `Settings` object:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "DkvTPKfDPorE",
"outputId": "024047f3-5bd8-49f1-a14c-fe04ad510d1b"
},
"outputs": [],
"source": [
"%pip install llama-index llama-index-llms-mistralai llama-index-embeddings-mistralai nest_asyncio --upgrade"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "VBud26iuq3Yj"
},
"outputs": [],
"source": [
"# Set the Mistral API key\n",
"import os\n",
"os.environ[\"MISTRAL_API_KEY\"] = \"***Your-Mistral-API-Key***\"\n",
"\n",
"# Ensures that sync and async code can be used together without issues\n",
"import nest_asyncio\n",
"nest_asyncio.apply()\n",
"\n",
"# Import and set up llama index\n",
"from llama_index.llms.mistralai import MistralAI\n",
"from llama_index.embeddings.mistralai import MistralAIEmbedding\n",
"from llama_index.core import Settings\n",
"\n",
"# Define your LLM and embedding model\n",
"llm = MistralAI(model=\"open-mixtral-8x22b\", temperature=0.1)\n",
"embed_model = MistralAIEmbedding(model_name=\"mistral-embed\")\n",
"\n",
"# Set the LLM and embedding model in the Settings object\n",
"Settings.llm = llm\n",
"Settings.embed_model = embed_model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JSFSNvH3yNXy"
},
"source": [
"### Step 2: Initialize Langfuse\n",
"\n",
"Next, we initialize the Langfuse client. [Sign up](https://cloud.langfuse.com/auth/sign-up) for Langfuse if you haven't already. Copy your [API keys](https://langfuse.com/faq/all/where-are-langfuse-api-keys) from your project settings and add them to your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 687
},
"collapsed": true,
"id": "FVTuaFX47x6b",
"outputId": "ed0deb2c-3a96-4ae8-e26f-d76237718c8f"
},
"outputs": [],
"source": [
"%pip install langfuse"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "-ikOqOQlpquS"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# get keys for your project from https://cloud.langfuse.com\n",
"LANGFUSE_SECRET_KEY=\"sk-lf-...\"\n",
"LANGFUSE_PUBLIC_KEY=\"pk-lf-...\"\n",
"LANGFUSE_HOST=\"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
"# LANGFUSE_HOST=\"https://us.cloud.langfuse.com\" # 🇺🇸 US region"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "hosC7NU8vHlI"
},
"outputs": [],
"source": [
"from langfuse import Langfuse\n",
"\n",
"langfuse = Langfuse()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LMFXe7RgpuFK"
},
"source": [
"Lastly, we register Langfuse's `LlamaIndexCallbackHandler` in the LlamaIndex `Settings.callback_manager` at the root of the app.\n",
"\n",
"Find out more about the Langfuse's LlamaIndex integration [here](https://langfuse.com/docs/integrations/llama-index/get-started)."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"id": "GFa1p-zxBVZX"
},
"outputs": [],
"source": [
"from llama_index.core import Settings\n",
"from llama_index.core.callbacks import CallbackManager\n",
"from langfuse.llama_index import LlamaIndexCallbackHandler\n",
"\n",
"langfuse_callback_handler = LlamaIndexCallbackHandler()\n",
"Settings.callback_manager = CallbackManager([langfuse_callback_handler])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ygzctbdrjJC"
},
"source": [
"### Step 3: Download data\n",
"\n",
"We download the file we want to use for RAG. In this example, we use a [hedgehog care guide](https://www.pro-igel.de/downloads/merkblaetter_engl/wildtier_engl.pdf) pdf file to enable the language model to answer questions about caring for hedgehogs 🦔."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "dG5sRzbJCt81",
"outputId": "589c44ed-9ebd-4706-d52e-782a5ec8f854"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-09-20 13:16:39-- https://www.pro-igel.de/downloads/merkblaetter_engl/wildtier_engl.pdf\n",
"Resolving www.pro-igel.de (www.pro-igel.de)... 152.53.23.200\n",
"Connecting to www.pro-igel.de (www.pro-igel.de)|152.53.23.200|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 1160174 (1.1M) [application/pdf]\n",
"Saving to: ‘./hedgehog.pdf’\n",
"\n",
"./hedgehog.pdf 100%[===================>] 1.11M 2.03MB/s in 0.5s \n",
"\n",
"2024-09-20 13:16:40 (2.03 MB/s) - ‘./hedgehog.pdf’ saved [1160174/1160174]\n",
"\n"
]
}
],
"source": [
"!wget 'https://www.pro-igel.de/downloads/merkblaetter_engl/wildtier_engl.pdf' -O './hedgehog.pdf'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cgfxuV2ccGwH"
},
"source": [
"Next, we load the pdf using the LlamaIndex [`SimpleDirectoryReader`](https://docs.llamaindex.ai/en/stable/module_guides/loading/simpledirectoryreader/)."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"id": "siRG6MDkII4f"
},
"outputs": [],
"source": [
"from llama_index.core import SimpleDirectoryReader\n",
"\n",
"hedgehog_docs = SimpleDirectoryReader(input_files=[\"./hedgehog.pdf\"]).load_data()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sFOhOuTFy95m"
},
"source": [
"### Step 4: Build RAG on the hedgehog doc\n",
"\n",
"Next, we create vector embeddings of the hedgehog document using [`VectorStoreIndex`](https://docs.llamaindex.ai/en/stable/module_guides/indexing/vector_store_index/) and then convert it into a [queryable engine](https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/) to retrieve information based on queries."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"id": "JTYXBIO4C4xF"
},
"outputs": [],
"source": [
"from llama_index.core import VectorStoreIndex\n",
"\n",
"hedgehog_index = VectorStoreIndex.from_documents(hedgehog_docs)\n",
"hedgehog_query_engine = hedgehog_index.as_query_engine(similarity_top_k=5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FF3tzeQ0zQjF"
},
"source": [
"Finally, to put everything together, we query the engine and print a response:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZEKGTDjt7Yci",
"outputId": "dd95e7a3-dd61-4d09-f5d4-b1b680a8339c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hedgehogs that require help are those that are sick, injured, and helpless, such as orphaned hoglets. These hedgehogs in need may be temporarily taken into human care and must be released into the wild as soon as they can survive there independently.\n"
]
}
],
"source": [
"response = hedgehog_query_engine.query(\"Which hedgehogs require help?\")\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7KL7zUSfd9Ue"
},
"source": [
"All steps of the LLM chain are now tracked in Langfuse.\n",
"\n",
"Example trace in Langfuse: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/367db23d-5b03-446b-bc73-36e289596c00"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w1nBZzaUe9OG"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Uz2TF32Wr6ZC"
},
"source": [
"### Step 5: (Optional) Implement user feedback to see how your application is performing\n",
"\n",
"To monitor the quality of your hedgehog chat application, you can use [Langfuse Scores](https://langfuse.com/docs/scores/overview) to store user feedback (e.g. thumps up/down or comments). These scores can then be analysed in PostHog.\n",
"\n",
"Scores are used to evaluate single observations or entire traces. You can create them via the annotation workflow in the Langfuse UI, run model-based evaluation or ingest via the SDK as we do it in this example.\n",
"\n",
"To get the context of the current observation, we use the [`observe()` decorator](https://langfuse.com/docs/sdk/python/decorators) and apply it to the hedgehog_helper() function."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "K9TbweErr5po",
"outputId": "a5733ef2-18f4-4d1d-c958-c4717da2668e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Based on the provided context, it is not recommended to keep wild hedgehogs as pets. The Federal Nature Conservation Act protects hedgehogs as a native mammal species, making it illegal to chase, catch, injure, kill, or take their nesting and refuge places. Exceptions apply only to sick, injured, and helpless hedgehogs, which may be temporarily taken into human care and released into the wild as soon as they can survive independently. It is important to respect the natural habitats and behaviors of wild animals, including hedgehogs.\n"
]
},
{
"data": {
"text/plain": [
"<langfuse.client.StatefulClient at 0x7c7cd656e2f0>"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langfuse.decorators import langfuse_context, observe\n",
"\n",
"# Langfuse observe() decorator to automatically create a trace for the top-level function and spans for any nested functions.\n",
"@observe()\n",
"def hedgehog_helper(user_message):\n",
"\n",
" response = hedgehog_query_engine.query(user_message)\n",
" trace_id = langfuse_context.get_current_trace_id()\n",
"\n",
" print(response)\n",
"\n",
" return trace_id\n",
"\n",
"trace_id = hedgehog_helper(\"Can I keep the hedgehog as a pet?\")\n",
"\n",
"# Score the trace, e.g. to add user feedback using the trace_id\n",
"langfuse.score(\n",
" trace_id = trace_id,\n",
" name=\"user-explicit-feedback\",\n",
" value=0.9,\n",
" data_type=\"NUMERIC\", # optional, inferred if not provided\n",
" comment=\"Good to know!\", # optional\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eP00ojqjzkx_"
},
"source": [
"### Step 6: See your data in PostHog\n",
"\n",
"Finally, we connect PostHog to our Langfuse account. Below is a summary of the steps to take (or see the [docs](https://posthog.com/docs/ai-engineering/langfuse-posthog) for full details):\n",
"\n",
"1. [Sign up](https://us.posthog.com/) for your free PostHog account if you haven't already\n",
"2. Copy both your project API key and host from your [project settings](https://us.posthog.com/project/settings/project-details).\n",
"3. In your [Langfuse dashboard](https://cloud.langfuse.com/), click on **Settings** and scroll down to the **Integrations** section to find the PostHog integration.\n",
"4. Click **Configure** and paste in your PostHog host and project API key (you can find these in your [PostHog project settings](https://us.posthog.com/settings/project)).\n",
"5. Click **Enabled** and then **Save**.\n",
"\n",
" Langfuse will then begin exporting your data to PostHog once a day.\n",
"\n",
"**Using the Langfuse dashboard template:**\n",
"\n",
"Once you've installed the integration, [dashboard templates](https://posthog.com/docs/ai-engineering/langfuse-posthog#using-the-langfuse-dashboard-template) help you quickly set up relevant insights.\n",
"\n",
"For our hedgehog chat application, we are using the template dashboard shown below. This enables you to analyze model cost, user feedback, and latency in PostHog.\n",
"\n",
"To create your own dashboard from a template:\n",
"\n",
"1. Go to the [dashboard](https://us.posthog.com/dashboard) tab in PostHog.\n",
"2. Click the **New dashboard** button in the top right.\n",
"3. Select **LLM metrics – Langfuse** from the list of templates."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8-4RemMiapyG"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tJQkey3IEm7O"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DIPyaNV9jhSX"
},
"source": [
"# Feedback\n",
"---\n",
"\n",
"If you have any feedback or requests, please create a GitHub [Issue](https://langfuse.com/issue) or share your idea with the community on [Discord](https://discord.langfuse.com/)."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}