← Back to Cookbook
camel graph rag
Details
File: third_party/CAMEL_AI/camel_graph_rag.ipynb
Type: Jupyter Notebook
Use Cases: Graph RAG, RAG
Integrations: Camel
Content
Notebook content (JSON format):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "DJeeABnkOsPK"
},
"source": [
"# 🐫 Using CAMEL to Do Graph RAG with Mistral Models"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l4VANToobUzu"
},
"source": [
"This cookbook walks you through the process of performing graph-based Retrieval-Augmented Generation (RAG) using **CAMEL**, powered by the advanced **Mistral** models. Specifically, we'll utilize the **Mistral Large 2** model to extract and structure knowledge from a given content source, and store this information in a **Neo4j** graph database. Subsequently, we can leverage a hybrid approach, combining vector retrieval and knowledge graph retrieval, to query and explore the stored knowledge."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "slaeSyWkbIzc"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X5mWln9dCDzE"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8SV7XmyKbwdc"
},
"source": [
"## 📦 Installation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-qAA1j0Sb2Ve"
},
"source": [
"First, install the CAMEL package with all its dependencies:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"collapsed": true,
"id": "os7Mcfw2Svww",
"outputId": "5b757430-90e5-4a23-e294-dcd75d3dfb14"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: camel-ai[all] in /usr/local/lib/python3.10/dist-packages (0.1.6.0)\n",
"Requirement already satisfied: anthropic<0.30.0,>=0.29.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.29.2)\n",
"Requirement already satisfied: colorama<1,>=0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.4.6)\n",
"Requirement already satisfied: curl_cffi==0.6.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.6.2)\n",
"Requirement already satisfied: docstring-parser<0.16,>=0.15 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.15)\n",
"Requirement already satisfied: groq<0.6.0,>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.5.0)\n",
"Requirement already satisfied: ipykernel<7.0.0,>=6.0.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (6.29.5)\n",
"Requirement already satisfied: jsonschema<5,>=4 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.21.1)\n",
"Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.26.4)\n",
"Requirement already satisfied: openai<2.0.0,>=1.2.3 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.37.1)\n",
"Requirement already satisfied: pathlib<2.0.0,>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.0.1)\n",
"Requirement already satisfied: protobuf<5,>=4 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.25.4)\n",
"Requirement already satisfied: pydantic<3,>=1.9 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.6.1)\n",
"Requirement already satisfied: tiktoken<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.7.0)\n",
"Requirement already satisfied: PyMuPDF<2.0.0,>=1.22.5 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.24.9)\n",
"Requirement already satisfied: accelerate<1,>=0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.32.1)\n",
"Requirement already satisfied: beautifulsoup4<5,>=4 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.12.3)\n",
"Requirement already satisfied: cohere<5.0,>=4.56 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.57)\n",
"Requirement already satisfied: datasets<3,>=2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.20.0)\n",
"Requirement already satisfied: diffusers<1,>=0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.29.2)\n",
"Requirement already satisfied: discord.py<3.0.0,>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.4.0)\n",
"Requirement already satisfied: docker<8.0.0,>=7.1.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (7.1.0)\n",
"Requirement already satisfied: docx2txt<0.9,>=0.8 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.8)\n",
"Requirement already satisfied: duckduckgo-search<7.0.0,>=6.1.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (6.2.5)\n",
"Requirement already satisfied: google-generativeai<0.7.0,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.6.0)\n",
"Requirement already satisfied: googlemaps<5.0.0,>=4.10.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.10.0)\n",
"Requirement already satisfied: imageio<3.0.0,>=2.34.2 in /usr/local/lib/python3.10/dist-packages (from imageio[pyav]<3.0.0,>=2.34.2; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2.34.2)\n",
"Requirement already satisfied: jupyter_client<9.0.0,>=8.6.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (8.6.2)\n",
"Requirement already satisfied: litellm<2.0.0,>=1.38.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.41.1)\n",
"Requirement already satisfied: mistral-common<2.0.0,>=1.3.3 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.3.3)\n",
"Requirement already satisfied: mistralai<0.5.0,>=0.4.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.4.2)\n",
"Requirement already satisfied: neo4j<6.0.0,>=5.18.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (5.23.0)\n",
"Requirement already satisfied: newspaper3k<0.3.0,>=0.2.8 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.2.8)\n",
"Requirement already satisfied: openapi-spec-validator<0.8.0,>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.7.1)\n",
"Requirement already satisfied: opencv-python<5,>=4 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.10.0.84)\n",
"Requirement already satisfied: pillow<11.0.0,>=10.2.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (10.4.0)\n",
"Requirement already satisfied: prance<24.0.0.0,>=23.6.21.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (23.6.21.0)\n",
"Requirement already satisfied: pyTelegramBotAPI<5.0.0,>=4.18.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.21.0)\n",
"Requirement already satisfied: pydub<0.26.0,>=0.25.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.25.1)\n",
"Requirement already satisfied: pygithub<3.0.0,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.3.0)\n",
"Requirement already satisfied: pymilvus<3.0.0,>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.4.4)\n",
"Requirement already satisfied: pyowm<4.0.0,>=3.3.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (3.3.0)\n",
"Requirement already satisfied: qdrant-client<2.0.0,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.10.1)\n",
"Requirement already satisfied: rank-bm25<0.3.0,>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.2.2)\n",
"Requirement already satisfied: redis<6.0.0,>=5.0.6 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (5.0.8)\n",
"Requirement already satisfied: requests_oauthlib<2.0.0,>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.3.1)\n",
"Requirement already satisfied: sentence-transformers<4.0.0,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (3.0.1)\n",
"Requirement already satisfied: sentencepiece<1,>=0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.2.0)\n",
"Requirement already satisfied: slack-sdk<4.0.0,>=3.27.2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (3.31.0)\n",
"Requirement already satisfied: soundfile<1,>=0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (0.12.1)\n",
"Requirement already satisfied: torch<3,>=2 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (2.3.1+cu121)\n",
"Requirement already satisfied: transformers<5,>=4 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (4.42.4)\n",
"Requirement already satisfied: unstructured<0.11.0,>=0.10.30 in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.10.30)\n",
"Requirement already satisfied: wikipedia<2,>=1 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (1.4.0)\n",
"Requirement already satisfied: wolframalpha<6.0.0,>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from camel-ai[all]) (5.1.3)\n",
"Requirement already satisfied: cffi>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from curl_cffi==0.6.2->camel-ai[all]) (1.16.0)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from curl_cffi==0.6.2->camel-ai[all]) (2024.7.4)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate<1,>=0->camel-ai[all]) (24.1)\n",
"Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate<1,>=0->camel-ai[all]) (5.9.5)\n",
"Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate<1,>=0->camel-ai[all]) (6.0.1)\n",
"Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate<1,>=0->camel-ai[all]) (0.23.5)\n",
"Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate<1,>=0->camel-ai[all]) (0.4.3)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (3.7.1)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (1.7.0)\n",
"Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (0.27.0)\n",
"Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (0.5.0)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (1.3.1)\n",
"Requirement already satisfied: tokenizers>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (0.19.1)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.30.0,>=0.29.0->camel-ai[all]) (4.12.2)\n",
"Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4<5,>=4->camel-ai[all]) (2.5)\n",
"Requirement already satisfied: aiohttp<4.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (3.9.5)\n",
"Requirement already satisfied: backoff<3.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (2.2.1)\n",
"Requirement already satisfied: fastavro<2.0,>=1.8 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (1.9.5)\n",
"Requirement already satisfied: importlib_metadata<7.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (6.11.0)\n",
"Requirement already satisfied: requests<3.0.0,>=2.25.0 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (2.32.3)\n",
"Requirement already satisfied: urllib3<3,>=1.26 in /usr/local/lib/python3.10/dist-packages (from cohere<5.0,>=4.56->camel-ai[all]) (2.0.7)\n",
"Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (3.15.4)\n",
"Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (17.0.0)\n",
"Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (0.6)\n",
"Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (0.3.8)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (2.1.4)\n",
"Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (4.66.4)\n",
"Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (3.4.1)\n",
"Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets<3,>=2->camel-ai[all]) (0.70.16)\n",
"Requirement already satisfied: fsspec<=2024.5.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.5.0,>=2023.1.0->datasets<3,>=2->camel-ai[all]) (2024.5.0)\n",
"Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from diffusers<1,>=0->camel-ai[all]) (2024.5.15)\n",
"Requirement already satisfied: click>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from duckduckgo-search<7.0.0,>=6.1.0->camel-ai[all]) (8.1.7)\n",
"Requirement already satisfied: primp>=0.5.5 in /usr/local/lib/python3.10/dist-packages (from duckduckgo-search<7.0.0,>=6.1.0->camel-ai[all]) (0.5.5)\n",
"Requirement already satisfied: google-ai-generativelanguage==0.6.4 in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (0.6.4)\n",
"Requirement already satisfied: google-api-core in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (2.19.1)\n",
"Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (2.137.0)\n",
"Requirement already satisfied: google-auth>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (2.27.0)\n",
"Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/dist-packages (from google-ai-generativelanguage==0.6.4->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (1.24.0)\n",
"Requirement already satisfied: av in /usr/local/lib/python3.10/dist-packages (from imageio[pyav]<3.0.0,>=2.34.2; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (12.3.0)\n",
"Requirement already satisfied: comm>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.2.2)\n",
"Requirement already satisfied: debugpy>=1.6.5 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (1.6.6)\n",
"Requirement already satisfied: ipython>=7.23.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (7.34.0)\n",
"Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (5.7.2)\n",
"Requirement already satisfied: matplotlib-inline>=0.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.1.7)\n",
"Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (1.6.0)\n",
"Requirement already satisfied: pyzmq>=24 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (24.0.1)\n",
"Requirement already satisfied: tornado>=6.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (6.3.3)\n",
"Requirement already satisfied: traitlets>=5.4.0 in /usr/local/lib/python3.10/dist-packages (from ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (5.7.1)\n",
"Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5,>=4->camel-ai[all]) (23.2.0)\n",
"Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5,>=4->camel-ai[all]) (2023.12.1)\n",
"Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5,>=4->camel-ai[all]) (0.35.1)\n",
"Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5,>=4->camel-ai[all]) (0.19.1)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from jupyter_client<9.0.0,>=8.6.2->camel-ai[all]) (2.8.2)\n",
"Requirement already satisfied: ijson in /usr/local/lib/python3.10/dist-packages (from litellm<2.0.0,>=1.38.1->camel-ai[all]) (3.3.0)\n",
"Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /usr/local/lib/python3.10/dist-packages (from litellm<2.0.0,>=1.38.1->camel-ai[all]) (3.1.4)\n",
"Requirement already satisfied: python-dotenv>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from litellm<2.0.0,>=1.38.1->camel-ai[all]) (1.0.1)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9->camel-ai[all]) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9->camel-ai[all]) (2.16.2)\n",
"Requirement already satisfied: orjson<3.11,>=3.9.10 in /usr/local/lib/python3.10/dist-packages (from mistralai<0.5.0,>=0.4.2->camel-ai[all]) (3.10.6)\n",
"Requirement already satisfied: pytz in /usr/local/lib/python3.10/dist-packages (from neo4j<6.0.0,>=5.18.0->camel-ai[all]) (2024.1)\n",
"Requirement already satisfied: cssselect>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (1.2.0)\n",
"Requirement already satisfied: lxml>=3.6.0 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (4.9.4)\n",
"Requirement already satisfied: nltk>=3.2.1 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (3.8.1)\n",
"Requirement already satisfied: feedparser>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (6.0.11)\n",
"Requirement already satisfied: tldextract>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (5.1.2)\n",
"Requirement already satisfied: feedfinder2>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (0.0.4)\n",
"Requirement already satisfied: jieba3k>=0.35.1 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (0.35.1)\n",
"Requirement already satisfied: tinysegmenter==0.3 in /usr/local/lib/python3.10/dist-packages (from newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (0.3)\n",
"Requirement already satisfied: jsonschema-path<0.4.0,>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from openapi-spec-validator<0.8.0,>=0.7.1->camel-ai[all]) (0.3.3)\n",
"Requirement already satisfied: lazy-object-proxy<2.0.0,>=1.7.1 in /usr/local/lib/python3.10/dist-packages (from openapi-spec-validator<0.8.0,>=0.7.1->camel-ai[all]) (1.10.0)\n",
"Requirement already satisfied: openapi-schema-validator<0.7.0,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from openapi-spec-validator<0.8.0,>=0.7.1->camel-ai[all]) (0.6.2)\n",
"Requirement already satisfied: chardet>=3.0 in /usr/local/lib/python3.10/dist-packages (from prance<24.0.0.0,>=23.6.21.0->camel-ai[all]) (5.2.0)\n",
"Requirement already satisfied: ruamel.yaml>=0.17.10 in /usr/local/lib/python3.10/dist-packages (from prance<24.0.0.0,>=23.6.21.0->camel-ai[all]) (0.18.6)\n",
"Requirement already satisfied: six~=1.15 in /usr/local/lib/python3.10/dist-packages (from prance<24.0.0.0,>=23.6.21.0->camel-ai[all]) (1.16.0)\n",
"Requirement already satisfied: pynacl>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from pygithub<3.0.0,>=2.3.0->camel-ai[all]) (1.5.0)\n",
"Requirement already satisfied: pyjwt>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from pyjwt[crypto]>=2.4.0->pygithub<3.0.0,>=2.3.0->camel-ai[all]) (2.8.0)\n",
"Requirement already satisfied: Deprecated in /usr/local/lib/python3.10/dist-packages (from pygithub<3.0.0,>=2.3.0->camel-ai[all]) (1.2.14)\n",
"Requirement already satisfied: setuptools>=67 in /usr/local/lib/python3.10/dist-packages (from pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (71.0.4)\n",
"Requirement already satisfied: grpcio<=1.63.0,>=1.49.1 in /usr/local/lib/python3.10/dist-packages (from pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (1.63.0)\n",
"Requirement already satisfied: environs<=9.5.0 in /usr/local/lib/python3.10/dist-packages (from pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (9.5.0)\n",
"Requirement already satisfied: ujson>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (5.10.0)\n",
"Requirement already satisfied: milvus-lite<2.5.0,>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (2.4.8)\n",
"Requirement already satisfied: PyMuPDFb==1.24.9 in /usr/local/lib/python3.10/dist-packages (from PyMuPDF<2.0.0,>=1.22.5->camel-ai[all]) (1.24.9)\n",
"Requirement already satisfied: geojson<3,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from pyowm<4.0.0,>=3.3.0->camel-ai[all]) (2.5.0)\n",
"Requirement already satisfied: PySocks<2,>=1.7.1 in /usr/local/lib/python3.10/dist-packages (from pyowm<4.0.0,>=3.3.0->camel-ai[all]) (1.7.1)\n",
"Requirement already satisfied: grpcio-tools>=1.41.0 in /usr/local/lib/python3.10/dist-packages (from qdrant-client<2.0.0,>=1.9.0->camel-ai[all]) (1.62.2)\n",
"Requirement already satisfied: portalocker<3.0.0,>=2.7.0 in /usr/local/lib/python3.10/dist-packages (from qdrant-client<2.0.0,>=1.9.0->camel-ai[all]) (2.10.1)\n",
"Requirement already satisfied: async-timeout>=4.0.3 in /usr/local/lib/python3.10/dist-packages (from redis<6.0.0,>=5.0.6->camel-ai[all]) (4.0.3)\n",
"Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests_oauthlib<2.0.0,>=1.3.1->camel-ai[all]) (3.2.2)\n",
"Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<4.0.0,>=3.0.1->camel-ai[all]) (1.3.2)\n",
"Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<4.0.0,>=3.0.1->camel-ai[all]) (1.13.1)\n",
"Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (1.13.1)\n",
"Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (3.3)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (2.20.5)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (12.1.105)\n",
"Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch<3,>=2->camel-ai[all]) (2.3.1)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch<3,>=2->camel-ai[all]) (12.5.82)\n",
"Requirement already satisfied: filetype in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.2.0)\n",
"Requirement already satisfied: python-magic in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.4.27)\n",
"Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.9.0)\n",
"Requirement already satisfied: emoji in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2.12.1)\n",
"Requirement already satisfied: dataclasses-json in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.6.7)\n",
"Requirement already satisfied: python-iso639 in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2024.4.27)\n",
"Requirement already satisfied: langdetect in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.0.9)\n",
"Requirement already satisfied: rapidfuzz in /usr/local/lib/python3.10/dist-packages (from unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (3.9.5)\n",
"Requirement already satisfied: pdfminer.six in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (20231228)\n",
"Requirement already satisfied: python-pptx<=0.6.23 in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.6.23)\n",
"Requirement already satisfied: pdf2image in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.17.0)\n",
"Requirement already satisfied: python-docx>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.1.2)\n",
"Requirement already satisfied: unstructured.pytesseract>=0.3.12 in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.3.12)\n",
"Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (3.6)\n",
"Requirement already satisfied: msg-parser in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.2.0)\n",
"Requirement already satisfied: xlrd in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2.0.1)\n",
"Requirement already satisfied: onnx in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.16.1)\n",
"Requirement already satisfied: unstructured-inference==0.7.11 in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.7.11)\n",
"Requirement already satisfied: openpyxl in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (3.1.5)\n",
"Requirement already satisfied: pypandoc in /usr/local/lib/python3.10/dist-packages (from unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.13)\n",
"Requirement already satisfied: layoutparser[layoutmodels,tesseract] in /usr/local/lib/python3.10/dist-packages (from unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.3.4)\n",
"Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.0.9)\n",
"Requirement already satisfied: onnxruntime<1.16 in /usr/local/lib/python3.10/dist-packages (from unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.15.1)\n",
"Requirement already satisfied: xmltodict in /usr/local/lib/python3.10/dist-packages (from wolframalpha<6.0.0,>=5.0.0->camel-ai[all]) (0.13.0)\n",
"Requirement already satisfied: more-itertools in /usr/local/lib/python3.10/dist-packages (from wolframalpha<6.0.0,>=5.0.0->camel-ai[all]) (10.3.0)\n",
"Requirement already satisfied: jaraco.context in /usr/local/lib/python3.10/dist-packages (from wolframalpha<6.0.0,>=5.0.0->camel-ai[all]) (5.3.0)\n",
"Requirement already satisfied: multidict in /usr/local/lib/python3.10/dist-packages (from wolframalpha<6.0.0,>=5.0.0->camel-ai[all]) (6.0.5)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0,>=3.0->cohere<5.0,>=4.56->camel-ai[all]) (1.3.1)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0,>=3.0->cohere<5.0,>=4.56->camel-ai[all]) (1.4.1)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0,>=3.0->cohere<5.0,>=4.56->camel-ai[all]) (1.9.4)\n",
"Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic<0.30.0,>=0.29.0->camel-ai[all]) (3.7)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic<0.30.0,>=0.29.0->camel-ai[all]) (1.2.2)\n",
"Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12.0->curl_cffi==0.6.2->camel-ai[all]) (2.22)\n",
"Requirement already satisfied: marshmallow>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from environs<=9.5.0->pymilvus<3.0.0,>=2.4.0->camel-ai[all]) (3.21.3)\n",
"Requirement already satisfied: sgmllib3k in /usr/local/lib/python3.10/dist-packages (from feedparser>=5.2.1->newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (1.0.0)\n",
"Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (1.63.2)\n",
"Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (5.4.0)\n",
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (0.4.0)\n",
"Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (4.9)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->anthropic<0.30.0,>=0.29.0->camel-ai[all]) (1.0.5)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic<0.30.0,>=0.29.0->camel-ai[all]) (0.14.0)\n",
"Requirement already satisfied: h2<5,>=3 in /usr/local/lib/python3.10/dist-packages (from httpx[http2]>=0.20.0->qdrant-client<2.0.0,>=1.9.0->camel-ai[all]) (4.1.0)\n",
"Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib_metadata<7.0,>=6.0->cohere<5.0,>=4.56->camel-ai[all]) (3.19.2)\n",
"Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.19.1)\n",
"Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (4.4.2)\n",
"Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.7.5)\n",
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (3.0.47)\n",
"Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (2.16.1)\n",
"Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.2.0)\n",
"Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (4.9.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2<4.0.0,>=3.1.2->litellm<2.0.0,>=1.38.1->camel-ai[all]) (2.1.5)\n",
"Requirement already satisfied: pathable<0.5.0,>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema-path<0.4.0,>=0.3.1->openapi-spec-validator<0.8.0,>=0.7.1->camel-ai[all]) (0.4.3)\n",
"Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (4.2.2)\n",
"Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk>=3.2.1->newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (1.4.2)\n",
"Requirement already satisfied: rfc3339-validator in /usr/local/lib/python3.10/dist-packages (from openapi-schema-validator<0.7.0,>=0.6.0->openapi-spec-validator<0.8.0,>=0.7.1->camel-ai[all]) (0.1.4)\n",
"Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets<3,>=2->camel-ai[all]) (2024.1)\n",
"Requirement already satisfied: cryptography>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from pyjwt[crypto]>=2.4.0->pygithub<3.0.0,>=2.3.0->camel-ai[all]) (42.0.8)\n",
"Requirement already satisfied: XlsxWriter>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from python-pptx<=0.6.23->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (3.2.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.25.0->cohere<5.0,>=4.56->camel-ai[all]) (3.3.2)\n",
"Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib/python3.10/dist-packages (from ruamel.yaml>=0.17.10->prance<24.0.0.0,>=23.6.21.0->camel-ai[all]) (0.2.8)\n",
"Requirement already satisfied: requests-file>=1.4 in /usr/local/lib/python3.10/dist-packages (from tldextract>=2.0.1->newspaper3k<0.3.0,>=0.2.8->camel-ai[all]) (2.1.0)\n",
"Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json->unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.9.0)\n",
"Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from Deprecated->pygithub<3.0.0,>=2.3.0->camel-ai[all]) (1.14.1)\n",
"Requirement already satisfied: httplib2<1.dev0,>=0.19.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (0.22.0)\n",
"Requirement already satisfied: google-auth-httplib2<1.0.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (0.2.0)\n",
"Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (4.1.1)\n",
"Requirement already satisfied: backports.tarfile in /usr/local/lib/python3.10/dist-packages (from jaraco.context->wolframalpha<6.0.0,>=5.0.0->camel-ai[all]) (1.2.0)\n",
"Requirement already satisfied: olefile>=0.46 in /usr/local/lib/python3.10/dist-packages (from msg-parser->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.47)\n",
"Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.10/dist-packages (from openpyxl->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.1.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers<4.0.0,>=3.0.1->camel-ai[all]) (3.5.0)\n",
"Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch<3,>=2->camel-ai[all]) (1.3.0)\n",
"Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.1->google-ai-generativelanguage==0.6.4->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (1.48.2)\n",
"Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.10/dist-packages (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client<2.0.0,>=1.9.0->camel-ai[all]) (6.0.1)\n",
"Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.10/dist-packages (from h2<5,>=3->httpx[http2]>=0.20.0->qdrant-client<2.0.0,>=1.9.0->camel-ai[all]) (4.0.0)\n",
"Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/local/lib/python3.10/dist-packages (from httplib2<1.dev0,>=0.19.0->google-api-python-client->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (3.1.2)\n",
"Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.8.4)\n",
"Requirement already satisfied: coloredlogs in /usr/local/lib/python3.10/dist-packages (from onnxruntime<1.16->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (15.0.1)\n",
"Requirement already satisfied: flatbuffers in /usr/local/lib/python3.10/dist-packages (from onnxruntime<1.16->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (24.3.25)\n",
"Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.7.0)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.23.1->ipykernel<7.0.0,>=6.0.0->camel-ai[all]) (0.2.13)\n",
"Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=2.15.0->google-generativeai<0.7.0,>=0.6.0->camel-ai[all]) (0.6.0)\n",
"Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->unstructured<0.11.0,>=0.10.30->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.0.0)\n",
"Requirement already satisfied: iopath in /usr/local/lib/python3.10/dist-packages (from layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.1.10)\n",
"Requirement already satisfied: pdfplumber in /usr/local/lib/python3.10/dist-packages (from layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.11.2)\n",
"Requirement already satisfied: pytesseract in /usr/local/lib/python3.10/dist-packages (from layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.3.10)\n",
"Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.18.1+cu121)\n",
"Requirement already satisfied: effdet in /usr/local/lib/python3.10/dist-packages (from layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.4.1)\n",
"Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs->onnxruntime<1.16->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (10.0)\n",
"Requirement already satisfied: timm>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.0.8)\n",
"Requirement already satisfied: pycocotools>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2.0.8)\n",
"Requirement already satisfied: omegaconf>=2.0 in /usr/local/lib/python3.10/dist-packages (from effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (2.3.0)\n",
"Requirement already satisfied: pypdfium2>=4.18.0 in /usr/local/lib/python3.10/dist-packages (from pdfplumber->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (4.30.0)\n",
"Requirement already satisfied: antlr4-python3-runtime==4.9.* in /usr/local/lib/python3.10/dist-packages (from omegaconf>=2.0->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (4.9.3)\n",
"Requirement already satisfied: matplotlib>=2.1.0 in /usr/local/lib/python3.10/dist-packages (from pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (3.7.1)\n",
"Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.2.1)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (0.12.1)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (4.53.1)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->unstructured-inference==0.7.11->unstructured[all-docs]<0.11.0,>=0.10.30; extra == \"tools\" or extra == \"all\"->camel-ai[all]) (1.4.5)\n"
]
}
],
"source": [
"pip install camel-ai[all]==0.1.6.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X5SDX1kPb6bz"
},
"source": [
"## 🔧 Setup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "StvpXTi7b7pF"
},
"source": [
"Import the required modules from CAMEL-AI:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7PcVsBYvzfSI"
},
"outputs": [],
"source": [
"from camel.models import ModelFactory\n",
"from camel.types import ModelPlatformType, ModelType\n",
"from camel.configs import MistralConfig, OllamaConfig\n",
"from camel.loaders import UnstructuredIO\n",
"from camel.storages import Neo4jGraph\n",
"from camel.retrievers import AutoRetriever\n",
"from camel.embeddings import MistralEmbedding\n",
"from camel.types import StorageType, RoleType\n",
"from camel.agents import ChatAgent, KnowledgeGraphAgent\n",
"from camel.messages import BaseMessage"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DahHkqSOb-T3"
},
"source": [
"## 🔑 Setting Up API Keys\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "odHOpz0XcAjA"
},
"source": [
"For secure access to Mistral AI's services, we'll prompt for the API key."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "B0v7xEFFoabv",
"outputId": "e90f0ac7-86f5-40ec-c9fa-2553c59065d6"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Enter your API key: ··········\n"
]
}
],
"source": [
"import os\n",
"from getpass import getpass\n",
"\n",
"# Prompt for the API key securely\n",
"mistral_api_key = getpass('Enter your API key: ')\n",
"os.environ[\"MISTRAL_API_KEY\"] = mistral_api_key"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bPPZx6WwcDzG"
},
"source": [
"## 🗄️ Configuring Neo4j Graph Database"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "m4pJ-V2FcG6o"
},
"source": [
"Set up your Neo4j instance by providing the URL, username, and password, [here](https://neo4j.com/docs/aura/auradb/getting-started/create-database/) is the guidance, check your credentials in the downloaded .txt file. Note that you may need to wait up to 60 seconds if the instance has just been set up."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iH7NuLvq3p2o"
},
"outputs": [],
"source": [
"# Set Neo4j instance\n",
"n4j = Neo4jGraph(\n",
" url=\"Your_URI\",\n",
" username=\"Your_Username\",\n",
" password=\"Your_Password\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vmzruZl1cJQD"
},
"source": [
"## 🧠 Creating the Model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "flTRxefgcOxp"
},
"source": [
"Set up Mistral Large 2 model using the CAMEL ModelFactory:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PDhfqzA2zYmm"
},
"outputs": [],
"source": [
"# Set up model\n",
"mistral_large_2 = ModelFactory.create(\n",
" model_platform=ModelPlatformType.MISTRAL,\n",
" model_type=ModelType.MISTRAL_LARGE,\n",
" model_config_dict=MistralConfig(temperature=0.2).__dict__,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PPyHfsGyraJf"
},
"outputs": [],
"source": [
"# You can also set up model locally by using ollama\n",
"mistral_large_2_local = ModelFactory.create(\n",
" model_platform=ModelPlatformType.OLLAMA,\n",
" model_type=\"mistral-large\",\n",
" model_config_dict=OllamaConfig(temperature=0.2).__dict__,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61PwBl9XcWJ_"
},
"source": [
"## 🤖 Generate a Knowledge Graph Using CAMEL's Agent"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nFl_Paah4OdP"
},
"source": [
"Set up instances for knowledge graph agent:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LkUn3Ph0S1Z0"
},
"outputs": [],
"source": [
"# Set instance\n",
"uio = UnstructuredIO()\n",
"kg_agent = KnowledgeGraphAgent(model=mistral_large_2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7EUMC-nck0d"
},
"source": [
"Provide an example text input that the knowledge graph agent will process:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "syY548PW4d2t"
},
"outputs": [],
"source": [
"# Set example text input\n",
"text_example = \"\"\"\n",
"CAMEL has developed a knowledge graph agent can run with Mistral AI's most\n",
"advanced model, the Mistral Large 2. This knowledge graph agent is capable\n",
"of extracting entities and relationships from given content and create knowledge\n",
"graphs automaticlly.\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bKSxhv0ucof9"
},
"source": [
"Create an element from the text and use the knowledge graph agent to extract node and relationship information:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cDB_iDCl4h6F"
},
"outputs": [],
"source": [
"# Create an element from given text\n",
"element_example = uio.create_element_from_text(text=text_example)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WlmeLofo4kDE",
"outputId": "9adb40a6-1c22-4228-a7d6-63c72ae868bf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sure, let's go through the process of extracting nodes and relationships from the given content and structuring them into Node and Relationship objects.\n",
"\n",
"### Content:\n",
"\"CAMEL has developed a knowledge graph agent that can run with Mistral AI's most advanced model, the Mistral Large 2. This knowledge graph agent is capable of extracting entities and relationships from given content and creating knowledge graphs automatically.\"\n",
"\n",
"### Step-by-Step Extraction:\n",
"\n",
"1. **Identify Entities (Nodes):**\n",
" - \"CAMEL\"\n",
" - \"knowledge graph agent\"\n",
" - \"Mistral AI\"\n",
" - \"Mistral Large 2\"\n",
" - \"entities\"\n",
" - \"relationships\"\n",
" - \"content\"\n",
" - \"knowledge graphs\"\n",
"\n",
"2. **Categorize Entities:**\n",
" - \"CAMEL\": Organization\n",
" - \"knowledge graph agent\": Software\n",
" - \"Mistral AI\": Organization\n",
" - \"Mistral Large 2\": Model\n",
" - \"entities\": Concept\n",
" - \"relationships\": Concept\n",
" - \"content\": Concept\n",
" - \"knowledge graphs\": Concept\n",
"\n",
"3. **Identify Relationships:**\n",
" - \"CAMEL\" has developed \"knowledge graph agent\"\n",
" - \"knowledge graph agent\" can run with \"Mistral AI\"\n",
" - \"knowledge graph agent\" can run with \"Mistral Large 2\"\n",
" - \"knowledge graph agent\" is capable of extracting \"entities\"\n",
" - \"knowledge graph agent\" is capable of extracting \"relationships\"\n",
" - \"knowledge graph agent\" is capable of creating \"knowledge graphs\"\n",
" - \"knowledge graph agent\" extracts from \"content\"\n",
"\n",
"### Structured Output:\n",
"\n",
"#### Nodes:\n",
"```python\n",
"Node(id='CAMEL', type='Organization')\n",
"Node(id='knowledge graph agent', type='Software')\n",
"Node(id='Mistral AI', type='Organization')\n",
"Node(id='Mistral Large 2', type='Model')\n",
"Node(id='entities', type='Concept')\n",
"Node(id='relationships', type='Concept')\n",
"Node(id='content', type='Concept')\n",
"Node(id='knowledge graphs', type='Concept')\n",
"```\n",
"\n",
"#### Relationships:\n",
"```python\n",
"Relationship(subj=Node(id='CAMEL', type='Organization'), obj=Node(id='knowledge graph agent', type='Software'), type='Developed')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='Mistral AI', type='Organization'), type='RunsWith')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='Mistral Large 2', type='Model'), type='RunsWith')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='entities', type='Concept'), type='Extracts')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='relationships', type='Concept'), type='Extracts')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='knowledge graphs', type='Concept'), type='Creates')\n",
"Relationship(subj=Node(id='knowledge graph agent', type='Software'), obj=Node(id='content', type='Concept'), type='ExtractsFrom')\n",
"```\n",
"\n",
"This structured output captures the entities and their relationships as described in the content. Each node has a unique identifier and a type, and each relationship specifies the subject, object, and type of the relationship.\n"
]
}
],
"source": [
"# Let Knowledge Graph Agent extract node and relationship information\n",
"ans_element = kg_agent.run(element_example, parse_graph_elements=False)\n",
"print(ans_element)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "J0JzGaFN4pRh",
"outputId": "2ce40f15-06e1-462f-d781-5691519d5654"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GraphElement(nodes=[Node(id='CAMEL', type='Organization', properties={'source': 'agent_created'}), Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), Node(id='Mistral AI', type='Organization', properties={'source': 'agent_created'}), Node(id='Mistral Large 2', type='Model', properties={'source': 'agent_created'}), Node(id='entities', type='Concept', properties={'source': 'agent_created'}), Node(id='relationships', type='Concept', properties={'source': 'agent_created'}), Node(id='content', type='Concept', properties={'source': 'agent_created'}), Node(id='knowledge graphs', type='Concept', properties={'source': 'agent_created'})], relationships=[Relationship(subj=Node(id='CAMEL', type='Organization', properties={'source': 'agent_created'}), obj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), type='Developed', properties={'source': 'agent_created'}), Relationship(subj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), obj=Node(id='Mistral AI', type='Organization', properties={'source': 'agent_created'}), type='CanRunWith', properties={'source': 'agent_created'}), Relationship(subj=Node(id='Mistral AI', type='Organization', properties={'source': 'agent_created'}), obj=Node(id='Mistral Large 2', type='Model', properties={'source': 'agent_created'}), type='HasModel', properties={'source': 'agent_created'}), Relationship(subj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), obj=Node(id='entities', type='Concept', properties={'source': 'agent_created'}), type='CapableOfExtracting', properties={'source': 'agent_created'}), Relationship(subj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), obj=Node(id='relationships', type='Concept', properties={'source': 'agent_created'}), type='CapableOfExtracting', properties={'source': 'agent_created'}), Relationship(subj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), obj=Node(id='content', type='Concept', properties={'source': 'agent_created'}), type='ExtractsFrom', properties={'source': 'agent_created'}), Relationship(subj=Node(id='knowledge graph agent', type='Software', properties={'source': 'agent_created'}), obj=Node(id='knowledge graphs', type='Concept', properties={'source': 'agent_created'}), type='Creates', properties={'source': 'agent_created'})], source=<unstructured.documents.elements.Text object at 0x7fe47c8b8f10>)\n"
]
}
],
"source": [
"# Check graph element\n",
"graph_elements = kg_agent.run(element_example, parse_graph_elements=True)\n",
"print(graph_elements)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sqEfkLI-c23g"
},
"source": [
"Add the extracted graph elements to the Neo4j database:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "89ZF99QSAaGJ"
},
"outputs": [],
"source": [
"# Add the element to neo4j database\n",
"n4j.add_graph_elements(graph_elements=[graph_elements])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B6C8wSo6x27Z"
},
"source": [
"### 🎉 Now you can go to [here](https://workspace-preview.neo4j.io/connection/connect) to check the knowledge graph built with CAMEL's Knowledge Graph Agent and Mistral AI's Mistral Large 2 model!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LckRFGpN6Za4"
},
"source": [
"## 🗃️ Running Graph RAG with CAMEL"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vh4jrhPSyaZw"
},
"source": [
"*Next we will showcase how to run RAG in a hybrid approach, combining vector retrieval and knowledge graph retrieval, to query and explore the stored knowledge.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A_MiPGmOeDN3"
},
"source": [
"Set up a vector retriever with local storage and embedding model from Mistral AI:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AwpMPpvtr6O7"
},
"outputs": [],
"source": [
"# Set retriever\n",
"camel_retriever = AutoRetriever(\n",
" vector_storage_local_path=\"local_data/embedding_storage\",\n",
" storage_type=StorageType.QDRANT,\n",
" embedding_model=MistralEmbedding(),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BpAPOf7VeFjx"
},
"source": [
"Provide an example user query:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PTZ-8nX359bb"
},
"outputs": [],
"source": [
"# Set one user query\n",
"query=\"what's the relationship between Mistral Large 2 and Mistral AI? What kind of feature does Mistral Large 2 has?\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Yq6kGaEJeI53"
},
"source": [
"Retrieve related content using the vector retriever, here we take Mistral AI's news in the website as example conetent, you can also set the local file path here:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "hLQGgp4DJAlu",
"outputId": "9ce02b93-4bc6-443f-8a62-0ee4579b6fdb"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package punkt to /root/nltk_data...\n",
"[nltk_data] Unzipping tokenizers/punkt.zip.\n",
"[nltk_data] Downloading package averaged_perceptron_tagger to\n",
"[nltk_data] /root/nltk_data...\n",
"[nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Original Query:\n",
"{ what's the relationship between Mistral Large 2 and Mistral AI? What kind of feature does Mistral Large 2 has? }\n",
"Retrieved Context:\n",
"Mistral Large 2 is designed for single-node inference with long-context applications in mind – its size of 123 billion parameters allows it to run at large throughput on a single node.\n",
"We are releasing Mistral Large 2 under the Mistral Research License, that allows usage and modification for research and non-commercial usages. For commercial usage of Mistral Large 2 requiring self-deployment, a Mistral Commercial License must be acquired by contacting us.\n"
]
}
],
"source": [
"# Get related content by using vector retriever\n",
"vector_result = camel_retriever.run_vector_retriever(\n",
" query=query,\n",
" content_input_paths=\"https://mistral.ai/news/mistral-large-2407/\",\n",
")\n",
"\n",
"# Show the result from vector search\n",
"print(vector_result)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w188PLzteMkw"
},
"source": [
"Parse content from the specified URL and create knowledge graph data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DyuusYoo5-Vw"
},
"outputs": [],
"source": [
"# Parse conetent from mistral website and create knowledge graph data by using\n",
"# the Knowledge Graph Agent, store the information into graph database.\n",
"\n",
"elements = uio.parse_file_or_url(\n",
" input_path=\"https://mistral.ai/news/mistral-large-2407/\"\n",
")\n",
"chunk_elements = uio.chunk_elements(\n",
" chunk_type=\"chunk_by_title\", elements=elements\n",
")\n",
"\n",
"graph_elements = []\n",
"for chunk in chunk_elements:\n",
" graph_element = kg_agent.run(chunk, parse_graph_elements=True)\n",
" n4j.add_graph_elements(graph_elements=[graph_element])\n",
" graph_elements.append(graph_element)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1UgtR5UWeSD0"
},
"source": [
"Create an element from the user query:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZDYCLClxeT68"
},
"outputs": [],
"source": [
"# Create an element from user query\n",
"query_element = uio.create_element_from_text(text=query)\n",
"\n",
"# Let Knowledge Graph Agent extract node and relationship information from the qyery\n",
"ans_element = kg_agent.run(query_element, parse_graph_elements=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zD9hH7QAeVNT"
},
"source": [
"Match entities from the query in the knowledge graph storage content:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ttL5pnye66Wl",
"outputId": "d3313aaa-ac23-42bd-b81d-329a10942ca9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['Node Mistral Large 2 (label: Model) has relationship HASFEATURE with Node 128k context window (label: Feature)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node French (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node German (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Spanish (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Italian (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Portuguese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Arabic (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Hindi (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Russian (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Chinese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Japanese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Korean (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Python (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Java (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node C (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node C++ (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node JavaScript (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship SUPPORTS with Node Bash (label: CodingLanguage)', 'Node Mistral Large 2 (label: Model) has relationship DESIGNEDFOR with Node Single-node inference (label: Concept)', 'Node Mistral Large 2 (label: Model) has relationship DESIGNEDFOR with Node Long-context applications (label: Concept)', 'Node Mistral Large 2 (label: Model) has relationship HAS with Node 123 billion parameters (label: Attribute)', 'Node Mistral Large 2 (label: Model) has relationship RUNSAT with Node Large throughput (label: Attribute)', 'Node Mistral Large 2 (label: Model) has relationship RELEASEDUNDER with Node Mistral Research License (label: License)', 'Node Mistral Large 2 (label: Model) has relationship SETSFRONTIERIN with Node Performance (label: Metric)', 'Node Mistral Large 2 (label: Model) has relationship SETSFRONTIERIN with Node Cost of Serving (label: Metric)', 'Node Mistral Large 2 (label: Model) has relationship SETSFRONTIERON with Node Evaluation Metrics (label: Concept)', 'Node Mistral Large 2 (label: Model) has relationship SETSNEWPOINTON with Node Performance/Cost Pareto Front (label: Concept)', 'Node Mistral Large 2 (label: Model) has relationship BENCHMARKEDTHROUGH with Node evaluation pipeline (label: Process)', 'Node Mistral Large 2 (label: Model) has relationship HASCAPABILITY with Node instruction-following capabilities (label: Capability)', 'Node Mistral Large 2 (label: Model) has relationship HASCAPABILITY with Node conversational capabilities (label: Capability)', 'Node Mistral Large 2 (label: Model) has relationship FOLLOWS with Node precise instructions (label: InstructionType)', 'Node Mistral Large 2 (label: Model) has relationship HANDLES with Node long multi-turn conversations (label: ConversationType)', 'Node Mistral Large 2 (label: Model) has relationship PERFORMSON with Node MT-Bench (label: Benchmark)', 'Node Mistral Large 2 (label: Model) has relationship PERFORMSON with Node Wild Bench (label: Benchmark)', 'Node Mistral Large 2 (label: Model) has relationship PERFORMSON with Node Arena Hard benchmarks (label: Benchmark)', 'Node Mistral Large 2 (label: Model) has relationship PERFORMSON with Node general alignment benchmarks (label: Benchmark)', 'Node Mistral Large 2 (label: Model) has relationship TRAINEDON with Node Multilingual Data (label: Concept)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node French (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node German (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Spanish (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Italian (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Portuguese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Arabic (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Hindi (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Russian (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Chinese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Japanese (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Korean (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node English (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship EXCELSIN with Node Dutch (label: Language)', 'Node Mistral Large 2 (label: Model) has relationship PERFORMANCERESULTS with Node MMLU Benchmark (label: Benchmark)', 'Node Mistral Large 2 (label: Model) has relationship COMPAREDTO with Node Mistral Large (label: Model)', 'Node Mistral Large 2 (label: Model) has relationship COMPAREDTO with Node Llama (label: Model)', 'Node Mistral Large 2 (label: Model) has relationship EQUIPPEDWITH with Node Enhanced Function Calling (label: Skill)', 'Node Mistral Large 2 (label: Model) has relationship EQUIPPEDWITH with Node Retrieval Skills (label: Skill)', 'Node Mistral Large 2 (label: Model) has relationship UNDERGONE with Node Training (label: Process)', 'Node Mistral Large 2 (label: Model) has relationship EXECUTES with Node Parallel Function Calls (label: Skill)', 'Node Mistral Large 2 (label: Model) has relationship EXECUTES with Node Sequential Function Calls (label: Skill)', 'Node Mistral Large 2 (label: Model) has relationship SERVESAS with Node Power Engine (label: Role)', 'Node Mistral Large 2 (label: Product) has relationship EXPOSEDON with Node la Plateforme (label: Platform)', 'Node Mistral Large 2 (label: Product) has relationship ENRICHEDWITH with Node new features (label: Concept)', 'Node Mistral AI (label: Organization) has relationship HASMODEL with Node Mistral Large 2 (label: Model)', 'Node Mistral AI (label: Organization) has relationship PROVIDES with Node Mistral Large 2 (label: Model)', 'Node Mistral AI (label: Organization) has relationship PARTNERSWITH with Node Google Cloud Platform (label: CloudServiceProvider)', 'Node Mistral AI (label: Organization) has relationship PROVIDESMODELSON with Node Vertex AI (label: Service)', 'Node Mistral AI (label: Organization) has relationship PROVIDESMODELSON with Node Azure AI Studio (label: Service)', 'Node Mistral AI (label: Organization) has relationship PROVIDESMODELSON with Node Amazon Bedrock (label: Service)', 'Node Mistral AI (label: Organization) has relationship PROVIDESMODELSON with Node IBM watsonx.ai (label: Service)']\n"
]
}
],
"source": [
"# Match the enetity got from query in the knowledge graph storage content\n",
"kg_result = []\n",
"for node in ans_element.nodes:\n",
" n4j_query = f\"\"\"\n",
"MATCH (n {{id: '{node.id}'}})-[r]->(m)\n",
"RETURN 'Node ' + n.id + ' (label: ' + labels(n)[0] + ') has relationship ' + type(r) + ' with Node ' + m.id + ' (label: ' + labels(m)[0] + ')' AS Description\n",
"UNION\n",
"MATCH (n)<-[r]-(m {{id: '{node.id}'}})\n",
"RETURN 'Node ' + m.id + ' (label: ' + labels(m)[0] + ') has relationship ' + type(r) + ' with Node ' + n.id + ' (label: ' + labels(n)[0] + ')' AS Description\n",
"\"\"\"\n",
" result = n4j.query(query=n4j_query)\n",
" kg_result.extend(result)\n",
"\n",
"kg_result = [item['Description'] for item in kg_result]\n",
"\n",
"# Show the result from knowledge graph database\n",
"print(kg_result)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sjKr29w8eZ4n"
},
"source": [
"Combine results from the vector search and knowledge graph entity search:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P9ne7UoB6GY5"
},
"outputs": [],
"source": [
"# combine result from vector seach and knowledge graph entity search\n",
"comined_results = vector_result + \"\\n\".join(kg_result)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q-NaDHypebKZ"
},
"source": [
"Set up an assistant agent to answer questions based on the retrieved context:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LxEUi36F7kq1",
"outputId": "24aae0bd-2f39-47c4-8f1e-ef99637701a7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Mistral Large 2 is a model developed and provided by Mistral AI. It is designed for single-node inference with long-context applications in mind, featuring 123 billion parameters that allow it to run at large throughput on a single node.\n",
"\n",
"Key features of Mistral Large 2 include:\n",
"\n",
"1. **Long-Context Applications**: It is designed to handle long-context applications effectively.\n",
"2. **Multilingual Support**: It supports a wide range of languages including French, German, Spanish, Italian, Portuguese, Arabic, Hindi, Russian, Chinese, Japanese, Korean, and English.\n",
"3. **Coding Language Support**: It supports various coding languages such as Python, Java, C, C++, JavaScript, and Bash.\n",
"4. **Instruction-Following and Conversational Capabilities**: It has capabilities for following precise instructions and handling long multi-turn conversations.\n",
"5. **Benchmark Performance**: It performs on various benchmarks including MT-Bench, Wild Bench, Arena Hard benchmarks, and general alignment benchmarks.\n",
"6. **Enhanced Function Calling and Retrieval Skills**: It is equipped with enhanced function calling and retrieval skills, capable of executing parallel and sequential function calls.\n",
"7. **Licensing**: It is released under the Mistral Research License for non-commercial use, with a commercial license available for self-deployment.\n",
"\n",
"Mistral AI provides Mistral Large 2 and partners with various cloud service providers like Google Cloud Platform, offering the model on services such as Vertex AI, Azure AI Studio, Amazon Bedrock, and IBM watsonx.ai.\n"
]
}
],
"source": [
"# Set agent\n",
"sys_msg = BaseMessage.make_assistant_message(\n",
" role_name=\"CAMEL Agent\",\n",
" content=\"\"\"You are a helpful assistant to answer question,\n",
" I will give you the Original Query and Retrieved Context,\n",
" answer the Original Query based on the Retrieved Context.\"\"\",\n",
")\n",
"\n",
"camel_agent = ChatAgent(system_message=sys_msg,\n",
" model=mistral_large_2)\n",
"\n",
"# Pass the retrieved infomation to agent\n",
"user_prompt=f\"\"\"\n",
"The Original Query is {query}\n",
"The Retrieved Context is {comined_results}\n",
"\"\"\"\n",
"\n",
"user_msg = BaseMessage.make_user_message(\n",
" role_name=\"CAMEL User\", content=user_prompt\n",
")\n",
"\n",
"# Get response\n",
"agent_response = camel_agent.step(user_msg)\n",
"\n",
"print(agent_response.msg.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZMWitnLdlYCT"
},
"source": [
"## 🌟 Highlights"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w2OwDpJ0lg8o"
},
"source": [
"* Automated Knowledge Extraction: The Knowledge Graph Agent automates the extraction of entities and relationships, making the process efficient and effective.\n",
"\n",
"* Mistral AI Integration: This cookbook showcases the integration of Mistral AI's advanced models, particularly the Mistral Large 2, with CAMEL-AI to create a powerful knowledge graph system.\n",
"\n",
"\n",
"* Secure and Scalable: Using CAMEL-AI's robust architecture and Neo4j for graph storage ensures that the solution is both secure and scalable."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tlTJulaQlbyW"
},
"source": [
"By following this cookbook, you can leverage the cutting-edge capabilities of **CAMEL AI** and **Mistral AI** to build sophisticated knowledge graphs, facilitating advanced data analysis and retrieval tasks."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}