import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
Introduction
Mistral AI's Large model is available on the IBM watsonx.ai platform as a fully managed solution, as well as an on-premise deployment.
Getting started
The following solutions outline the steps to query Mistral Large on the SaaS version of IBM watsonx.ai.
Pre-requisites
The following items are required:
- An IBM watsonx project (
IBM_CLOUD_PROJECT_ID) - A Service ID with an access policy enabling the use of the Watson Lachine Learning service.
To enable access to the API, you must make sure that:
- Your Service ID has been added to the project as
EDITOR, - You have generated an API key (
IBM_CLOUD_API_KEY) for your Service ID.
Querying the model (chat completion)
You can query Mistral Large using either IBM's SDK or plain HTTP calls.
:::warning
The examples below leverage the mistral-common Python package to properly format
the user messages with special tokens. It is strongly recommended to avoid passing
raw strings and handle special tokens manually: this might result in silent
tokenization errors that would highly degrade the quality of the model output.
:::
1 - `httpx` (tested with `0.27.2`)
2 - `ibm-watsonx-ai` (tested with `1.1.11`)
3 - `mistral-common` (tested with `1.4.4`)
4
5 In the following snippet, your API key will be used to generate an IAM token,
6 then the call to the model is performed using this token for authentication.
7
8 ```python
9 from ibm_watsonx_ai import Credentials
10 from ibm_watsonx_ai.foundation_models import ModelInference
11 from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
12 from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
13 from mistral_common.protocol.instruct.request import ChatCompletionRequest
14 from mistral_common.protocol.instruct.messages import UserMessage
15
16 import os
17 import httpx
18
19 IBM_CLOUD_REGIONS = {
20 "dallas": "us-south",
21 "london": "eu-gb",
22 "frankfurt": "eu-de",
23 "tokyo": "jp-tok"
24 }
25
26 IBM_CLOUD_PROJECT_ID = "xxx-xxx-xxx" # Replace with your project id
27
28
29 def get_iam_token(api_key: str) -> str:
30 """
31 Return an IAM access token generated from an API key.
32 """
33
34 headers = {"Content-Type": "application/x-www-form-urlencoded"}
35 data = f"apikey={api_key}&grant_type=urn:ibm:params:oauth:grant-type:apikey"
36 resp = httpx.post(
37 url="https://iam.cloud.ibm.com/identity/token",
38 headers=headers,
39 data=data,
40 )
41 token = resp.json().get("access_token")
42 return token
43
44
45 def format_user_message(raw_user_msg: str) -> str:
46 """
47 Return a formatted prompt using the official Mistral tokenizer.
48 """
49
50 tokenizer = MistralTokenizer.v3() # Use v3 for Mistral Large
51 tokenized = tokenizer.encode_chat_completion(
52 ChatCompletionRequest(
53 messages=[UserMessage(content=raw_user_msg)], model="mistral-large"
54 )
55 )
56 return tokenized.text
57
58
59 region = "frankfurt" # Define the region of your choice here
60 api_key = os.environ["IBM_API_KEY"]
61 access_token = get_iam_token(api_key=api_key)
62 credentials = Credentials(url=f"https://{IBM_CLOUD_REGIONS[region]}.ml.cloud.ibm.com",
63 token=access_token)
64
65 params = {GenParams.MAX_NEW_TOKENS: 256, GenParams.TEMPERATURE: 0.0}
66 model_inference = ModelInference(
67 project_id=IBM_CLOUD_PROJECT_ID,
68 model_id="mistralai/mistral-large",
69 params=params,
70 credentials=credentials,
71 )
72 user_msg_content = "Who is the best French painter? Answer in one short sentence."
73 resp = model_inference.generate_text(prompt=format_user_message(user_msg_content))
74 print(resp)
75
76 ```
77
78</TabItem>Going further
For more information and examples, you can check:
- The IBM watsonx.ai Python SDK documentation
- This IBM Developer tutorial on how to use Mistral Large with IBM watsonx.ai flows engine.