Models
Groq
Example
Groq Params
| Parameter | Type | Default | Description |
|---|---|---|---|
name | str | "Groq" | Name of the Groq model |
model | str | "mixtral-8x7b-32768" | The specific Groq model to use |
frequency_penalty | float | - | Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. |
logit_bias | Any | - | Modify the likelihood of specified tokens appearing in the completion. Accepts a json object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. |
logprobs | int | - | - |
max_tokens | int | - | The maximum number of tokens to generate in the chat completion. |
presence_penalty | float | - | Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. |
response_format | Dict[str, Any] | - | An object specifying the format that the model must output. Setting to { "type": "json_object" } enables JSON mode, which guarantees the message the model generates is valid JSON. |
seed | int | - | If specified, the system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. |
stop | Union[str, List[str]] | - | Up to 4 sequences where the API will stop generating further tokens. |
temperature | float | - | What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. |
top_logprobs | int | - | - |
top_p | float | - | An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. |
user | str | - | A unique identifier representing your end-user, which can help OpenAI to monitor and detect abuse. |
extra_headers | Any | - | - |
extra_query | Any | - | - |
api_key | str | - | API key for Groq |
organization | str | - | - |
base_url | str | - | Base URL for the Groq API |
timeout | float | - | - |
max_retries | int | - | - |
default_headers | Any | - | - |
default_query | Any | - | - |
groq_client | GroqClient | - | Custom Groq client, if provided |
LLM Params
Groq is a subclass of the LLM class and has access to the same params
| Parameter | Type | Default | Description |
|---|---|---|---|
model | str | - | ID of the model to use. |
name | str | - | Name for this LLM. Note: This is not sent to the LLM API. |
metrics | Dict[str, Any] | - | Metrics collected for this LLM. Note: This is not sent to the LLM API. |
response_format | Any | - | Format of the response. |
tools | List[Union[Tool, Dict]] | - | A list of tools provided to the LLM. Tools are functions the model may generate JSON inputs for. If you provide a dict, it is not called by the model. Always add tools using the add_tool() method. |
tool_choice | Union[str, Dict[str, Any]] | - | Controls which (if any) function is called by the model. "none" means the model will not call a function and instead generates a message. "auto" means the model can pick between generating a message or calling a function. Specifying a particular function via {"type": "function", "function": {"name": "my_function"}} forces the model to call that function. "none" is the default when no functions are present. "auto" is the default if functions are present. |
run_tools | bool | True | If True, runs tools. |
show_tool_calls | bool | - | If True, shows tool calls in the response. |
functions | Dict[str, Function] | - | Functions extracted from the tools. Note: These are not sent to the LLM API and are only used for execution. |
function_call_limit | int | 20 | Maximum number of function calls allowed. |
function_call_stack | List[FunctionCall] | - | Stack of function calls. |
system_prompt | str | - | System prompt provided to the LLM. |
instructions | str | - | Instructions provided to the LLM. |