Product updates

Release 2.7.10

New Features:

• ClickUp Tool: Added support for managing tasks on ClickUp.

Improvements:

• Postgres custom table schema: Add custom table schema parameter for Postgres tools.

New Examples:

• LLM-OS v1: Add a streamlit application example for LLM-OS. A application that uses an LLM with advanced operating-system level capabilities.
• GeoBuddy: An AI-powered geography agent that analyzes images to predict locations based on visible cues like landmarks, architecture, and cultural symbols.

Bug Fixes:

• Text Search Compatibility: Resolved an issue where Text Search was not forward-compatible with recent versions.
• Custom Function Parsing: Fixed an issue where function parsing for custom functions did not pass all parameters to the AI.
• Claude Memory Handling: Addressed an issue in Claude where memory was not correctly handled during streaming.
• Gemini RepeatedComposite and other parsing issues: Gemini messages were incompatible with memory and storage in some cases. This has been resolved.

Tools and Fixes

New Features

• RedditTools: Added RedditTools with cookbook examples.
• TelegramTools: Introduced TelegramTools with a cookbook example.

Improvements

• GeminiEmbedder Defaults: Updated the defaults on the GeminiEmbedder to use a more up-to-date model and incorporate recommendations from their docs

Bug Fixes:

• Playground Imports: Fixed imports of docx being required for using the playground.
• AzureOpenAI Compatibility: Resolved compatibility issues with AzureOpenAIEmbedder and AzureOpenAIChat.
• SlackTool Exceptions: Fixed cases where SlackTool threw an exception when the “user” field was blank or not available.
• Response Models: Fixed cases where Response Models didn’t work with show_tool_calls, ensuring the response model is used if one is available.
• Claude Memory Blocks: Fixed an issue where memory blocks in raw Claude fromat with TextBlock, etc.
• Model Switching: Fixed cases where switching models from ChatGPT to Gemini with the same history caused issues.

Support for OpenAI o1

New Features:

• Google Calendar Tool Added

Bug Fixes:

• Fixed get_embedding_and_usage function of FastEmbedEmbedder.
• Fixed developer as a role being incorrect for OpenAILike models.
• Fixed insert function of LanceDb .

New Example

• New Marketing Workflow example added

Improvements and Fixes

This release introduces new capabilities, such as image-to-image generation, MongoDB vector support, and enhanced chunking methods, alongside significant bug fixes and usability enhancements.

New Features:

• Image-to-Image Generation Support: Extended FalTools to support image-to-image generation.
• MongoDB Integrations: Added MongoDB support across agent storage, knowledge base, agent memory, and workflow storage.
• Download Image Utility: Introduced a utility to download agent-generated images directly for enhanced workflows.

Improvements:

• Audio Transcript Handling: Refined the handling of audio transcripts to ensure accurate message output when processing audio files.
• Optimized Chunking: Enhanced fixed and recursive chunking methods to improve document processing performance.
• HuggingFace Embedder: Resolved compatibility issues with HuggingFace embedder for better reliability.
• Ollama Embedder: Upgraded the Ollama embedder for more robust and efficient embedding capabilities.
• OpenAI o1 Model Specification Support: Added support for OpenAI o1 model specifications, enabling compatibility with the latest OpenAI features.
• Spider Tool Improvements: Addressed performance issues with the Spider tool to enhance reliability.
• LanceDB Enhancements: Added support for accessing existing database tables in LanceDB.
• Github tool repo creation: Added support for repo creation via the Github tool.
• ScrapeGraph tool addition: Adds integration with ScrapeGraph AI for smart web scraping and markdown conversion capabilities.

Bug Fixes:

• OpenAI rejecting functions with too large description: This issue was discovered with Composio tools, but affected all tools. Resolved and made tool definitions better match JSON Schema.
• Google Embedder Compatibility: Fixed an issue with Pydantic 2.10.x that caused the rejection of the Google embedder, restoring full functionality.
• ChromaDB Upsert Issue: Resolved bugs that caused errors during upsert operations in ChromaDB.
• Async Error Messaging: Improved error messages for unsupported async operations in models.
• Gemini Tool Parameters: Fixed an issue with parameter handling in Gemini tools.

Extending Vector database support

This update adds new vector database integrations, improvements to DuckDuckGo tool and Gemini model and new cookbooks.

New Features:

• Cassandra Integration as a Vector Database: Introduced support for Apache Cassandra as a Vector Database, leveraging CassIO for vector storage and retrieval.
• ClickHouse as a Vector Database: Added support for ClickHouse as a Vector Database.

Improvements

• DuckDuckGo Modifier: Added a modifier parameter to the DuckDuckGo tool, allowing users to refine searches with site-specific queries, file type filters, and safe search toggles.
• Enhanced Error Handling for python-docx: Improved error messaging for scenarios where the python-docx library is not installed, providing clearer guidance and smoother debugging.

Bug Fixes

• Qdrant Default Embedder: Removed OpenAI as the default embedder for Qdrant, enabling user-configurable embeddings.
• Gemini Import Fix: Fixed an issue where missing OpenAI library dependencies caused errors when using Gemini.

Confluence Tool & Fixes

This update introduces the Confluence tool for collaboration, improves tool compatibility, and resolves deep copy issues for Ollama chat agents.

New Features

• Confluence Tool: Added a new tool using the Atlassian Confluence SDK, enabling operations such as listing pages in a space, creating and updating pages, retrieving page content by title, and getting space details.

Improvements

• Tool Compatibility: Enhanced older custom functions with manually specified descriptions and parameters to align with the updated tool-building system.

Bug Fixes

• Deep Copy for Ollama Chat Agents: Addressed an issue where manually set clients caused errors during agent model copying, ensuring all properties are properly handled.

Audio Support and Enhancements

This release enhances multimodal capabilities with audio support, improves session page performance, and fixes various bugs for better stability and usability.

New Features

• Audio Response Support: Added support for audio responses, enhancing multimodal interaction capabilities.
• Audio Generation Tools: Integrated Eleven Labs for Audio Generation.
• Cohere Embedder: Introduced a new Cohere Embedder class with a corresponding cookbook example to demonstrate its usage.
• File Agent Storage Support: Added JSON and YAML as storage options for agent session data.

Improvements

• Version Checker for OpenAI: Added a warning for users with OpenAI versions below 1.52.0 to ensure compatibility with features like audio in ChatCompletionMessage.
• Agent Response Handling: Enhanced processing of agent responses to support lists, improving handling of multi-item outputs.

Bug Fixes

• AWS Bedrock Tool Descriptions: Fixed an issue where the transfer tool description was missing, causing incompatibility with AWS Bedrock Claude.
• Response Content Handling: Resolved crashes on the session page caused by non-string response content.
• Deep Copy Agent Memory: Addressed deep copy errors when using agent memory on the playground.
• Session Page Enhancements: Fixed the refresh button.
• Fix Tool Parsing for Ollama: Fixed JSON schema tool parsing by transforming ['string', 'null'] parameters to 'string' for compatibility.
• Response Parsing for Gemini Tool: Improved response parsing to handle unserializable objects in tool_calls for Gemini on the playground.
• Memory Handling for Google Provider: Fixed an issue in monitoring_data where memory was removed for all providers, causing blank titles on Phidata.app; now only modifies memory for Google provider.
• RecursiveChunking ID Conflict: Resolved an issue in RecursiveChunking where processing large files with multiple chunks caused duplicate chunk record IDs, leading to psycopg.errors.UniqueViolation.

Multi-Modal Tools

This update introduces new multimodal tools, enhances the multimodal capabilities of existing models, and includes several quality-of-life improvements.

New Features

• Giphy Tool: Added Giphy integration to enhance creative collaboration.
• Native image upload support for Claude: Added native support for uploading images natively to the Anthropic API.
• Youtube Knowledge base: Added support for new YouTube knowledge base, allowing it to be loaded directly using YouTube video links.

Improvements

• API key validation: Added API key validation for model classes.
• Gemini audio: Improved native audio upload support for Gemini Model.
• YoutubeTools: A new Youtube tool which allows generation of timestamps
• Workspace Configuration Flexibility: Refactored type-checking logic using isinstance() to enhance flexibility and maintainability.
• Web Crawler Stability: Switched to crawl4ai async to resolve issues and improve performance.
• Memory Optimization: Improved memory usage in large-scale workflows for better efficiency.
• User Interface: Refined the UI for better tracking of team activities.

Bug Fixes

• Role-Based Access Control Bugs: Resolved issues affecting access permissions.
• Gemini Functions: Fixed errors when functions had no specified parameters.

Gemini Improvements

Improvements

• Improvements to Gemini Multimodality.

Gemini 2 Update

New Features

• Gemini Multimodal Support: Added support for multimodal (image, video, text) input processing with Gemini.
• Mem0 Integration Example: Introduced a cookbook example demonstrating Mem0 integration for Agent memory.
• CSV URL Knowledgebase: Added functionality to create and manage knowledgebases from CSV URLs.

Improvements

• Vertex AI Gemini 2 Update: The Vertex AI class has been updated to support the Gemini 2 model.

Bug Fixes

• Structured Output Fix: Resolved issues with Ollama structured output to ensure consistent data formatting.

From Text to Media: A Multi-Modal Leap

This update introduces image and video multi-modal support for the agent playground and adds image and video generation tools like FAL, replicate, and ModelLabs.

Highlights

• Support for video: Agents now natively support video.
• Multi-Modal rendering on Agent Playground: Agent Playground can now render images and videos.

New Features

• New Tools: Added Replicate, FAL, and ModelLabs tools to generate video and images.

Improvements

• Various cookbook examples were added to cover real-world agent use cases.

Enhanced Agent Visibility, RAG Improvements, and Workflow Tools

This update introduces exciting new features, performance enhancements, and crucial fixes to ensure better usability and functionality.

Highlights

• Multimodal Agents: Agents now natively support image and audio with video coming soon.
• Agentic Workflows are now generally available: Build deterministic multi-agent pipelines using Workflows.
• Agent can share state between tool calls: New session_state variable allows Agents to maintain and share state across function calls.
• Agents in teams can respond directly to the user: now responses of team members do not need to go through the team leader.
• Context Injection: Agents can be provided context that is resolved in real-time during execution.
• Human-in-the-loop: Tool calls can now be confirmed (or approved) before being executed.
• Advanced chunking: Support for Semantic and Agentic chunking (and more).

New Features

• Show Reasoning in the Playground: Provides users with insights into the agent’s thought process and decision-making, offering a behind-the-scenes look at how the agent operates within the Playground.
• Show Tool Call in the Playground: Display tool call results and metrics on hover within the Playground.
• Sessions Page Enhancements: Added context, reasoning, and tools integration to the Sessions page.
• Delete Custom Endpoint for Playground: Introduced the ability to delete endpoints directly within the Playground.
• Show References in the Playground: You can now view the sources used for RAG.
• Chunking Strategies in RAG: Introduced five levels of chunking strategies in RAG: Semantic Chunking, Fixed Chunking, Agentic Chunking, Document Chunking, and Recursive Chunking.
• Unified Reranker for Vector Databases: Implemented a unified reranking feature across various vector databases, including LanceDB, with support for CohereReranker.
• Milvus Vector Database Integration: Added support for Milvus as a vector database option.
• Support for Multi-Modalities: Added support for processing audio, video, and image data to enhance agent capabilities.
• Context Injection: Improved context handling to enhance agent responses and usability.
• Pre-hook and post-hook for function calls: enabling users to validate arguments, add human-in-the-loop flows and validate results of tool calls.

Improvements

• Duplicate Endpoint Prevention: Enhanced endpoint creation logic to prevent duplicates.
• Workflow Session Logging: Improved logging mechanisms for workflow sessions.
• Ollama Tool Call Streaming: Updated the Ollama tool to improve call streaming capabilities.
• Logging Updates for Gemini: Enhanced logging functionality for Gemini models.
• Ollama LLM Class Updates: Resolved issues in tool calling to improve usage reliability.
• Product Manager Agent Workflow Example: Added a new example to demonstrate the practical use and capabilities of workflow.
• Dynamic Prompt: System prompt, user prompt and instructions can now be passed as functions resolved during run-time.

Bug Fixes

• Session Read-All Fix: Fixed an issue where titles were not created when users provided a list of messages instead of a single message.
• Ollama Tool Response Issue: Addressed inconsistencies where Ollama tools always returned empty responses.

Breaking Changes

• Agent.add_context is now Agent.add_references as the context terminology is now used for context injection. Similarly, context_format is now references_format.

New Integrations

This update is packed with new integrations, significant improvements to existing tools, and crucial fixes to enhance functionality and user experience.

New Features

• InternLM Support: Added feature request integration for InternLM25.
• Google Vertex AI Integration: Expanded compatibility with Google Model Vertex AI.
• Discord Integration Tool: Seamlessly connect and collaborate within Discord.
• Baidu Search Tool: Introducing Baidu Search integration for enhanced search capabilities in Phidata workflows.

Improvements

• Agent Monitoring Sync: Optimized synchronization between Reasoning Agent and Main Agent
• Knowledge Cookbooks: Improved usability with enhanced S3, embedder, and document cookbooks; added an example using LanceDB as a vector database.
• License Clarifications: Updated license for better transparency and understanding.
• Default Model Adjustments: Refined the default model example for OpenAI; new guidance to avoid errors when changing models.
• Qwen2.5 Coding Agent: Performance enhancements and stability improvements.
• Installation Improvements: Added support for “psycopg[binary]” in installation commands

Bug Fixes

• Phi Tool Templates: Integrated new templates and added a comprehensive cookbook.
• Knowledge Base Cookbooks: Addressed inconsistencies for a seamless experience.
• Context Creation: Fixed an issue where empty contexts couldn’t be created without a knowledge base.
• Ollama Knowledge Example: Resolved errors for improved functionality and reliability.