Agents: Build Custom AI Assistants

LibreChat’s AI Agents feature provides a flexible framework for creating custom AI assistants powered by various model providers.

This feature is similar to OpenAI’s Assistants API and ChatGPT’s GPTs, but with broader model support and a no-code implementation, letting you build sophisticated assistants with specialized capabilities.

Getting Started

To create a new agent, select “Agents” from the endpoint menu and open the Agent Builder panel found in the Side Panel.

The creation form includes:

• Avatar: Upload a custom avatar to personalize your agent
• Name: Choose a distinctive name for your agent
• Description: Optional details about your agent’s purpose
• Instructions: System instructions that define your agent’s behavior
• Model: Select from available providers and models

Existing agents can be selected from the top dropdown of the Side Panel.

• Also by mention with ”@” in the chat input.

Model Configuration

The model parameters interface allows fine-tuning of your agent’s responses:

• Temperature (0-1 scale for response creativity)
• Max context tokens
• Max output tokens
• Additional provider-specific settings

Agent Capabilities

Code Interpreter

When enabled, the Code Interpreter capability allows your agent to:

• Execute code in multiple languages, including:
  • Python, JavaScript, TypeScript, Go, C, C++, Java, PHP, Rust, and Fortran
• Process files securely through the LibreChat Code Interpreter API
• Run code without local setup, configuration, or sandbox deployment
• Handle file uploads and downloads seamlessly
• More info about the Code Interpreter API
  • Requires an API Subscription from code.librechat.ai

File Search

The File Search capability enables:

• RAG (Retrieval-Augmented Generation) functionality
• Semantic search across uploaded documents
• Context-aware responses based on file contents
• File attachment support at both agent and chat thread levels

Tools

Agents can also be enhanced with various built-in tools:

• DALL-E-3: Image generation from text descriptions

• Tavily Search: Advanced search API with diverse data source integration

• Calculator: Mathematical calculations

• Google Search: Access to web search functionality

• Stable Diffusion: Text-to-image generation

• Azure AI Search: Information retrieval

• Traversaal: A robust search API for LLM Agents

• Wolfram: Computational and mathematical capabilities

• More tool integrations are planned, including Model Context Provider support.

• Tools can be disabled using the librechat.yaml configuration file:

  • More info

Actions

With the Actions capability, you can dynamically create tools from OpenAPI specs to add to your Agents.

Clicking the button above will open a form where you can input the OpenAPI spec URL and create an action:

• Actions can be disabled using the librechat.yaml configuration file:
  • More info
• Individual domains can be whitelisted for agent actions:
  • More info

File Management

Agents support three distinct file upload categories:

1. Image Upload: For visual content processing
2. File Search Upload: Documents for RAG capabilities
3. Code Interpreter Upload: Files for code processing

Files can be attached directly to the agent configuration or within individual chat threads.

Sharing and Permissions

Administrator Controls

Administrators have access to global permission settings:

• Enable/disable agent sharing across all users
• Control agent usage permissions
• Manage agent creation rights
• Configure platform-wide settings

The use of agents for all users can also be disabled via config, more info.

User-Level Sharing

Individual users can:

• Share their agents with all users (if enabled)
• Control editing permissions for shared agents
• Manage access to their created agents

Notes

• Instructions, model parameters, attached files, and tools are only exposed to the user if they have editing permissions
  • An agent may leak any attached data, whether instructions or files, through conversation—make sure your instructions are robust against this
• Only original authors and administrators can delete shared agents
• Agents are private to authors unless shared

Optional Configuration

LibreChat allows admins to configure the use of agents via the librechat.yaml file:

• Disable Agents for all users (including admins): more info
• Customize agent capabilities using: more info

Best Practices

• Provide clear, specific instructions for your agent
• Carefully consider which tools are necessary for your use case
• Organize files appropriately across the three upload categories
• Review permission settings before sharing agents
• Test your agent thoroughly before deploying to other users

Recap

1. Select “Agents” from the endpoint dropdown menu
2. Open the Agent Builder panel
3. Fill out the required agent details
4. Configure desired capabilities (Code Interpreter, File Search)
5. Add necessary tools and files
6. Set sharing permissions if desired
7. Create and start using your agent

What’s next?

LibreChat Agents usher in a new era for the app where future pipelines can be streamlined via Agents for specific tasks and workflows across your experience in LibreChat.

Future updates will include:

• General improvements to the current Agent experience
• Multi-agent orchestration for complex workflows
• Ability to customize agents for various functions: titling (chat thread naming), memory management (user context/history), and prompt enhancement (input assistance/predictions)
• More tools, tool parameters, and Model Context Provider support for tools and file resources.

Furthermore, the update introduces a new paradigm for LibreChat, as its underlying architecture provides a much needed refresh for the app, optimizing both the user experience and overall app performance.

To highlight one notable optimization, an AI generation of roughly 1000 tokens will transfer about 1 MB of data using traditional endpoints (at the time of writing, any endpoint option besides Agents and AWS Bedrock).

Using an agent, the same generation will transfer about about 52 kb of data, a 95% reduction in data transfer, which is that much less of a load on the server and the user’s device.

AI Agents in LibreChat provide a powerful way to create specialized assistants without coding knowledge while maintaining the flexibility to work with your preferred AI models and providers.

#LibreChat #AIAssistants #NoCode #OpenSource