ExUtcp

Elixir implementation of the Universal Tool Calling Protocol (UTCP).

Introduction

The Universal Tool Calling Protocol (UTCP) is a standard for defining and interacting with tools across communication protocols. UTCP emphasizes scalability, interoperability, and ease of use.

Key characteristics:

• Scalability: Handles large numbers of tools and providers without performance degradation
• Interoperability: Supports multiple provider types including HTTP, WebSockets, gRPC, and CLI tools
• Ease of Use: Built on simple, well-defined patterns

Features

• Transports: HTTP, CLI, WebSocket, gRPC, GraphQL, MCP, TCP/UDP, WebRTC
• Streaming support across all transports
• OpenAPI Converter: Automatic API discovery and tool generation
• Variable substitution via environment variables or .env files
• In-memory repository for providers and tools
• Authentication: API Key, Basic, OAuth2
• Connection pooling and lifecycle management
• Test configuration with integration test exclusion by default
• Advanced Search: Multiple algorithms with fuzzy matching and semantic search
• Monitoring and Metrics: Telemetry, PromEx, health checks, and performance monitoring
• Comprehensive test suite with 1473+ tests

Installation

Add ex_utcp to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_utcp, "~> 0.3.2"}
  ]
end

Getting Started

Basic Usage

alias ExUtcp.{Client, Config}

# Create a client configuration
config = Config.new(providers_file_path: "providers.json")

# Start a UTCP client
{:ok, client} = Client.start_link(config)

# Search for tools
{:ok, tools} = Client.search_tools(client, "", 10)

# Call a tool
{:ok, result} = Client.call_tool(client, "provider.tool_name", %{"arg" => "value"})

Programmatic Provider Registration

alias ExUtcp.{Client, Config, Providers}

# Create a client
config = Config.new()
{:ok, client} = Client.start_link(config)

# Create an HTTP provider
provider = Providers.new_http_provider([
  name: "my_api",
  url: "https://api.example.com/tools",
  http_method: "POST"
])

# Register the provider
{:ok, tools} = Client.register_tool_provider(client, provider)

# Call a discovered tool
{:ok, result} = Client.call_tool(client, "my_api.echo", %{"message" => "Hello!"})

CLI Provider Example

alias ExUtcp.{Client, Config, Providers}

# Create a client
config = Config.new()
{:ok, client} = Client.start_link(config)

# Create a CLI provider
provider = Providers.new_cli_provider([
  name: "my_script",
  command_name: "python my_script.py",
  working_dir: "/path/to/script"
])

# Register the provider
{:ok, tools} = Client.register_tool_provider(client, provider)

# Call a tool
{:ok, result} = Client.call_tool(client, "my_script.greet", %{"name" => "World"})

Configuration

Provider Configuration File

Create a providers.json file to define your providers:

{
  "providers": [
    {
      "name": "http_api",
      "type": "http",
      "http_method": "POST",
      "url": "https://api.example.com/tools",
      "content_type": "application/json",
      "headers": {
        "User-Agent": "ExUtcp/0.2.0"
      },
      "auth": {
        "type": "api_key",
        "api_key": "${API_KEY}",
        "location": "header",
        "var_name": "Authorization"
      }
    },
    {
      "name": "cli_tool",
      "type": "cli",
      "command_name": "python my_tool.py",
      "working_dir": "/opt/tools",
      "env_vars": {
        "PYTHONPATH": "/opt/tools"
      }
    }
  ]
}

Variable Substitution

UTCP supports variable substitution using ${VAR} or $VAR syntax:

# Load variables from .env file
{:ok, env_vars} = Config.load_from_env_file(".env")

config = Config.new(
  variables: env_vars,
  providers_file_path: "providers.json"
)

OpenAPI Converter

The OpenAPI Converter automatically discovers and converts OpenAPI specifications into UTCP tools.

Basic Usage

alias ExUtcp.{Client, Config}

# Create a client
{:ok, client} = Client.start_link(%{providers_file_path: nil, variables: %{}})

# Convert OpenAPI spec from URL
{:ok, tools} = Client.convert_openapi(client, "https://api.example.com/openapi.json")

# Convert OpenAPI spec from file
{:ok, tools} = Client.convert_openapi(client, "path/to/spec.yaml")

# Convert with options
{:ok, tools} = Client.convert_openapi(client, spec, %{
  prefix: "my_api",
  auth: %{type: "api_key", api_key: "Bearer ${API_KEY}"}
})

Supported Formats

• OpenAPI 2.0 (Swagger)
• OpenAPI 3.0
• JSON and YAML specifications
• URL and file-based specifications

Authentication Mapping

The converter automatically maps OpenAPI security schemes to UTCP authentication:

• API Key authentication
• HTTP Basic authentication
• HTTP Bearer authentication
• OAuth2 flows
• OpenID Connect

Architecture

The library is organized into several main components:

• ExUtcp.Client - Main client interface
• ExUtcp.Config - Configuration management
• ExUtcp.Providers - Provider implementations for different protocols
• ExUtcp.Transports - Transport layer implementations
• ExUtcp.Tools - Tool definitions and management
• ExUtcp.Repository - Tool and provider storage
• ExUtcp.OpenApiConverter - OpenAPI specification conversion

Implementation Status

Gap Analysis: UTCP Implementations Comparison

Priority Recommendations

High Priority

• [x] OpenAPI Converter: Automatic API discovery and tool generation
• [x] TCP/UDP Transport: Low-level network protocols
• [x] Advanced Search: Sophisticated search algorithms
• [x] Monitoring: Metrics and health checks
• [x] WebRTC Transport: Peer-to-peer communication

Medium Priority

• [ ] Batch Operations: Multiple tool calls

Low Priority

• [ ] Custom Variable Loaders: Beyond .env files
• [ ] API Documentation Generation

Implementation Status

Completed Features

• 8 transports: HTTP, CLI, WebSocket, gRPC, GraphQL, MCP, TCP/UDP, WebRTC
• Streaming support across all transports
• OpenAPI Converter: Automatic API discovery and tool generation
• Authentication: API Key, Basic, OAuth2
• Connection pooling and lifecycle management
• Error recovery with retry logic
• Test configuration with integration test exclusion by default
• Advanced Search: Multiple algorithms with fuzzy matching and semantic search
• Monitoring and Metrics: Telemetry, PromEx, health checks, and performance monitoring
• 497+ tests with comprehensive coverage
• Production examples for all transports

Missing Features

• Batch Operations: Multiple tool calls

Roadmap

Phase 1: Core Transports (Completed)

• [x] HTTP/HTTPS, CLI, WebSocket, gRPC, GraphQL, MCP

Phase 2: Enhanced Features (Completed)

• [x] OpenAPI Converter
• [x] TCP/UDP Transport
• [x] Advanced Search
• [x] Monitoring and Metrics

Phase 3: Extended Protocols (Completed)

• [x] WebRTC Transport

Phase 4: Enterprise Features

• [ ] Batch Operations
• [ ] Custom Variable Loaders
• [ ] API Documentation Generation

Supported Transports

Implemented

• HTTP/HTTPS: REST API integration with OpenAPI support
• CLI: Command-line tool integration
• WebSocket: Real-time communication
• gRPC: High-performance RPC calls with Protocol Buffers
• GraphQL: GraphQL API integration with HTTP/HTTPS
• MCP: Model Context Protocol integration with JSON-RPC 2.0
• TCP/UDP: Low-level network protocols with connection management
• WebRTC: Peer-to-peer communication with data channels and NAT traversal

Planned

• Additional enterprise features and optimizations

Examples

See examples/ directory:

• http_client.exs - HTTP provider
• cli_client.exs - CLI provider
• websocket_client.exs - WebSocket provider
• grpc_client.exs - gRPC provider
• graphql_example.exs - GraphQL provider
• mcp_example.exs - MCP provider
• tcp_udp_example.exs - TCP/UDP provider
• streaming_examples.exs - Streaming examples
• openapi_example.exs - OpenAPI Converter examples
• search_example.exs - Advanced search examples
• monitoring_example.exs - Monitoring and metrics examples
• webrtc_example.exs - WebRTC peer-to-peer examples

Testing

# Unit tests only (default - excludes integration tests)
mix test

# All tests including integration tests
mix test --include integration

# Integration tests only
mix test --only integration

The test suite is configured to exclude integration tests by default for faster development cycles. Integration tests require external services and are run separately.

Advanced Search

ExUtcp provides sophisticated search capabilities for discovering tools and providers:

Search Algorithms

• Exact Search: Precise matching for tool and provider names
• Fuzzy Search: Approximate matching using FuzzyCompare library for handling typos and variations
• Semantic Search: Intelligent matching using Haystack full-text search and keyword analysis
• Combined Search: Merges results from all algorithms for comprehensive discovery

Search Features

• Multi-field Search: Search across tool names, descriptions, parameters, and responses
• Advanced Filtering: Filter by provider, transport type, tags, and capabilities
• Result Ranking: Intelligent scoring based on relevance, popularity, quality, and context
• Security Scanning: TruffleHog integration for detecting sensitive data in search results
• Search Suggestions: Auto-complete and suggestion system for improved user experience
• Similar Tool Discovery: Find related tools based on semantic similarity

Basic Usage

# Start client
{:ok, client} = ExUtcp.Client.start_link()

# Search with different algorithms
exact_results = ExUtcp.Client.search_tools(client, "get_user", %{algorithm: :exact})
fuzzy_results = ExUtcp.Client.search_tools(client, "get_usr", %{algorithm: :fuzzy, threshold: 0.6})
semantic_results = ExUtcp.Client.search_tools(client, "user management", %{algorithm: :semantic})

# Advanced search with filters and security scanning
advanced_results = ExUtcp.Client.search_tools(client, "api", %{
  algorithm: :combined,
  filters: %{transports: [:http, :websocket], providers: ["my_api"]},
  security_scan: true,
  filter_sensitive: true,
  limit: 10
})

# Get search suggestions
suggestions = ExUtcp.Client.get_search_suggestions(client, "us", limit: 5)

# Find similar tools
similar_tools = ExUtcp.Client.find_similar_tools(client, "get_user", limit: 3)

Monitoring and Metrics

ExUtcp provides comprehensive monitoring capabilities for production deployments:

Monitoring Features

• Telemetry Integration: Automatic telemetry events for all UTCP operations
• Prometheus Metrics: PromEx integration for metrics collection and visualization
• Health Checks: System health monitoring for transports and components
• Performance Monitoring: Operation timing, statistical analysis, and alerting
• Custom Metrics: Support for application-specific metrics and dashboards

Telemetry Events

ExUtcp emits telemetry events for:

• Tool calls with duration, success/failure, and metadata
• Search operations with algorithm, filters, and result counts
• Provider registration and deregistration
• Connection establishment and lifecycle events
• System health and performance metrics

Basic Usage

# Start monitoring system
ExUtcp.Monitoring.start()

# Get system metrics
metrics = ExUtcp.Client.get_monitoring_metrics(client)

# Get health status
health = ExUtcp.Client.get_health_status(client)

# Get performance summary
performance = ExUtcp.Client.get_performance_summary(client)

# Record custom metrics
ExUtcp.Monitoring.Performance.record_custom_metric("api_requests", :counter, 1, %{endpoint: "/users"})

Prometheus Integration

Configure PromEx in your application:

# config/config.exs
config :ex_utcp, ExUtcp.Monitoring.PromEx,
  disabled: false,
  manual_metrics_start_delay: :no_delay,
  drop_metrics_groups: [],
  grafana: [
    host: "http://localhost:3000",
    username: "admin",
    password: "admin"
  ]

Comparison with Python UTCP

ExUtcp is an independent Elixir implementation of the UTCP specification.

Key Differences:

• ExUtcp: 8 transports (includes WebSocket, gRPC, GraphQL, TCP/UDP, WebRTC)
• Python UTCP: 6 transports (HTTP, SSE, CLI, MCP, Text, Streamable HTTP)
• ExUtcp: Advanced search, monitoring, and health checks
• Python UTCP: Plugin architecture, official implementation

Both implementations follow the UTCP specification and excel in different areas.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Run the test suite
6. Submit a pull request

License

MIT License - see LICENSE file for details.

Links

• UTCP Website
• Go Implementation
• Python Implementation
• Hex Package
• HexDocs