@spec initialize_oorl_object(Object.object_id(), map()) :: {:ok, oorl_state()}

Initializes an OORL object with learning capabilities.

Sets up a complete OORL learning system for an object including policy networks, value functions, social learning capabilities, and meta-learning features. This is the entry point for enabling advanced learning capabilities on any AAOS object.

Parameters

• object_id - Unique identifier for the learning object
• learning_config - Configuration options map with the following keys:
  • :policy_type - Policy representation (:neural, :tabular, default: :neural)
  • :social_learning_enabled - Enable social learning (default: true)
  • :meta_learning_enabled - Enable meta-learning (default: true)
  • :curiosity_driven - Enable curiosity-driven exploration (default: true)
  • :coalition_participation - Allow coalition membership (default: true)
  • :learning_rate - Base learning rate (default: 0.01)
  • :exploration_rate - Initial exploration rate (default: 0.1)
  • :discount_factor - Future reward discount (default: 0.95)

Returns

• {:ok, oorl_state} - Successfully initialized OORL state structure

OORL State Structure

The returned state includes:

• Policy Network: Decision-making policy (neural or tabular)
• Value Function: State value estimation function
• Experience Buffer: Replay buffer for learning
• Social Learning Graph: Network of social connections
• Meta-Learning State: Strategy adaptation mechanisms
• Goal Hierarchy: Multi-objective goal structure
• Reward Function: Multi-component reward specification
• Exploration Strategy: Exploration/exploitation balance

Examples

# Initialize with neural policy
iex> {:ok, state} = OORL.initialize_oorl_object("agent_1", %{
...>   policy_type: :neural,
...>   learning_rate: 0.001,
...>   social_learning_enabled: true
...> })
iex> state.policy_network.type
:neural

# Initialize tabular policy for discrete environments
iex> {:ok, state} = OORL.initialize_oorl_object("discrete_agent", %{
...>   policy_type: :tabular,
...>   exploration_rate: 0.2
...> })
iex> state.policy_network.type
:tabular

# Initialize with meta-learning disabled
iex> {:ok, state} = OORL.initialize_oorl_object("simple_agent", %{
...>   meta_learning_enabled: false,
...>   curiosity_driven: false
...> })
iex> state.exploration_strategy.type
:epsilon_greedy

Configuration Guidelines

Policy Type Selection

• Neural: Continuous state/action spaces, complex patterns
• Tabular: Discrete spaces, interpretable policies
• Hybrid: Mixed discrete/continuous environments

Learning Rates

• High (0.1-0.5): Fast changing environments
• Medium (0.01-0.1): Typical applications
• Low (0.001-0.01): Stable environments, fine-tuning

Social Learning

• Enable for multi-agent environments
• Disable for single-agent optimization
• Consider computational overhead

Performance Impact

• Initialization time: ~5-10ms
• Memory usage: ~5-50KB depending on configuration
• Neural networks: Higher memory, better generalization
• Tabular policies: Lower memory, exact solutions

Error Conditions

Initialization may fail due to:

• Invalid configuration parameters
• Insufficient system resources
• Conflicting option combinations

@spec learning_step(
  Object.object_id(),
  any(),
  any(),
  float(),
  any(),
  social_context()
) ::
  {:ok,
   %{
     policy_update: map(),
     social_updates: map(),
     meta_updates: map(),
     total_learning_signal: float()
   }}
  | {:error, atom()}

Performs a single learning step for an OORL object.

Processes a complete learning experience including individual policy updates, social learning integration, and meta-learning adaptation. This is the core learning function that integrates multiple levels of learning in a single operation.

Parameters

• object_id - ID of the learning object (must be OORL-enabled)
• state - Current environment state (any serializable term)
• action - Action taken by the object
• reward - Numerical reward signal received
• next_state - Resulting environment state after action
• social_context - Social learning context containing:
  • :observed_actions - Actions observed from peer objects
  • :peer_rewards - Reward signals from peer objects
  • :coalition_membership - Active coalition memberships
  • :interaction_dyads - Active interaction dyads
  • :message_history - Recent communication history

Returns

• {:ok, learning_results} - Successful learning with detailed results:
  • :policy_update - Individual policy learning results
  • :social_updates - Social learning integration results
  • :meta_updates - Meta-learning adaptation results
  • :total_learning_signal - Aggregate learning signal strength
• {:error, reason} - Learning step failed due to:
  • :object_not_found - Object not registered
  • :invalid_state - State format invalid
  • :learning_disabled - OORL not enabled for object
  • :resource_exhausted - Insufficient computational resources

Learning Process

Each learning step involves:

1. Experience Creation: Package (state, action, reward, next_state)
2. Individual Learning: Update policy using RL algorithm
3. Social Learning: Integrate peer observations and rewards
4. Meta-Learning: Adapt learning strategy based on performance
5. Result Aggregation: Combine learning signals from all levels

Examples

# Basic learning step
iex> social_context = %{
...>   peer_rewards: [{"agent_2", 0.8}],
...>   interaction_dyads: ["dyad_1"]
...> }
iex> {:ok, results} = OORL.learning_step(
...>   "agent_1", 
...>   %{position: {0, 0}}, 
...>   :move_right, 
...>   1.0, 
...>   %{position: {1, 0}},
...>   social_context
...> )
iex> results.total_learning_signal
0.35

# Learning with rich social context
iex> rich_context = %{
...>   observed_actions: [
...>     %{object_id: "agent_2", action: :explore, outcome: :success},
...>     %{object_id: "agent_3", action: :exploit, outcome: :failure}
...>   ],
...>   peer_rewards: [{"agent_2", 1.2}, {"agent_3", -0.5}],
...>   coalition_membership: ["coalition_alpha"],
...>   interaction_dyads: ["dyad_2", "dyad_3"]
...> }
iex> {:ok, results} = OORL.learning_step(
...>   "social_agent", current_state, action, reward, next_state, rich_context
...> )
iex> results.social_updates.peer_influence
0.25

Learning Algorithms

The learning step uses different algorithms based on policy type:

Neural Policies

• Policy gradient with social regularization
• Experience replay with peer experiences
• Neural network parameter updates

Tabular Policies

• Q-learning with social Q-value sharing
• Direct state-action value updates
• Exploration bonus from peer actions

Social Learning Integration

Social learning enhances individual learning through:

• Imitation: Copy successful actions from high-performing peers
• Advice Taking: Weight peer rewards in policy updates
• Coordination: Align actions with coalition objectives
• Knowledge Transfer: Share learned policies across similar states

Performance Characteristics

• Learning step time: 1-10ms depending on complexity
• Memory usage: Temporary allocations for experience processing
• Convergence: 2-5x faster with effective social learning
• Scalability: Linear with number of peer objects in context

Meta-Learning Adaptation

Meta-learning continuously adapts:

• Learning rates based on convergence speed
• Exploration strategies based on environment dynamics
• Social weights based on peer performance
• Reward function components based on goal achievement