blogatto - v5.0.1

blogatto/config/robots

Configuration for generating the robots.txt file.

When a RobotsConfig is provided to the main Config, the build pipeline generates a robots.txt file in the output directory using the webls library.

Example

import blogatto/config/robots

let bot =
  robots.Robot(
    user_agent: "*",
    allowed_routes: ["/"],
    disallowed_routes: ["/admin/"],
  )

let cfg =
  robots.RobotsConfig(
    sitemap_url: "https://example.com/sitemap.xml",
    robots: [bot],
  )

Types

Robot

</>

A crawl policy for a specific user agent.

Use "*" as the user_agent for a catch-all policy that applies to all crawlers not matched by a more specific entry.

pub type Robot {
  Robot(
    user_agent: String,
    allowed_routes: List(String),
    disallowed_routes: List(String),
  )
}

Constructors 

  • Robot(
  user_agent: String,
  allowed_routes: List(String),
  disallowed_routes: List(String),
)

    Arguments

    user_agent

    The crawler user agent this policy applies to (e.g., "*", "googlebot").

    allowed_routes

    URL paths the crawler is allowed to access (e.g., ["/", "/blog/"]).

    disallowed_routes

    URL paths the crawler must not access (e.g., ["/admin/", "/private/"]).

RobotsConfig

</>

Configuration for robots.txt generation.

Contains the sitemap URL advertised to crawlers and a list of per-user-agent crawl policies.

pub type RobotsConfig {
  RobotsConfig(sitemap_url: String, robots: List(Robot))
}

Constructors 

  • RobotsConfig(sitemap_url: String, robots: List(Robot))

    Arguments

    sitemap_url

    The full URL of the sitemap for crawlers (e.g., "https://example.com/sitemap.xml").

    robots

    Crawl policies, one per user agent.

Values

new

</> 
pub fn new(sitemap_url: String) -> RobotsConfig

Create a RobotsConfig with a sitemap URL and no crawl policies.

robot

</> 
pub fn robot(config: RobotsConfig, robot: Robot) -> RobotsConfig

Add a crawl policy for a user agent.

✨ Search Document