Attributes

• code (type: String.t, default: nil) -
• detailedReason (type: integer(), default: nil) - Each of the above not_crawled_reason will have a set of detailed reason defined in crawler/trawler/trawler_enums.proto.
• internalStatus (type: GoogleApi.ContentWarehouse.V1.Model.UtilStatusProto.t, default: nil) - The status returned when RPCs are used to internally fetch the image (eg. from FIFE).
• isTerminal (type: boolean(), default: nil) - Specifies if the current crawl state is terminal.
• noIndexAfterTimestamp (type: String.t, default: nil) - Time in seconds since epoch after which this image should be considered unavailable.
• notCrawledReason (type: String.t, default: nil) -
• overrodeTerminalState (type: boolean(), default: nil) - When true, it means that a non-terminal state has overwrote a terminal one.
• repid (type: String.t, default: nil) - The repid for the urls. This repid is the id given to the dupe cluster this url belongs to.
• robotedAgents (type: String.t, default: nil) - A comma separated list of user agents for which this image should be considered roboted. All images are crawled using googlebot-images and this exists here purely for informative reasons.
• url (type: String.t, default: nil) - The url at which we crawled this content. With us starting to use repids the crawl table key no longer is suggestive of the url. In addition this is used in Amarna to detect race conditions between a reference changing its crawl directive and the original crawl job finishing.
• urlDeleted (type: boolean(), default: nil) - Set to true if the url is taken down by clients. This indicates that this crawl state is used to fast remove the crawl result of the url instead of waiting for Multiverse crawl results. For more information, please refer to go/amarna-url-deletion.