LUXAIRY

For shops

LuxairyBot

LuxairyBot is the web crawler of LUXAIRY, a fashion search service for AI assistants. It reads the product information your shop publishes for search engines, so that assistants can find your products and send shoppers to your own product page.

How to recognise it

User agent Mozilla/5.0 (compatible; LuxairyBot/0.1; +https://www.luxairy.si/bot)
robots.txt tokenLuxairyBot
Former nameUnigeoBot. Rules you wrote for it are still obeyed.

LuxairyBot always says who it is. It never uses a browser user agent.

What it does

  • It reads your robots.txt and sitemaps, then fetches product pages listed in the sitemap.
  • It extracts only the structured product data your shop publishes for search engines (schema.org Product and Offer: name, brand, price, currency, availability, sizes, colours and image links).
  • It does not store page HTML. It does not log in, fill in forms, add to cart, run JavaScript or collect personal data.
  • Pages are revisited about once a week. Conditional requests (If-None-Match, If-Modified-Since) avoid downloading pages that have not changed.
  • Shoppers are sent to your product page. Links may carry an affiliate code only where your shop runs an affiliate programme with a network we work with.

How it treats your site

  • At most one request at a time to your site, and at least 2 seconds between requests. A larger Crawl-delay in robots.txt is used instead.
  • On 429 Too Many Requests or 503 Service Unavailable it backs off and honours Retry-After. After repeated server errors it pauses your site for 24 hours.
  • A limited number of pages per shop and run.
  • Only ports 80 and 443, and only the hosts of the shops it crawls.

Where it runs

LuxairyBot runs in Amazon Web Services, Region eu-north-1 (Stockholm). Its fixed IP address will be listed here when crawling moves there.

How to opt out

LuxairyBot follows robots.txt (RFC 9309) and the W3C TDM Reservation Protocol.

Block the whole site:

User-agent: LuxairyBot
Disallow: /

Block only some sections:

User-agent: LuxairyBot
Disallow: /outlet/

Slow it down:

User-agent: LuxairyBot
Crawl-delay: 10

A text-and-data-mining reservation is respected too: /.well-known/tdmrep.json, the TDM-Reservation: 1 response header or <meta name="tdm-reservation" content="1">. A reserved page is not used; a reserved site is treated as a full opt-out.

robots.txt is read again at least every 24 hours. When your site blocks LuxairyBot, it stops crawling and deletes the product data it collected from your site at its next visit. Rules for User-agent: * apply too, unless you have a LuxairyBot group.

Contact

Questions, problems or opt-out and removal requests: info@luxairy.si. Please include your domain, and write from an address at that domain or tell us how to verify that you represent the site.

If you think LuxairyBot is causing load on your site, include a time window and an example request from your logs.