For shops
LuxairyBot
LuxairyBot is the web crawler of LUXAIRY, a fashion search service for AI assistants. It reads the product information your shop publishes for search engines, so that assistants can find your products and send shoppers to your own product page.
How to recognise it
| User agent | Mozilla/5.0 (compatible; LuxairyBot/0.1; +https://www.luxairy.si/bot) |
|---|---|
| robots.txt token | LuxairyBot |
| Former name | UnigeoBot. Rules you wrote for it are still obeyed. |
LuxairyBot always says who it is. It never uses a browser user agent.
What it does
- It reads your
robots.txtand sitemaps, then fetches product pages listed in the sitemap. - It extracts only the structured product data your shop publishes for search engines
(schema.org
ProductandOffer: name, brand, price, currency, availability, sizes, colours and image links). - It does not store page HTML. It does not log in, fill in forms, add to cart, run JavaScript or collect personal data.
- Pages are revisited about once a week. Conditional requests (
If-None-Match,If-Modified-Since) avoid downloading pages that have not changed. - Shoppers are sent to your product page. Links may carry an affiliate code only where your shop runs an affiliate programme with a network we work with.
How it treats your site
- At most one request at a time to your site, and at least 2 seconds between requests. A larger
Crawl-delayin robots.txt is used instead. - On
429 Too Many Requestsor503 Service Unavailableit backs off and honoursRetry-After. After repeated server errors it pauses your site for 24 hours. - A limited number of pages per shop and run.
- Only ports 80 and 443, and only the hosts of the shops it crawls.
Where it runs
LuxairyBot runs in Amazon Web Services, Region eu-north-1 (Stockholm). Its fixed IP address will be listed here when crawling moves there.
How to opt out
LuxairyBot follows robots.txt (RFC 9309) and the W3C TDM Reservation Protocol.
Block the whole site:
User-agent: LuxairyBot
Disallow: /
Block only some sections:
User-agent: LuxairyBot
Disallow: /outlet/
Slow it down:
User-agent: LuxairyBot
Crawl-delay: 10
A text-and-data-mining reservation is respected too: /.well-known/tdmrep.json, the
TDM-Reservation: 1 response header or <meta name="tdm-reservation"
content="1">. A reserved page is not used; a reserved site is treated as a full opt-out.
robots.txt is read again at least every 24 hours. When your site blocks LuxairyBot, it stops crawling and
deletes the product data it collected from your site at its next visit. Rules for
User-agent: * apply too, unless you have a LuxairyBot group.
Contact
Questions, problems or opt-out and removal requests: info@luxairy.si. Please include your domain, and write from an address at that domain or tell us how to verify that you represent the site.
If you think LuxairyBot is causing load on your site, include a time window and an example request from your logs.