Product AI’s fetcher
If you saw Product AI in your logs, someone asked us about a product: yours, or one that a deeper search thought your store might sell. Here is what we read and how to recognise us.
What it reads
When someone asks Product AI about a product, we read that one product page and a few related public pages, such as the product’s JSON and the store’s best sellers, to build their report. If the store uses Judge.me, Okendo, Yotpo or Loox for its reviews, that service receives the store’s and the product’s IDs so we can read recent reviews. If we can’t read the product page, we ask Judge.me whether the store is theirs. To find suppliers we also run a few searches on supplier sites such as AliExpress and CJ, and an AI model compares a few listing photos and titles with the product.
The deeper search
When a report doesn’t prove a match, a computer we run may search further for that one report. It uses an ordinary Chrome browser, which starts each search from an empty profile, to open public pages: search results, public social media posts, the shops they link to and supplier catalogues. If the picture someone gave us, or a photo on the page they gave us, shows a shop’s web address or name, it can look for that shop directly. It reads a shop it finds with plain requests too, for its public product list and data such as its best sellers and reviews. It never signs in. If a page asks for a login, a CAPTCHA or an age check, it leaves that page and the report says the site wasn’t reached. The report says so too when it couldn’t read a shop it was linked to.
What it doesn’t do
We don’t crawl stores or index catalogues, and nothing runs unless someone has asked about a product.
How to recognise it
Our fetcher’s requests are signed with Web Bot Auth (HTTP Message Signatures). Our public key is published at /.well-known/http-message-signatures-directory. Requests use a standard browser user agent, so check the Signature-Agent header and signature rather than the user agent. Pages the deeper search opens in its browser carry no signature, and look like any other visit from Chrome.
Limits
For the store someone asks about, we keep to at most 12 requests per report for its pages and data, plus at most 3 to each review service it uses, and we honour Retry-After and rate limits. Its product photos load on top of that, so we can compare them. The deeper search opens at most 45 pages for a report, plus up to 30 public social media posts read from inside one of them, and works on one report at a time. When a site answers its browser that it’s rate limiting us, the browser asks that site for nothing more on that report.
Contact
To ask about a request, or to have a report that shows your store or your photos taken down, email us at support@productai.shop.