STORSINT-bot
Our crawler, and how to stop it.
If you see STORSINT-bot/1.0 in your server logs, that's us. Storsint Consulting AB builds websites for companies, and our crawler occasionally visits companies' websites to see how they work: whether they work on phones, use a secure connection (HTTPS) and make their contact details easy to find.
What it does
- Pages
- The home page and at most four subpages, such as contact, about, quote and services
- Rate
- At most one request a second to a site
- Timeout
- Gives up after ten seconds without a response
- Size
- Skips pages larger than 2 MB
- robots.txt
- Always obeyed
- Content
- Public pages and their HTML only: no logins, forms, images or files
- Visits
- Usually once, and once more a week later if the site couldn't be reached
We also ask Google PageSpeed Insights to measure how fast the home page loads. That visit comes from Google (Chrome-Lighthouse), not from STORSINT-bot.
What we do with what we see
We use it to understand which companies we can help and to write a relevant email, if we get in touch. Our privacy policy explains how we handle the data.
Stopping it
Add this to your robots.txt:
User-agent: STORSINT-bot
Disallow: /Or email your domain to [email protected] and we'll take it off our lists.