About our bot

Travel The List, a guide to events worth travelling for, reads the public pages of official event organisers with a web crawler that identifies itself as TravelTheListBot. It keeps our dates, ticket windows and links correct.

What it reads

Public event pages on official sites, each site’s terms of use, and its robots.txt and /.well-known/tdmrep.json. It never logs in, never fills in forms and never tries to get past a block or a challenge page.

How often

When we first add a site, we may read its pages once a day for about a week while we test. After that, most pages once every few weeks; weekly in the two months before an event and in the month before tickets go on sale; about twice a day while tickets are on sale. Never more than one request per second to each host name, and slower if your Crawl-delay asks for it.

What we do with it

We note facts such as dates and ticket windows, write them in our own words and link to your page. We don’t republish your text or images. Automated tools, including AI models, read the text to find those facts.

How to opt out

Any of these stops us reading a page:

robots.txt: User-agent: TravelTheListBot then Disallow: /. We also respect blocks addressed to common AI crawlers.
The TDMRep reservation (tdm-reservation: 1 in tdmrep.json, a response header or a meta tag).
noai in a robots, ai or TravelTheListBot meta tag, or in the X-Robots-Tag header.

Our identity

Every request it makes carries this user agent:

Mozilla/5.0 (compatible; TravelTheListBot/1.0; +https://travelthelist.com/about/our-bot)

We run mainly on cloud servers in Frankfurt, and occasionally, for testing, from our own connection. Our IP addresses are not fixed, so robots.txt is the reliable way to control us.