The web is being read by machines now. aitools.hjlabs.in gives you three free, privacy-first tools to control and measure that: generate an llms.txt to guide AI crawlers, build an AI-focused robots.txt to allow or block GPTBot, ClaudeBot, CCBot, Google-Extended & PerplexityBot, and count GPT/Claude tokens in any text. No signup, no upload.
Each tool is a single-purpose, no-nonsense utility. Pick what you need — they all run entirely in your browser.
Publish a curated llms.txt map of your key pages. Enter your site name, description and key pages — get a clean, spec-compliant markdown file to drop at your domain root. Adoption is still low — we explain below exactly what it does and does not do today.
Decide which AI crawlers may train on or fetch your content. Toggle GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more, then copy a ready-to-publish robots.txt. The fastest way to block AI crawlers — or selectively allow them.
Paste any prompt or document and instantly see an estimated GPT and Claude token count, plus characters, words and an approximate API cost. Perfect for staying under context limits and budgeting prompts. Runs locally — your text never leaves the page.
Open Token Counter →For two decades, the contract between websites and machines was simple: a robots.txt file and an XML sitemap told search-engine crawlers what to index, and that was that. Large language models broke that contract. ChatGPT, Claude, Gemini, Perplexity and a growing fleet of AI agents now read the open web to train models, to answer questions in real time, and to cite sources inside generated answers. Your site is no longer just being ranked — it is being read, summarized, and quoted by machines. That shift created an entirely new category of decisions for every site owner, and almost no tooling exists to make those decisions easy. That is the gap aitools.hjlabs.in fills.
The llms.txt proposal is a single markdown file at your domain root giving language models a curated map of your most important content: here is what this site is, here are the key docs, the pricing page, the API reference.
Be realistic about adoption. No major AI provider has confirmed it reads llms.txt. Google's John Mueller has stated plainly that no AI system currently uses it, and an Ahrefs study of 137,000 domains found that 97% of published llms.txt files received zero requests. Anyone selling llms.txt as a ranking or citation lever is overstating the evidence.
So why generate one? It costs nothing, it is trivially reversible, and it is a reasonable hedge if adoption arrives. Treat it as optional housekeeping — not as an AI-visibility strategy. What genuinely drives AI citations today is covered in section 4. Our llms.txt generator emits a spec-compliant file in seconds if you want one.
Not every site wants its content used to train the next foundation model — and publishers in particular are drawing hard lines. The mechanism is still robots.txt, but the user-agents are new: GPTBot (OpenAI training), ClaudeBot (Anthropic), CCBot (Common Crawl, which feeds many models), Google-Extended (Gemini training), PerplexityBot, Bytespider (TikTok/ByteDance) and others. Getting the exact agent strings and directives right is fiddly, and a single typo silently fails. Our AI robots.txt generator gives you a clean toggle per crawler and emits a correct file — whether you want to block AI crawlers entirely or allow some and deny others.
Everything you send to or receive from an LLM is billed and bounded in tokens, not words. Underestimate and your prompt gets truncated; overestimate and you waste money or hit context-window errors. Our token counter gives an instant client-side estimate of GPT and Claude tokens for any text, alongside characters, words, and an approximate API cost — so you can budget a prompt or check that a document fits a model's context window before you ever call the API.
If llms.txt is not the lever, what is? The measurable shift is that being cited inside an AI answer has decoupled from ranking. The share of AI Overview citations coming from the organic top 10 fell from roughly 76% in mid-2024 to between 17% and 38% by early 2026 — so ranking #3 no longer implies being quoted. That matters commercially, because brands cited in an AI Overview see materially higher click-through than uncited brands on the same query.
What correlates with getting quoted is pedestrian and unglamorous:
FAQPage and HowTo JSON-LD make the question-answer boundary explicit. Our schema generator emits both.None of this requires a new file at your domain root. It requires writing the answer down plainly and letting it be checked.
Every tool here runs 100% in your browser. Your site structure, your robots rules, your prompts and documents are processed locally with JavaScript and never transmitted to any server. There is no account, no tracking of your inputs, and no upload. This is the same privacy-first, edge-deployed approach behind our sister tools at fmt.hjlabs.in and pixel.hjlabs.in.
hjLabs.in operates a family of web properties, and we maintain llms.txt and AI-crawler policies across all of them. These tools are the internal utilities we built for ourselves, polished and made free for everyone. The AI-era web is still being defined — getting these three files right today is one of the cheapest, highest-leverage things a site owner can do for generative engine optimization (GEO) and AI visibility.