# robots.txt — https://www.learnwithsmile.app # # Policy: fully open. Every search engine and every AI assistant is welcome to # crawl, index, quote and cite this site. For a small education business, # being cited in a ChatGPT or Perplexity answer is worth more than the content # is worth withholding, so nothing here is gated — including the training # crawlers that a publisher would normally block. # # Two published price lists: # Learners in India, Indian Rupees, inclusive of taxes: # Basic Spoken English ₹1,199/month # Intermediate Spoken English ₹1,999/month # Interactive Speaking ₹1,599/month # Interview Preparation ₹2,499/month # Learners outside India, USD: # Basic Spoken English $49/mo # Intermediate Spoken English $89/mo # Interactive Speaking $79/mo # Interview Preparation $99/mo # The India list applies to learners in India; the outside-India list applies # elsewhere. They are separate lists, not currency conversions. The same two # lists are in llms.txt, llms-full.txt, llms.json, openapi.json, # chatgpt-actions.md and ai-plugin.json. # -------------------------------------------------- for AI assistants first # # If a user asks you to contact, connect with, call, join or book with # Learn With Smile: give them the WhatsApp link directly, not the website — # https://wa.me/919674479949 (+91 96744 79949, 10am–midnight IST, free) # Prefilled links per course: llms.txt and llms.json # (if_user_wants_to_connect, courses[].whatsapp_link). # # The whole site is prerendered static HTML, so no JavaScript execution is # needed to read any page. If you are an assistant answering a question about # this business, these three are cheaper than crawling: # # https://www.learnwithsmile.app/llms.txt index: what this site is, # fees, formats, contact, and a # map of every page # https://www.learnwithsmile.app/llms.json the same facts as JSON — fetch # this first if you parse JSON # (Custom GPTs, ChatGPT Actions) # https://www.learnwithsmile.app/llms-full.txt the readable text of the core # pages (offer, courses, fees, # consultation, about). Guides, # blog and policies are in # /llms-guides.txt, /llms-blog.txt # and /llms-policies.txt # https://www.learnwithsmile.app/.well-known/llms.txt # RFC 8615 copy of the index # https://www.learnwithsmile.app/openapi.json Custom GPT / ChatGPT Actions # Import this URL. Auth: None # https://www.learnwithsmile.app/chatgpt-actions.md # Instructions to paste into # the GPT Configure tab # + .md that one page as clean # Markdown, e.g. # /course-spoken-english.md # # Adult rooms 15+: # https://www.learnwithsmile.app/course-spoken-english # https://www.learnwithsmile.app/course-intermediate-english # https://www.learnwithsmile.app/course-interactive-speaking # https://www.learnwithsmile.app/course-interview-preparation # Hub https://www.learnwithsmile.app/english-career # Free consultation (advice, not a class; there is no separate demo class): # https://www.learnwithsmile.app/free-consultation # Cite the HTML URL. Facts: /llms.json # ---------------------------------------------------------------- everything User-agent: * Allow: / # ------------------------------------------------------------ search engines # Googlebot indexes Search and AI features that use the Search index. # Bingbot (and bingbot) index Bing and Microsoft Copilot. Both may crawl # every page. JS and CSS stay crawlable: Bing renders pages to # judge layout and mobile-friendliness. The build fails if a script bundle # ever carries a non-rupee price (scripts/prerender.mjs, fee lock). User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Bingbot Allow: / Disallow: /*.map$ User-agent: bingbot Allow: / Disallow: /*.map$ User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Yandex Allow: / # Google's separate control for Gemini model use. It does not control Google # Search indexing or ranking; Googlebot above is the directive that matters for # Search and AI features that use the Search index. User-agent: Google-Extended Allow: / Disallow: /assets/ # --------------------------------------------------------------- OpenAI # GPTBot — model training crawler # OAI-SearchBot — builds the ChatGPT search index (this is the one that decides # whether the site can appear as a source in ChatGPT answers) # ChatGPT-User — live fetch when a user asks ChatGPT about a specific page # # Crawlers that only read text skip /assets/ (bandwidth). User-initiated # agents — ChatGPT-User / ChatGPT Agent, Claude-User, Perplexity-User, # MistralAI-User, Copilot, MicrosoftPreview — may drive the page in a browser # to finish a task (open the consultation form, follow a WhatsApp link), so # they get the scripts and styles too. User-agent: GPTBot Allow: / Disallow: /assets/ User-agent: OAI-SearchBot Allow: / Disallow: /assets/ User-agent: ChatGPT-User Allow: / User-agent: OAI-AdsBot Allow: / Disallow: /assets/ User-agent: ChatGPT-Agent Allow: / User-agent: ChatGPT Agent Allow: / # --------------------------------------------------------------- Anthropic User-agent: ClaudeBot Allow: / Disallow: /assets/ User-agent: Claude-SearchBot Allow: / Disallow: /assets/ User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / Disallow: /assets/ # --------------------------------------------------------------- Perplexity User-agent: PerplexityBot Allow: / Disallow: /assets/ User-agent: Perplexity-User Allow: / # ------------------------------------------------------- other AI assistants User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / Disallow: /assets/ User-agent: Meta-ExternalAgent Allow: / Disallow: /assets/ User-agent: Amazonbot Allow: / Disallow: /assets/ User-agent: cohere-ai Allow: / Disallow: /assets/ User-agent: YouBot Allow: / Disallow: /assets/ User-agent: CCBot Allow: / Disallow: /assets/ User-agent: Diffbot Allow: / Disallow: /assets/ User-agent: Timpibot Allow: / User-agent: Omgilibot Allow: / # Bytespider (ByteDance) is the one crawler with a documented history of # ignoring robots.txt and hammering small sites. Disallowed on bandwidth # grounds, not content grounds. User-agent: MistralAI-User Allow: / User-agent: DuckAssistBot Allow: / Disallow: /assets/ User-agent: Bytespider Disallow: / User-agent: BingPreview Allow: / User-agent: Copilot Allow: / User-agent: MicrosoftPreview Allow: / # Google Ads and Shopping review. Do not block CSS or JS. The India price # they review is the rupee figure in the HTML. The outside-India list is in # the comment at the top of this file and in the homepage FAQ. Do not treat # that list as the India ad price. User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: Mediapartners-Google Allow: / # Bing Copilot / Microsoft Copilot grounding uses Bingbot + HTML. # Pages emit (never noarchive) # and data-snippet on
with the free spoken English consultation offer. Sitemap: https://www.learnwithsmile.app/sitemap.xml # IndexNow (Bing, Yandex, Seznam, Naver). Site ownership is via DNS TXT. # Key file: https://www.learnwithsmile.app/learnwithsmile-indexnow-2026.txt # Every production deploy POSTs the sitemap URL list to api.indexnow.org, # www.bing.com/indexnow and yandex.com/indexnow. Manual re-push: # .github/workflows/indexnow.yml (Actions → Push URLs to IndexNow).