Next.js Sitemap and Robots.txt: Complete SEO Configuration Guide — SEO INSIGHTS
Next.js Sitemap and Robots.txt: Complete SEO Configuration Guide
Learn how to configure a Next.js sitemap and robots.txt for SEO, including sitemap.ts, robots.ts, crawling rules, validation, and Google Search Console.
A Next.js website can be fast, modern, and technically impressive, but that does not automatically mean search engines will discover every important page.
Imagine you have just launched a Next.js website.
Your homepage works perfectly. Your blog posts load quickly. Dynamic routes generate pages from a database. Categories work. Your metadata looks good.
Then you search for your newly published article.
Nothing.
You check another article.
Still nothing.
At this point, many beginners start changing titles, adding keywords, or installing SEO tools.
But there is a more fundamental question:
Can search engines efficiently discover and crawl the URLs that matter?
This is where a sitemap and robots.txt become important.
Next.js provides dedicated metadata-file conventions for generating both. In the App Router, you can create app/sitemap.ts for a programmatically generated sitemap and app/robots.ts for crawler rules. Next.js serves the resulting files at /sitemap.xml and /robots.txt. ( Next.js)
Let’s look at how to configure them properly.
What Are Sitemap.xml and Robots.txt?
Before writing code, understand the different jobs these two files perform.
What Is sitemap.xml?
An XML sitemap is a file that lists URLs you want search engines to discover and crawl.
For example:
https://example.com/blog/nextjs-seo
A sitemap can also provide information such as the last significant modification date of a URL.
Google recommends using fully qualified, absolute URLs and including the URLs you want to appear in search results, generally corresponding to your preferred canonical URLs. ( Google for Developers)
A sitemap is therefore best thought of as a discovery and URL-information mechanism.
It does not guarantee that a page will be indexed.
Google describes sitemap submission as a hint rather than a guarantee that URLs will be crawled or indexed. ( Google for Developers)
What Is robots.txt?
The robots.txt file provides instructions to crawlers about which URLs they may access.
A basic example is:
User-Agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
You can also disallow particular paths:
User-Agent: *
Allow: /
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
Google explains that robots.txt is primarily a mechanism for managing crawler access and crawl traffic. It is not a reliable way to prevent a page from appearing in Google Search. If you need a page excluded from search results, use an appropriate method such as noindex or authentication. ( Google for Developers)
That distinction is one of the most important concepts in technical SEO.
Sitemap vs Robots.txt: What’s the Difference?
Think about it this way:
Sitemap: “Here are the important URLs on my website.”
Robots.txt: “Here are the parts crawlers are allowed or not allowed to access.”
They complement each other rather than replacing each other.
Why Sitemap and Robots.txt Matter for Next.js SEO
Next.js gives developers considerable flexibility.
A site might contain:
- Static pages
- Dynamic blog posts
- Product pages
- Category pages
- Documentation
- User-generated content
- API routes
- Private dashboards
Search engines do not need every URL in the application.
Your sitemap should focus on the URLs that are useful and eligible for search.
Your robots rules should prevent unnecessary crawler access where appropriate.
This becomes particularly important when your Next.js application uses dynamic routes.
How to Create a Next.js Sitemap
Modern Next.js App Router projects can use the special sitemap.ts metadata file.
Create:
app/
├── layout.tsx
├── page.tsx
├── sitemap.ts
└── robots.ts
Next.js recognizes sitemap.ts as a special metadata file and generates the XML sitemap. ( Next.js)
Step 1: Create sitemap.ts
Add the following file:
// app/sitemap.ts
import type { MetadataRoute } from ‘next’
export default function sitemap(): MetadataRoute.Sitemap {
return [
{
url: ‘https://example.com',
lastModified: new Date(),
},
{
url: ‘https://example.com/about',
lastModified: new Date(),
},
{
url: ‘https://example.com/blog',
lastModified: new Date(),
},
]
}
After deployment, Next.js generates:
https://example.com/sitemap.xml
The MetadataRoute.Sitemap type provides a structured way to define sitemap entries. ( Next.js)
Creating a Dynamic Next.js Sitemap
A static sitemap is fine for a small website.
But suppose your website has 2,000 blog posts stored in a database.
You don’t want to manually maintain thousands of URLs.
Instead, retrieve your content and generate the sitemap dynamically.
For example:
// app/sitemap.ts
import type { MetadataRoute } from ‘next’
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const posts = await getPosts()
return posts.map((post) => ({
url: `https://example.com/blog/${post.slug}`,
lastModified: post.updatedAt,
}))
}
Here, getPosts() represents your own database or content-fetching function.
The important idea is simple:
Your application’s source of truth generates the sitemap.
That reduces the chance of forgetting newly published URLs.
Use Accurate lastModified Values
One of the most useful sitemap fields is lastModified.
For example:
{
url: ‘https://example.com/blog/nextjs-seo',
lastModified: post.updatedAt,
}
Avoid using the current date for every URL simply because the sitemap was generated today.
If an article was genuinely updated six months ago, don’t make it appear as though it was updated today.
Google says it can use lastmod when the value is consistently accurate and verifiable. It should represent the last significant modification to the page. ( Google for Developers)
Significant changes can include updates to:
A minor change to a copyright year, for example, shouldn’t normally be treated as a significant content update. ( Google for Developers)
Should You Use changeFrequency and priority?
Next.js supports sitemap fields such as:
{
url: ‘https://example.com/blog',
lastModified: new Date(),
changeFrequency: ‘weekly’,
priority: 0.8,
}
However, don’t assume these values will improve Google rankings.
Google explicitly states that it ignores the priority and changefreq values in XML sitemaps. Google can use an accurate lastmod value as a crawl scheduling signal. ( Google for Developers)
Therefore, if you’re building a modern Next.js sitemap, concentrate on providing:
- Correct URLs
- Canonical URLs
- Accurate lastModified values where available
rather than trying to manipulate crawl behavior through arbitrary priority numbers.
Adding Database Content to a Sitemap
A common Next.js application might have routes such as:
/blog/[slug]
/product/[slug]
/category/[slug]
The sitemap can be generated from your database.
For example:
import type { MetadataRoute } from ‘next’
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const posts = await db.post.findMany({
select: {
slug: true,
updatedAt: true,
},
})
return posts.map((post) => ({
url: `https://example.com/blog/${post.slug}`,
lastModified: post.updatedAt,
}))
}
The exact database query depends on whether you use PostgreSQL, MySQL, MongoDB, an ORM, or another data source.
The SEO principle remains the same:
Generate sitemap URLs from the same reliable data source that generates your public pages.
How to Create Next.js robots.txt
The App Router also supports a special robots.ts file.
Create:
app/robots.ts
Then add:
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: ‘*’,
allow: ‘/’,
},
sitemap: ‘https://example.com/sitemap.xml',
}
}
Next.js will generate the corresponding:
https://example.com/robots.txt
The current Next.js documentation identifies robots.ts as a special metadata route and supports crawler rules, sitemap URLs, and multiple user-agent configurations. ( Next.js)
How to Block a Private Route
Suppose your application contains:
/dashboard
/admin
/private
You may not want general crawlers accessing those areas.
You could configure:
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: ‘*’,
allow: ‘/’,
disallow: [‘/admin/’, ‘/private/’],
},
sitemap: ‘https://example.com/sitemap.xml',
}
}
The generated file will contain corresponding Disallow rules.
However, remember that robots rules are about crawler access, not authentication.
Never rely on robots.txt to protect confidential information.
If information is genuinely private, protect it with authentication or appropriate server-side access controls.
Allowing and Blocking Specific Crawlers
Next.js also supports separate rules for different user agents.
For example:
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: ‘Googlebot’,
allow: ‘/’,
disallow: [‘/private/’],
},
{
userAgent: [‘Applebot’, ‘Bingbot’],
allow: ‘/’,
},
],
sitemap: ‘https://example.com/sitemap.xml',
}
}
Only use specialized crawler rules when you have a genuine reason.
For most beginner websites, a simple:
User-Agent: *
Allow: /
is easier to maintain.
Connecting robots.txt to Your Sitemap
One of the simplest configurations is:
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: ‘*’,
allow: ‘/’,
},
sitemap: ‘https://example.com/sitemap.xml',
}
}
This produces a Sitemap directive in the robots file.
Google supports sitemap declarations in robots.txt, and the sitemap URL should be an absolute URL. ( Google for Developers)
For example:
User-Agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
A Practical Next.js SEO Configuration
For a typical public website, you could start with:
app/sitemap.ts
import type { MetadataRoute } from ‘next’
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const posts = await getPosts()
return [
{
url: ‘https://example.com',
lastModified: new Date(),
},
{
url: ‘https://example.com/about',
lastModified: new Date(),
},
…posts.map((post) => ({
url: `https://example.com/blog/${post.slug}`,
lastModified: post.updatedAt,
})),
]
}
app/robots.ts
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: ‘*’,
allow: ‘/’,
disallow: [‘/admin/’, ‘/private/’],
},
sitemap: ‘https://example.com/sitemap.xml',
}
}
Get SEO Insights’s stories in your inbox
Join Medium for free to get updates from this writer.
This provides a straightforward foundation for a content-driven Next.js site.
Which URLs Should Go Into a Next.js Sitemap?
This is where many developers make mistakes.
Don’t automatically put every route generated by your application into the sitemap.
Focus on URLs that:
- Are publicly accessible
- Provide useful content
- You want search engines to discover
- Are canonical versions
- Are eligible to appear in search results
For example, a blog might include:
/
/about
/blog
/blog/nextjs-seo
/blog/technical-seo
/contact
But it might exclude:
/admin
/login
/account
/private
Google recommends using canonical URLs in sitemaps rather than listing multiple URLs that represent the same content. ( Google for Developers)
Don’t Put Noindex Pages in Your Sitemap
Suppose you have a page with:
<meta name=”robots” content=”noindex”>
You generally don’t want to simultaneously tell search engines through your sitemap:
“This is one of the URLs I want you to consider for search.”
Keep your sitemap focused on URLs that you actually want discovered and considered for indexing.
Also remember that noindex must be accessible to crawlers for Google to see the directive. If a page is blocked in robots.txt, Google cannot retrieve the page and therefore cannot see its noindex instruction. ( Google for Developers)
Sitemap and Canonical URLs
Consider this example:
https://example.com/product/shoes
https://example.com/product/shoes?color=red
https://example.com/product/shoes?color=blue
If the first URL is the canonical page and the query parameters create variations that shouldn’t be independently indexed, your sitemap should generally focus on the canonical URL.
Google recommends using the sitemap to indicate the canonical URLs you prefer to appear in search. ( Google for Developers)
A sitemap should therefore not become a dumping ground for every URL your application can generate.
How Large Can a Sitemap Be?
Google’s sitemap limits allow a single sitemap to contain up to:
If your sitemap exceeds those limits, split it into multiple sitemap files or use a sitemap index. ( Google for Developers)
For a large Next.js application, this matters.
Imagine an e-commerce website with hundreds of thousands of products.
A single:
app/sitemap.ts
may not be enough.
Next.js supports multiple sitemaps and the generateSitemaps function for larger applications. ( Next.js)
Creating Multiple Next.js Sitemaps
For large applications, Next.js provides generateSitemaps().
A simplified example is:
import type { MetadataRoute } from ‘next’
export async function generateSitemaps() {
return [{ id: 0 }, { id: 1 }]
}
export default async function sitemap({
id,
}: {
id: Promise<string>
}): Promise<MetadataRoute.Sitemap> {
const sitemapId = await id
const products = await getProductsForSitemap(Number(sitemapId))
return products.map((product) => ({
url: `https://example.com/products/${product.slug}`,
lastModified: product.updatedAt,
}))
}
The exact implementation should be adapted to your database and URL architecture.
Next.js documents the use of multiple sitemap files for large applications and notes Google’s 50,000-URL limit per sitemap. ( Next.js)
How to Test Your Next.js Sitemap
After deploying your application, open:
https://example.com/sitemap.xml
Check that:
- The URL loads successfully.
- The response is XML.
- URLs are absolute.
- URLs use HTTPS if your site uses HTTPS.
- Important pages are present.
- Unwanted private pages are absent.
- URLs don’t contain accidental staging domains.
- URLs don’t point to deleted pages.
- lastmod values are sensible.
A particularly important check is the domain.
If you moved from:
to:
make sure your production sitemap does not continue generating staging URLs.
How to Test robots.txt
Open:
https://example.com/robots.txt
You might expect to see:
User-Agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Check every Disallow rule carefully.
A single overly broad rule can prevent crawlers from accessing URLs you actually want discovered.
For example:
Disallow: /
blocks the entire site from crawling under that rule.
For a normal public website, that would usually be an obvious configuration mistake.
Submit the Sitemap to Google Search Console
After deploying the site, add your sitemap through Google Search Console.
The typical sitemap URL is:
https://example.com/sitemap.xml
Google’s Search Console sitemap report can show whether Google has accessed the sitemap and whether processing errors were detected. ( Google for Developers)
Remember that submitting a sitemap is not an indexing guarantee.
It tells Google about URLs; it doesn’t force those URLs into search results.
Sitemap and robots.txt SEO Checklist
Before launching your Next.js website, run through this checklist.
Sitemap checklist
- app/sitemap.ts exists
- Sitemap returns valid XML
- Production domain is used
- URLs are absolute
- HTTPS URLs are used where appropriate
- Canonical URLs are included
- Important public pages are included
- Private/admin URLs are excluded
- lastModified values are accurate
- Duplicate URLs are avoided
- Large sitemaps are split when necessary
Robots.txt checklist
- app/robots.ts exists
- /robots.txt loads
- Public content is crawlable
- Private areas are handled appropriately
- Important CSS/JavaScript resources aren’t unnecessarily blocked
- Sitemap URL is included
- Production domain is used
- No accidental Disallow: / exists
Search Console checklist
- Site property is verified
- Sitemap submitted
- Sitemap processing checked
- Important URLs inspected
- Indexing issues monitored
- Canonicalization reviewed where necessary
Common Next.js Sitemap and Robots.txt Mistakes
Mistake 1: Using an Old Sitemap Approach Without Understanding Your Router
Next.js has evolved.
For an App Router project, the current metadata-file convention supports:
app/sitemap.ts
app/robots.ts
Next.js automatically handles the corresponding metadata routes. ( Next.js)
Older tutorials may demonstrate custom route handlers or Pages Router techniques. Those approaches aren’t automatically wrong, but you should first check whether the current metadata-file convention is simpler for your project.
Mistake 2: Blocking the Entire Website
A development configuration might contain:
Disallow: /
This can be useful in some private or testing environments.
But accidentally deploying it to production can prevent crawlers from accessing your public content.
Always inspect the production /robots.txt.
Mistake 3: Thinking robots.txt Means “Do Not Index”
This is perhaps the most common misunderstanding.
robots.txt controls crawling.
It is not a reliable indexing-removal mechanism.
If a URL must not appear in Google Search, use an appropriate noindex directive or another supported method, and make sure the crawler can access the page to see the noindex instruction. ( Google for Developers)
Mistake 4: Adding Every URL to the Sitemap
More URLs are not automatically better.
If your application generates thousands of parameter combinations, temporary pages, private routes, or duplicate URLs, don’t automatically include them all.
A sitemap should represent the important URLs you want search engines to discover.
Mistake 5: Setting Every lastModified Date to Today
This creates misleading information.
If you set:
lastModified: new Date()
for every page on every request, every URL can appear to have changed whenever the sitemap is generated.
Use actual content modification data when you have it.
Google specifically recommends that lastmod accurately represent significant changes. ( Google for Developers)
Sitemap and Robots.txt for a Next.js Blog
For a typical blog, the architecture could look like this:
app/
├── page.tsx
├── about/
│ └── page.tsx
├── blog/
│ ├── page.tsx
│ └── [slug]/
│ └── page.tsx
├── sitemap.ts
└── robots.ts
The sitemap might include:
/
/about
/blog
/blog/nextjs-seo
/blog/technical-seo
/blog/website-performance
The robots file might be:
User-Agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
Whether /api/ should be disallowed depends on your application’s architecture and what those endpoints expose. Don’t copy the rule blindly.
The correct configuration is the one that matches your actual site.
Do Sitemap and Robots.txt Improve Rankings Directly?
It is better to think of these files as technical SEO infrastructure rather than ranking shortcuts.
A sitemap helps search engines discover and understand the URLs you consider important.
robots.txt helps manage crawler access.
Neither file can compensate for:
- Poor content
- Weak search intent alignment
- Technical accessibility problems
- Poor internal linking
- Duplicate content
- Poor page experience
- Lack of useful information
Google also makes clear that submitting a sitemap does not guarantee crawling or indexing. ( Google for Developers)
The goal is therefore not to “trick” a search engine.
The goal is to make your website’s architecture easier to understand and crawl.
Final Next.js SEO Configuration Example
For a straightforward production site, you can use this as a starting point.
app/sitemap.ts
import type { MetadataRoute } from ‘next’
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const posts = await getPosts()
return [
{
url: ‘https://example.com',
lastModified: new Date(),
},
{
url: ‘https://example.com/about',
lastModified: new Date(),
},
{
url: ‘https://example.com/blog',
lastModified: new Date(),
},
…posts.map((post) => ({
url: `https://example.com/blog/${post.slug}`,
lastModified: post.updatedAt,
})),
]
}
app/robots.ts
import type { MetadataRoute } from ‘next’
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: ‘*’,
allow: ‘/’,
disallow: [‘/admin/’, ‘/private/’],
},
sitemap: ‘https://example.com/sitemap.xml',
}
}
Replace the example domain, database function, and routes with those used by your own application.
Key Takeaways
- Next.js sitemap and robots.txt configuration is straightforward with the modern App Router metadata-file conventions.
- Use app/sitemap.ts to generate a sitemap that becomes available at /sitemap.xml. (Next.js)
- Use app/robots.ts to generate crawler instructions at /robots.txt. (Next.js)
- Include important, canonical, publicly accessible URLs in your sitemap.
- Use accurate lastModified information instead of changing every date automatically.
- Don’t use robots.txt as a replacement for noindex when the goal is to keep a page out of search results.
- Test both files in production and submit your sitemap through Google Search Console.
- For large websites, split sitemaps when the 50,000-URL or 50 MB limits are reached. (Google for Developers)
Frequently Asked Questions
1. How do I create a sitemap in Next.js?
In an App Router project, create app/sitemap.ts and export a function returning a MetadataRoute.Sitemap array. Next.js uses this metadata route to generate /sitemap.xml. ( Next.js)
2. How do I create robots.txt in Next.js?
Create app/robots.ts and return a MetadataRoute.Robots object containing your crawler rules and sitemap URL. Next.js generates the /robots.txt response automatically. ( Next.js)
3. Should I include the sitemap URL in robots.txt?
Yes, you can include an absolute sitemap URL using the sitemap field in your Next.js robots configuration. Google supports sitemap declarations in robots.txt. ( Next.js)
4. Does robots.txt prevent a page from being indexed?
Not reliably. robots.txt primarily controls whether crawlers can access URLs. A blocked URL can potentially still appear in search results if its URL is discovered elsewhere. To prevent indexing, use noindex or another appropriate method while ensuring the crawler can access the directive. ( Google for Developers)
5. Does submitting a Next.js sitemap guarantee Google will index my pages?
No. A sitemap helps Google discover URLs, but sitemap submission is only a hint and does not guarantee that Google will crawl or index every URL. ( Google for Developers)
Conclusion
A Next.js application gives you a strong technical foundation, but good SEO still depends on making your site’s architecture understandable to search engines.
That’s why sitemap.xml and robots.txt should be treated as essential parts of your technical SEO setup.
For an App Router project, the implementation can be remarkably simple:
app/sitemap.ts
app/robots.ts
From there, generate your sitemap from reliable content data, include the canonical URLs that matter, use truthful lastModified values, and keep your crawler rules as simple as your site allows.
Most importantly, remember the difference between the two:
The sitemap helps search engines discover the URLs you want them to know about. robots.txt controls which URLs crawlers are allowed to access.
Once those fundamentals are correct, validate the production files, submit the sitemap in Google Search Console, and continue monitoring how your Next.js website is crawled and indexed.
Also Read:
Next.js SEO: Complete Guide for Developers (2026) Next.js Metadata API: Complete SEO Guide to Titles, Descriptions and Open Graph Next.js PostgreSQL : How to Connect PostgreSQL to Next.js: Step-by-Step Tutorial
Originally published at https://seoinsights.co.in on October 4, 2026.
