I am going to keep this post short as I didn't have much time this week.
Sitemaps are very important for a website as they provide a guide or map for web crawlers like Google to easily find content & index them easier, anything you want the search engines to find must be included in the sitemap & anything not included is not indexed.
Creating a sitemap is easy but how would you do it if you have dynamic content like a blog post or maybe you have an e-commerce website.
Lucky for us, it's really simple I will go through the steps I went through while adding a sitemap to my site.
Fun fact: You don't need a sitemap if your website is properly linked, doesn't have any dynamic content & has less than 500 pages.
Also, sitemaps cannot include more than 50,000 URLs.
The Sitemap provides very helpful information to the crawlers like URL or location for the page, last modified date, how frequently the data is updated & the priority for this page.
Here's the sitemap for my website to explain the structure in more detail:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://fayeed.dev</loc>
<lastmod>2020-07-29</lastmod>
</url>
<url>
<loc>https://fayeed.dev/blog</loc>
<lastmod>2020-07-29</lastmod>
</url>
<url>
<loc>https://fayeed.dev/blog/building-my-personal-website-using-nextjs-ssg</loc>
<lastmod>2020-08-06</lastmod>
</url>
<url>
<loc>https://fayeed.dev/blog/how-to-make-inline-links-in-flutter</loc>
<lastmod>2019-08-11</lastmod>
</url>
<url>
<loc>https://fayeed.dev/blog/create-a-chat-app-in-minutes-in-flutter</loc>
<lastmod>2019-08-05</lastmod>
</url>
<url>
<loc>https://fayeed.dev/projects</loc>
<lastmod>2020-07-29</lastmod>
</url>
</urlset>
Sitemaps only use six different type of tags urlset
, <url>
, <loc>
, <lastmod>
, <changefreq>
& <prority>
.
This is the top most tag & provides reference to the sitemap protocol to the search engines. All the url tags lives inside this tag.
Parent tag for each url & all the remaining tags are children.
<loc>
- This specifies the location for the each page & must be start with a protocol like http
or https
.<lastmod>
- This is a optional tag that specifies the last date this page was modified.<changefreq>
- This is also a optional tag and its purpose it to show how frequently is this page content likely to change, think of reddit.<prority>
- The priority of this url relative to others. Google acutally ignore this tag so need to use this.Well creating a sitemap is as easy as creating a new XML file in your project public/static root folder called sitemap.xml
and follow the xml
structure mentioned above.
You do also need to create a robot.txt
file in the same folder & add a link to your sitemap there, this will allow the crawlers to easily find the sitemap if it is somewhere else other than the root of the URL.
User-agent: *
Sitemap: https://example.com/sitemap.xml
If you have some form of dynamic content on your NextJs website like a blog, adding each URL to the sitemap can be kind of a pain. To easily resolve this we can create a helper function for this.
import fs from "fs";
import { PostData } from "../types/post_data";
export const generateSitemap = async (posts: PostData[]) => {
const sitemap = `
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>${BASE_URL}</loc>
</url>
<url>
<loc>${BASE_URL}/blog</loc>
</url>
${[...posts]
.map((post) => `<url><loc>${post.canonicalUrl}</loc></url>`)
.join("")}
<url>
<loc>${BASE_URL}/projects</loc>
</url>
</urlset>`.replace(",", "");
// writes sitemap.xml to public directory
const path = `${process.cwd()}/public/sitemap.xml`;
fs.writeFileSync(path, sitemap, "utf8");
console.log(`generated sitemap`);
return sitemap;
};
What this little helper function does it goes through, all the posts that we pass it, generates a sitemap string which we then write to our public folder using the fs
module. In my case, I already had another helper function which gets all the markdown file & all the props in it I just passed it to this function.
And then in your getStaticProps
method in your landing page you can just call this helper function & it will create the sitemap for you.
export const getStaticProps = async () => {
// gets all the markdown files & returns the content in proper structure
const posts = await loadBlogPosts();
await generateSitemap(posts);
};
And don't forget to create Robot.txt
file
User-agent: *
Sitemap: https://example.com/sitemap.xml