I am going to keep this post short as I didn't have much time this week.

Sitemaps are very important for a website as they provide a guide or map for web crawlers like Google to easily find content & index them easier, anything you want the search engines to find must be included in the sitemap & anything not included is not indexed.

Creating a sitemap is easy but how would you do it if you have dynamic content like a blog post or maybe you have an e-commerce website.

Lucky for us, it's really simple I will go through the steps I went through while adding a sitemap to my site.

Fun fact: You don't need a sitemap if your website is properly linked, doesn't have any dynamic content & has less than 500 pages.

Also, sitemaps cannot include more than 50,000 URLs.

What is the purpsose of the sitemap?

The Sitemap provides very helpful information to the crawlers like URL or location for the page, last modified date, how frequently the data is updated & the priority for this page.

Here's the sitemap for my website to explain the structure in more detail:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://fayeed.dev</loc>
    <lastmod>2020-07-29</lastmod>
  </url>
  <url>
    <loc>https://fayeed.dev/blog</loc>
    <lastmod>2020-07-29</lastmod>
  </url>
  <url>
    <loc>https://fayeed.dev/blog/building-my-personal-website-using-nextjs-ssg</loc>
    <lastmod>2020-08-06</lastmod>
  </url>
  <url>
    <loc>https://fayeed.dev/blog/how-to-make-inline-links-in-flutter</loc>
    <lastmod>2019-08-11</lastmod>
  </url>
  <url>
    <loc>https://fayeed.dev/blog/create-a-chat-app-in-minutes-in-flutter</loc>
    <lastmod>2019-08-05</lastmod>
  </url>
  <url>
    <loc>https://fayeed.dev/projects</loc>
    <lastmod>2020-07-29</lastmod>
  </url>
</urlset>

Sitemaps only use six different type of tags urlset, <url>, <loc>, <lastmod>, <changefreq> & <prority>.

urlset

This is the top most tag & provides reference to the sitemap protocol to the search engines. All the url tags lives inside this tag.

url

Parent tag for each url & all the remaining tags are children.

• <loc> - This specifies the location for the each page & must be start with a protocol like http or https.
• <lastmod> - This is a optional tag that specifies the last date this page was modified.
• <changefreq> - This is also a optional tag and its purpose it to show how frequently is this page content likely to change, think of reddit.
• <prority> - The priority of this url relative to others. Google acutally ignore this tag so need to use this.

How do I create a sitemap for my site?

Well creating a sitemap is as easy as creating a new XML file in your project public/static root folder called sitemap.xml and follow the xml structure mentioned above.

You do also need to create a robot.txt file in the same folder & add a link to your sitemap there, this will allow the crawlers to easily find the sitemap if it is somewhere else other than the root of the URL.

User-agent: *
Sitemap: https://example.com/sitemap.xml

How do I handle dynamic content?

If you have some form of dynamic content on your NextJs website like a blog, adding each URL to the sitemap can be kind of a pain. To easily resolve this we can create a helper function for this.

import fs from "fs";
import { PostData } from "../types/post_data";

export const generateSitemap = async (posts: PostData[]) => {
  const sitemap = `
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
      <loc>${BASE_URL}</loc>
  </url>
  <url>
    <loc>${BASE_URL}/blog</loc>
  </url>
  ${[...posts]
    .map((post) => `<url><loc>${post.canonicalUrl}</loc></url>`)
    .join("")}
  <url>
    <loc>${BASE_URL}/projects</loc>
  </url>
</urlset>`.replace(",", "");

  // writes sitemap.xml to public directory
  const path = `${process.cwd()}/public/sitemap.xml`;

  fs.writeFileSync(path, sitemap, "utf8");

  console.log(`generated sitemap`);

  return sitemap;
};

What this little helper function does it goes through, all the posts that we pass it, generates a sitemap string which we then write to our public folder using the fs module. In my case, I already had another helper function which gets all the markdown file & all the props in it I just passed it to this function.

And then in your getStaticProps method in your landing page you can just call this helper function & it will create the sitemap for you.

export const getStaticProps = async () => {
  // gets all the markdown files & returns the content in proper structure
  const posts = await loadBlogPosts();

  await generateSitemap(posts);
};

And don't forget to create Robot.txt file

User-agent: *
Sitemap: https://example.com/sitemap.xml