Adding a Sitemap and Robots.txt to My Blog

After adding remark and rehype plugins to the blog and finishing the related development work, I checked the Google GA I had attached to the blog and it looked like basically only I had visited it (1-2 visitors 🥲). So I added the approach provided by Next.js so the site could show up through SEO.

Creating a sitemap#

I made it using Generating a sitemap using code (.js, .ts) from the explanations in NEXT.js - sitemap.xml. At first, I also tried applying Generating multiple sitemaps, and I created sitemap.ts files for the posts and books menus, which have detail pages, but it did not work properly. So while looking for another approach, I ended up changing the app/sitemap.ts file.

import { allBooks, allPosts } from '@/.contentlayer/generated'
import { BASE_URL } from '@/app/lib/definitions'
import { MetadataRoute } from 'next'
 
export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: `${BASE_URL}/`,
      lastModified: new Date(),
      changeFrequency: 'yearly',
      priority: 1,
    },
    {
      url: `${BASE_URL}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: `${BASE_URL}/tags`,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.5,
    },
    {
      url: `${BASE_URL}/get912000won`,
      lastModified: new Date(),
    },
    ...allPosts.map((post) => ({
      url: `${BASE_URL}/posts/${post.slug}`,
      lastModified: post.date,
    })),
    ...allBooks.map((book) => ({
      url: `${BASE_URL}/books/${book.slug}`,
      lastModified:
        book.start_read_date === book.finish_read_date
          ? book.start_read_date
          : book.finish_read_date,
    })),
  ]
}

I use contentlayer, so I already have objects that contain information about the posts written in markdown. That made it easy to import allPosts and allBooks and add the post data.

Creating robots.txt#

I used Generate a Robots file from the explanations in NEXT.js - robots.txt. It is even easier than setting up the sitemap.

I created the following in app/robots.ts.

import { MetadataRoute } from 'next'
 
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
    },
    sitemap: 'https://get6.github.io/sitemap.xml',
  }
}

Registering with search engine sites#

What I learned from an AI answer was that creating these files is not the end of it. You also need to add the site to search engine services so it can actually be crawled.

So you need to register it directly with several search engine sites. There seem to be services that automatically submit your site to many search engines, but I have not tested whether they do it properly. Still, I will share links below for some well-known sites on the internet.

1. Google - Google Search Console
2. Naver - Search Advisor
3. Bing - Webmasters Tools
4. Baidu - 百度站长工具
5. Yandex - Webmaster

For now, I only added Google and Naver. All of these sites basically require you to proceed after logging in. If there is a site you want to use, sign up and continue from there.

For both Google and Naver, I chose the method where you add an HTML tag. I think the method of uploading an HTML file is more for web servers such as NGINX.

Normally, you would just add the tag between the header tags, but since I use Next.js, I had to configure it separately.

export const metadata: Metadata = {
  // ...other settings omitted
  verification: {
    google: 'the content part from the tag they told you to copy',
    other: {
      // Put the name part from the tag they told you to copy on the left.
      'naver-site-verification':
        'the content part from the tag they told you to copy',
    },
  },
}

Once the tags are deployed to your hosting environment, just click the ownership verification button and you are done.
After that, it could also be useful to look through the webmaster tools provided by each site and use the collected data to see which posts are the most popular.

God gave us the gift of life; it is up to us to give ourselves the gift of living well.

— Voltaire