{"id":11526,"date":"2024-06-26T18:52:27","date_gmt":"2024-06-26T18:52:27","guid":{"rendered":"https:\/\/mobilemall.co\/blog\/?p=11526"},"modified":"2025-11-11T09:53:28","modified_gmt":"2025-11-11T09:53:28","slug":"custom-sitemap-for-big-sites","status":"publish","type":"post","link":"https:\/\/mobilemall.co\/blog\/custom-sitemap-for-big-sites\/","title":{"rendered":"How to Create Manual Sitemaps For Large Custom Websites"},"content":{"rendered":"\n

I recently received a message on LinkedIn about creating sitemaps for large websites with thousands of pages. Free sitemap generators like xml-sitemaps.com and mysitemapgenerator.com only work for a limited number of URLs, If you are doing SEO for custom websites not using any CMS will can require your technical SEO skills, here you can’t take benefit of SEO plugins that create automatically for you, so you’ll need a different solution for bigger sites custom sites.<\/p>\n\n\n\n

I know two ways to create sitemaps for large sites: manual creation using CSV and Notepad, and automated creation using Python. Today, I’ll share the Python method, which requires basic Python knowledge.<\/p>\n\n\n\n

I’ll share three Python scripts:<\/h2>\n\n\n\n
    \n
  1. Single File Script<\/strong>: Creates one XML file for a list of URLs (suitable for up to 25,000 URLs).<\/li>\n\n\n\n
  2. Category Script<\/strong>: Divides sitemaps into categories (works for even 1 million URLs).<\/li>\n\n\n\n
  3. Alphabetical Script<\/strong>: Divides sitemaps into 26 files, one for each letter of the alphabet (also works for 1 million URLs).<\/li>\n<\/ol>\n\n\n\n

    You can run this code in Google Colab or In Your Local System by Pycharm!<\/em><\/strong><\/p>\n\n\n\n

    Script 1:<\/h3>\n\n\n\n
    import pandas as pd\nfrom lxml import etree\n\n\ndf = pd.read_csv('list.csv')  # Assuming the URLs are in the first column\n\n# XML Sitemap structure\nurlset = etree.Element(\"urlset\", xmlns=\"http:\/\/www.sitemaps.org\/schemas\/sitemap\/0.9\")\n\nfor url in df.iloc[:, 0]:  # Loop through the first column containing URLs\n    url_element = etree.SubElement(urlset, \"url\")\n    loc = etree.SubElement(url_element, \"loc\")\n    loc.text = url\n    changefreq = etree.SubElement(url_element, \"changefreq\")\n    changefreq.text = \"weekly\"\n    priority = etree.SubElement(url_element, \"priority\")\n    priority.text = \"1.0\"\n\n\nsitemap = etree.tostring(urlset, pretty_print=True, xml_declaration=True, encoding='UTF-8')\n\n# Save the sitemap\nwith open(\"sitemap.xml\", \"wb\") as file:\n    file.write(sitemap)\n\nprint(\"Sitemap generated and saved as sitemap.xml\")\n<\/code><\/pre>\n\n\n\n
    REQUIRED CSV<\/em><\/p>\n\n\n\n
    List.csv just place the list of urls in column A, <\/strong>(this list will have your list of urls that you want to get to get in the sitemap) now if you wonder where can i get my website list of urls, you need to export it from your database<\/em>!<\/p>\n\n\n\n
    Script 2<\/h3>\n\n\n\n
    import csv\nimport os\nfrom collections import defaultdict\nfrom xml.etree.ElementTree import Element, SubElement, tostring\nfrom xml.dom import minidom\n\ndef prettify(elem):\n    \"\"\"Return a pretty-printed XML string for the Element.\"\"\"\n    rough_string = tostring(elem, 'utf-8')\n    reparsed = minidom.parseString(rough_string)\n    return reparsed.toprettyxml(indent=\" \")\n\ndef create_sitemap(urls):\n    \"\"\"Create sitemap XML for a list of URLs.\"\"\"\n    urlset = Element('urlset', xmlns='http:\/\/www.sitemaps.org\/schemas\/sitemap\/0.9')\n    for url in urls:\n        url_elem = SubElement(urlset, 'url')\n        loc = SubElement(url_elem, 'loc')\n        loc.text = url\n        changefreq = SubElement(url_elem, 'changefreq')\n        changefreq.text = 'daily'\n    return prettify(urlset)\n\ndef read_urls_from_csv(file_path):\n    \"\"\"Read URLs from a CSV file.\"\"\"\n    with open(file_path, newline='', encoding='utf-8') as csvfile:\n        reader = csv.reader(csvfile)\n        return [(row[0], row[1]) for row in reader if len(row) >= 2]  # Ensure the row has at least two columns\n\ndef main():\n    csv_file_path = 'list.csv'  # Path to your CSV file\n    url_data = read_urls_from_csv(csv_file_path)\n\n    # Organize URLs by category\n    url_dict = defaultdict(list)\n    for category, url in url_data:\n        url_dict[category].append(url)\n\n    # Create and save sitemaps\n    for category, urls in url_dict.items():\n        sitemap = create_sitemap(urls)\n        file_name = f'{category}.xml'\n        with open(file_name, 'w', encoding='utf-8') as file:\n            file.write(sitemap)\n        print(f'Sitemap for {category} saved as {file_name}')\n\nif __name__ == '__main__':\n    main()\n<\/code><\/pre>\n\n\n\n
    REQUIRED CSV<\/em><\/p>\n\n\n\n
    List.csv just place the Title\/Value of categores in Column A & urls next to respective Category column B, <\/strong>(you need to export database, and in csv you can apply filter of two columns category and list of slugs\/urls of your website)!<\/p>\n\n\n\n
    Script 3<\/h3>\n\n\n\n
    import csv\nimport os\nfrom collections import defaultdict\nfrom xml.etree.ElementTree import Element, SubElement, tostring\nfrom xml.dom import minidom\n\ndef prettify(elem):\n    \"\"\"Return a pretty-printed XML string for the Element.\"\"\"\n    rough_string = tostring(elem, 'utf-8')\n    reparsed = minidom.parseString(rough_string)\n    return reparsed.toprettyxml(indent=\" \")\n\ndef create_sitemap(urls):\n    \"\"\"Create sitemap XML for a list of URLs.\"\"\"\n    urlset = Element('urlset', xmlns='http:\/\/www.sitemaps.org\/schemas\/sitemap\/0.9')\n    for url in urls:\n        url_elem = SubElement(urlset, 'url')\n        loc = SubElement(url_elem, 'loc')\n        loc.text = url\n        changefreq = SubElement(url_elem, 'changefreq')\n        changefreq.text = 'daily'\n    return prettify(urlset)\n\ndef read_urls_from_csv(file_path):\n    \"\"\"Read URLs from a CSV file.\"\"\"\n    with open(file_path, newline='', encoding='utf-8') as csvfile:\n        reader = csv.reader(csvfile)\n        return [row[0] for row in reader if row]  # Ensure the row is not empty\n\ndef main():\n    csv_file_path = 'list.csv'  # Path to your CSV file\n    urls = read_urls_from_csv(csv_file_path)\n\n    # Organize URLs by first letter after domain\n    url_dict = defaultdict(list)\n    for url in urls:\n        parts = url.split('\/')\n        if parts[-1]:  # Ensure the last part is not empty\n            letter = parts[-1][0].lower()\n        else:\n            letter = parts[-2][0].lower() if len(parts) > 1 else 'other'\n        url_dict[letter].append(url)\n\n    # Create and save sitemaps\n    for letter, urls in url_dict.items():\n        sitemap = create_sitemap(urls)\n        file_name = f'stores-{letter}.xml'  # Corrected file name with underscore\n        with open(file_name, 'w', encoding='utf-8') as file:\n            file.write(sitemap)\n        print(f'Sitemap for {letter} saved as {file_name}')\n\nif __name__ == '__main__':\n    main()\n<\/code><\/pre>\n\n\n\n
    REQUIRED CSV<\/em><\/p>\n\n\n\n
    List.csv just place the urls Column A<\/strong>, script will use it’s own logic to create sitemaps by aplahabet order(just export list of urls from your database).<\/p>\n