Create recursive sitemap with robots.txt url extraction

[ghost]

Hey mate,

i think it would be great to extend pysitemaps so it can extract urls matching specific locations.

for example the user could use a robots.txt in order to find valid urls on the server, or a config file .

What do you think?

[seowings]

Thanks for your comments. This certainly looks like a good/useful feature for following use cases.

• Create Sitemap only for a "Path" specified by user
• Sitemaps must respect robots.txt

I am currently looking into this, and will come with an extention.

[ghost]

great.... i will be watching your repo and feel free to contact me in case you have any questions :)

[seowings]

@9967819 Can you help me with some example data so that I can finalize a new build.

For example,

• Input data for the test e.g. list of urls, or paths to exclue or sample robots.txt file.
• desired output e.g. possible sitemap.

[ghost]

@9967819 Can you help me with some example data so that I can finalize a new build.

For example,

• Input data for the test e.g. list of urls, or paths to exclue or sample robots.txt file.
• desired output e.g. possible sitemap.

Sorry for the late reply. Here is my sample robots.txt file.

Also there should be a way to exclude static files (css and js) when generating the sitemap file. Note that it makes sense that the sitemap only includes urls matching the content of the robots.txt input. For testing, i suggest you make a testcase using your local development server.