Robots - To index or not to index?
I'm having a look around the site at the moment and trying to think what I can do to help search engines spider MilkHub better. After reading about sitemap files I've started creating my sitemap.xml and I'll be fleshing that out a bit over the next week or so and pinging that to the search enginges. What else can I do?
Earlier this week Google were talking about Dynamic URLs vs. static URLs and this got me thinking... If you look at my blog I have the main listing, archive break downs for each year and month (and if you look hard enough by day too), and there's also listings broken down by each tag. All of these listings have their own URLs with pagination handled by the query. I'd like to know...
- Do search engines like all of these different listings?
- Would it help search engines if I asked them not to index filtered listings like mine? Perhaps this would help them concentrate on my main listing.
- Would search engines prefer one URL (+ query) for all of these listings or are they happier with the clean URLs?
Comments
No comments have been made, why not be the first?