Hi there,

I need advice on having my website's 3000+ dynamic .asp pages crawled and cataloged by search engines. I've been reading about IIS Isapi-Rewrite. Is that what I need to do to make crawlers more willing to crawl my pages? How does this work? Do I install it on the server and it does the rest? Any advice and guidance is much appreciated!

Thanks,
Nicole

There's been lots of dicussion about this, on lots of forums.
The truth of the matter is that the search engines can and will happily crawl pages without your needing to make URL's more meaningful to users.

The advice is to not use more than 2 - 3 parameters in the URL, and to not use session id numbers in the URL. Just in terms of getting crawled, thats about it.

You may want to go down the SEFU route as part of your SEO plan - keywords in the URL's, etc..., and that might actually be a good idea, but its not something you need to do just to get crawled.

I put up a dynamically driven ASP.Net site only 3 weeks ago. The URL's include parameters, etc - no search engine friendly URLs used at all. Yet, within about 10 days, Google had indexed every single page. All 932 of them. The site is actually getting a fair bit of search engine traffic already - mainly from Google - and the site is far from complete.

So, keep the URL's as short as you can (things after the ".asp?"), and avoid using the session id in the URL (Google say they won't include such pages in their index).

__________________
www.bsolveit.com
Web Design & SEO for Business Growth

I agree that search engines will crawl dynamic urls as BSolveIT says. However, all my experiences indicates that meaningful (static) urls are best. See also:
11 Best Practices for URLs

Isapi Rewrite is one method, but not everyone has this option. You can also use custom 404. Take a look at ex. my Strictly CSS signature, that site is using custom 404. Some of the URLs have to many hyphens IMO, but I haven't bothered to do something with it.

__________________
Strictly CSS - directory for CSS designed web sites | Katalog og firmasøk

I posted a page about URL Rewrites a while back, but I need to expand it. URL Rewriting isn't really that tricky, but you need to plan for it and fully understand what your trying to do.

I think it is a great idea, even if the spiders will crawl your site without it. I like it because it gives pages meaningful names and hides your underlying infrastructure.

The other great thing is if you decide to change your underlying infrastructre, say asp to asp.net, then you can maintain all those legacy URLs without needing to do any ugly hacks.

Just a side note: I was using the free mod_rewrite ISAPI filter, but it was crashing on a clients site. I went with the ISAPI filter from Helix (I belive) and I haven't had any problems at all. I use mod_rewrite on a staging server with no problems however, and I even had a site that got 90,000 hits in one day without crashing. I think it has something to do with with a Windows update. Just stress test the filter before putting it into production.

__________________
AJAX Example Sites is now here! With a nice
AJAX Chat Application Tutorial

Yup, the software just doesn't work on some of our servers even if they look identical in terms of hardware and software.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.

Paz,

I'm glad to see that I'm not the only one that had that problem. Our staging server is setup identically to our production server, but for some reason it crashes on the production server. I guess you can't complain too much about free code.

__________________
AJAX Example Sites is now here! With a nice
AJAX Chat Application Tutorial

Understanding crawlers and spiders is quite a brainful, but I have a question about a .NET website.....

I was wondering if anyone here has had a website spidered that uses a default.aspx .master page.

I was wondering if the tags are placed in the master page or the content page.

Maybe using a url redirect would be the best way to handle this.

Any solutions are helpful

cplmckenzie

The new directory site I mentioned I've been buiding is in ASP.Net 2.0, using master pages. http://www.addurlfast.com

I haven't got very far with it yet, as I just don't get much spare time at the moment. However, Google spidered and indexed 900+ pages within 3 - 4 days of my putting the "work in progress" site up on the server.

When you ask about the tags, I assume you mean the meta tags? When using master pages, you need to add the meta tags via the code behind page in the prerender event:



I've put up a few other .net sites, using master pages, for clients in the last couple of weeks. It does take a while to get used to the different way of doing things, and it's not always ideal, but if you plan the site carefully you'll be fine.

__________________
www.bsolveit.com
Web Design & SEO for Business Growth

Hey BSolveIt,

Thanks for the reply. It certainly removes some of the clouds.

Ok, so per your code example,

Dim meta_description As New HtmlMeta
meta_description.Name = "description"
Dim meta_keywords As New HtmlMeta
meta_keywords.Name = "keywords"

Then when I assign the meta_ a value as above, how do I place the description text and the keywords list in the assignment string?

Would meta_description have a value property as in...

meta_description.Value = "DESCRIPTION TEXT"
meta_keywords.Value = "KEYWORD LIST"

Would that be the way to code that?

regards...

Ahhh.. you would think so, but.... no.

Here's a fuller listing:



That's about it.
Obviously, you can add other Meta tags in this way too.

Good luck!

__________________
www.bsolveit.com
Web Design & SEO for Business Growth