Submitted by simonyarwood on Sat, 07/01/2006 - 21:29.
Hi,
over the past few months, I have listed sites on google, yet google only ever lists the index page for each of my sites, no other pages are listed, what am I doing wrong?
Submitted by simonyarwood on Sat, 07/01/2006 - 23:53.
hello e-strategist,
Yes, I have the robots.txt file and I submitted the links to Google sitemaps,
I did this a few months back and since then google has put my latest index page up but (I looked at the cached page) but no other pages are listed?
Submitted by e-strategist on Sun, 07/02/2006 - 00:44.
have you checked your robots.txt file. Are you sure that you havent blocked google to crawl your pages. Sorry I'm sure that you have done this but I'm asking to be sure.
Do you have inappropriate links at your homepage. For example your site is about Cars but you have links to hotel websites or something like that.
Since Bigdaddy update Google is looking for such infos at websites.
The GoogleGut Matt Cutts has some important info about that:
Submitted by simonyarwood on Sun, 07/02/2006 - 08:42.
in the HTML I have
my robots.txt file reads:
###############################
#
#
User-agent: *
#
# list folders robots are not allowed to index
#
Disallow: /cgi-bin/
Disallow: /client_ads/
Disallow: /colorschemes/
Disallow: /mark/
Disallow: /resources/
Disallow: /images/
#
# list specific files robots are not allowed to index
#
#
#
###############################
Submitted by simonyarwood on Sun, 07/02/2006 - 08:46.
Quote:
Do you have inappropriate links at your homepage. For example your site is about Cars but you have links to hotel websites or something like that.
I do have links on my index page that don't relate to the content of the site, one is for the guy that put the site together and the other link is for the hosting company.
In what you say, would it be best to remove all such external links from the index page and have them on the otherpages instead?
Submitted by Jeremy Palmer on Mon, 07/03/2006 - 04:35.
Hi Simon,
There has been a lot of chatter on other forums that google is having major problems indexing content right now (possibly one of the reasons they're pushing sitemaps). From what I've seen, I would agree with them.
Many of my sites have been dropping pages for no apparent reason. Many of my new sites are also having a hard time getting indexed.
A few questions:
Does your site have any JavaScript? If so, how are you using it?
Is your code clean? Do you think there could be something tripping up the crawler? http://validator.w3.org/detailed.html
It's difficult for any site to pass validation (quityourdayjob.com has 36 errors), however, you may have an error that the crawler can't overcome. Just a thought...
Submitted by simonyarwood on Mon, 07/03/2006 - 12:40.
My theory on this is as follows:
Google drops your site or fails to list pages you submitted,
you need the traffic to help you make money, so what do you do?
PAY FOR IT!! with Adwords:evil:
Yet again, I did another test on Google.
I submitted several sites at the same time, ALL the sites related to business were either not listed or just had the index page listed. I listed one site that was a personal site, nothing to do with making money, guess what!, Google listed and indexed all the pages for that site.
I think this proves a point.
I know what I am going to do, I will put links to the other sites and see if Google will index the business sites that way, .
Submitted by Jeremy Palmer on Mon, 07/03/2006 - 19:57.
Hi Simon,
I think it has to do with your document using strict XHTML. If you are using the DOCTYPE XHTML every tag must have a closing tag. I don't think this is enough to trip up the crawler though...
If the XHTML tag does not have a corresponding closing tag (e.g.
) you can generally use the syntax with the closing / just before the >
Does your site have any JavaScript? If so, how are you using it?
Is your code clean? Do you think there could be something tripping up the crawler? http://validator.w3.org/detailed.html
It's difficult for any site to pass validation (quityourdayjob.com has 36 errors), however, you may have an error that the crawler can't overcome. Just a thought...
Notice that you mentioned QYDJ has 36 errors? I'm glad to read this, as when I remove the affliliate links I'm squeaky clean. Can the links supplied by the merchant create havoc for a crawler?
over the past few months, I have listed sites on google, yet google only ever lists the index page for each of my sites, no other pages are listed, what am I doing wrong?
Simon
I remeber watching Mat Cutts podcast regarding big sites. He is a Google engeneere but also I believe he is a kind of spokesmen for Google. In a nutshell what he said in one of his podacsts that if you have huges sites with tousands of pages, to submit them to Google over time in groups- I think he meant Sitemap. Now with data feeddriven sites if your very fist site map will contain more that 1000 links it may raise some eyebrowes.
None of my sites have pages in thousands. Never the less I have developed this strategy (unfortunately I have no valid proof that it will work for anyone). Whenever I am ready for my site to be crawled, before I submit the sitemap to google I write at least three articles. I mostly use EzineArtiles. What I have noticed is that if Google discovers the site on it's own the initial indexing is rather quick. I wait few days so at least 10 or more pages get into Googles index and only then do I submit a Sitemap. It works for me. I have this site that i have submitted about a month ago and using this method I have already 300 pages indexed and some of them are already ranking well for my keywords. Hope it helps. It's just my theory, but I think it matters if your site is discovered via link from a highly ranked and traffiked site.
Hi Simon,
Do you have a robots.txt file? have you submitted your links to Google sitemaps?
hello e-strategist,
Yes, I have the robots.txt file and I submitted the links to Google sitemaps,
I did this a few months back and since then google has put my latest index page up but (I looked at the cached page) but no other pages are listed?
Simon
have you checked your robots.txt file. Are you sure that you havent blocked google to crawl your pages. Sorry I'm sure that you have done this but I'm asking to be sure.
Do you have inappropriate links at your homepage. For example your site is about Cars but you have links to hotel websites or something like that.
Since Bigdaddy update Google is looking for such infos at websites.
The GoogleGut Matt Cutts has some important info about that:
http://www.mattcutts.com/blog/
You might also try this site by Aaron Wall:
http://www.seobook.com/
Aaron also has an ebook that I have purchased and helped me quite a bit with SEO optimization.
in the HTML I have
my robots.txt file reads:
###############################
#
#
User-agent: *
#
# list folders robots are not allowed to index
#
Disallow: /cgi-bin/
Disallow: /client_ads/
Disallow: /colorschemes/
Disallow: /mark/
Disallow: /resources/
Disallow: /images/
#
# list specific files robots are not allowed to index
#
#
#
###############################
google says they located the robots.txt file OK.
Simon
I do have links on my index page that don't relate to the content of the site, one is for the guy that put the site together and the other link is for the hosting company.
In what you say, would it be best to remove all such external links from the index page and have them on the otherpages instead?
Simon
Hi Simon,
There has been a lot of chatter on other forums that google is having major problems indexing content right now (possibly one of the reasons they're pushing sitemaps). From what I've seen, I would agree with them.
Many of my sites have been dropping pages for no apparent reason. Many of my new sites are also having a hard time getting indexed.
A few questions:
Does your site have any JavaScript? If so, how are you using it?
Is your code clean? Do you think there could be something tripping up the crawler?
http://validator.w3.org/detailed.html
It's difficult for any site to pass validation (quityourdayjob.com has 36 errors), however, you may have an error that the crawler can't overcome. Just a thought...
My theory on this is as follows:
Google drops your site or fails to list pages you submitted,
you need the traffic to help you make money, so what do you do?
PAY FOR IT!! with Adwords:evil:
Yet again, I did another test on Google.
I submitted several sites at the same time, ALL the sites related to business were either not listed or just had the index page listed. I listed one site that was a personal site, nothing to do with making money, guess what!, Google listed and indexed all the pages for that site.
I think this proves a point.
I know what I am going to do, I will put links to the other sites and see if Google will index the business sites that way, .
Simon
Hello Jeremy,
did the validation, I had 10 errors on the index page, non related to java script.
one problem the validator showed was
# Error Line 6 column 43: end tag for "meta" omitted, but OMITTAG NO was specified.
You may have neglected to close an element, or perhaps you meant to "self-close" an element, that is, ending it with "/>" instead of ">".
is there a problem here?, I can't see it
Simon
Hi Simon,
I think it has to do with your document using strict XHTML. If you are using the DOCTYPE XHTML every tag must have a closing tag. I don't think this is enough to trip up the crawler though...
If the XHTML tag does not have a corresponding closing tag (e.g.
) you can generally use the syntax with the closing / just before the >Hope that makes sense.
Best,
Jeremy
Notice that you mentioned QYDJ has 36 errors? I'm glad to read this, as when I remove the affliliate links I'm squeaky clean. Can the links supplied by the merchant create havoc for a crawler?
Cheers
J.
I remeber watching Mat Cutts podcast regarding big sites. He is a Google engeneere but also I believe he is a kind of spokesmen for Google. In a nutshell what he said in one of his podacsts that if you have huges sites with tousands of pages, to submit them to Google over time in groups- I think he meant Sitemap. Now with data feeddriven sites if your very fist site map will contain more that 1000 links it may raise some eyebrowes.
None of my sites have pages in thousands. Never the less I have developed this strategy (unfortunately I have no valid proof that it will work for anyone). Whenever I am ready for my site to be crawled, before I submit the sitemap to google I write at least three articles. I mostly use EzineArtiles. What I have noticed is that if Google discovers the site on it's own the initial indexing is rather quick. I wait few days so at least 10 or more pages get into Googles index and only then do I submit a Sitemap. It works for me. I have this site that i have submitted about a month ago and using this method I have already 300 pages indexed and some of them are already ranking well for my keywords. Hope it helps. It's just my theory, but I think it matters if your site is discovered via link from a highly ranked and traffiked site.