Discovery

• Initial Accessibility Assessment
  • Is the site indexed?
  • Are the URLs from the XML Sitemap indexed?
  • Are the right pages indexed (without duplicates)?
• Page Discovery
  • At least one internal link to every page
  • Most important pages linked from home page
  • Comprehensive HTML Sitemaps
  • Good external links
  • XML sitemap
  • Links work without JavaScript, Flash, or images enabled
• Page Accessibility
  • Single URL for every page (no duplicates)
  • URL exists for every page (URL changes for each new page)
  • Pages have extractable content (particularly images, AJAX, JavaScript, Flash, Silverlight)
  • URLs don’t have too many parameters
  • Does Google Webmaster Tools report any crawl errors?
  • Do all versions of home page redirect to single version (including www and non-www)?
  • Are pages blocked with robots.txt or meta tags?
  • Redirects are via 301 (not 302)
  • Not found page returns a status 404/410
• Indexing Drop Diagnosics
  • Is the site penalized? (Microsoft Live Search Webmaster Center)
  • Did the site change infrastructure? (CMS, redirecting pages, language)
  • Substantial change in site content or external linking

Source: Jane And Robot