How to get Google to index your site is one of the questions we get asked most often but what is indexing and why does it matter? To answer the question, we first need to look at “How do search engines work”?
How do search engines work?
Google, and other search engines, use algorithms to provide their search results. An algorithm is a set of rules that are followed to complete a task. In this case the task is to find an answer to a search enquiry. Google’s algorithms use multiple factors to provide the result your customers are looking for. Factors include website content, how recent the content is, location, links, valid code and more.
Google uses “spiders” or “bots” (little bits of software or code) to “crawl” websites (hence, “spider”). If the bot locates a new website, page or post Google analyses and lists it. Essentially, Google inspects the website and grades the overall user experience. Google attempts to find the answers customers/members/site users are searching for.
(By the way, keywords still matter and we’ll come to them in another post.)
Of course, none of that will matter if the spiders can’t even inform Google that your site or new article/page is there to begin with. This is where indexing comes in.
So, what exactly is indexing?
Website indexing is the systematic way a Googlebot (spider) gathers and processes all the information from pages and sites.. The bot reports new findings which are then added to Google’s searchable index, but only if the pages are quality content and don’t trigger alarm bells by violating Google’s user-oriented mandate. The bot collects
- the page content
- the location on the webpage where search terms are placed
- it also examines titles tags and alt attributes for images.
The algorithm provides an answer to your customers search by deciding where to rank your page amongst all the other pages related to those keywords.
How does Google discover new content, posts or websites?
The process begins with content/pages/websites that have already been indexed via earlier crawl sessions. The bot also adds sitemap data provided by the site owner. (More on this in another post). As the spider browses previously crawled sites, it discovers links in those pages to add to the list of pages to be crawled.
What’s a sitemap?
Have you ever been on holiday and picked up a flyer to direct you to tourist destinations? That’s what a sitemap is. It’s a way to direct tourists (in this case the spider) to destinations (articles/pages/posts/links).
A sitemap is a file where you can list the web pages of your site to tell Google and other search engines about the structure of your site content. Search engine spiders like Googlebot read this file to more accurately crawl your site.
Do I need a sitemap?
To quote Google directly…
If your site’s pages are properly linked, our web crawlers can usually discover most of your site. Even so, a sitemap can improve the crawling of your site, particularly if your site meets one of the following criteria:
- Your site is large. Thus, it’s more likely crawlers might overlook crawling some of your new or recently updated pages.
- Your site has a large archive of content pages that are isolated or well not linked to each other. If you site pages do not naturally reference each other, you can list them in a sitemap to ensure that Google does not overlook some of your pages.
- Your site is new and has few external links to it. Web crawlers crawl the web by following links from one page to another. Thus, Google might not discover your pages if no other sites link to them.Your site uses rich media content, is shown in Google News, or uses other sitemaps-compatible annotations. Google can take additional information from sitemaps into account for search, where appropriate.
Where do I start?
Create a Sitemap
A sitemap is a special document (XML) which is accessible through via website’s server. The sitemap lists each page on your website. It tells search engines when new pages have been added and how often to check back for changes on specific pages. Popular web platforms like WordPress and Joomla have a range of plugins (bits of extra software) to automatically create and update your sitemap for you as well as submitting it to Google and other search engines. If you need help, just call us on 0191 303 7141 or get in touch via the contact page.
Submit your sitemap to Google Search Console (formerly known as Google Webmaster Tools
Once you have your sitemap you need to submit it Google Search Console. If you don’t already have one, simply create a free Google Account, then sign up for Webmaster Tools. We’ll cover this in a following post and as part of our “7 days of some of our best tips to make your website work for you” give away.
Install Google Analytics
Google analytics is essential to track how your visitors are using your website but it can also give Google the heads up that new content is available.
Submit your website URL to the search engines
This is a bit old school but it can’t hurt and only takes a moment. To submit your website URL to Google, sign into your Google Account and head over to the submit URL console. You might as well also submit your website to Bing.
Create or update social profiles
Are you thinking about social media? You should, it’s useful in many ways. As mentioned previously, spiders finds your website via links. One way to get some quick links is by creating social networking profiles for your new website or adding a link to your new website to pre-existing profiles. This includes Twitter profiles, Facebook pages, Google+ profiles or pages, LinkedIn profiles or company pages, Pinterest profiles, and YouTube channels.
Create Offsite Content
Share your knowledge – offsite content includes submitting guest posts to blogs in your niche, articles to quality article directories and press releases for distribution. Please note this is about quality content from quality sites – you don’t want spammy content from spammy sites because that will simply tar you with the same brush.