There are several invisible aspects which make a website work as efficiently as possible and help it to be found. Here we discuss some of them:
ROBOTS.TXT
This is a text file created by a web developer to explain to search engine robots how to crawl the pages on a website. The robots.txt file is part of a group of web standards – REP or robots exclusion protocol – which are used by websites to communicate with web crawlers and other web robots.
The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned; in theory, the robots read this information first. If this file doesn’t exist, then web robots assume that the website owner does not wish to place any limitations on crawling the entire site.
XML SITEMAP
All websites should include an XML sitemap, which informs which URLs are available for crawling. As the name suggests, it is a map of a website which is there to point search engines, principally Google, to all the pages, helping search engines to track and understand a website’s structure.
XML sitemaps have an important role to play in SEO, helping search engines to find important website pages quickly, such as news pages – particularly when they aren’t easy to find.
META TAGS
There are a variety of different tags which can be included within your website structure, including meta tags. These are snippets of text that describe a page's content and help explain to search engines what a web page is about. Again, like the other ‘invisible’ parts of a website, meta tags don't appear on the page itself, but only in the page's source code. They exist in HTML and are only visible to search engines (or web developers like us, who know where to look for them).
Meta tags impact how a site appears in the search engine results page – key to whether people will actually discover your website or not. They will therefore impact your traffic and engagement rates, which can improve your SEO rankings.
There are other meta tags too, including OG (Open Graph) meta tags. These are snippets of code that affect how URLs are displayed when shared on social media. Again, if you know where to look, you can find them in the head section of a website with an OG in the front.
Including OG meta tags should mean that content you share on social media will be easier to understand and view, making it more likely that people will click through to your website. They are understood by the main social media platforms, including Facebook; if you include an OG meta tag, it will help Facebook understand better what your content is about.
Twitter has its own form of meta tags, referred to as Twitter cards – a meta data specification used by the channel to display rich text, imagery and video when links are shared.
CANONICAL TAGS
Finally, a canonical tag is a way of telling search engines about any duplicate content on your site, for example where a page is visible on more than one URL, something which Google doesn’t like. If you don’t highlight this, then Google will most probably assume you are trying to ‘trick’ its system and you might be given a lower score, which will impact your SEO ranking.
If you have a valid reason for sharing duplicate content, canonical tags are a way of explaining this. Basically, you are telling search engines that a specific URL represents the master copy of a page.
A canonical tag is a snippet of HTML code that defines the main version for duplicate, near-duplicate and similar pages. It will tell Google which version of a page to index and rank. If you don’t do this, Google will simply decide for itself…
LANGUAGE TAGS
Another ‘invisible’ part of a website are language tags and we’ll be covering those in our next blog.
If you want to discover more about these invisible elements on your website, then get in touch on tel: 020 3397 3222.