You’ve made an awesome page, you’ve published it and you’re raring to go, but no one has found your page in days, weeks… no one is sharing it. Why?
1. Errors in your robots.txt file
The disallow tool makes content not appear in indexing, so it’s important that the posts and pages you want to be shown are not under the disallow tool.
To disallow individual pages from being indexed then use the following code:
User-agent: *
Disallow: /directory/name-of-file.html
To disallow selected directories, use this:
User-agent: *
Disallow: /first-directory/
Disallow: /second-directory/
2. Your page privacy settings are stopping readers from accessing it
Private or password protected pages/areas of a website are not going to get many, if any at all, viewers to read your content. Ensure your content is able to be seen by everyone because search engine robots won’t be able to bypass your privacy controls.
3. Cookies or JavaScript
If you hide content away that’s only available to those that accept cookies then search engines won’t index your content. To put it simply, search engines robots will do the bare minimum to get by and if you’re making it hard for them to find your page(s) then there’s no hope of it getting indexed instantly.
Some webpages also have complicated JavaScript, making it difficult for those robots to crawl through and find your pages; although this is limited to a select number of websites it can still apply to a number of sites that use such difficult scripts.
4. HTTP Status Code
Ensure your website/webpage has a 200 status code, the code that says your webpage is good to visit. If you want to redirect users then you can use a 301 status code and if you want the page to not be seen you can use a 403 status code; 404s, 401s and 406s are also codes that you can use that won’t be indexed by search engines.
5. Meta Robots Noindex Tags
The meta robots noindex tag allows users to tell search engine robots that a page should not be indexed, meaning if yours is using a noindex tag then you’re not getting indexed. If you’re using this code in your <head> section then you’re not going to get your post indexed:
You can use this if you want to share out your posts and get people to read them: