{"id":487,"date":"2019-04-10T03:42:51","date_gmt":"2019-04-10T03:42:51","guid":{"rendered":"https:\/\/s-oneill.com\/sov02\/?p=487"},"modified":"2020-10-17T00:39:07","modified_gmt":"2020-10-17T00:39:07","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/s-oneill.com\/sov02\/robots-txt\/","title":{"rendered":"Robots.txt"},"content":{"rendered":"\n<p>An appropriate way to start off SEO posts is a short on on robots.txt. This is the file that web crawlers and bots are supposed to look to for instructions when indexing your site, including an instruction not to index. Apparently there are some naughty bots out there that ignore any type of instructions but the big players like Google attempt to follow the robots.txt directions.<\/p>\n\n\n\n<p>For a quick review, check out <a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"http:\/\/www.robotstxt.org\/\" target=\"_blank\">http:\/\/www.robotstxt.org\/<\/a>  They list examples including this one that specifies directories not to index.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">User-agent: * <br>Disallow: \/cgi-bin\/ <br>Disallow: \/tmp\/ <br>Disallow: \/~joe\/<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>An appropriate way to start off SEO posts is a short on on robots.txt. This is the file that web crawlers and bots are supposed to look to for instructions when indexing your site, including an instruction not to index. Apparently there are some naughty bots out there that ignore any type of instructions but [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[35,47],"tags":[],"class_list":["post-487","post","type-post","status-publish","format-standard","hentry","category-developer","category-seo"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Robots.txt - s-oneill.com<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt - s-oneill.com\" \/>\n<meta property=\"og:description\" content=\"An appropriate way to start off SEO posts is a short on on robots.txt. This is the file that web crawlers and bots are supposed to look to for instructions when indexing your site, including an instruction not to index. Apparently there are some naughty bots out there that ignore any type of instructions but [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"s-oneill.com\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-10T03:42:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-10-17T00:39:07+00:00\" \/>\n<meta name=\"author\" content=\"Me(sean)\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Me(sean)\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\"},\"author\":{\"name\":\"Me(sean)\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/ab11148c6ea273be80dddd8baec1a545\"},\"headline\":\"Robots.txt\",\"datePublished\":\"2019-04-10T03:42:51+00:00\",\"dateModified\":\"2020-10-17T00:39:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\"},\"wordCount\":92,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a\"},\"articleSection\":[\"Developer\",\"SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/s-oneill.com\/sov02\/robots-txt\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\",\"url\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\",\"name\":\"Robots.txt - s-oneill.com\",\"isPartOf\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/#website\"},\"datePublished\":\"2019-04-10T03:42:51+00:00\",\"dateModified\":\"2020-10-17T00:39:07+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/s-oneill.com\/sov02\/robots-txt\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/robots-txt\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/s-oneill.com\/sov02\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/#website\",\"url\":\"https:\/\/s-oneill.com\/sov02\/\",\"name\":\"s-oneill.com\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/s-oneill.com\/sov02\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a\",\"name\":\"sean o\",\"logo\":{\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/image\/\"},\"description\":\"Web dev over 15 years.\",\"sameAs\":[\"http:\/\/s-oneill.com\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/ab11148c6ea273be80dddd8baec1a545\",\"name\":\"Me(sean)\",\"url\":\"https:\/\/s-oneill.com\/sov02\/author\/user01\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots.txt - s-oneill.com","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/s-oneill.com\/sov02\/robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt - s-oneill.com","og_description":"An appropriate way to start off SEO posts is a short on on robots.txt. This is the file that web crawlers and bots are supposed to look to for instructions when indexing your site, including an instruction not to index. Apparently there are some naughty bots out there that ignore any type of instructions but [&hellip;]","og_url":"https:\/\/s-oneill.com\/sov02\/robots-txt\/","og_site_name":"s-oneill.com","article_published_time":"2019-04-10T03:42:51+00:00","article_modified_time":"2020-10-17T00:39:07+00:00","author":"Me(sean)","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Me(sean)"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/#article","isPartOf":{"@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/"},"author":{"name":"Me(sean)","@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/ab11148c6ea273be80dddd8baec1a545"},"headline":"Robots.txt","datePublished":"2019-04-10T03:42:51+00:00","dateModified":"2020-10-17T00:39:07+00:00","mainEntityOfPage":{"@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/"},"wordCount":92,"commentCount":0,"publisher":{"@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a"},"articleSection":["Developer","SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/s-oneill.com\/sov02\/robots-txt\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/","url":"https:\/\/s-oneill.com\/sov02\/robots-txt\/","name":"Robots.txt - s-oneill.com","isPartOf":{"@id":"https:\/\/s-oneill.com\/sov02\/#website"},"datePublished":"2019-04-10T03:42:51+00:00","dateModified":"2020-10-17T00:39:07+00:00","breadcrumb":{"@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/s-oneill.com\/sov02\/robots-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/s-oneill.com\/sov02\/robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/s-oneill.com\/sov02\/"},{"@type":"ListItem","position":2,"name":"Robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/s-oneill.com\/sov02\/#website","url":"https:\/\/s-oneill.com\/sov02\/","name":"s-oneill.com","description":"","publisher":{"@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/s-oneill.com\/sov02\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/a9217b5797251a243dfd504d7ad7d00a","name":"sean o","logo":{"@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/image\/"},"description":"Web dev over 15 years.","sameAs":["http:\/\/s-oneill.com"]},{"@type":"Person","@id":"https:\/\/s-oneill.com\/sov02\/#\/schema\/person\/ab11148c6ea273be80dddd8baec1a545","name":"Me(sean)","url":"https:\/\/s-oneill.com\/sov02\/author\/user01\/"}]}},"_links":{"self":[{"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/posts\/487","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/comments?post=487"}],"version-history":[{"count":2,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/posts\/487\/revisions"}],"predecessor-version":[{"id":489,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/posts\/487\/revisions\/489"}],"wp:attachment":[{"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/media?parent=487"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/categories?post=487"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/s-oneill.com\/sov02\/wp-json\/wp\/v2\/tags?post=487"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}