add setting to be added to article frontmatter to block web crawlers (#…

…736) Signed-off-by: pcnorth <[email protected]>
datatrails · Dec 13, 2023 · 1da59c2 · 1da59c2
1 parent a0c4575
commit 1da59c2
Show file tree

Hide file tree

Showing 21 changed files with 68 additions and 3 deletions.
diff --git a/content/archive/_index.md b/content/archive/_index.md
@@ -9,6 +9,8 @@ images: []
 weight: 999
 toc: true
 aliases: 
+sitemap_exclude: true
+robotsdisallow: true
 ---
 This section is intended to be a space for articles that are no longer needed but still have content that we do not want to delete.
 

diff --git a/content/archive/verify-receipts/index.md b/content/archive/verify-receipts/index.md
@@ -9,7 +9,9 @@ images: []
 menu: 
   archive:
     parent: "archive"
-weight: 10
+weight: 
+sitemap_exclude: true
+robotsdisallow: true
 ---
 # This article is deprecated
 

diff --git a/content/contributing/_index.md b/content/contributing/_index.md
@@ -5,4 +5,6 @@ date: 2021-05-20T12:03:27+01:00
 lastmod: 2021-05-20T12:03:27+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/contributing/adding-content/_index.md b/content/contributing/adding-content/_index.md
@@ -5,4 +5,6 @@ date: 2021-05-20T18:01:01+01:00
 lastmod: 2021-05-20T18:01:01+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/contributing/adding-content/adding-articles-to-datatrails-docs/index.md b/content/contributing/adding-content/adding-articles-to-datatrails-docs/index.md
@@ -10,6 +10,8 @@ menu:
 weight: 3
 images: []
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 {{< note >}}

diff --git a/...contributing/adding-content/adding-content-sections-to-datatrails-docs/index.md b/...contributing/adding-content/adding-content-sections-to-datatrails-docs/index.md
@@ -10,6 +10,8 @@ menu:
 weight: 4
 images: []
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 {{< note >}}

diff --git a/content/contributing/adding-content/creating-your-branch/index.md b/content/contributing/adding-content/creating-your-branch/index.md
@@ -11,6 +11,8 @@ menu:
     parent: "adding-content"
 weight: 2
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 As the main branch of these docs is protected you will need to create a separate branch and submit a PR to make any changes

diff --git a/content/contributing/formatting-content/_index.md b/content/contributing/formatting-content/_index.md
@@ -5,4 +5,6 @@ date: 2021-05-20T19:14:54+01:00
 lastmod: 2021-05-20T19:14:54+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/contributing/formatting-content/advanced-formatting/index.md b/content/contributing/formatting-content/advanced-formatting/index.md
@@ -11,6 +11,8 @@ menu:
     parent: "formatting-content"
 weight: 7
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 As Doks is based on Hugo it has many more extensible formatting options, including some custom tools we have added ourselves to enhance the delivery of documentation overall.

diff --git a/content/contributing/formatting-content/basic-markdown-formatting/index.md b/content/contributing/formatting-content/basic-markdown-formatting/index.md
@@ -11,6 +11,8 @@ menu:
     parent: "formatting-content"
 weight: 6
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 Doks uses Goldmark markdown which is primarily Github-flavoured.

diff --git a/content/contributing/formatting-content/style-guide/index.md b/content/contributing/formatting-content/style-guide/index.md
@@ -11,6 +11,8 @@ menu:
     parent: "formatting-content"
 weight: 8
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 It is important in any documentation to have a readable, sane and consistent experience, to that end we have defined some guidelines to ensure all content meets the same standard.

diff --git a/content/contributing/getting-started/_index.md b/content/contributing/getting-started/_index.md
@@ -5,4 +5,6 @@ date: 2021-05-20T12:03:27+01:00
 lastmod: 2021-05-20T12:03:27+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/contributing/getting-started/pulling-and-building-rkvst-docs/index.md b/content/contributing/getting-started/pulling-and-building-rkvst-docs/index.md
@@ -10,6 +10,8 @@ menu:
 weight: 1
 images: []
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 {{< note >}}
 DataTrails Docs depend on npm version 14.

diff --git a/content/contributing/playground/_index.md b/content/contributing/playground/_index.md
@@ -7,4 +7,6 @@ lastmod: 2021-05-22T19:47:46+01:00
 draft: false
 images: []
 weight: 999
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/contributing/playground/openapi-example-render/index.md b/content/contributing/playground/openapi-example-render/index.md
@@ -11,6 +11,8 @@ menu:
     parent: "playground"
 weight: 999
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 {{< openapi url="https://raw.githubusercontent.com/rkvst/archivist-docs/master/doc/openapi/subjectsv1.swagger.json" >}}
diff --git a/content/contributing/playground/test-page/index.md b/content/contributing/playground/test-page/index.md
@@ -11,4 +11,6 @@ menu:
     parent: "playground"
 weight: 998
 toc: true
+sitemap_exclude: true
+robotsdisallow: true
 ---
diff --git a/content/sales/_index.md b/content/sales/_index.md
@@ -0,0 +1,14 @@
+---
+title: "Sales"
+description: ""
+lead: ""
+date: 2023-12-11T09:25:37+01:00
+lastmod: 2023-12-11T09:25:37+01:00
+draft: false
+images: []
+weight: 999
+toc: true
+aliases: 
+sitemap_exclude: true
+robotsdisallow: true
+---
diff --git a/content/sales/contactus/index.md b/content/sales/contactus/index.md
@@ -5,6 +5,8 @@ date: 2021-05-18T18:55:49+01:00
 lastmod: 2021-05-18T18:55:49+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 <a rel="noopener" data-ux-btn="secondary" data-ux="ButtonSecondary" data-aid="CTA_BUTTON_RENDERED" href="http://datatrails.ai/beta" target="_blank" data-tccl="ux2.INTRODUCTION.introduction2.Group.Default.Button.Secondary.47772.click,click" data-typography="ButtonAlpha" class="btn btn-primary">Request a Demo</a>
diff --git a/content/support/_index.md b/content/support/_index.md
@@ -0,0 +1,14 @@
+---
+title: "Support"
+description: ""
+lead: ""
+date: 2023-12-11T09:25:37+01:00
+lastmod: 2023-12-11T09:25:37+01:00
+draft: false
+images: []
+weight: 999
+toc: true
+aliases: 
+sitemap_exclude: true
+robotsdisallow: true
+---
diff --git a/content/support/contactus/index.md b/content/support/contactus/index.md
@@ -5,6 +5,8 @@ date: 2021-05-18T18:55:36+01:00
 lastmod: 2021-05-18T18:55:36+01:00
 draft: false
 images: []
+sitemap_exclude: true
+robotsdisallow: true
 ---
 
 For any queries please contact [email protected]
diff --git a/layouts/robots.txt b/layouts/robots.txt
@@ -2,6 +2,7 @@ User-agent: *
 {{ if eq (hugo.Environment) "production" -}}
 Allow: /
 {{ else -}}
-Disallow: /
+{{ range where .Data.Pages "Params.robotsdisallow" true }}
+Disallow: {{ .RelPermalink }}{{ end }}
 {{ end }}
-Sitemap: {{ "sitemap.xml" | absURL -}}
+Sitemap: {{ "sitemap.xml" | absURL -}}