Adding a robots.txt

[RossBugginsNHS]

Description

Added a robots.txt

Context

Requests that seach engines to ignore certain areas.

It will also allow us to link to a sitemap in the future if we automate generating one.

I have created it with the following:

User-agent: *
Disallow: /schema/*
Disallow: /schemas/*
Disallow: /events/*
Disallow: /templates/*

As in future these first 3 paths will not direct to web cms, but to schema repositories.
The 4th (templates) is an example of what we may want to do - ie include other areas to not attempt to be crawled.

Type of changes

• Refactoring (non-breaking change)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would change existing functionality)
• Bug fix (non-breaking change which fixes an issue)

Checklist

• I am familiar with the contributing guidelines
• I have followed the code style of the project
• I have added tests to cover my changes
• I have updated the documentation accordingly
• This PR is a result of pair or mob programming

---

Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.

• I confirm that neither PII/PID nor sensitive data are included in this PR and the codebase changes.