[Task]: Only enable full robots.txt & sitemap on the main domain

[diox]

Description

It's technically possible for AMO to be reached through alternative domains (and more of these will be added in the future). We shouldn't enable /robots.txt and the sitemap when domains other than the main addons.mozilla.org one are being used.

There is already a mechanism for the robots.txt part of this, the check needs to be flipped to take other domains into account:

service = request.META['SERVER_NAME'] == settings.SERVICES_DOMAIN
if _service or not settings.ENGAGE_ROBOTS:

Needs to be changed to something like (check SERVER_NAME is updated correctly when other domains are used!):

is_main_domain = request.META['SERVER_NAME'] == settings.DOMAIN
if not is_main_domain or not settings.ENGAGE_ROBOTS:

Note that currently settings.ENGAGE_ROBOTS is also False on dev & stage, by design. That makes this annoying to test for QA.

Acceptance Criteria

Milestones/checkpoints

Preview Give feedback

1. robots.txt returns Disallow: / for User-agent * for every domain other than addons.mozilla.org
2. sitemap isn't accessible through other domains than addons.mozilla.org

Checks

• If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task