robots.txt (or sitemap) for downloads.haskell.org #214
Description
Hi, I'm not sure this is the right place to open this issue, but I'm not sure where the code is for downloads.haskell.org, so I hope it's an acceptable discussion place or that somebody can direct me to the proper venue.
The SEO situation for GHC seems a bit hairy at the moment, and I was wondering if just adding a robots.txt to that site could yield some improvement.
Say, for example, you Google for "haskell type families". One of the first results I get is
https://downloads.haskell.org/~ghc/7.8.2/docs/html/users_guide/type-families.html
whereas the better result from haskell.org would be
https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#ghc-flag--XTypeFamilies
It seems unlikely that one would want to search old versions of the manual, so could we clear this up with a robots file with Disallow entries for the older GHC versions' docs?