|
| 1 | +# Infrastructure to Count Crate Downloads |
| 2 | + |
| 3 | +This module creates the infrastructure that enables [crates.io] to count crate |
| 4 | +downloads asynchronously using request logs from our Content Delivery Networks. |
| 5 | + |
| 6 | +Whenever a new archive with request logs is uploaded to S3, S3 pushes an event |
| 7 | +into a SQS queue. [crates.io] monitors the queue and processes incoming events. |
| 8 | +From the event, it can determine what files to fetch from S3, download and then |
| 9 | +parse them, and update the download counts in the database. |
| 10 | + |
| 11 | +```mermaid |
| 12 | +sequenceDiagram |
| 13 | + static.crates.io ->> S3: Uploads logs |
| 14 | + S3 ->> SQS: Queues event |
| 15 | + crates.io ->> SQS: Pulls event from queue |
| 16 | + crates.io ->> S3: Fetches new log file |
| 17 | + crates.io ->> crates.io: Parses log file |
| 18 | + crates.io ->> crates.io: Updates download counts |
| 19 | +``` |
| 20 | + |
| 21 | +See [rust-lang/simpleinfra#372] for a detailed discussion of the design. |
| 22 | + |
| 23 | +## AWS Accounts |
| 24 | + |
| 25 | +The infrastructure for [crates.io] has historically been deployed to the |
| 26 | +`legacy` AWS account. For this infrastructure, new accounts have been created |
| 27 | +that follow the new convention of specialized and isolated accounts for |
| 28 | +services. |
| 29 | + |
| 30 | +This requires the S3 bucket with the request logs in the `legacy` account to |
| 31 | +push events into the SQS queue in a different account. And the [crates.io] |
| 32 | +application needs a second set of AWS credentials to pull events from the |
| 33 | +queue. |
| 34 | + |
| 35 | +[crates.io]: https://crates.io |
| 36 | +[rust-lang/simpleinfra#372]: https://github.com/rust-lang/simpleinfra/issues/372 |
0 commit comments