Skip to content

Commit a938704

Browse files
committed
Add README to crates-io-logs module
1 parent 7065ff6 commit a938704

File tree

1 file changed

+36
-0
lines changed

1 file changed

+36
-0
lines changed
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Infrastructure to Count Crate Downloads
2+
3+
This module creates the infrastructure that enables [crates.io] to count crate
4+
downloads asynchronously using request logs from our Content Delivery Networks.
5+
6+
Whenever a new archive with request logs is uploaded to S3, S3 pushes an event
7+
into a SQS queue. [crates.io] monitors the queue and processes incoming events.
8+
From the event, it can determine what files to fetch from S3, download and then
9+
parse them, and update the download counts in the database.
10+
11+
```mermaid
12+
sequenceDiagram
13+
static.crates.io ->> S3: Uploads logs
14+
S3 ->> SQS: Queues event
15+
crates.io ->> SQS: Pulls event from queue
16+
crates.io ->> S3: Fetches new log file
17+
crates.io ->> crates.io: Parses log file
18+
crates.io ->> crates.io: Updates download counts
19+
```
20+
21+
See [rust-lang/simpleinfra#372] for a detailed discussion of the design.
22+
23+
## AWS Accounts
24+
25+
The infrastructure for [crates.io] has historically been deployed to the
26+
`legacy` AWS account. For this infrastructure, new accounts have been created
27+
that follow the new convention of specialized and isolated accounts for
28+
services.
29+
30+
This requires the S3 bucket with the request logs in the `legacy` account to
31+
push events into the SQS queue in a different account. And the [crates.io]
32+
application needs a second set of AWS credentials to pull events from the
33+
queue.
34+
35+
[crates.io]: https://crates.io
36+
[rust-lang/simpleinfra#372]: https://github.com/rust-lang/simpleinfra/issues/372

0 commit comments

Comments
 (0)