Skip to content

Commit 3a62e47

Browse files
committed
Created HOWITWORKS; cleanup of README
Addresses a request from @chrisboulton in GitHub issue chrisboulton#149 Slight grammar cleanup and content update in README.md Mention of HOWITWORKS.md in README.md, referring those who want to know more that direction Expanded and slightly cleaner version of a comment made in chrisboulton#149 that prompted this commit/PR was placed in HOWITWORKS.md Signed-off-by: Daniel Hunsaker <[email protected]>
1 parent 02809d6 commit 3a62e47

File tree

2 files changed

+243
-43
lines changed

2 files changed

+243
-43
lines changed

HOWITWORKS.md

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
*For an overview of how to __use__ php-resque, see `README.md`.*
2+
3+
The following is a step-by-step breakdown of how php-resque operates.
4+
5+
## Enqueue Job ##
6+
7+
What happens when you call `Resque::enqueue()`?
8+
9+
1. `Resque::enqueue()` calls `Resque_Job::create()` with the same arguments it
10+
received.
11+
2. `Resque_Job::create()` checks that your `$args` (the third argument) are
12+
either `null` or in an array
13+
3. `Resque_Job::create()` generates a job ID (a "token" in most of the docs)
14+
4. `Resque_Job::create()` pushes the job to the requested queue (first
15+
argument)
16+
5. `Resque_Job::create()`, if status monitoring is enabled for the job (fourth
17+
argument), calls `Resque_Job_Status::create()` with the job ID as its only
18+
argument
19+
6. `Resque_Job_Status::create()` creates a key in Redis with the job ID in its
20+
name, and the current status (as well as a couple of timestamps) as its
21+
value, then returns control to `Resque_Job::create()`
22+
7. `Resque_Job::create()` returns control to `Resque::enqueue()`, with the job
23+
ID as a return value
24+
8. `Resque::enqueue()` triggers the `afterEnqueue` event, then returns control
25+
to your application, again with the job ID as its return value
26+
27+
## Workers At Work ##
28+
29+
How do the workers process the queues?
30+
31+
1. `Resque_Worker::work()`, the main loop of the worker process, calls
32+
`Resque_Worker->reserve()` to check for a job
33+
2. `Resque_Worker->reserve()` checks whether to use blocking pops or not (from
34+
`BLOCKING`), then acts accordingly:
35+
* Blocking Pop
36+
1. `Resque_Worker->reserve()` calls `Resque_Job::reserveBlocking()` with
37+
the entire queue list and the timeout (from `INTERVAL`) as arguments
38+
2. `Resque_Job::reserveBlocking()` calls `Resque::blpop()` (which in turn
39+
calls Redis' `blpop`, after prepping the queue list for the call, then
40+
processes the response for consistency with other aspects of the
41+
library, before finally returning control [and the queue/content of the
42+
retrieved job, if any] to `Resque_Job::reserveBlocking()`)
43+
3. `Resque_Job::reserveBlocking()` checks whether the job content is an
44+
array (it should contain the job's type [class], payload [args], and
45+
ID), and aborts processing if not
46+
4. `Resque_Job::reserveBlocking()` creates a new `Resque_Job` object with
47+
the queue and content as constructor arguments to initialize the job
48+
itself, and returns it, along with control of the process, to
49+
`Resque_Worker->reserve()`
50+
* Queue Polling
51+
1. `Resque_Worker->reserve()` iterates through the queue list, calling
52+
`Resque_Job::reserve()` with the current queue's name as the sole
53+
argument on each pass
54+
2. `Resque_Job::reserve()` passes the queue name on to `Resque::pop()`,
55+
which in turn calls Redis' `lpop` with the same argument, then returns
56+
control (and the job content, if any) to `Resque_Job::reserve()`
57+
3. `Resque_Job::reserve()` checks whether the job content is an array (as
58+
before, it should contain the job's type [class], payload [args], and
59+
ID), and aborts processing if not
60+
4. `Resque_Job::reserve()` creates a new `Resque_Job` object in the same
61+
manner as above, and also returns this object (along with control of
62+
the process) to `Resque_Worker->reserve()`
63+
3. In either case, `Resque_Worker->reserve()` returns the new `Resque_Job`
64+
object, along with control, up to `Resque_Worker::work()`; if no job is
65+
found, it simply returns `FALSE`
66+
* No Jobs
67+
1. If blocking mode is not enabled, `Resque_Worker::work()` sleeps for
68+
`INTERVAL` seconds; it calls `usleep()` for this, so fractional seconds
69+
*are* supported
70+
* Job Reserved
71+
1. `Resque_Worker::work()` triggers a `beforeFork` event
72+
2. `Resque_Worker::work()` calls `Resque_Worker->workingOn()` with the new
73+
`Resque_Job` object as its argument
74+
3. `Resque_Worker->workingOn()` does some reference assignments to help keep
75+
track of the worker/job relationship, then updates the job status from
76+
`WAITING` to `RUNNING`
77+
4. `Resque_Worker->workingOn()` stores the new `Resque_Job` object's payload
78+
in a Redis key associated to the worker itself (this is to prevent the job
79+
from being lost indefinitely, but does rely on that PID never being
80+
allocated on that host to a different worker process), then returns control
81+
to `Resque_Worker::work()`
82+
5. `Resque_Worker::work()` forks a child process to run the actual `perform()`
83+
6. The next steps differ between the worker and the child, now running in
84+
separate processes:
85+
* Worker
86+
1. The worker waits for the job process to complete
87+
2. If the exit status is not 0, the worker calls `Resque_Job->fail()` with
88+
a `Resque_Job_DirtyExitException` as its only argument.
89+
3. `Resque_Job->fail()` triggers an `onFailure` event
90+
4. `Resque_Job->fail()` updates the job status from `RUNNING` to `FAILED`
91+
5. `Resque_Job->fail()` calls `Resque_Failure::create()` with the job
92+
payload, the `Resque_Job_DirtyExitException`, the internal ID of the
93+
worker, and the queue name as arguments
94+
6. `Resque_Failure::create()` creates a new object of whatever type has
95+
been set as the `Resque_Failure` "backend" handler; by default, this is
96+
a `Resque_Failure_Redis` object, whose constructor simply collects the
97+
data passed into `Resque_Failure::create()` and pushes it into Redis
98+
in the `failed` queue
99+
7. `Resque_Job->fail()` increments two failure counters in Redis: one for
100+
a total count, and one for the worker
101+
8. `Resque_Job->fail()` returns control to the worker (still in
102+
`Resque_Worker::work()`) without a value
103+
* Job
104+
1. The job calls `Resque_Worker->perform()` with the `Resque_Job` as its
105+
only argument.
106+
2. `Resque_Worker->perform()` sets up a `try...catch` block so it can
107+
properly handle exceptions by marking jobs as failed (by calling
108+
`Resque_Job->fail()`, as above)
109+
3. Inside the `try...catch`, `Resque_Worker->perform()` triggers an
110+
`afterFork` event
111+
4. Still inside the `try...catch`, `Resque_Worker->perform()` calls
112+
`Resque_Job->perform()` with no arguments
113+
5. `Resque_Job->perform()` calls `Resque_Job->getInstance()` with no
114+
arguments
115+
6. If `Resque_Job->getInstance()` has already been called, it returns the
116+
existing instance; otherwise:
117+
7. `Resque_Job->getInstance()` checks that the job's class (type) exists
118+
and has a `perform()` method; if not, in either case, it throws an
119+
exception which will be caught by `Resque_Worker->perform()`
120+
8. `Resque_Job->getInstance()` creates an instance of the job's class, and
121+
initializes it with a reference to the `Resque_Job` itself, the job's
122+
arguments (which it gets by calling `Resque_Job->getArguments()`, which
123+
in turn simply returns the value of `args[0]`, or an empty array if no
124+
arguments were passed), and the queue name
125+
9. `Resque_Job->getInstance()` returns control, along with the job class
126+
instance, to `Resque_Job->perform()`
127+
10. `Resque_Job->perform()` sets up its own `try...catch` block to handle
128+
`Resque_Job_DontPerform` exceptions; any other exceptions are passed
129+
up to `Resque_Worker->perform()`
130+
11. `Resque_Job->perform()` triggers a `beforePerform` event
131+
12. `Resque_Job->perform()` calls `setUp()` on the instance, if it exists
132+
13. `Resque_Job->perform()` calls `perform()` on the instance
133+
14. `Resque_Job->perform()` calls `tearDown()` on the instance, if it
134+
exists
135+
15. `Resque_Job->perform()` triggers an `afterPerform` event
136+
16. The `try...catch` block ends, suppressing `Resque_Job_DontPerform`
137+
exceptions by returning control, and the value `FALSE`, to
138+
`Resque_Worker->perform()`; any other situation returns the value
139+
`TRUE` along with control, instead
140+
17. The `try...catch` block in `Resque_Worker->perform()` ends
141+
18. `Resque_Worker->perform()` updates the job status from `RUNNING` to
142+
`COMPLETE`, then returns control, with no value, to the worker (again
143+
still in `Resque_Worker::work()`)
144+
19. `Resque_Worker::work()` calls `exit(0)` to terminate the job process
145+
cleanly
146+
* SPECIAL CASE: Non-forking OS (Windows)
147+
1. Same as the job above, except it doesn't call `exit(0)` when done
148+
7. `Resque_Worker::work()` calls `Resque_Worker->doneWorking()` with no
149+
arguments
150+
8. `Resque_Worker->doneWorking()` increments two processed counters in Redis:
151+
one for a total count, and one for the worker
152+
9. `Resque_Worker->doneWorking()` deletes the Redis key set in
153+
`Resque_Worker->workingOn()`, then returns control, with no value, to
154+
`Resque_Worker::work()`
155+
4. `Resque_Worker::work()` returns control to the beginning of the main loop,
156+
where it will wait for the next job to become available, and start this
157+
process all over again

0 commit comments

Comments
 (0)