Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added possibility to dump object space to file before kill for memory limit #42

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ This module automatically restarts the Unicorn workers, based on the number of r

If `verbose` is set to true, then after every request, your log will show the requests left before restart. This logging is done at the `info` level.

### Unicorn::WorkerKiller::Oom(memory_limit_min=(1024**3), memory_limit_max=(2*(1024**3)), check_cycle = 16, verbose = false)
### Unicorn::WorkerKiller::Oom(memory_limit_min=(1024**3), memory_limit_max=(2*(1024**3)), check_cycle = 16, verbose = false, object_space_dump = false)

This module automatically restarts the Unicorn workers, based on its memory size.

Expand All @@ -43,6 +43,8 @@ The memory size check is done in every `check_cycle` requests.

If `verbose` is set to true, then every memory size check will be shown in your logs. This logging is done at the `info` level.

If `object_space_dump` is set to true, before killing a process for memory reason, memory object space is dumped to file. By default the file name is `/tmp/unicorn_worker_$PID.json` and can be changed in `Unicorn::WorkerKiller::Configuration` class. Dump file is generated through `ObjectSpace.dump_all` that is available from Ruby 2.1

# Special Thanks

- [@hotchpotch](http://github.com/hotchpotch/) for the [original idea](https://gist.github.com/hotchpotch/1258681)
Expand Down
4 changes: 3 additions & 1 deletion lib/unicorn/configuration.rb
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
module Unicorn::WorkerKiller
class Configuration
attr_accessor :max_quit, :max_term, :sleep_interval
attr_accessor :max_quit, :max_term, :sleep_interval, :object_space_dump_file, :object_space_dump_lock

def initialize
self.max_quit = 10
self.max_term = 15
self.sleep_interval = 1
self.object_space_dump_file = '/tmp/unicorn_worker_$PID.json'
self.object_space_dump_lock = '/tmp/unicorn_worker_dump.lock'
end
end
end
35 changes: 31 additions & 4 deletions lib/unicorn/worker_killer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ class << self
# send TERM signals until `configuration.max_term`. Finally, send a KILL
# signal. A single signal is sent per request.
# @see http://unicorn.bogomips.org/SIGNALS.html
def self.kill_self(logger, start_time)
def self.kill_self(logger, start_time, first_signal=:QUIT)
alive_sec = (Time.now - start_time).round
worker_pid = Process.pid

@@kill_attempts ||= 0
@@kill_attempts += 1

sig = :QUIT
sig = first_signal
sig = :TERM if @@kill_attempts > configuration.max_quit
sig = :KILL if @@kill_attempts > configuration.max_term

Expand All @@ -32,14 +32,19 @@ module Oom
# affect the request.
#
# @see https://github.com/defunkt/unicorn/blob/master/lib/unicorn/oob_gc.rb#L40
def self.new(app, memory_limit_min = (1024**3), memory_limit_max = (2*(1024**3)), check_cycle = 16, verbose = false)
def self.new(app, memory_limit_min = (1024**3), memory_limit_max = (2*(1024**3)), check_cycle = 16, verbose = false, object_space_dump=false)
ObjectSpace.each_object(Unicorn::HttpServer) do |s|
s.extend(self)
s.instance_variable_set(:@_worker_memory_limit_min, memory_limit_min)
s.instance_variable_set(:@_worker_memory_limit_max, memory_limit_max)
s.instance_variable_set(:@_worker_check_cycle, check_cycle)
s.instance_variable_set(:@_worker_check_count, 0)
s.instance_variable_set(:@_verbose, verbose)
s.instance_variable_set(:@_object_space_dump, object_space_dump)
end
if object_space_dump
require 'objspace'
trap_sigabrt
end
app # pretend to be Rack middleware since it was in the past
end
Expand All @@ -60,11 +65,33 @@ def process_client(client)
logger.info "#{self}: worker (pid: #{Process.pid}) using #{rss} bytes." if @_verbose
if rss > @_worker_memory_limit
logger.warn "#{self}: worker (pid: #{Process.pid}) exceeds memory limit (#{rss} bytes > #{@_worker_memory_limit} bytes)"
Unicorn::WorkerKiller.kill_self(logger, @_worker_process_start)
first_signal = @_object_space_dump ? :SIGABRT : :QUIT
Unicorn::WorkerKiller.kill_self(logger, @_worker_process_start, first_signal)
end
@_worker_check_count = 0
end
end

def self.trap_sigabrt
trap(:SIGABRT) do
if lock_for_dump
dump_file = Unicorn::WorkerKiller.configuration.object_space_dump_file.gsub(/\$PID/, Process.pid.to_s)
GC.start
ObjectSpace.dump_all(output: File.open(dump_file, 'w'))
release_lock_for_dump
end
Process.kill(:QUIT, Process.pid)
end
end

def self.lock_for_dump
@lock_for_dump_file = File.open(Unicorn::WorkerKiller.configuration.object_space_dump_lock, File::RDWR|File::CREAT, 0644)
@lock_for_dump_file.flock(File::LOCK_NB|File::LOCK_EX)
end

def self.release_lock_for_dump
@lock_for_dump_file.close
end
end

module MaxRequests
Expand Down