Creating a background task for Rails

First you will need somewhere to store your settings. Create a file called example.yml in the configdirectory and a file called example.rb in the config/initializers directory. A minimal example.yml file contains this

development:
    pid_file: /var/run/example.pid
    sleep_for: 10

# Same for production and test

The example.rb file should look like this

EXAMPLE = YAML.load_file("#{RAILS_ROOT}/config/example.yml")[RAILS_ENV]

You can now access all your settings from the global EXAMPLE hash. Now we need to create a script to run, the skeleton of which is stored in the scripts directory (using the same set_pid code from a previous post)

def set_pid(filename)
  # Code from the previous post
end

logger = Logger.new("#{RAILS_ROOT}/log/example.log", 'daily')
logger.formatter = Logger::Formatter.new()

if set_pid(EXAMPLE['pid_file'])
  logger.info "Example started, polling every #{EXAMPLE['sleep_for']} seconds"
else
  logger.error "Example is already running"
  exit
end

$continue = true

trap(:SIGTERM) { $continue = false }

while $continue
  # Do your stuff here

  sleep POLL['sleep_for']
end

logger.info "Shutting down"

Finally we need to make sure that the process is started up when the machine starts up we need this script (example._ctl) in /etc/init.d

#!/bin/sh

PID=/var/run/example.pid

# Find the name of the script
NAME=`basename $0`

script_result=0

start()
{
    su --login example_user --command "cd /home/application/path/current ; /usr/bin/env ruby script/runner -e production script/example.rb &"

    echo "Starting $NAME service: OK"
}

stop()
{
    if [ -r $PID ]
    then
        kill `cat $PID`
        STATUS="OK"
    else
        STATUS="FAILED"
        script_result=1
    fi

    echo "Stopping $NAME service: $STATUS"
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac

exit $script_result

Scraping a users Twitter

I needed to get all the urls from a user's tweets, this is what I came up with.

#!/usr/bin/env ruby

require 'rubygems'

require 'open-uri'
require 'json'
require 'hpricot'

def get_page(url)
  x = open(url).read
  y = JSON::parse(x)

  doc = Hpricot(y['#timeline'])
  doc.search('li').each do |l|
    status = l.attributes['id']
    l.search("span.entry-content/a.web").each do |c| 
      puts "#{status} = #{c.attributes['href']}"
    end
  end

  doc = Hpricot(y['#pagination'])
  return doc.search('a').first.attributes['href']
end

user = 'twitter-user'

next_page = get_page("http://twitter.com/#{user}?page=1&format=json")

while next_page
  # So we don't get banned
  sleep 1
  next_page = get_page("http://twitter.com#{next_page}")
end

Capturing stdout for tests in Ruby

I found this on Thinking Digitally that allows me to capture stdout and test that it is correct.

require 'stringio'

def capture_stdout
  out = StringIO.new
  $stdout = out
  yield
  return out
ensure
  $stdout = STDOUT
end

Which can be used as follows

class ATest < Test::Unit::TestCase
  def zxc(text, count)
    count.times {puts text.upcase}
  end

  def test_stdout_capture
     out = capture_stdout do
        zxc('yes', 3)
     end

     assert_equal "YES\nYES\nYES\n", out.string
  end
end

Finding the bad parts in a log file

This little script makes things easier by extracting the errors out of a rails log file

#!/usr/bin/env ruby

text = ''
completed = false
total = 0
good = 0

ARGF.each do |l|
 if l =~ /^Processing /
  puts text if completed == false and text.size > 0
  text = ''
  completed = false
  total += 1
 elsif l =~ /^Completed /
  completed = true
  good += 1
 end
 text << l
end

puts text if completed == false and text.size > 0

$stderr.puts "Read #{total} transactions, #{total - good} were errors"

To use it you do this: striplog.rb development.log > development.log.errors. It will print out a summary line at the end to show you how many errors it found

More fun with pid files

Another way of handling pid files is a more ruby like way is:

#!/usr/bin/env ruby

class PidFile
  def initialize(filename, &code)
    unless set_pid(filename)
      raise RuntimeError.new("The process is already running")
    end

    code.call

    File.delete(filename)
  end

  private

  def set_pid(filename)
    # Same code as before
  end
end

And call it thus

PidFile.new("/var/run/app.pid") do
  puts "What we want to do"
end

Checking for processes and pid files

If you have a periodic task that might still be running when you try to run it and don't want both copies of the program running then you need to create a pid file and check it when your process starts.

# Returns true if it was able to create the pid file

def set_pid(filename)
  if File.exist?(filename)
    File.open(filename, "r").each do |l|
      old_pid = l.chomp.to_i

      begin
        Process.kill(0, old_pid)
        # The process is running
        return false
      rescue Errno::EPERM
        # The process is running and you don't have permission
        return false
      rescue
        # the process is not running or something
      end
    end

    File.delete(filename)
  end

  # Create the new process
  f = File.new(filename, "w")
  f.puts $$
  f.close

  return true
end

At the start of you program call it as follows

unless set_pid("/var/run/app.pid")
  puts "This process is already running!"
  exit
end

This is good for OS X and Linux, not sure if it will work for Windows. Probably will

A rake task to install crons

Assuming the cron tasks you want to run are stored in script/crons/{daily,hourly,monthly,weekly}/... this rake task will create the necessary crons on the system

namespace :crons do
  desc "Install the crons we need to run this site"
  task :install do
    text = ''

    template = "@%s (cd %s ; /usr/local/bin/ruby script/runner -e production script/crons/%s/%s)\n"

    Dir["#{RAILS_ROOT}/script/crons/*/*"].each do |file|
      if file =~ /crons\/daily/
        text << template % ['daily', RAILS_ROOT, 'daily', File.basename(file)]
      elsif file =~ /crons\/hourly/
        text << template % ['hourly', RAILS_ROOT, 'hourly', File.basename(file)]
      elsif file =~ /crons\/monthly/
        text << template % ['monthly', RAILS_ROOT, 'monthly', File.basename(file)]
      elsif file =~ /crons\/weekly/
        text << template % ['weekly', RAILS_ROOT, 'weekly', File.basename(file)]
      else
        puts "Unknown file #{file}"
      end
    end

    cron_file = "#{RAILS_ROOT}/script/crons/crontab.txt"

    if File.exist?(cron_file)
      File.delete(cron_file)
    end

    unless text == ''
      f = File.new(cron_file, "w")
      f.puts text
      f.close
    end
  end

  desc "Remove any previously installed crons"
  task :remove do
    cron_file = "#{RAILS_ROOT}/script/crons/crontab.txt"

    if File.exist?(cron_file)
      File.delete(cron_file)
    end
  end
end

To run it just go rake crons:install and it will create the crontabs as a text file. Then we can use the capistrano task to set them up

namespace :deploy do
  namespace :crons do
    desc "Installing the crons"
    task :install, :roles => :db do
      run "(cd #{deploy_to}/current ; rake crons:install)"
      run "cat #{deploy_to}/current/script/crons/crontab.txt | crontab -"
    end
    desc "Remove the crons"
    task :remove, :roles => :db do
      run "crontab -r"
      run "(cd #{deploy_to}/current ; rake crons:remove)"
    end
  end
end

You could probably do it all in the rake task if you wanted. You might want to set up some hooks to run this after a deploy