Extracting timings from Rails logs

Just a little tool to extract the timings for requests from the logs

#!/usr/bin/env ruby

ip_address = nil
datetime = nil

ARGV.each do |filename|
  File.open(filename, "r").each do |line|
    if line =~ /^Processing .*for ([\d\.]+) at (\d\d\d\d\-\d\d\-\d\d \d\d:\d\d:\d\d)/
      ip_address = $1
      datetime = $2
    elsif line =~ /^Completed in (\d+)ms .View: (\d+), DB: (\d+).*http(.*)\]/
      puts "#{ip_address}\t#{datetime}\t#{$1}\t#{$2}\t#{$3}\thttp#{$4}"
    end
  end
end

Give it bunch of logs on the command line and it will output tab separated lines such as:

1.1.1.1 2010-04-05 04:03:03 7 1 4 http://www.example.com/fred

Where the values are as follows:

1. The ip from which the request was made


2. The data and time of the request


3. The overall processing time in ms


4. The ms timing for the view


5. The ms timing for the db


6. The requested url

Gamers and Programmers

I am a gamer. That is to say that for entertainment I play computer games the way other people watch TV or go to the cinema, I don't watch TV or go to the cinema.

I am also a programmer which means that the first two or three months of playing a new game involves trying to create a mental model of how the software works. More importantly I try to work out how I would create my own game. After returning to playing Civilisation IV I ended up with a half built system for a similar game.

I have recently started to play visual novels and I want to write my own. Which of course means writing my own engine :)

Which of course will have to be cross platform (including my iPod Touch) and I suspect will be in C++.

I already have a good idea as to where 2010 is going to go...

Reassigning STDERR without warning messages

There are times when some library feels that it is necessary to print a warning you have no interest in to STDERR (the constant assigned to the stderr stream on Ruby). All well and good until you run cron jobs. Then you can an email each time. So lets reassign STDERR, do something that will try and write to STDERR and then put it back. Without generating any unnecessary messages.

# Get a copy of $stderr so we can restore it later
$stderr_backup = $stderr.dup

# Send everything for $stderr to /dev/null
$stderr.reopen("/dev/null", "w")

# Now do something that will write to $stderr or STDERR

# Restore the original $stderr. Note that this will result in a warning
# being sent to the existing $stderr which we will not see
STDERR = $stderr_backup.dup

# Now restore the global $stderr
$stderr = $stderr_backup.dup

Swap the last two lines round and you will get a warning message printed that you are reassigning a constant. Miss out the last line and STDERR.puts will write to the stderr stream but $stderr.puts will still be writing to /dev/null

Crons for Rails

Well all the stuff that has been building up in this blog has now turned into a plugin for Rails that automates the installation and setup of crons for rails applications. It's available on GitHub at http://github.com/PeterHickman/Crons-for-Rails

Creating a background task for Rails

First you will need somewhere to store your settings. Create a file called example.yml in the configdirectory and a file called example.rb in the config/initializers directory. A minimal example.yml file contains this

development:
    pid_file: /var/run/example.pid
    sleep_for: 10

# Same for production and test

The example.rb file should look like this

EXAMPLE = YAML.load_file("#{RAILS_ROOT}/config/example.yml")[RAILS_ENV]

You can now access all your settings from the global EXAMPLE hash. Now we need to create a script to run, the skeleton of which is stored in the scripts directory (using the same set_pid code from a previous post)

def set_pid(filename)
  # Code from the previous post
end

logger = Logger.new("#{RAILS_ROOT}/log/example.log", 'daily')
logger.formatter = Logger::Formatter.new()

if set_pid(EXAMPLE['pid_file'])
  logger.info "Example started, polling every #{EXAMPLE['sleep_for']} seconds"
else
  logger.error "Example is already running"
  exit
end

$continue = true

trap(:SIGTERM) { $continue = false }

while $continue
  # Do your stuff here

  sleep POLL['sleep_for']
end

logger.info "Shutting down"

Finally we need to make sure that the process is started up when the machine starts up we need this script (example._ctl) in /etc/init.d

#!/bin/sh

PID=/var/run/example.pid

# Find the name of the script
NAME=`basename $0`

script_result=0

start()
{
    su --login example_user --command "cd /home/application/path/current ; /usr/bin/env ruby script/runner -e production script/example.rb &"

    echo "Starting $NAME service: OK"
}

stop()
{
    if [ -r $PID ]
    then
        kill `cat $PID`
        STATUS="OK"
    else
        STATUS="FAILED"
        script_result=1
    fi

    echo "Stopping $NAME service: $STATUS"
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac

exit $script_result

Scraping a users Twitter

I needed to get all the urls from a user's tweets, this is what I came up with.

#!/usr/bin/env ruby

require 'rubygems'

require 'open-uri'
require 'json'
require 'hpricot'

def get_page(url)
  x = open(url).read
  y = JSON::parse(x)

  doc = Hpricot(y['#timeline'])
  doc.search('li').each do |l|
    status = l.attributes['id']
    l.search("span.entry-content/a.web").each do |c| 
      puts "#{status} = #{c.attributes['href']}"
    end
  end

  doc = Hpricot(y['#pagination'])
  return doc.search('a').first.attributes['href']
end

user = 'twitter-user'

next_page = get_page("http://twitter.com/#{user}?page=1&format=json")

while next_page
  # So we don't get banned
  sleep 1
  next_page = get_page("http://twitter.com#{next_page}")
end

Capturing stdout for tests in Ruby

I found this on Thinking Digitally that allows me to capture stdout and test that it is correct.

require 'stringio'

def capture_stdout
  out = StringIO.new
  $stdout = out
  yield
  return out
ensure
  $stdout = STDOUT
end

Which can be used as follows

class ATest < Test::Unit::TestCase
  def zxc(text, count)
    count.times {puts text.upcase}
  end

  def test_stdout_capture
     out = capture_stdout do
        zxc('yes', 3)
     end

     assert_equal "YES\nYES\nYES\n", out.string
  end
end