Thursday 3 December 2009

Reassigning STDERR without warning messages

There are times when some library feels that it is necessary to print a warning you have no interest in to STDERR (the constant assigned to the stderr stream on Ruby). All well and good until you run cron jobs. Then you can an email each time. So lets reassign STDERR, do something that will try and write to STDERR and then put it back. Without generating any unnecessary messages.

# Get a copy of $stderr so we can restore it later
$stderr_backup = $stderr.dup

# Send everything for $stderr to /dev/null
$stderr.reopen("/dev/null", "w")

# Now do something that will write to $stderr or STDERR

# Restore the original $stderr. Note that this will result in a warning
# being sent to the existing $stderr which we will not see
STDERR = $stderr_backup.dup

# Now restore the global $stderr
$stderr = $stderr_backup.dup

Swap the last two lines round and you will get a warning message printed that you are reassigning a constant. Miss out the last line and STDERR.puts will write to the stderr stream but $stderr.puts will still be writing to /dev/null

Friday 27 November 2009

Crons for Rails

Well all the stuff that has been building up in this blog has now turned into a plugin for Rails that automates the installation and setup of crons for rails applications. It's available on GitHub at http://github.com/PeterHickman/Crons-for-Rails

Wednesday 21 October 2009

Creating a background task for Rails

First you will need somewhere to store your settings. Create a file called example.yml in the configdirectory and a file called example.rb in the config/initializers directory. A minimal example.yml file contains this

development:
pid_file: /var/run/example.pid
sleep_for: 10

# Same for production and test

The example.rb file should look like this

EXAMPLE = YAML.load_file("#{RAILS_ROOT}/config/example.yml")[RAILS_ENV]

You can now access all your settings from the global EXAMPLE hash. Now we need to create a script to run, the skeleton of which is stored in the scripts directory (using the same set_pid code from a previous post)
def set_pid(filename)
# Code from the previous post
end

logger = Logger.new("#{RAILS_ROOT}/log/example.log", 'daily')
logger.formatter = Logger::Formatter.new()

if set_pid(EXAMPLE['pid_file'])
logger.info "Example started, polling every #{EXAMPLE['sleep_for']} seconds"
else
logger.error "Example is already running"
exit
end

$continue = true

trap(:SIGTERM) { $continue = false }

while $continue
# Do your stuff here

sleep POLL['sleep_for']
end

logger.info "Shutting down"

Finally we need to make sure that the process is started up when the machine starts up we need this script (example._ctl) in /etc/init.d
#!/bin/sh

PID=/var/run/example.pid

# Find the name of the script
NAME=`basename $0`

script_result=0

start()
{
su --login example_user --command "cd /home/application/path/current ; /usr/bin/env ruby script/runner -e production script/example.rb &"

echo "Starting $NAME service: OK"
}

stop()
{
if [ -r $PID ]
then
kill `cat $PID`
STATUS="OK"
else
STATUS="FAILED"
script_result=1
fi

echo "Stopping $NAME service: $STATUS"
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
esac

exit $script_result

Wednesday 14 October 2009

Scraping a users Twitter

I needed to get all the urls from a user's tweets, this is what I came up with.

#!/usr/bin/env ruby

require 'rubygems'

require 'open-uri'
require 'json'
require 'hpricot'

def get_page(url)
x = open(url).read
y = JSON::parse(x)

doc = Hpricot(y['#timeline'])
doc.search('li').each do |l|
status = l.attributes['id']
l.search("span.entry-content/a.web").each do |c|
puts "#{status} = #{c.attributes['href']}"
end
end

doc = Hpricot(y['#pagination'])
return doc.search('a').first.attributes['href']
end

user = 'twitter-user'

next_page = get_page("http://twitter.com/#{user}?page=1&format=json")

while next_page
# So we don't get banned
sleep 1
next_page = get_page("http://twitter.com#{next_page}")
end

Tuesday 8 September 2009

Capturing stdout for tests in Ruby

I found this on Thinking Digitally that allows me to capture stdout and test that it is correct.
require 'stringio'

def capture_stdout
out = StringIO.new
$stdout = out
yield
return out
ensure
$stdout = STDOUT
end

Which can be used as follows
class ATest < Test::Unit::TestCase
def zxc(text, count)
count.times {puts text.upcase}
end

def test_stdout_capture
out = capture_stdout do
zxc('yes', 3)
end

assert_equal "YES\nYES\nYES\n", out.string
end
end

Monday 7 September 2009

Finding the bad parts in a log file

This little script makes things easier by extracting the errors out of a rails log file
#!/usr/bin/env ruby

text = ''
completed = false
total = 0
good = 0

ARGF.each do |l|
if l =~ /^Processing /
puts text if completed == false and text.size > 0
text = ''
completed = false
total += 1
elsif l =~ /^Completed /
completed = true
good += 1
end
text << l
end

puts text if completed == false and text.size > 0

$stderr.puts "Read #{total} transactions, #{total - good} were errors"

To use it you do this: striplog.rb development.log > development.log.errors. It will print out a summary line at the end to show you how many errors it found

More fun with pid files

Another way of handling pid files is a more ruby like way is:
#!/usr/bin/env ruby

class PidFile
def initialize(filename, &code)
unless set_pid(filename)
raise RuntimeError.new("The process is already running")
end

code.call

File.delete(filename)
end

private

def set_pid(filename)
# Same code as before
end
end

And call it thus
PidFile.new("/var/run/app.pid") do
puts "What we want to do"
end

Friday 4 September 2009

Checking for processes and pid files

If you have a periodic task that might still be running when you try to run it and don't want both copies of the program running then you need to create a pid file and check it when your process starts.
# Returns true if it was able to create the pid file

def set_pid(filename)
if File.exist?(filename)
File.open(filename, "r").each do |l|
old_pid = l.chomp.to_i

begin
Process.kill(0, old_pid)
# The process is running
return false
rescue Errno::EPERM
# The process is running and you don't have permission
return false
rescue
# the process is not running or something
end
end

File.delete(filename)
end

# Create the new process
f = File.new(filename, "w")
f.puts $$
f.close

return true
end

At the start of you program call it as follows
unless set_pid("/var/run/app.pid")
puts "This process is already running!"
exit
end

This is good for OS X and Linux, not sure if it will work for Windows. Probably will

Thursday 3 September 2009

A rake task to install crons

Assuming the cron tasks you want to run are stored in script/crons/{daily,hourly,monthly,weekly}/... this rake task will create the necessary crons on the system
namespace :crons do
desc "Install the crons we need to run this site"
task :install do
text = ''

template = "@%s (cd %s ; /usr/local/bin/ruby script/runner -e production script/crons/%s/%s)\n"

Dir["#{RAILS_ROOT}/script/crons/*/*"].each do |file|
if file =~ /crons\/daily/
text << template % ['daily', RAILS_ROOT, 'daily', File.basename(file)]
elsif file =~ /crons\/hourly/
text << template % ['hourly', RAILS_ROOT, 'hourly', File.basename(file)]
elsif file =~ /crons\/monthly/
text << template % ['monthly', RAILS_ROOT, 'monthly', File.basename(file)]
elsif file =~ /crons\/weekly/
text << template % ['weekly', RAILS_ROOT, 'weekly', File.basename(file)]
else
puts "Unknown file #{file}"
end
end

cron_file = "#{RAILS_ROOT}/script/crons/crontab.txt"

if File.exist?(cron_file)
File.delete(cron_file)
end

unless text == ''
f = File.new(cron_file, "w")
f.puts text
f.close
end
end

desc "Remove any previously installed crons"
task :remove do
cron_file = "#{RAILS_ROOT}/script/crons/crontab.txt"

if File.exist?(cron_file)
File.delete(cron_file)
end
end
end

To run it just go rake crons:install and it will create the crontabs as a text file. Then we can use the capistrano task to set them up
namespace :deploy do
namespace :crons do
desc "Installing the crons"
task :install, :roles => :db do
run "(cd #{deploy_to}/current ; rake crons:install)"
run "cat #{deploy_to}/current/script/crons/crontab.txt | crontab -"
end
desc "Remove the crons"
task :remove, :roles => :db do
run "crontab -r"
run "(cd #{deploy_to}/current ; rake crons:remove)"
end
end
end

You could probably do it all in the rake task if you wanted. You might want to set up some hooks to run this after a deploy

Purging sessions in rails

Keeping it simple we first create a script:

logger = Logger.new("#{RAILS_ROOT}/log/sessions_purge.log")
logger.formatter = Logger::Formatter.new()

older_than = Date.today - 1

logger.info "Starting to purge sessions older than #{older_than}"

counter = 0
ActiveRecord::SessionStore::Session.find_by_sql(["SELECT * FROM sessions WHERE updated_at < ?", older_than]).each do |r|
counter += 1
begin
r.destroy
rescue Exception => e
logger.error "Unable to remove #{r.key} #{e}"
end
end

logger.info "Removed #{counter} sessions"

Which needs to be run from the users crontab

@daily (cd /home/project ; /usr/local/bin/ruby script/runner -e production script/crons/session_purge.rb)

If ActiveRecord::SessionStore::Session doesn't work for you then create a model for the sessions, ruby script/generate model session, and replace it with Session

Wednesday 12 August 2009

It does grow on trees

Whilst it is true that money doesn't grow on trees it is also irrelevant. Money is only a way of exchanging wealth, the other being barter.

Wealth grows on trees. Imagine I own a field, the field can cost me say £10. We can say that the field is worth £10 or more correctly that I was prepared to pay £10. But just what is the field worth?
  • I could turn it into a car park and charge £1 a car a day
  • I could rent it to a farmer for his cows to graze in
  • I could plant a tree and sell the apple from it
The field may have cost me £10 but with no more outlay I can create more wealth than the £10 I spent on the field. Wealth is measured in money but wealth is not money.

The Web is not Print

When you design for print you get to choose the size, shape, weight and finish of the stock it will be printed on which means that you will know how your design will come out. Web is different...
  • You cannot control the size of the screen or its resolution. You have lost control of the layout and how the fonts will come out
  • You cannot assume the screen is correctly calibrated, you have lost control of the colours. Subtle colour differences are lost
  • You cannot assume the fonts are available or being rendered in the size you want, how the system is configured is unknown to you
As a designer you no longer have the same degree of control on the web that you had in print.

Friday 3 July 2009

...and in last place. Me!

There are only two programming languages that I have tried to get to grips with and failed. This is not complete failure but not being able to get much beyond the simple applications. Trying to build something bigger always fails big time.

The languages are Forth and Rebol.

God knows when I started to learn Forth, possibly the 80's, maybe earlier. I had a friend who wrote games in Forth. I could modify his code but sit me own with a blank screen and a spec and I was completely lost.

Rebol just plain confused me and I am no great fan of integrated environments (a la Smalltalk).

Oh well, can't win them all.

Monday 22 June 2009

A late news years resolution

I have decided to stop learning new programming languages unless I have a use for it that no other language will cope with. This year so far I have taught myself Objective C, Erlang and Haskell.

Not that I've actually had a use for them. I can get what I want done in C, C++, Java, Perl, Python, Ruby, Tcl/Tk or Lua (or any other language that I might know). Learning programming languages has been getting in the way of me writing code. Also no new technologies for the same reason.

My Amazon wish list is now three pages shorter.

My resolution is therefore to write the actual applications!

Interesting things are the bane of my life.

Wednesday 17 June 2009

"All this useless beauty"

There is a phrase "Photoshop designer" which I use to identify those designers who design websites in photoshop which the client then signs off without it ever passing the desk of the person who has to code it. Now I know that it is possible to go from a psd file to nice clean html. There are even companies who do nothing but that.

But why all this pain?
  • What font is that?
  • What is the hex for that colour?
  • Spacing and paddings, are they in ems, px or %?
  • How does the page behave when resized?
  • How does the page behave when there is real data in there?

Imagine that an architect delivered his design for a building as a couple of water colours and 1/50th scale model. Sure you could reverse engineer quite a bit from that but the architect is on your side so he will also give you blueprints and a list of materials.

The "Photoshop designer" on the other hand is your enemy, why else would he give you a jpeg full of "Lorem ipsum dolor sit amet, consectetur adipisicing elit..." and expect you to build a website.

Because he hates you.

Tuesday 16 June 2009

Looking good, feeling bad

What do you feel about the following line of output?

20 tests, 20 assertions, 0 failures, 0 errors

Happy, sad or nervous?

Well Ruby helpfully generates the scaffolding for tests whenever you create a new controller. They look like this:

require 'test_helper'

class HomeControllerTest < ActionController::TestCase
# Replace this with your real tests.
test "the truth" do
assert true
end
end

So I have just tested that true does really equal true, twenty times. That warm fuzzy feeling is changing into a nervous tick. I propose a new rule, tests should actually test something or delete them.