eightbitraptor.com Rawr.

On my way to Wales...

Android Development without Eclipse

I started having a bash at some Android programming recently, nothing amazing, just flexing the old Java muscles again (for the first time since university, back when you had to avoid the Velociraptors to get to lectures), and it’s actually pretty fun.

What’s not fun however is the behemoth that is Eclipse. I swear, before I got fed up and threw it away, I spent longer bashing my head at Eclipse than I actually spent setting up the toolchain, writing some code and getting it on my phone.

Whilst the Android developer website is amazingly helpful I thought it might help to have a quick reference to get started. And the most important step is to download the sdk from here, unzip this to a directory where it isn’t going to get in your way, I use ~/code/android-sdk and add ~/code/android-sdk/tools to your path.

The next important thing is to install the SDK components for whatever android version you’re using and create an Android Virtual Device to use in the emulator. This is all covered in the Android install guides and when your done you should be good to go at creating a project.

The basic workflow is this:

Note: If you are reinstalling a new version of your app you’ll need to add the -r flag to android install otherwise it will whinge that the app apready exists.

The steps to getting your app on an actual phone are pretty much the same however:

and breath a breath of fresh air at being able to use $EDITOR to make cool things again.

posted on July 27 2010, and tagged with android programming java cli

RSS Awesomeness, Creating feeds with Sinatra & Builder

So I was reading the Design Monkey blog this evening and suddenly a vast sense of inadequacy welled up inside me. Was it because I had such vastly inferior design skills? Or maybe far less comical writing ability?

Hell No.

The design monkey provides that most awesome of things (as I’m sure do many others), an RSS feed of his eloquent prose, so not to be outdone and armed with the best of tools, I set out creating one with Ruby, builder and Sinatra

This turned out to take about 10 minutes and be amazingly less impressive than it sounds but here goes anyway. First you should create a builder template for your RSS feed. Mine looks like this:

  xml.instruct! :xml, :version => "1.0"
  xml.rss :version => "2.0" do
    xml.channel do
      xml.title "eightbitraptor"
      xml.description "The personal blog of developer, music lover and recovering sysadmin Matt House"
      xml.link "http://eightbitraptor.com/posts"

      for post in locals[:posts]
        xml.item do
          xml.title post.title
          xml.description to_html(post.body)
          xml.pubDate pretty_date(post.published)
          xml.link post_url(post)
        end
      end
    end
  end

This is saved in my views folder as feed.builder so that in my main Application I can define my route like so:

  get '/feed.xml' do
    builder :feed, :locals => { :posts => Post.all }
  end

Where a Post class does all the magical awesome (but is essentially just a way of grouping together titles and bodies as referenced in the builder template).

And that is pretty much all you need to do to generate functional rss feeds with Sinatra. If you want to dig deeper, especially into what my mystical Post object is, then the code is on github as per usual.

Get the source

posted on July 26 2010, and tagged with ruby programming

Running MPD on OSX

After finally getting fed up and giving iTunes the boot, I got round to making MPD work on my Mac. and unfortunately, apt-get install it ain’t!

First step is to actually get hold of and install mpd, if you’re using sensible and using homebrew that’s as easy as:

brew install mpd

Which will pull in all of the required dependancies and compile them all for you. Then comes the mpd config file. This is all pretty standard stuff, you can adapt from the standard and massively verbose example included with the mpd sources. Mine lives at /usr/local/Cellar/mpd/0.15.9/share/doc/mpd/mpdconf.example. The stuff you need to care about is:

music_directory
playlist_directory
log_file
db_file
pid_file

mixer_type "software"

Make sure these paths are all writeable by the user that you intend to run mpd as. In my case, I run mpd as the mpd user, and I made the mpd user and my normal user account members of group mpd.

What this amounts to is a music and playlist directory that the mpd user can read from and that I can add songs to. If you run a multi user system it’s probably a good idea to put this somewhere outside of your home dir.

A special point regarding the mixer_type line: I have found this necessary when running on Snow Leopard to avoid mplayer crashing hard when trying to skip playing tracks, but as is normal with these things YMMV.

Once this has been set up you should be able to start mpd with

mpd --create-db

and watch it chug away for a while depending on how much music you have.

Client

I use the excellent Theremin, which is an OSX native MPD client and does the job admirably. If that’s not your style there are an excellent array of decent mpd clients out there.

Last.fm

Last fm Scrobbling is acheived by the use of the lastfmsubmitd daemon, and it’s built in client lastmp. It’s dead easy to set up. Clone the sources from Github and follow the instructions in the INSTALL file. The client scrobbler lives inside the contrib folder of the checkout.

I installed lastfmsubmitd to /usr/local/bin and created it’s config file, and then simply copied the contrib/lastmp script to /usr/local/bin.

One gotcha if you’re not familiar with running Python stuff (I’m not) is that lastmp will bail out complaining it can’t import libmpdclient2. this is easily fixed with:

easy_install py-libmpdclient2@

which will ramble on about installing eggs, I guess these are pythons equivalent of gems.

Both of these daemons apparently need to be running to actually make scrobbling happen so I normally wrap these up in /usr/local/bin/music_starter, which looks like

  #! /bin/sh
  /usr/local/bin/mpd && \
  /usr/local/bin/lastfmsubmitd && \
  /usr/local/bin/lastmp

Tying it all together

You can start the whole kit and caboodle on boot by creating the following plist file and adding it to launchctl:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Disabled</key>
	<false/>
	<key>Label</key>
	<string>com.eightbitraptor.mpd</string>
	<key>Program</key>
  <string>/usr/local/bin/music_starter</string>
</dict>
</plist>

I put this in /Library/LaunchDaemons/com.eightbitraptor.mpd.plist. Add it to launchd and start it like this:

sudo launchctl load -w /Library/LaunchDaemons/com.eightbitraptor.mpd.plist
sudo launchctl start com.eightbitraptor.mpd.plist

And Job done! Now you too can get rid of stinking iTunes. Now all that’s left is to find something comparable to “mp3tagedit”:

posted on July 21 2010, and tagged with sysadmin music

Configuring rubygems

After spending ages meticulously installing gems with

matthouse [~] > sudo gem install some-random-gem --no-ri --no-rdoc

I finally got round to trying to find a better way. It turns out that rubygems has it’s own config file, which is cool. It’s located at ~/.gemrc and is a simple yaml file. My gemrc now looks like this:

:update_sources: true
:sources: 
- http://gems.rubyforge.org/
- http://gems.github.com
:backtrace: false
:verbose: true
gem: --no-ri --no-rdoc

You can find out more about the options by running

matthouse [~] > gem help environment

The options in my config file work as follows

posted on August 26 2009, and tagged with ruby sysadmin

Defining a cron job that happens on machine reboot

I have an app that uses the Sphinx search engine and it’s partner Rails plugin thinking-sphinx. This is a neat little search engine that can be embedded into Rails projects.

The only downside is that you need to start the server before the app will run properly. You start thinking sphinx by running

rake ts:start

I started writing a full init script for this until I realised that writing a massive init script that I can manage with chkconfig (our servers are Redhat Enterprise) is slightly overkill for one freaking rake task.

After some research I found out that our venerable friend cron supports running tasks on reboot, you simply need to define your job with @reboot instead of the standard time information.

Nothing is as enlightening as an example so here goes nothing

matthouse@sakura $ crontab -l
@reboot echo "cron is ace" >> /tmp/somefile

to read more check man 5 crontab and read all the way to the bottom

posted on August 20 2009, and tagged with ruby sysadmin linux

Managing Ubuntu instances with Puppet

My machines at home can be pretty varied. I like to test out distributions and tend to just format my main dev box and try out new and exciting things pretty frequently. But one thing I get really sick of is the time spent making my environment personal, getting things like commonly used software installed, and my dot files in the correct places.

I needed a way of managing my infrastructure that doesn’t involve me sitting down for a couple of hours after every install grabbing packages, compiling software etc.

And that’s where Puppet comes in :)

What is Puppet

Puppet is a configuration management framework. It’s kind of a logical step forward from Cfengine. It is written in Ruby, by Luke Kanies of Reductive Labs and basically consists of 3 things:

Puppet makes deploying identical environments across one or multiple machines one hell of a lot easier than it would be normally, it also lets you track changes and keep specific configurations in one central repository so that you can see at a glance, what state your machines are in and what package versions they have installed etc.

It also deals with abstracting the operating system layer so that your puppet recipes can be OS agnostic, the packages are handled automatically your default package manager.

Why Puppet? What about the alternatives

There are a couple of alternatives to Puppet. Cfengine, which Puppet is meant to be a replacement for, owing to Luke Kanies experience at writing custom Cfengine plugins. The Puppet Wiki has a very good article on why Puppet is not Cfengine and how it compares, which makes very interesting reading.

The other main alternative is Chef. Chef is also written in Ruby and is definitely worth keeping an eye on. It tries to alleviate some of the main issues in Puppet, mainly that instead of a custom DSL the recipes are written in Ruby, which lets you do something like this:

node[:gems].each do |gem|
  gem_package gem[:name] do
    version gem[:version]
    source gem[:source]
    action :install
  end
end

where gems is just a massive hash of ruby gems, versions and sources. Currently in Puppet, installing a named gem with a specific version requires a seperate package declaration for each package, which isn’t very DRY and can clutter up your recipes a bit.

At the moment Chef is in early stages and seems to be a bit of a pain to use for anything other than Ubuntu so I prefer to concentrate on using Puppet, at least until Chef gets a bit more mature.

Installing Puppet and setting up the Puppet Master

Puppet is pretty easy to set up, and on a blank install of Ubuntu the process is pretty much as follows:

I prefer to use a custom rubygems install rather than the provided package. This is due to Debian’s lack of support for Rubygems because they have apt and don’t like having multiple package providers.

I disagree with this because Rubygems, like CPAN or Python Eggs, is a very specific package manager for a very specific purpose, but this is a topic for another post/rant! Normally I’d compile and install Rubygems using the excellent checkinstall program, but seeing as how I like to automate my system, I’ve included a deb with my Puppet repository. So we can install it directly from there.

So lets install ruby, clone the Puppet repository, install Rubygems and then install Puppet.

sudo apt-get install ruby git-core
cd /etc/ && git clone git://github.com/shadowaspect/puppet.git
sudo dpkg -i /etc/puppet/files/deb/rubygems1.8-1.3.5_i386.deb
sudo gem install puppet

Now we need to look at starting the server with the correct certificate name, my recipes assume that your puppetmaster host is called puppet, so make sure that it is configured in your hosts file and start up the puppetmaster with:

sudo puppetmasterd --certname puppet

So now we have a repository of Puppet recipes and our puppetmaster running you can run puppet on the machine with

sudo puppetd --test --verbose

This will fail the first time due to an invalid client certificate, to rectify this you need to have the puppetmaster sign the client cert, you can do this by running (on the master)

sudo puppetca -l # for a list of unsigned certs
sudo puppetca -s <certname> # from the above list

Writing Recipes

Puppet pulls it’s configuration from the files located in /etc/puppet. It basically assumes a couple of things:

Puppet has several useful constructs available for writing recipes, stuff like classes and basic inheritance, and also defines for creating your own custom functions, you can see examples of this by checking out my github repository, but a basic node definition would look like this:

node mattsmachine{
  package{ "apt":
    ensure => installed,
  }
  file{"/etc/apt/sources.list":
    ensure => present,
    owner => root,
    require => Package["apt"]
  }
}

This noddy little example just makes sure that apt is installed and that the sources.list is present on the machine. You can do so much more awesome stuff than this, for which I advise you to check out the excellent tutorials on the Puppet site, and have a browse around my custom Puppet repository. Get the source

Where next?

Obviously the initial install of Puppet (installing Ruby and Git by hand, compiling rubygems etc) can be pretty time consuming in itself. At work we use RedHat, and so we have created a custom kickstart file that completely automates the process of installing the operating system and performing the first puppet run. All you have to do is boot the machine from the network, point the installer at the kickstart file and go and grab a coffee while it installs the OS (from a network share), configures the network, installs ruby and puppet, locates the puppetmaster and runs puppet to install the rest of the system depending on the role it’s destined to perform.

This is exceptionally neat, and I think there are definitely options out there to make something like this work with Ubuntu/Debian either via the Debian preseed files, or tools like Fai, so expect more posts on the topic in the future.

posted on July 28 2009, and tagged with ubuntu linux sysadmin config management

Multiple SSHD servers on ArchLinux and OpenID Delegation

A reasonably productive sunday morning has led to me having a couple of interesting things to talk about :) The first is running multiple ssh daemons on the same machine, and the second id about OpenID delegation, so that you can identify yourself on OpenID enabled sites with your own personal URL, rather than the URL of your OpenID provider.

Multiple SSH Servers and why you should bother?

The use case I found for running multiple SSH servers is one of security through obscurity. For instance I wanted to have my server accessible from other machines on my local network using standard password authentication so that I can log in and manage stuff remotely with relatively few restrictions. To this end I set up the standard SSH server listening on port 22.

This works great internally and because port 22 is blocked on my firewall I don’t have to worry (too much) about people outside of the network.

But what if I want to access from outside my network, for instance I want to ssh in from work and grab some music that I left at home? I don’t really want to open port 22 and similarly I don’t really want to restrict the internal users too much. So the best solution is to run a seperate sshd listening on a higher port with restricted security settings.

This is actually pretty easy to get set up. But there are a couple of gotchas and these instructions pertain to Arch Linux, but the basic principles are the same for any distro.

Port 55225 # A new higher port to listen on
PidFile /var/run/sshd-external.pid # Make sure that this new instance uses a seperate pid
PubkeyAuthentication yes # enable Public Key Auth
PasswordAuthentication no # and don't let users use passwords
AllowUsers extern # Allow access only to the extern user

#
# Parameters to be passed to sshd
#
SSHD_ARGS="-f /etc/ssh-external/sshd_config -h /etc/ssh-external/ssh_host_rsa_key"

There are a couple of extra things that you can do to improve security in addition to the standard key based auth above, one of which is to restrict your external user to a restrictive shell (ie. /bin/bash --restrictive) or a chroot jail, I leave this as an excercise for the reader.

Delegating OpenID Authentication to your personal website.

This is going to be a much shorter section that the previous one! I love OpenID but one thing I didn’t like is having to sign in using the URL to an openID provider like Flickr or LiveJournal. If I’m signing in using my identity then why the hell can’t I use my own URL?

It turns out that you can, and most OpenID providers will play nice and give you instructions on how to set it up. Basically it just involves some new information in the head of your websites markup. That extra info looks a lot like this:

<link rel="openid.server" href="http://server.myid.net/server" /> 
<link rel="openid.delegate" href="http://username.myid.net/" /> 
<meta http-equiv="X-XRDS-Location" content="http://username.myid.net/xrds" />

Obviously replace username with your actual OpenID providers username!

This small bit of markup now allows me to sign in to openID enabled sites with theshadowaspect.com as my identity.

posted on July 12 2009, and tagged with sysadmin linux ssh openid

Debugging Ruby using -rdebug

After a particularly annoying hour or two spent tracking down a problem in one of our apps recently, I spent my time wishing that Ruby had a decent debugger! Well it turns out that it does, and it’s pretty nifty.

Debuggers can be amazingly helpful when a particular bit of code is borking and you’re not sure why, and they save the effort of modifying your code to output debug symbols and variables all the time. You can run the debugger on a script like so:

[08:08]sakura mtmp $ ruby -rdebug testscript.rb 
Debug.rb
Emacs support available.
/opt/local/lib/ruby/vendor_ruby/1.8/ubygems.rb:10:require 'rubygems'
(rdb:1)

This also works with Rails projects, simply run the debugger against script/server.

First and foremost, use the h key from the debug console for help

You can set breakpoints using the syntax below. Breakpoints are lines at which the program execution will pause so you can poke around the environment.

(rdb:1) b testscript.rb:10
Set breakpoint 1 at testscript.rb 10
(rdb:1) b testscript.rb:20
Set breakpoint 2 at testscript.rb:20
(rdb:1)

This is the kind of output you’d see when you hit a breakpoint: we can see the line at which the execution stopped (we defined a breakpoint above as being on testscript.rb line 20), we can also see 5 lines of context each side of the Breakpoint, from here we can step through the code or just continue the exectution as normal.

(rdb:1) c
Breakpoint 2, toplevel at testscript.rb:20
testscript.rb:20:some_object = DoNothing.new(1)
(rdb:1) l
[15, 24] in testscript.rb
   15    def has_been_run?
   16      return @times_run
   17    end
   18  end
   19  
=> 20  some_object = DoNothing.new(1)
   21  5.times do
   22    some_object.run
   23  end
   24  
(rdb:1)

You can grab information about variables that are currently in scope and what their values are, the code below grabs all the instance variables belonging to the object some_object. Certain commands can also be used to grab the values of all globals and all constants that are defined in your program.

(rdb:1) v i some_object
  @do_nothing_for => 1
  @times_run => 0

This is just a basic introduction to the Ruby debugger, there is a ton of stuff that you can do with it, such as thread manipulation: stopping and starting the threads of your program at will, setting watchpoints and catchpoints, and loads more.

posted on July 09 2009, and tagged with ruby programming

Always Learning! Enumerable#inject and Hash#merge in Ruby

I learned something cool today. While discussing neat one liners in Ruby, someone suggested a potential challenge, it went something like this:

Given an array of numbers, how would you convert that into a hash where the key is the number and the value is the number of times that it appears in the array. so for example you start with the array

[1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5]

And what you want to end up with is something like this:

{5=>5, 6=>1, 1=>4, 2=>3, 3=>4, 4=>1}

So how would you do it? The solution I came up with after a couple of minutes was this:

h = Hash.new(0)
[1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5].each { |i| h[i] += 1 }
puts h

This is a fairly simple solution, first you create a new hash and initialise every value with 0, then we iterate over each value of the array, passing the value to a block which finds the hash entry with the correct key and increments its value (this is acheivable because we initialise our hash with a Fixnum as a default value). I thought this was pretty neat but it turns out, as with most things in Ruby, that there is a much cooler way :)

It looks like this:

p [1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5].inject(Hash.new(0)){ |memo,i| memo.merge({i => memo[i] += 1}) }

So how is this working?

The first thing we need to look at is Enumerable#inject. This takes an argument and a block. Inject then calls the block once for every element in the Collection we are calling it for, in this case this is our array of values. The argument we give to the inject function will be yielded as the first argument to the block (memo), the second argument in the block(i) is the value in our collection we are working on.

An important thing to note with inject is that the return value of the block is passed into the block on the subsequent call, replacing the old value of our first block argument (memo in this case).

This lets us do something funky inside the block using the Hash#merge function. merge takes a hash as an argument and merges it’s values into the hash that it’s called on. A couple of examples to illustrate this (in irb):

irb(main):007:0> {"first" => 2}.merge({"second" => 2})
=> {"second"=>2, "first"=>2}
irb(main):001:0> {"first" => 2}.merge({"first" => 4})
=> {"first"=>4}

So what is our merge function doing? It’s passing in an empty hash initialised with zeros, and for every entry in the array it’s merging in a new hash that just contains the incremented value for that particular key.

The p at the beginning is just a shorthand way of saying puts, so that our final hash is printed to the terminal

Clever huh?

posted on May 14 2009, and tagged with ruby programming tutorial

Rails Development with Ubuntu

Having recently purchased myself a new Dell Mini 9, I got a chance to set up a development environment for rails apps so that I can hack on the train to and from work. It’s not particularly hard but I thought I’d document it for reference and hopefully someone will find this useful.

First of all a note or 2 about the hardware, the machine is a 1.6GHz Intel Atom chip. A single core, cpu that supports hyper-threading (it shows up in /proc/cpuinfo as 2 seperate cpu’s but you can only control scaling on cpu0). It has a 4Gb SSD and 1Gb of RAM. This minimal spec turns out to be more than enough for a full Rails/mysql/mongrel development stack if you’re careful with a few defaults.

First of all I removed all unnecesary packages using a combination of apt-get and synaptic. apt-get remove is useful if you know what you want to kill (such as openoffice.org) but I find synaptic really useful for browsing through stuff you’re not sure about, libraries for example, and killing anything that doesn’t sound useful.

The next step is to install ruby, rubygems and a few useful dependancies. This is covered in detail elsewhere on the web so I’ll be brief, basically just:

sudo apt-get install ruby1.8 irb1.8 rdoc1.8 ri1.8 \
             readline build-essential libopenssl-ruby1.8
sudo ln -s /usr/bin/ruby /usr/bin/ruby1.8 #repeat for irb,ri and rdoc
wget http://www.rubygems.org/rubygems/rubygems-1.3.1.tar.gz
tar zxvf rubygems-1.3.1.tar.gz && cd rubygems-1.3.1
sudo ruby setup.rb

If you’d like a nice deb package that you can track with apt, you can replace the last step with:

sudo apt-get install checkinstall
checkinstall 'sudo ruby setup.rb'

Checkinstall is an awesome piece of software, it’s so awesome it deserves it’s own post so I’ll skip over the details, check it out.

After setting up this step you should now have a fully functioning rubygems installation, test it by issuing gem -v

Next step is to get mysql installed and configured. I did this using apt-get, because it’s easy:

sudo apt-get install mysql-server mysql-client libmysql-dev

Be aware that you need the mysql development headers in order to compile the mysql gem, which we will install next, alongside adding Github to our gems sources, installing rails and a bunch of other useful stuff.

sudo gem sources -a http://gems.github.com
sudo gem install rails mysql sqlite3-ruby capistrano hoe hpricot webby RedCloth rake

And that is pretty much all there is to it. you can now run your favourite editor (in my case gedit) and start writing Ruby.

There are a couple of Dell Mini specific issues that I had to do before I found this machine to be a truly comfortable development environment though.

That’s pretty much everything I’ve done so far. If I find anything else that works really well I’ll update this post. Also any comments/suggestions then email them to me :)

posted on March 31 2009, and tagged with ubuntu netbooks rails ruby programming

Fixing font rendering in Arch/Gnome

If you use Arch linux you may have noticed that the default font rendering in Gnome is a bit lacking, especially if you don’t have Xfce installed as well, this appears to be because Xfce installs gsfonts as one of it’s dependancies yet Gnome doesn’t. So the first to do is install it.

$ pacman -S gsfonts

The next thing I like to do is to install the liberation font set, and the microsoft web fonts. These are available using pacman and can be installed as follows:

$ pacman -S ttf-liberation ttf-ms-fonts

Then I like to install the versions of freetype, libxft and cairo with the cleartype patches applied. These are available from AUR, so grab them and install them. If you have yaourt you can simply do

yaourt -S cairo-cleartype freetype2-cleartype libxft-cleartype

Then install the packages that it creates (usually in /tmp/yaourt somehwhere). NB: Make sure that if you have installed any of the non cleartype versions of these packages using pacman that you remove them before installing the new versions.
If you don’t have yaourt installed for whatever reason you can grab the tarballs directly from the AUR repo and build the packages like this:

$ tar zxvf package.src.tar.gz
$ cd package
$ makepkg
$ pacman -U package-i686.pkg.tar.gz

The remaining step is to configure your font preferences in the Gnome Appearance dialog, my preferred settings are:

Now you should be able to restart X and enjoy some much better looking fonts

posted on January 28 2009, and tagged with linux sysadmin arch linux fonts