Tommy Chheng : Programming Blog

Must have Apps for Mac Developers

I recently got the new 15 inch Macbook Pro with Snow Leopard and decided to clean out my set of mac apps to the bare necessities. Here’s what I got it down to:

general purpose software:
Firefox: I use this along with two most-have plugins: ubiquity, firebug.
iwork: Need Pages/Numbers/Keynote.
adium: much more customizable than ichat.
VLC: plays every type of video.
colloquy: irc chat client.
transmission: torrents!
photoshop cs4: Need to edit images from time to time.
skype: Making the calls.
flickr uploadr: Easier to upload photos than their web interface.
unrarx: the builtin uncompress software doesn’t work with rar.

dev editors:
smultron: a great low footprint text editor.
textmate: Best editor for any type of programming language.
eclipse: The java IDE. i use hadoop and learning clojure so java is back in. Eclipse has great debugging and junit integration.

utilities:
quicksliver: quick app launcher.
sequel pro: Browsing mySQL DBs.
gitx: I mostly use command line for git, but i like the visual diffs in gitx.
iterm: I like this better than terminal. *mostly because of the full screen ability and preferred shortcut keys.
geektool: I keep a cat of the syslog on my desktop to make sure no background process is going haywire.
virtualbox: Free opensource VM client. For checking website compatibility in IE7.

tech(ok, you don’t need all of these, but they are great tech to try out):
java: 1.6 is already builtin.
git: distributed source control, must have for any coding project or documents you write. build from source.
mysql: need to run a local mysql server. get the 64 bit version!
ruby on rails: My choice of web development framework.
couchdbx: my choice of the nosql db. this is a self-contained package.
clojure: functional programming language and a modern lisp dialect built on the jvm.
hadoop: run a local copy of jobs for testing before deploying to ec2.
pig: the pig command line is vital for testing pig scripts locally.

CouchDB Install Problems on Snow Leopard: Trace/BPT trap

Just got the new Mac Snow Leopard this weekend! With the OS switching from 32bit to 64bit, i had to recompile much of my development software. CouchDB was particularly troubling.

I deleted all my macports, installed new macports 1.8, built 64bit versions of spidermonkey, icu and erlang using macports. Erlang looked like it was working:

→$erl
Erlang R13B01 (erts-5.7.2)  [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [kernel-poll:false]
Eshell V5.7.2  (abort with ^G)
1> 1+1.
2
2> q().

I rebuilt couchdb from the latest sources using

./bootstrap; ./configure; make && sudo make install

When I tried to start CouchDB:

→$sudo couchdb
Apache CouchDB 0.10.0a802973 (LogLevel=info) is starting.
Trace/BPT trap

Ouch…no idea what this “Trace/BPT trap” error means, but luckily Benoit Chesneau on the CouchDB user mailing list found out it was a MacPorts build issue and not the code itself. Installing CouchDB and its dependencies from the directly from source solved the issue.  The exact solutions to rebuild from the sources rather macports are shown on the CouchDB wiki install page.

Twitter OAuth Ruby Gem PIN-based Authentication API Change

I have been using the moomerman-twitter_oauth gem for to allow users to login to our web app via Twitter Connect. Unfortunately, it stopped working when a change in the Twitter API happened. During the OAuth authentication process, instead of being redirected back to our web app, it would show a screen that said:

"You've successfully granted access...enter the following PIN to complete the process"

After browsing the net and getting help from the Twitter API team, I learned that Twitter recently made a change to their OAuth process to allow this PIN type authentication for applications. See for more info: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/472500cfe9e7cdb9/848f834227d3e64d?pli=1

The oauth ruby gem defaults to using PIN-based process instead of the regular web app redirect process. To fix this, explicitly set the oauth_callback url parameter when getting the request token:

    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )
    request_token = @twitter_client.request_token(:oauth_callback => oauth_confirm_url)

If you are getting a (OAuth::Unauthorized) “401 Unauthorized” error after adding the oauth_callback parameter, try altering your oauth callback method to explicitly state the oauth_verifier as well:

  def oauth_callback
    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )

    @twitter_access_token = @twitter_client.authorize(
      session[:request_token],
      session[:request_token_secret],
       auth_verifier =>params[:oauth_verifier]
    )

Apparently, this was posted on the Twitter API Development Group in late May, but it would have been nice if Twitter DMed every web app signed up on the twitter app list of the change. I guess this is a warning for any Twitter API consumers to follow the dev list closely..

OS Process Timed out CouchDB error and fix

If you come across an OS Process Timed out using a view in CouchDB, you can adjust the os_process_timeout setting in the couchdb futon utility.

[Tue, 21 Jul 2009 23:57:45 GMT] [error] [<0.2804.0>] Uncaught error in HTTP request: {exit,
                                {{bad_return_value,
                                  {os_process_error,"OS process timed out."}},
                                 {gen_server,call,
                                  [<0.2808.0>,
                                   {prompt,
                                    [<<"reduce">>,
                                     [<<"function(keys, values, rereduce) {\n     return sum(values);\n   }">>],
                                     [[[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1]]]},
                                   infinity]}}}

Thanks to Dustin on the couchdb user mailing list for pointing this out.

Installing CouchDB on Ubuntu: Problems and fixes

Got around to install CouchDB from source on a Ubuntu VM. CouchDB is schemaless “document” database server with a RESTful HTTP/JSON API. I’m mostly experimenting it with because of the schemaless design. The map/reduce views took a bit to get used to after being so well acustomed to SQL queries in the past, but I like what I see so far.

While installing it, I ran into a few problems:

Install Erlang(I was having trouble with the 64-bit erlang and openssl package, i ended up just using the 32 bit version):

sudo apt-get install build-essential erlang

First error message:

configure: error: Could not find the js library.
Is the Mozilla SpiderMonkey library installed?

Uh oh, let’s install spidermonkey:

sudo add-apt-repository ppa:launchpad/ppa
sudo apt-get install libmozjs-dev

Another error pops up:

checking for icu-config... no
*** The icu-config script could not be found. Make sure it is
*** in your path, and that taglib is properly installed.
*** Or see http://ibm.com/software/globalization/icu/
configure: error: Library requirements (ICU) not met.

Oh no, international character support needed, install icu:

sudo apt-get install libicu-dev

And then another error appears:

checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.

Hmm, curl needed:

sudo apt-get install libcurl4-openssl-dev

Now for Couchdb install commands (stolen from the couchdb wiki)

svn co http://svn.apache.org/repos/asf/couchdb/trunk couchdb
cd couchdb
./bootstrap
./configure
make
sudo make install
make clean
make distclean
sudo -i
adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
chown -R couchdb:couchdb /usr/local/var/lib/couchdb
chown -R couchdb:couchdb /usr/local/var/log/couchdb
chown -R couchdb:couchdb /usr/local/var/run
chown -R couchdb:couchdb /usr/local/etc/couchdb
chmod -R 0770 /usr/local/var/lib/couchdb
chmod -R 0770 /usr/local/var/log/couchdb
chmod -R 0770 /usr/local/var/run
chmod -R 0770 /usr/local/etc/couchdb
cp /usr/local/etc/init.d/couchdb /etc/init.d/
update-rc.d couchdb defaults

Start it up with:

sudo /etc/init.d/couchdb start

CouchDB has a really great development community behind it. Subscribe to the mailing list, read the wiki, follow the couchdb twitter and read the book

Ruby on Rails 2.3.2 Upgrade Gotchas

While cleaning/re-factoring the PeopleJar codebase, we decided to make the move from Rails 2.2.2 to Rails 2.3.2. Rack integration was the biggest single “under the hood” change. Adam Wiggins of Heroku explains the benefits in his Rails Metal, Rack and Sinatra presentation from RailsConf. He has a great example of incorporating a Sinatra app with a Rails app.

No upgrades would be complete without some incompatibilities. As noted on the official Ruby on Rails release notes, test Sessions, Cookies, File uploads, JSON/XML APIs. We found conflicts in the facebooker and ar-extensions gems. Luckily, the developers already pushed out fixes so you can just do a gem update.

There was one other cryptic error in one of our ajax calls:

  Status: 500 Internal Server Error
  private method `split' called for #
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/mime_type.rb:206:in `method_missing'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:51:in `media_type'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:117:in `parseable_data?'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:138:in `POST'
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/request.rb:428:in `request_parameters'

We could fix this with some good old monkey patching! Add this to an initializer in your config/initializers directory. this is a reported bug at the rails lighthouse and should be fixed in the next release.

module Mime
  class Type
    def split(*args)
      to_s.split(*args)
    end
  end
end

Overall, the upgrade went pretty well: just a few gem upgrades and minor tweaks. If you are on Rails 2.2.2 or older, give the upgrade a shot!

Organizing Models in Rails without Namespacing

I was doing some code cleanup and noticed we have over 50 models in our models directory. Not good. The first thing I thought was to add namespacing. I googled the topic and turns out namespacing models is a bad idea because of how Rails collapses the model namespaces. Adding sub directories is a better and cleaner solution. Read Josh Susser’s post on the topic.

To use sub directories, you must add the directories to Rails’ config.load_paths in your environment.rb

   config.load_paths << "#{Rails.root}/app/models/profile"

To make this more flexible, you can use this snippet to add all sub directories:

  #Add all sub directories in models to load paths
  Dir.entries("#{Rails.root}/app/models").each do |file|
    next if file.eql?('.') || file.eql?('..')
    full_path = "#{Rails.root}/app/models/#{file}"
    config.load_paths << full_path if File.directory?(full_path)
  end

Easy Javascript Validation with Javascript Lint For Rails Testing

I found this great Javascript Lint tool. It checks for common(missing semicolons) and not so common(use of the void type) javascript errors. A full featured Selenium test suite would be the ideal for full javascript testing coverage but Javascript Lint is a no hassle addition.

I wrote a simple Rails integration test case to scan all the javascript files in the public/javascripts folder. Download the javascript lint tool and add the file below to your integration test suite. You easily adapt this to Rspec if you would like.

require 'test_helper'

class JavascriptTest < ActionController::IntegrationTest
  # Download jsl from http://www.javascriptlint.com and add the jsl to your PATH environment variable
  def setup
    @js_paths = File.join(Rails.root, 'public', 'javascripts', '*.js')
  end

  test "validate javascript files for errors" do
    Dir[@js_paths].each do |js_path|
      output = %x[jsl -process #{js_path}]

      is_valid_js_file = output.include?("0 error(s)") ?  true : false
      assert(is_valid_js_file, "JSLint on #{js_path} should return no errors: \n #{output}")
    end
  end
end

Here’s an example of the error output:

  2) Failure:
test_validate_javascript_files_for_errors(JavascriptTest)
    [/test/integration/javascript_test.rb:14:in `test_validate_javascript_files_for_errors'
     /test/integration/javascript_test.rb:10:in `each'
     /test/integration/javascript_test.rb:10:in `test_validate_javascript_files_for_errors']:
JSLint on ..../public/javascripts/application.js should return no errors:
 JavaScript Lint 0.3.0 (JavaScript-C 1.5 2004-09-24)
Developed by Matthias Miller (http://www.JavaScriptLint.com)

application.js
.../public/javascripts/application.js(3): lint warning: missing semicolon
        num = ;
........^

Deploying a Sinatra app on Nginx/Passenger with Capistrano and Git

Need a quick web app done in Ruby? While Ruby on Rails is a great framework, it is overkill for a simple one page application. Enter: Sinatra.

Sinatra is a light-weight Ruby framework intended for fast web applications. In fact, here is all you need for a hello world app:


# myapp.rb
require 'rubygems'
require 'sinatra'
get '/' do
'Hello world!'
end


You can read more about Sinatra syntax on their documentation page.

This guide will get you started from the ground up to setting up and deploying a sinatra app on Nginx/Passenger with Capistrano and Git. I choose Passenger because it allows you to run multiple ruby applications easier than the traditional Mongrel setup.

Install Passenger/Nginx

Install the passenger gem:
gem install passenger

Install Nginx and Passenger Module:
passenger-install-nginx-module

The instructions from the install scripts are pretty much self-explanatory. More information on the official Phusion Passenger page. I also recommend installing Phusion’s Ruby Enterprise Edition for more memory savings.

Make sure your nginx.conf file is configured to include external files, notice the include statements at the end:


user deploy;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /opt/nginx/conf/mime.types;
default_type application/octet-stream;
passenger_root /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/passenger-2.2.2;
passenger_ruby /usr/bin/ruby;
passenger_max_pool_size 2;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
server_names_hash_bucket_size 128;
include /opt/nginx/conf/conf.d/*.conf;
include /opt/nginx/conf/sites-enabled/*;
}


Create a Nginx entry file in /opt/nginx/conf/sites-available/APP_NAME:


server {
listen 80;
server_name YOURDOMAINNAME.HERE;
root /u/apps/production/APP_NAME/current/public;
passenger_enabled on;
}


Symlink this in the sites-enabled folder:

ln -s /opt/nginx/conf/sites-available/APP_NAME /opt/nginx/conf/sites-enabled/APP_NAME


A basic Sinatra template

Now, let’s get a basic Sinatra app on the server. To speed things along, clone this Sinatra template app:
git clone git://github.com/tc/sinatra-template.git

This includes a Rack config file(config.ru) and a cap deploy script.

This was forked from zapnap/sinatra-template but I removed datamapper since I didn’t intend to use a database. I also added a cap template script.

Edit the deploy.rb file and add in your git url/username/app_name


#========================
#CONFIG
#========================
set :application, "APP_NAME"
set :scm, :git
set :git_enable_submodules, 1
set :repository, "GIT_URL"
set :branch, "master"
set :ssh_options, { :forward_agent => true }
set :stage, :production
set :user, "deploy"
set :use_sudo, false
set :runner, "deploy"
set :deploy_to, "/u/apps/#{stage}/#{application}"
set :app_server, :passenger
set :domain, "DOMAIN_URL"
#========================
#ROLES
#========================
role :app, domain
role :web, domain
role :db, domain, :primary => true
#========================
#CUSTOM
#========================
namespace :deploy do
task :start, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
task :stop, :roles => :app do
# Do nothing.
end
desc "Restart Application"
task :restart, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
end


Deploying to Server

Let’s deploy the app onto the server now. I will assume you already have ssh key authentication setup on the server.

Now, setup your directories using

cap deploy:setup


and deploy your code from your git repo.

cap deploy


That’s it! Passenger is able to recognize Sinatra app through the RACK interface. You should be able to visit the domain and see your app up:

The cap script will automatically restart the app by touching the restart.txt file.

Google IO 2009 Thoughts

Last week, I attended the Google IO 2009. Here’s a list of things I found to be important:

• Web Programming is the new programming model of our time. Learn it or be at a disadvantage.
• Google Wave is a strong real time communication tool. It looks promising but also looks bloated with too many features. I don’t see it replacing e-mail or IM in the near future.
• Just how Google Maps/Gmail popularized Ajax, HTML 5 will need a compelling application to make it widespread.
• Microsoft Internet Explorer needs to support HTML5!
• The major HTML5 components:
  • canvas: ease of interactivity in the browser. A “paint” application can be written in a just few lines of code.
  • video: no more flash video! Lets you use Javascript to alter the video as well. There was a demo of an object tracking at IO.
  • geolocation: Straight from the browser. Will be interesting when laptops/netbooks start coming with GPS
  • app cache: imagine developing apps using HTML/JS instead of traditional compiled languages. Should greatly improve software quality and app development-to-market time.
• Google Web Elements are quick copy and paste elements to make your site more interactive.
• Ignite Talk: Reading scifi is one way of predicting the future not because they will be true but because many of the world’s shapers also read them.
• Ignite Talk: Tim Ferris says to list your fears in great detail to realize greater possibilities.
• Ignite Talk: Patterns and anti-patterns:
  • make the right thing easy and the wrong things hard
  • allow users to party with others without you
  • what role do you play for your users

Google IO was seriously the best ran conference I’ve been to: great hospitality, great presentations and not to mention, very green(even with compostable bins). Here are some io2009 photos from my flickr.