Tommy Chheng

I have been using the moomerman-twitter_oauth gem for to allow users to login to our web app via Twitter Connect. Unfortunately, it stopped working when a change in the Twitter API happened. During the OAuth authentication process, instead of being redirected back to our web app, it would show a screen that said:

"You've successfully granted access...enter the following PIN to complete the process"

After browsing the net and getting help from the Twitter API team, I learned that Twitter recently made a change to their OAuth process to allow this PIN type authentication for applications. See for more info: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/472500cfe9e7cdb9/848f834227d3e64d?pli=1

The oauth ruby gem defaults to using PIN-based process instead of the regular web app redirect process. To fix this, explicitly set the oauth_callback url parameter when getting the request token:

    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )
    request_token = @twitter_client.request_token(:oauth_callback => oauth_confirm_url)

If you are getting a (OAuth::Unauthorized) “401 Unauthorized” error after adding the oauth_callback parameter, try altering your oauth callback method to explicitly state the oauth_verifier as well:

  def oauth_callback
    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )

    @twitter_access_token = @twitter_client.authorize(
      session[:request_token],
      session[:request_token_secret],
       auth_verifier =>params[:oauth_verifier]
    )

Apparently, this was posted on the Twitter API Development Group in late May, but it would have been nice if Twitter DMed every web app signed up on the twitter app list of the change. I guess this is a warning for any Twitter API consumers to follow the dev list closely..

If you come across an OS Process Timed out using a view in CouchDB, you can adjust the os_process_timeout setting in the couchdb futon utility.

[Tue, 21 Jul 2009 23:57:45 GMT] [error] [<0.2804.0>] Uncaught error in HTTP request: {exit,
                                {{bad_return_value,
                                  {os_process_error,"OS process timed out."}},
                                 {gen_server,call,
                                  [<0.2808.0>,
                                   {prompt,
                                    [<<"reduce">>,
                                     [<<"function(keys, values, rereduce) {\n     return sum(values);\n   }">>],
                                     [[[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1]]]},
                                   infinity]}}}

Thanks to Dustin on the couchdb user mailing list for pointing this out.

Got around to install CouchDB from source on a Ubuntu VM. CouchDB is schemaless “document” database server with a RESTful HTTP/JSON API. I’m mostly experimenting it with because of the schemaless design. The map/reduce views took a bit to get used to after being so well acustomed to SQL queries in the past, but I like what I see so far.

While installing it, I ran into a few problems:

Install Erlang(I was having trouble with the 64-bit erlang and openssl package, i ended up just using the 32 bit version):

sudo apt-get install build-essential erlang

First error message:

configure: error: Could not find the js library.
Is the Mozilla SpiderMonkey library installed?

Uh oh, let’s install spidermonkey:

sudo apt-get install libmozjs-dev

Another error pops up:

checking for icu-config... no
*** The icu-config script could not be found. Make sure it is
*** in your path, and that taglib is properly installed.
*** Or see http://ibm.com/software/globalization/icu/
configure: error: Library requirements (ICU) not met.

Oh no, international character support needed, install icu:

sudo apt-get install libicu-dev

And then another error appears:

checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.

Hmm, curl needed:

sudo apt-get install libcurl4-openssl-dev

Now for Couchdb install commands (stolen from the couchdb wiki)

svn co http://svn.apache.org/repos/asf/couchdb/trunk couchdb
cd couchdb
./bootstrap
./configure
make
sudo make install
make clean
make distclean
sudo -i
adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
chown -R couchdb:couchdb /usr/local/var/lib/couchdb
chown -R couchdb:couchdb /usr/local/var/log/couchdb
chown -R couchdb:couchdb /usr/local/var/run
chown -R couchdb:couchdb /usr/local/etc/couchdb
chmod -R 0770 /usr/local/var/lib/couchdb
chmod -R 0770 /usr/local/var/log/couchdb
chmod -R 0770 /usr/local/var/run
chmod -R 0770 /usr/local/etc/couchdb
cp /usr/local/etc/init.d/couchdb /etc/init.d/
update-rc.d couchdb defaults

Start it up with:

sudo /etc/init.d/couchdb start

CouchDB has a really great development community behind it. Subscribe to the mailing list, read the wiki, follow the couchdb twitter and read the book

While cleaning/re-factoring the PeopleJar codebase, we decided to make the move from Rails 2.2.2 to Rails 2.3.2. Rack integration was the biggest single “under the hood” change. Adam Wiggins of Heroku explains the benefits in his Rails Metal, Rack and Sinatra presentation from RailsConf. He has a great example of incorporating a Sinatra app with a Rails app.

No upgrades would be complete without some incompatibilities. As noted on the official Ruby on Rails release notes, test Sessions, Cookies, File uploads, JSON/XML APIs. We found conflicts in the facebooker and ar-extensions gems. Luckily, the developers already pushed out fixes so you can just do a gem update.

There was one other cryptic error in one of our ajax calls:

  Status: 500 Internal Server Error
  private method `split' called for #
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/mime_type.rb:206:in `method_missing'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:51:in `media_type'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:117:in `parseable_data?'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:138:in `POST'
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/request.rb:428:in `request_parameters'

We could fix this with some good old monkey patching! Add this to an initializer in your config/initializers directory. this is a reported bug at the rails lighthouse and should be fixed in the next release.

module Mime
  class Type
    def split(*args)
      to_s.split(*args)
    end
  end
end

Overall, the upgrade went pretty well: just a few gem upgrades and minor tweaks. If you are on Rails 2.2.2 or older, give the upgrade a shot!

I was doing some code cleanup and noticed we have over 50 models in our models directory. Not good. The first thing I thought was to add namespacing. I googled the topic and turns out namespacing models is a bad idea because of how Rails collapses the model namespaces. Adding sub directories is a better and cleaner solution. Read Josh Susser’s post on the topic.

To use sub directories, you must add the directories to Rails’ config.load_paths in your environment.rb

   config.load_paths << "#{Rails.root}/app/models/profile"

To make this more flexible, you can use this snippet to add all sub directories:

  #Add all sub directories in models to load paths
  Dir.entries("#{Rails.root}/app/models").each do |file|
    next if file.eql?('.') || file.eql?('..')
    full_path = "#{Rails.root}/app/models/#{file}"
    config.load_paths << full_path if File.directory?(full_path)
  end

I found this great Javascript Lint tool. It checks for common(missing semicolons) and not so common(use of the void type) javascript errors. A full featured Selenium test suite would be the ideal for full javascript testing coverage but Javascript Lint is a no hassle addition.

I wrote a simple Rails integration test case to scan all the javascript files in the public/javascripts folder. Download the javascript lint tool and add the file below to your integration test suite. You easily adapt this to Rspec if you would like.

require 'test_helper'

class JavascriptTest < ActionController::IntegrationTest
  # Download jsl from http://www.javascriptlint.com and add the jsl to your PATH environment variable
  def setup
    @js_paths = File.join(Rails.root, 'public', 'javascripts', '*.js')
  end

  test "validate javascript files for errors" do
    Dir[@js_paths].each do |js_path|
      output = %x[jsl -process #{js_path}]

      is_valid_js_file = output.include?("0 error(s)") ?  true : false
      assert(is_valid_js_file, "JSLint on #{js_path} should return no errors: \n #{output}")
    end
  end
end

Here's an example of the error output:

  2) Failure:
test_validate_javascript_files_for_errors(JavascriptTest)
    [/test/integration/javascript_test.rb:14:in `test_validate_javascript_files_for_errors'
     /test/integration/javascript_test.rb:10:in `each'
     /test/integration/javascript_test.rb:10:in `test_validate_javascript_files_for_errors']:
JSLint on ..../public/javascripts/application.js should return no errors:
 JavaScript Lint 0.3.0 (JavaScript-C 1.5 2004-09-24)
Developed by Matthias Miller (http://www.JavaScriptLint.com)

application.js
.../public/javascripts/application.js(3): lint warning: missing semicolon
        num = ;
........^

Need a quick web app done in Ruby? While Ruby on Rails is a great framework, it is overkill for a simple one page application. Enter: Sinatra.

Sinatra is a light-weight Ruby framework intended for fast web applications. In fact, here is all you need for a hello world app:


# myapp.rb
require 'rubygems'
require 'sinatra'
get '/' do
'Hello world!'
end


You can read more about Sinatra syntax on their documentation page.

This guide will get you started from the ground up to setting up and deploying a sinatra app on Nginx/Passenger with Capistrano and Git. I choose Passenger because it allows you to run multiple ruby applications easier than the traditional Mongrel setup.

Install Passenger/Nginx

Install the passenger gem:
gem install passenger

Install Nginx and Passenger Module:
passenger-install-nginx-module

The instructions from the install scripts are pretty much self-explanatory. More information on the official Phusion Passenger page. I also recommend installing Phusion’s Ruby Enterprise Edition for more memory savings.

Make sure your nginx.conf file is configured to include external files, notice the include statements at the end:


user deploy;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /opt/nginx/conf/mime.types;
default_type application/octet-stream;
passenger_root /opt/ruby-enterprise-1.8.6-20090421/lib/ruby/gems/1.8/gems/passenger-2.2.2;
passenger_ruby /usr/bin/ruby;
passenger_max_pool_size 2;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
server_names_hash_bucket_size 128;
include /opt/nginx/conf/conf.d/*.conf;
include /opt/nginx/conf/sites-enabled/*;
}


Create a Nginx entry file in /opt/nginx/conf/sites-available/APP_NAME:


server {
listen 80;
server_name YOURDOMAINNAME.HERE;
root /u/apps/production/APP_NAME/current/public;
passenger_enabled on;
}


Symlink this in the sites-enabled folder:

ln -s /opt/nginx/conf/sites-available/APP_NAME /opt/nginx/conf/sites-enabled/APP_NAME


A basic Sinatra template

Now, let’s get a basic Sinatra app on the server. To speed things along, clone this Sinatra template app:
git clone git://github.com/tc/sinatra-template.git

This includes a Rack config file(config.ru) and a cap deploy script.

This was forked from zapnap/sinatra-template but I removed datamapper since I didn’t intend to use a database. I also added a cap template script.

Edit the deploy.rb file and add in your git url/username/app_name


#========================
#CONFIG
#========================
set :application, "APP_NAME"
set :scm, :git
set :git_enable_submodules, 1
set :repository, "GIT_URL"
set :branch, "master"
set :ssh_options, { :forward_agent => true }
set :stage, :production
set :user, "deploy"
set :use_sudo, false
set :runner, "deploy"
set :deploy_to, "/u/apps/#{stage}/#{application}"
set :app_server, :passenger
set :domain, "DOMAIN_URL"
#========================
#ROLES
#========================
role :app, domain
role :web, domain
role :db, domain, :primary => true
#========================
#CUSTOM
#========================
namespace :deploy do
task :start, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
task :stop, :roles => :app do
# Do nothing.
end
desc "Restart Application"
task :restart, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
end


Deploying to Server

Let’s deploy the app onto the server now. I will assume you already have ssh key authentication setup on the server.

Now, setup your directories using

cap deploy:setup


and deploy your code from your git repo.

cap deploy


That’s it! Passenger is able to recognize Sinatra app through the RACK interface. You should be able to visit the domain and see your app up:

The cap script will automatically restart the app by touching the restart.txt file.

Last week, I attended the Google IO 2009. Here’s a list of things I found to be important:

• Web Programming is the new programming model of our time. Learn it or be at a disadvantage.
• Google Wave is a strong real time communication tool. It looks promising but also looks bloated with too many features. I don’t see it replacing e-mail or IM in the near future.
• Just how Google Maps/Gmail popularized Ajax, HTML 5 will need a compelling application to make it widespread.
• Microsoft Internet Explorer needs to support HTML5!
• The major HTML5 components:
  • canvas: ease of interactivity in the browser. A “paint” application can be written in a just few lines of code.
  • video: no more flash video! Lets you use Javascript to alter the video as well. There was a demo of an object tracking at IO.
  • geolocation: Straight from the browser. Will be interesting when laptops/netbooks start coming with GPS
  • app cache: imagine developing apps using HTML/JS instead of traditional compiled languages. Should greatly improve software quality and app development-to-market time.
• Google Web Elements are quick copy and paste elements to make your site more interactive.
• Ignite Talk: Reading scifi is one way of predicting the future not because they will be true but because many of the world’s shapers also read them.
• Ignite Talk: Tim Ferris says to list your fears in great detail to realize greater possibilities.
• Ignite Talk: Patterns and anti-patterns:
  • make the right thing easy and the wrong things hard
  • allow users to party with others without you
  • what role do you play for your users

Google IO was seriously the best ran conference I’ve been to: great hospitality, great presentations and not to mention, very green(even with compostable bins). Here are some io2009 photos from my flickr.

As researchers, we often wonder if what we are working will be beneficial. Additionally, we wonder if our limited time should be spent on one topic vs another. Over the upcoming months, I will be analyzing this problem and presenting possible solutions.

I will be working with Bill Tomlinson at UC Irvine on a new project called ResearchWatch. We are effectively researching “research.”

Our first goal will be to analyze the distribution of research today. We want to see which topics are at the heart of research and which are being possibly ignored. Could these ignored topics warrant more interest? Are the over-researched topics unnecessary?

We will be specifically analyzing the NSF Research Grant dataset for the past 10 years. We will cluster these into topics using various unsuprvised learning algorithms. This will give us a high level mapping of what the NSF has considered to be important to warrant research grants. Stay tuned for future updates…

Thanks to Amazon for supporting the academic community. They have provide us with a grant for various AWS services. Check out their recent blog post if you are a researcher and need a base of operations.

Here’s a guide to start hacking computer vision and image processing applications using OpenCV/Eclipse on Windows machines. While you can use Microsoft Visual Studio to program OpenCV apps, I find Eclipse a much easier to use IDE.

C++ Development Tools

• MinGW: A collection of freely available and freely distributable Windows specific header files and import libraries combined with GNU toolsets that allow one to produce native Windows programs that do not rely on any 3rd-party C runtime DLLs. MinGW is different from Cygwin because it uses the Windows C runtime libraries(mscvrt) rather than GNU’s libc.
• Msys: A Minimal SYStem to provide POSIX/Bourne configure scripts the ability to execute and create a Makefile used by make.
• Eclipse CDT: A IDE originally made for Java but includes an extensive plugin library. Now supports C/C++ and many other languages.

The latest version can be checked on the respective website. I used MinGW 5.1.3 and Msys 1.0.10. When installing MinGW, select the G++ and other compilers. Do NOT install the make in the MinGW setup. Msys will provide it.

MinGW does not include the GDB debugger so download gdb-6.6.tar.bz2 and install it to your MinGW directory. To uncompress it, open up the msys window and type in bunzip2 gdb-6.6.tar.bz2 and then tar -xvf gdb-6.6.tar. Copy all the contents to your MinGw folder.

OpenCV

The two big open source computer Vision/Image processing libraries in C/C++ are OpenCV and the Nasa Vision WorkBench.

OpenCV is the old-school C/C++ computer vision/image processing library. It is robust and contains many functions described in computer vision textbooks. I haven’t played with the newer NASA tool but it looks like it has a decent API as well.

Download and install OpenCV.

Linking OpenCV in Eclipse

You can setup Eclipse CDT to work with the OpenCV libraries. Create a new C++ project in Eclipse CDT. Select MinGw as the toolchain.

In the project properties, go to the C/C++ Build->Settings->GCC C++ Compiler, set the directories to:

• OpenCv\cv\include
• OpenCv\cxcore\include
• OpenCv\otherlibs\highgui
• OpenCv\otherlibs\cvcam\include
• OpenCv\cvaux\include

In the C++ Linker->Libraries, set:

• cv
• highgui
• cxcore

In Library search path, set:

• OpenCV\lib

Here’s a sample file to get you started. You should be able to compile this program and see an inverted image when you run it.


////////////////////////////////////////////////////////////////////////
//
// hello-world.cpp
//
// This is a simple, introductory OpenCV program. The program reads an
// image from a file, inverts it, and displays the result.
//
////////////////////////////////////////////////////////////////////////
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include <cv.h>
#include <highgui.h>
int main(int argc, char *argv[])
{
IplImage* img = 0;
int height,width,step,channels;
uchar *data;
int i,j,k;
if(argc<2){
printf("Usage: main \n\7");
exit(0);
}
// load an image
img=cvLoadImage(argv[1]);
if(!img){
printf("Could not load image file: %s\n",argv[1]);
exit(0);
}
// get the image data
height = img->height;
width = img->width;
step = img->widthStep;
channels = img->nChannels;
data = (uchar *)img->imageData;
printf("Processing a %dx%d image with %d channels\n",height,width,channels);
// create a window
cvNamedWindow("mainWin", CV_WINDOW_AUTOSIZE);
cvMoveWindow("mainWin", 100, 100);
// invert the image
for(i=0;i data[i*step+j*channels+k]=255-data[i*step+j*channels+k];
// show the image
cvShowImage("mainWin", img );
// wait for a key
cvWaitKey(0);
// release the image
cvReleaseImage(&img );
return 0;
}