Tommy Chheng

I recently became interested in domain specific languages(dsls) and just started reading “Language Implementation Patterns”. The author, Terence Parr, wrote this great tool called ANTLR which helps you in the scanning/parsing stage of constructing a compiler.

Sometimes you’ll see this error:

antlr warning " Tokens has non-LL(*) decision due to recursive rule invocations "

when you have an invalid LL grammar such as an invalid non-recursive rule(see more on the antlr mailing list), but you maybe getting this error due to a syntax mistake.

Unlike Jflex(a scanner) and CUP(a parser), ANTLR is a both a lexer/scanner and parser. Only use all caps for the lexing(scanning) rules! Parsing statements use lower case or camel case.

grammar test;
NUMBER	:	'0'..'9'+;
TERM	:	NUMBER | '(' EXPRESSION ')';
EXPRESSION	:	TERM (('+'|'-') TERM)*;

should be:

grammar test;
NUMBER	:	'0'..'9'+;
TERM	:	NUMBER | '(' expr ')';
expr	:	TERM (('+'|'-') TERM)*;

I decided it’s about time to learn a functional programming language. Clojure is an excellent choice to get started. I looked at Scala and Erlang as other possible functional programming languages to learn but I settled on Clojure because:

a) It’s a LISP dialect
b) Runs on JVM
c) Community support.

I was amazed at the amount of talk going on in the clojure community. The creator of clojure, Rick Hickey regularly answers questions on the clojure google groups.

To get started on the mac, Citizen428 has created an easy to use package. It has all the necessary jars and support for TextMate and Emacs. Download it from the github project

Take some time to read more from the Clojure website and google groups. They have plenty of answers.

Here’s a sample app:

(import '(javax.swing JFrame JLabel JTextField JButton)
        '(java.awt.event ActionListener)
        '(java.awt GridLayout))
(defn celsius []
  (let [frame (JFrame. "Celsius Converter")
        temp-text (JTextField.)
        celsius-label (JLabel. "Celsius")
        convert-button (JButton. "Convert")
        fahrenheit-label (JLabel. "Fahrenheit")]
    (.addActionListener convert-button
      (proxy [ActionListener] []
        (actionPerformed [evt]
          (let [c (Double/parseDouble (.getText temp-text))]
            (.setText fahrenheit-label
               (str (+ 32 (* 1.8 c)) " Fahrenheit"))))))
    (doto frame
      (.setLayout (GridLayout. 2 2 3 3))
      (.add temp-text)
      (.add celsius-label)
      (.add convert-button)
      (.add fahrenheit-label)
      (.setSize 300 80)
      (.setVisible true))))
(celsius)

I have been working with Pig as a more productive layer against Hadoop. Definitely check it out if you need to use map reduce. Textmate is my primary editor so I made a little bundle for syntax highlighting and script running. I’ll add snippets in the future.

Check it out at the github project page

Pig is a language designed to process large datasets. The execution layer runs uses Hadoop’s MapReduce framework.
Visit http://hadoop.apache.org/pig for more information on Pig.

This textmate bundle gives you syntax highlighting and the ability to run Pig scripts from Textmate.

Install:

mkdir -p ~/Library/Application\ Support/TextMate/Bundles
cd ~/Library/Application\ Support/TextMate/Bundles
git clone git://github.com/tc/pig-latin-tmbundle.git "PigLatin.tmbundle"
osascript -e 'tell app "TextMate" to reload bundles'

To run Pig scripts from Textmate:
Add the pig bin directory to your path env variable.
ex.

export PATH="/Users/tc/bin/pig-0.3.0/bin:$PATH"

Then, you can just press “Apple-R” to execute the active pig script in Textmate.

I recently got the new 15 inch Macbook Pro with Snow Leopard and decided to clean out my set of mac apps to the bare necessities. Here’s what I got it down to:

general purpose software:
Firefox: I use this along with two most-have plugins: ubiquity, firebug.
iwork: Need Pages/Numbers/Keynote.
adium: much more customizable than ichat.
VLC: plays every type of video.
colloquy: irc chat client.
transmission: torrents!
photoshop cs4: Need to edit images from time to time.
skype: Making the calls.
flickr uploadr: Easier to upload photos than their web interface.
unrarx: the builtin uncompress software doesn’t work with rar.

dev editors:
smultron: a great low footprint text editor.
textmate: Best editor for any type of programming language.
eclipse: The java IDE. i use hadoop and learning clojure so java is back in. Eclipse has great debugging and junit integration.

utilities:
quicksliver: quick app launcher.
sequel pro: Browsing mySQL DBs.
gitx: I mostly use command line for git, but i like the visual diffs in gitx.
iterm: I like this better than terminal. *mostly because of the full screen ability and preferred shortcut keys.
geektool: I keep a cat of the syslog on my desktop to make sure no background process is going haywire.
virtualbox: Free opensource VM client. For checking website compatibility in IE7.

tech(ok, you don’t need all of these, but they are great tech to try out):
java: 1.6 is already builtin.
git: distributed source control, must have for any coding project or documents you write. build from source.
mysql: need to run a local mysql server. get the 64 bit version!
ruby on rails: My choice of web development framework.
couchdbx: my choice of the nosql db. this is a self-contained package.
clojure: functional programming language and a modern lisp dialect built on the jvm.
hadoop: run a local copy of jobs for testing before deploying to ec2.
pig: the pig command line is vital for testing pig scripts locally.

Just got the new Mac Snow Leopard this weekend! With the OS switching from 32bit to 64bit, i had to recompile much of my development software. CouchDB was particularly troubling.

I deleted all my macports, installed new macports 1.8, built 64bit versions of spidermonkey, icu and erlang using macports. Erlang looked like it was working:

→$erl
Erlang R13B01 (erts-5.7.2) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [kernel-poll:false]
Eshell V5.7.2  (abort with ^G)
1> 1+1.
2
2> q().

I rebuilt couchdb from the latest sources using

./bootstrap; ./configure; make && sudo make install

When I tried to start CouchDB:

→$sudo couchdb
Apache CouchDB 0.10.0a802973 (LogLevel=info) is starting.
Trace/BPT trap

Ouch…no idea what this “Trace/BPT trap” error means, but luckily Benoit Chesneau on the CouchDB user mailing list found out it was a MacPorts build issue and not the code itself. Installing CouchDB and its dependencies from the directly from source solved the issue.  The exact solutions to rebuild from the sources rather macports are shown on the CouchDB wiki install page.

I have been using the moomerman-twitter_oauth gem for to allow users to login to our web app via Twitter Connect. Unfortunately, it stopped working when a change in the Twitter API happened. During the OAuth authentication process, instead of being redirected back to our web app, it would show a screen that said:

"You've successfully granted access...enter the following PIN to complete the process"

After browsing the net and getting help from the Twitter API team, I learned that Twitter recently made a change to their OAuth process to allow this PIN type authentication for applications. See for more info: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/472500cfe9e7cdb9/848f834227d3e64d?pli=1

The oauth ruby gem defaults to using PIN-based process instead of the regular web app redirect process. To fix this, explicitly set the oauth_callback url parameter when getting the request token:

    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )
    request_token = @twitter_client.request_token(:oauth_callback => oauth_confirm_url)

If you are getting a (OAuth::Unauthorized) “401 Unauthorized” error after adding the oauth_callback parameter, try altering your oauth callback method to explicitly state the oauth_verifier as well:

  def oauth_callback
    @twitter_client = TwitterOAuth::Client.new(
        :consumer_key => TWITTER_CONSUMER_KEY,
        :consumer_secret => TWITTER_CONSUMER_SECRET
    )

    @twitter_access_token = @twitter_client.authorize(
      session[:request_token],
      session[:request_token_secret],
       auth_verifier =>params[:oauth_verifier]
    )

Apparently, this was posted on the Twitter API Development Group in late May, but it would have been nice if Twitter DMed every web app signed up on the twitter app list of the change. I guess this is a warning for any Twitter API consumers to follow the dev list closely..

If you come across an OS Process Timed out using a view in CouchDB, you can adjust the os_process_timeout setting in the couchdb futon utility.

[Tue, 21 Jul 2009 23:57:45 GMT] [error] [<0.2804.0>] Uncaught error in HTTP request: {exit,
                                {{bad_return_value,
                                  {os_process_error,"OS process timed out."}},
                                 {gen_server,call,
                                  [<0.2808.0>,
                                   {prompt,
                                    [<<"reduce">>,
                                     [<<"function(keys, values, rereduce) {\n     return sum(values);\n   }">>],
                                     [[[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1],
                                      [[[<<"0832603">>,<<"and">>],
                                        <<"0832603">>],
                                       1]]]},
                                   infinity]}}}

Thanks to Dustin on the couchdb user mailing list for pointing this out.

Got around to install CouchDB from source on a Ubuntu VM. CouchDB is schemaless “document” database server with a RESTful HTTP/JSON API. I’m mostly experimenting it with because of the schemaless design. The map/reduce views took a bit to get used to after being so well acustomed to SQL queries in the past, but I like what I see so far.

While installing it, I ran into a few problems:

Install Erlang(I was having trouble with the 64-bit erlang and openssl package, i ended up just using the 32 bit version):

sudo apt-get install build-essential erlang

First error message:

configure: error: Could not find the js library.
Is the Mozilla SpiderMonkey library installed?

Uh oh, let’s install spidermonkey:

sudo apt-get install libmozjs-dev

Another error pops up:

checking for icu-config... no
*** The icu-config script could not be found. Make sure it is
*** in your path, and that taglib is properly installed.
*** Or see http://ibm.com/software/globalization/icu/
configure: error: Library requirements (ICU) not met.

Oh no, international character support needed, install icu:

sudo apt-get install libicu-dev

And then another error appears:

checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.

Hmm, curl needed:

sudo apt-get install libcurl4-openssl-dev

Now for Couchdb install commands (stolen from the couchdb wiki)

svn co http://svn.apache.org/repos/asf/couchdb/trunk couchdb
cd couchdb
./bootstrap
./configure
make
sudo make install
make clean
make distclean
sudo -i
adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
chown -R couchdb:couchdb /usr/local/var/lib/couchdb
chown -R couchdb:couchdb /usr/local/var/log/couchdb
chown -R couchdb:couchdb /usr/local/var/run
chown -R couchdb:couchdb /usr/local/etc/couchdb
chmod -R 0770 /usr/local/var/lib/couchdb
chmod -R 0770 /usr/local/var/log/couchdb
chmod -R 0770 /usr/local/var/run
chmod -R 0770 /usr/local/etc/couchdb
cp /usr/local/etc/init.d/couchdb /etc/init.d/
update-rc.d couchdb defaults

Start it up with:

sudo /etc/init.d/couchdb start

CouchDB has a really great development community behind it. Subscribe to the mailing list, read the wiki, follow the couchdb twitter and read the book

While cleaning/re-factoring the PeopleJar codebase, we decided to make the move from Rails 2.2.2 to Rails 2.3.2. Rack integration was the biggest single “under the hood” change. Adam Wiggins of Heroku explains the benefits in his Rails Metal, Rack and Sinatra presentation from RailsConf. He has a great example of incorporating a Sinatra app with a Rails app.

No upgrades would be complete without some incompatibilities. As noted on the official Ruby on Rails release notes, test Sessions, Cookies, File uploads, JSON/XML APIs. We found conflicts in the facebooker and ar-extensions gems. Luckily, the developers already pushed out fixes so you can just do a gem update.

There was one other cryptic error in one of our ajax calls:

  Status: 500 Internal Server Error
  private method `split' called for #
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/mime_type.rb:206:in `method_missing'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:51:in `media_type'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:117:in `parseable_data?'
    /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/request.rb:138:in `POST'
    /opt/local/lib/ruby/gems/1.8/gems/actionpack-2.3.2/lib/action_controller/request.rb:428:in `request_parameters'

We could fix this with some good old monkey patching! Add this to an initializer in your config/initializers directory. this is a reported bug at the rails lighthouse and should be fixed in the next release.

module Mime
  class Type
    def split(*args)
      to_s.split(*args)
    end
  end
end

Overall, the upgrade went pretty well: just a few gem upgrades and minor tweaks. If you are on Rails 2.2.2 or older, give the upgrade a shot!

I was doing some code cleanup and noticed we have over 50 models in our models directory. Not good. The first thing I thought was to add namespacing. I googled the topic and turns out namespacing models is a bad idea because of how Rails collapses the model namespaces. Adding sub directories is a better and cleaner solution. Read Josh Susser’s post on the topic.

To use sub directories, you must add the directories to Rails’ config.load_paths in your environment.rb

   config.load_paths << "#{Rails.root}/app/models/profile"

To make this more flexible, you can use this snippet to add all sub directories:

  #Add all sub directories in models to load paths
  Dir.entries("#{Rails.root}/app/models").each do |file|
    next if file.eql?('.') || file.eql?('..')
    full_path = "#{Rails.root}/app/models/#{file}"
    config.load_paths << full_path if File.directory?(full_path)
  end