I’m using IntelliJ IDEA 9.0.1 with the Scala plugin.
To get Processing working in IntelliJ:

1. Download the processing.org jars
2. Create a Scala project.
3. Add them to your libraries list in the modules setting.
4. Create a new scala object example below.

How is this useful? Scala’s syntax and its core data structures allow processing of data in a much more concise and understandable manner. Imagine you want to visualize xml or csv files with the Processing/Scala combo. Scala makes this easy with its xml and string manipulation libraries.

This example code is from Ben Fry’s Visualizing Data Book

import processing.core._
object Draw extends processing.core.PApplet {

  override def setup() {
    size(200, 200)
    noStroke( )
    fill(0, 102, 153, 204)
  }

  override def draw() {
    background(255)
    rect(width-mouseX, height-mouseY, 50, 50)
    rect(mouseX, mouseY, 50, 50)
  }

  def main(args: Array[String]) {
    val frame = new javax.swing.JFrame("Draw")
    frame.getContentPane().add(Draw)
    Draw.init

    frame.pack
    frame.setVisible(true)
  }
}

You may have noticed a lot of websites take advantage of this to enhance a service. Instapaper’s “Read Later” saves the current page and Pastefire’s “Pastefire this” saves the selected text.

To make your own, just write javascript inside a link tag as a one liner. Then just ask users to click and drag the link to the bookmark toolbar.

Here’s an example for my Next Sprocket site which will start you on a new open source task with the title and description partially filled out:
Sprocket it!


<a href="javascript:(function(){location.href='http://nextsprocket.com/tasks/new?task[title]='+encodeURIComponent(document.title)+'&task[description]=More details at '+encodeURIComponent(window.location.href)})()">Sprocket it!</a>


The task was for making gettext rails 3.0 compatibility and providing a sample Rails 3.0 app. Check out the solution here.

If you have a gem that you need upgraded or any open source problem, please post the task on Next Sprocket. Someone will get it done and it will be an open source benefit.

MongoDB For Natural Development

There were a few examples from Tim Morgan’s presentation at the LaRubyConf2010 that resonated how ActiveRecord can be a source of many headaches. I wager this is because the ActiveRecord ORM adds complexity to mapping from ruby objects to SQL world. The mapping isn’t one to one and complexity is the unfortunate byproduct.

On the other hand, MongoDB’s data modeling naturally maps to today’s popular programming languages. Essentially, most data are hashes(or associative arrays/dictionaries) and hashes are the basic data model for MongoDB. By having such a similar data representation, the ORM complexity level is extremely minimal compared to a typical ORM designed for SQL.

So use MongoDB because it can reduce the amount of complexity and bugs in your system.

So I started Next Sprocket with my buddy Nitin. Next Sprocket is the Open Source Developer’s Marketplace.

The basic idea is for any developer to post a task for a reward, let others contribute to the money pot and allow other developers to solve the problem. All the transactions are done via Paypal’s Adaptive Payments API.

You can help by posting a task up or you can vote for us in the Paypal Developer Challenge. The PayPal Developer Challenge voting ends on March 5. The top 10 will be judged by the likes of Marc Andreessen, Scott Cook and many others for the winner.

Assume you have a large list of xml records like:

<awardlist>
<award>
<awardnumber>0706313</awardnumber>
<title>Coherent Phonon Dynamics  in Semiconductors and Nanotubes</title>
<expirationdate>November 30, 2009</expirationdate>
</award>
<award>
<awardnumber>9909156</awardnumber>
<title>Sustainability of Arctic Communities: Advancing the Science of Integrated Assessment </title>
<expirationdate>November 30, 2009</expirationdate>
</award>
</awardlist>

Add a request handler to your solrconfig.xml file:

<requestHandler name="/dataimport">
<lst name="defaults">
<str name="config">xml-data-config.xml</str>
</lst>
</requestHandler>

Create the xml-data-config.xml file:

<dataConfig>
<dataSource type="FileDataSource" />
<document>
<entity name="nsfgrantsdir" rootEntity="false" dataSource="null"
processor="FileListEntityProcessor"
fileName="^.*\.xml$" recursive="false"
baseDir="/data/rw/data/nsf_grants_xml"
>
<entity name="nsf-grants"
pk="id"
datasource="nsfgrantsdir"
url="${nsfgrantsdir.fileAbsolutePath}"
processor="XPathEntityProcessor"
forEach="/awardslist/award"
transformer="DateFormatTransformer, RegexTransformer">
<field column="id" xpath="/awardslist/award/awardnumber" />
<field column="title_s" xpath="/awardslist/award/title"/>
<field column="expirationdate_dt" xpath="/awardslist/award/expirationdate" dateTimeFormat="MMMMM dd, yyyy" />
</entity>
</entity>
</document>
</dataConfig>

The first entity block will read all xml files in /data/rw/data/nsf_grants_xml and feed it into the second entity block for handling.

After done with the config files, start up solr and visit the dataimport admin page at http://localhost:8983/solr/admin/dataimport.jsp?handler=/dataimport and click the “Full-import”

This is where the import should just magically work for you. Unfortunately, I ran into many errors on the import. The first problem was the DateFormatTransformer couldn’t parse the correct date because it assumes the locale for the documents is the same locale on your machine. Feeling like a good open source citizen, I submitted a patch!

Then, I found xml data files contained illegal characters not allowed in XML:

Caused by: java.lang.RuntimeException: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR, code 11))


Now i have to strip all the illegal characters. I found this helpful Java code snippet from lucas song in the solr mailing list. Just add a FileUtils.readFileToString and FileUtils.writeStringToFile to quickly rewrite the xml files with validated characters.

public class XmlCharFilter {
public static String doFilter(String in) {
StringBuffer out = new StringBuffer(); // Used to hold the output.
char current; // Used to reference the current character.
if (in == null || ("".equals(in)))
return ""; // vacancy test.
for (int i = 0; i < in.length(); i++) {
current = in.charAt(i); // NOTE: No IndexOutOfBoundsException caught
// here; it should not happen.
if ((current == 0x9) || (current == 0xA) || (current == 0xD)
|| ((current >= 0x20) && (current <= 0xD7FF))
|| ((current >= 0xE000) && (current <= 0xFFFD))
|| ((current >= 0x10000) && (current <= 0x10FFFF)))
out.append(current);
}
return out.toString();
}
}


After filtering the xml for correct characters, the DIH import ran pretty fast, around 5 minutes for 120K documents of 400 MB on my Macbook pro laptop.

I recently purchased two Logitech mice: a VX Nano for my mobile use and a RX1500 for desktop use. The VX Nano came in retail packaging and the RX1500 came in OEM packaging. This photo illustrates the problem and solution to sustainable computer packaging. The OEM came with just what I need: the mouse. The retail packaging has everything and the kitchen sink. Who needs a driver CD? A manual for a mouse? All the extra cardboard?

Paypal Adaptive Payments API
The adaptive payments api gives you the opportunity to make preapproved payments, chained payments and parallel payments. The chained/parallel payments are great for commission-based apps or if you are trying to connect a buyer to multiple sellers with a single interface.

How to use with Rails:
Install:

sudo gem install paypal_adaptive

Setup your API info by adding a paypal_adaptive.yml to your config folder:

development:
  environment: "sandbox"
  username: "sandbox_username"
  password: "sandbox_password"
  signature: "sandbox_signature"
  application_id: "sandbox_app_id"

test:
  environment: "sandbox"
  username: "sandbox_username"
  password: "sandbox_password"
  signature: "sandbox_signature"
  application_id: "sandbox_app_id"

production:
  environment: "production"
  username: "my_production_username"
  password: "my_production_password"
  signature: "my_production_signature"
  application_id: "my_production_app_id"

Make the payment request:

pay_request = PaypalAdaptive::Request.new

data = {
"returnUrl" => "http://testserver.com/payments/completed_payment_request",
"requestEnvelope" => {"errorLanguage" => "en_US"},
"currencyCode"=>"USD",
"receiverList"=>{"receiver"=>
     [{"email"=>"testpp_1261697850_per@nextsprocket.com", "amount"=>"10.00"}]},
"cancelUrl"=>"http://testserver.com/payments/canceled_payment_request",
"actionType"=>"PAY",
"ipnNotificationUrl"=>"http://testserver.com/payments/ipn_notification"
}

pay_response = pay_request.pay(data)

if pay_response.success?
  redirect_to pp_response.approve_paypal_payment_url
else
  puts pay_response.errors.first['message']
  redirect_to failed_payment_url
end

Once the user goes to pp_response.approve_paypal_payment_url, they will be prompted to login to Paypal for payment.

Upon payment completion page, they will be redirected to http://testserver.com/payments/completed_payment_request.

They can also click cancel to go to http://testserver.com/payments/canceled_payment_request

The actual payment details will be sent to your server via “ipnNotificationUrl” You have to create a listener to receive POST messages from paypal. I added a Rails metal template in the templates folder which handles the callback.

Additionally, you can make calls to Paypal Adaptive’s other APIs:

payment_details, preapproval, preapproval_details,
cancel_preapproval, convert_currency, refund

Input is just a Hash just like the pay method. Refer to the Paypal Adaptive manual for more details.

public static > List getKeysSortedByValue(Map map) {
    final int size = map.size();
    final List> list = new ArrayList>(size);
    list.addAll(map.entrySet());
    final ValueComparator cmp = new ValueComparator();
    Collections.sort(list, cmp);
    final List keys = new ArrayList(size);
    for (int i = 0; i < size; i++) {
        keys.set(i, list.get(i).getKey());
    }
    return keys;
}
private static final class ValueComparator>
                                     implements Comparator> {
    public int compare(Map.Entry o1, Map.Entry o2) {
        return o1.getValue().compareTo(o2.getValue());
    }
}

Can you believe it?? >10 lines of verbose madness! The Ruby solution is:

>> x = {:a => 1, :b => 2, :c => 1, :d => 5}
>> x.keys.sort_by{|k| x[k]}
=> [:a, :c, :b, :d]


This makes you really appreciate closure support in languages. Java 7’s support of closure should bring more sanity to the Java world.

I’m also learning Clojure right now, the solution is just as nice as Ruby.

user=> (def x {:a 1 :b 2 :c 1 :d 5})
user=> (sort-by #(x %) (keys x))
(:a :c :b :d)


UTM?
When googling solutions for the conversion, I came across the UTM system. It looked related as it referenced the data in the same easting and northing points. UTM is a cartesian coordinate system but generalized for the entire world. It is not as specific as the State Plane Coordinate System.

WGS84?
WGS84 is the latest revision for the World Geodetic System widely used. Google Earth references lat/lng using this system and this is our target.

Convert from the California Zone VI State Plane Coordinate System to WGS84.
Now, back to the problem, our input is 6080411.905 ft easting 2169099.127 ft northing. We want its corresponding lat/lng.

Use cs2cs for the conversion
Download cs2cs from proj.4 cs2cs will convert coordinates from one system to another with the right projection parameters.

Find the corresponding projection for California Zone VI State Plane Coordinate System:

spatialreference.org has an extensive source of different coordinate systems.

Here’s the correct parameters for California VI as found on http://spatialreference.org/ref/esri/102646/

PROJCS["NAD_1983_StatePlane_California_VI_FIPS_0406_Feet",
    GEOGCS["GCS_North_American_1983",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS_1980",6378137,298.257222101]],
        PRIMEM["Greenwich",0],
        UNIT["Degree",0.017453292519943295]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["False_Easting",6561666.666666666],
    PARAMETER["False_Northing",1640416.666666667],
    PARAMETER["Central_Meridian",-116.25],
    PARAMETER["Standard_Parallel_1",32.78333333333333],
    PARAMETER["Standard_Parallel_2",33.88333333333333],
    PARAMETER["Latitude_Of_Origin",32.16666666666666],
    UNIT["Foot_US",0.30480060960121924],
    AUTHORITY["EPSG","102646"]]

Run cs2cs
Pipe the input coordinates to cs2cs with the correct parameters given above.
echo '6080411.905 2169099.127' | cs2cs -f %.16f +proj=lcc +lat_1=32.78333333333333 +lat_2=33.88333333333333 +lat_0=32.16666666666666 +lon_0=-116.25 +x_0=2000000 +y_0=500000.0000000002 +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 +no_defs +to +proj=latlon

Lat/Lng Output:
-117.8701429123491806 33.6091316264922995 0.0000000000000000

Alternatively, if you can’t use cs2cs or you rather implement the conversion directly with the projection parameters. The transformation equations here are listed at http://www.remotesensing.org/geotiff/proj_list/lambert_conic_conformal_2sp.html