Tommy Chheng

I ran into this method call quirk in Scala:

If the method has no parameters, you must call it without the parentheses unless the method declaration was defined with empty parentheses.

If you are new to Scala, you’ll probably see this manifest when using iterators.

scala> val wordListIter = wordList.iterator()
:9: error: wordList.iterator of type Iterator[(java.lang.String, java.lang.String)] does not take parameters
scala> val wordListIter = wordList.iterator
wordListIter: Iterator[(java.lang.String, java.lang.String)] = non-empty iterator
scala> val lines= Source.fromPath(filename).getLines
:6: error: missing arguments for method getLines in class Source;
follow this method with `_' if you want to treat it as a partially applied function
Error occured in an application involving default arguments.
val lines= Source.fromPath(filename).getLines
scala> val lines= Source.fromPath(filename).getLines()
lines: Iterator[String] = non-empty iterator

It would have been helpful if the empty parens are listed for the getLines scalaDocs

The convention ensures the uniform access principle. This convention is unlike languages like Ruby where you can call parameter-less methods with or without parentheses.

This except from page 212 of Programming in Scala explains the rationale:

instead of:
def width(): Int
the method is defined without parentheses:
def width: Int

Such parameterless methods are quite common in Scala. By contrast, methods defined with empty parentheses, such as def height(): Int, are called empty-paren methods. The recommended convention is to use a parameterless method whenever there are no parameters and the method accesses mutable state only by reading fields of the containing object (in particular, it does not change mutable state). This convention supports the uniform access principle which says that client code should not be affected by a decision to implement an attribute as a field or method.

One of the great things about scala is that it inter-operate with any existing Java library. Both ways, use scala classes from java and java from scala. Here’s an quick example of using Processing in Scala.

I’m using IntelliJ IDEA 9.0.1 with the Scala plugin.
To get Processing working in IntelliJ:

1. Download the processing.org jars
2. Create a Scala project.
3. Add them to your libraries list in the modules setting.
4. Create a new scala object example below.

How is this useful? Scala’s syntax and its core data structures allow processing of data in a much more concise and understandable manner. Imagine you want to visualize xml or csv files with the Processing/Scala combo. Scala makes this easy with its xml and string manipulation libraries.

This example code is from Ben Fry’s Visualizing Data Book

import processing.core._
object Draw extends processing.core.PApplet {

  override def setup() {
    size(200, 200)
    noStroke( )
    fill(0, 102, 153, 204)
  }

  override def draw() {
    background(255)
    rect(width-mouseX, height-mouseY, 50, 50)
    rect(mouseX, mouseY, 50, 50)
  }

  def main(args: Array[String]) {
    val frame = new javax.swing.JFrame("Draw")
    frame.getContentPane().add(Draw)
    Draw.init

    frame.pack
    frame.setVisible(true)
  }
}

There’s a niffy trick in browsers that you you execute Javascript code in the url location bar. (This is how a lot of security holes are created too!)
Try typing the below into your browser url bar:

javascript:alert('look how cool this is')

Cool, it runs javascript!

You may have noticed a lot of websites take advantage of this to enhance a service. Instapaper’s “Read Later” saves the current page and Pastefire’s “Pastefire this” saves the selected text.

To make your own, just write javascript inside a link tag as a one liner. Then just ask users to click and drag the link to the bookmark toolbar.

Here’s an example for my Next Sprocket site which will start you on a new open source task with the title and description partially filled out:
Sprocket it!


<a href="javascript:(function(){location.href='http://nextsprocket.com/tasks/new?task[title]='+encodeURIComponent(document.title)+'&task[description]=More details at '+encodeURIComponent(window.location.href)})()">Sprocket it!</a>


Great to wake up on Monday morning to see that someone has accepted and paid for the first open source task on our website Next Sprocket! Very happy to see our concept working.

The task was for making gettext rails 3.0 compatibility and providing a sample Rails 3.0 app. Check out the solution here.

If you have a gem that you need upgraded or any open source problem, please post the task on Next Sprocket. Someone will get it done and it will be an open source benefit.

Here’s the slides from a talk about MongoDB at the feb OC Ruby meetup.

MongoDB For Natural Development

There were a few examples from Tim Morgan’s presentation at the LaRubyConf2010 that resonated how ActiveRecord can be a source of many headaches. I wager this is because the ActiveRecord ORM adds complexity to mapping from ruby objects to SQL world. The mapping isn’t one to one and complexity is the unfortunate byproduct.

On the other hand, MongoDB’s data modeling naturally maps to today’s popular programming languages. Essentially, most data are hashes(or associative arrays/dictionaries) and hashes are the basic data model for MongoDB. By having such a similar data representation, the ORM complexity level is extremely minimal compared to a typical ORM designed for SQL.

So use MongoDB because it can reduce the amount of complexity and bugs in your system.

As a developer, I have come to appreciate how useful open source technologies is for me and everyone. Occasionally I come across a problem in an open source project that I can’t fix myself but willing to contribute some money to have it fixed. I am positive many other developers would too.

So I started Next Sprocket with my buddy Nitin. Next Sprocket is the Open Source Developer’s Marketplace.

The basic idea is for any developer to post a task for a reward, let others contribute to the money pot and allow other developers to solve the problem. All the transactions are done via Paypal’s Adaptive Payments API.

You can help by posting a task up or you can vote for us in the Paypal Developer Challenge. The PayPal Developer Challenge voting ends on March 5. The top 10 will be judged by the likes of Marc Andreessen, Scott Cook and many others for the winner.

I have an large set of xml files(110K documents at ~900MB) that i wanted to import into Solr as fast as possible.
Previously I tried just using Jruby/nokogiri and embedded solr connection but that was taking about 15 minutes. So i tried using Solr’s builtin DIH to import the xml files to speed up imports(and re-imports). Here’s how to do it:

Assume you have a large list of xml records like:

<awardlist>
<award>
<awardnumber>0706313</awardnumber>
<title>Coherent Phonon Dynamics  in Semiconductors and Nanotubes</title>
<expirationdate>November 30, 2009</expirationdate>
</award>
<award>
<awardnumber>9909156</awardnumber>
<title>Sustainability of Arctic Communities: Advancing the Science of Integrated Assessment </title>
<expirationdate>November 30, 2009</expirationdate>
</award>
</awardlist>

Add a request handler to your solrconfig.xml file:

<requestHandler name="/dataimport">
<lst name="defaults">
<str name="config">xml-data-config.xml</str>
</lst>
</requestHandler>

Create the xml-data-config.xml file:

<dataConfig>
<dataSource type="FileDataSource" />
<document>
<entity name="nsfgrantsdir" rootEntity="false" dataSource="null"
processor="FileListEntityProcessor"
fileName="^.*\.xml$" recursive="false"
baseDir="/data/rw/data/nsf_grants_xml"
>
<entity name="nsf-grants"
pk="id"
datasource="nsfgrantsdir"
url="${nsfgrantsdir.fileAbsolutePath}"
processor="XPathEntityProcessor"
forEach="/awardslist/award"
transformer="DateFormatTransformer, RegexTransformer">
<field column="id" xpath="/awardslist/award/awardnumber" />
<field column="title_s" xpath="/awardslist/award/title"/>
<field column="expirationdate_dt" xpath="/awardslist/award/expirationdate" dateTimeFormat="MMMMM dd, yyyy" />
</entity>
</entity>
</document>
</dataConfig>

The first entity block will read all xml files in /data/rw/data/nsf_grants_xml and feed it into the second entity block for handling.

After done with the config files, start up solr and visit the dataimport admin page at http://localhost:8983/solr/admin/dataimport.jsp?handler=/dataimport and click the “Full-import”

This is where the import should just magically work for you. Unfortunately, I ran into many errors on the import. The first problem was the DateFormatTransformer couldn’t parse the correct date because it assumes the locale for the documents is the same locale on your machine. Feeling like a good open source citizen, I submitted a patch!

Then, I found xml data files contained illegal characters not allowed in XML:

Caused by: java.lang.RuntimeException: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR, code 11))


Now i have to strip all the illegal characters. I found this helpful Java code snippet from lucas song in the solr mailing list. Just add a FileUtils.readFileToString and FileUtils.writeStringToFile to quickly rewrite the xml files with validated characters.

public class XmlCharFilter {
public static String doFilter(String in) {
StringBuffer out = new StringBuffer(); // Used to hold the output.
char current; // Used to reference the current character.
if (in == null || ("".equals(in)))
return ""; // vacancy test.
for (int i = 0; i < in.length(); i++) {
current = in.charAt(i); // NOTE: No IndexOutOfBoundsException caught
// here; it should not happen.
if ((current == 0x9) || (current == 0xA) || (current == 0xD)
|| ((current >= 0x20) && (current <= 0xD7FF))
|| ((current >= 0xE000) && (current <= 0xFFFD))
|| ((current >= 0x10000) && (current <= 0x10FFFF)))
out.append(current);
}
return out.toString();
}
}


After filtering the xml for correct characters, the DIH import ran pretty fast, around 5 minutes for 120K documents of 400 MB on my Macbook pro laptop.

I recently purchased two Logitech mice: a VX Nano for my mobile use and a RX1500 for desktop use. The VX Nano came in retail packaging and the RX1500 came in OEM packaging. This photo illustrates the problem and solution to sustainable computer packaging. The OEM came with just what I need: the mouse. The retail packaging has everything and the kitchen sink. Who needs a driver CD? A manual for a mouse? All the extra cardboard?

I have been tinkering with the new Paypal Adaptive Payments API and created a simple ruby gem to interface with it. Still pretty new but I’m using it with little problems so far. Submit bug reports if found. See the code at github

Paypal Adaptive Payments API
The adaptive payments api gives you the opportunity to make preapproved payments, chained payments and parallel payments. The chained/parallel payments are great for commission-based apps or if you are trying to connect a buyer to multiple sellers with a single interface.

How to use with Rails:
Install:

sudo gem install paypal_adaptive

Setup your API info by adding a paypal_adaptive.yml to your config folder:

development:
  environment: "sandbox"
  username: "sandbox_username"
  password: "sandbox_password"
  signature: "sandbox_signature"
  application_id: "sandbox_app_id"

test:
  environment: "sandbox"
  username: "sandbox_username"
  password: "sandbox_password"
  signature: "sandbox_signature"
  application_id: "sandbox_app_id"

production:
  environment: "production"
  username: "my_production_username"
  password: "my_production_password"
  signature: "my_production_signature"
  application_id: "my_production_app_id"

Make the payment request:

pay_request = PaypalAdaptive::Request.new

data = {
"returnUrl" => "http://testserver.com/payments/completed_payment_request",
"requestEnvelope" => {"errorLanguage" => "en_US"},
"currencyCode"=>"USD",
"receiverList"=>{"receiver"=>
     [{"email"=>"testpp_1261697850_per@nextsprocket.com", "amount"=>"10.00"}]},
"cancelUrl"=>"http://testserver.com/payments/canceled_payment_request",
"actionType"=>"PAY",
"ipnNotificationUrl"=>"http://testserver.com/payments/ipn_notification"
}

pay_response = pay_request.pay(data)

if pay_response.success?
  redirect_to pp_response.approve_paypal_payment_url
else
  puts pay_response.errors.first['message']
  redirect_to failed_payment_url
end

Once the user goes to pp_response.approve_paypal_payment_url, they will be prompted to login to Paypal for payment.

Upon payment completion page, they will be redirected to http://testserver.com/payments/completed_payment_request.

They can also click cancel to go to http://testserver.com/payments/canceled_payment_request

The actual payment details will be sent to your server via “ipnNotificationUrl” You have to create a listener to receive POST messages from paypal. I added a Rails metal template in the templates folder which handles the callback.

Additionally, you can make calls to Paypal Adaptive’s other APIs:

payment_details, preapproval, preapproval_details,
cancel_preapproval, convert_currency, refund

Input is just a Hash just like the pay method. Refer to the Paypal Adaptive manual for more details.

I’m working a graph problem in Java where I need to sort nodes by its edge count. I have a HashMap of nodes to edge count so I just need to sort a HashMap by its value.
A quick Google search brought up a few solutions. Below is a snippet of a typical solution from StackOverFlow

public static > List getKeysSortedByValue(Map map) {
    final int size = map.size();
    final List> list = new ArrayList>(size);
    list.addAll(map.entrySet());
    final ValueComparator cmp = new ValueComparator();
    Collections.sort(list, cmp);
    final List keys = new ArrayList(size);
    for (int i = 0; i < size; i++) {
        keys.set(i, list.get(i).getKey());
    }
    return keys;
}
private static final class ValueComparator>
                                     implements Comparator> {
    public int compare(Map.Entry o1, Map.Entry o2) {
        return o1.getValue().compareTo(o2.getValue());
    }
}

Can you believe it?? >10 lines of verbose madness! The Ruby solution is:

>> x = {:a => 1, :b => 2, :c => 1, :d => 5}
>> x.keys.sort_by{|k| x[k]}
=> [:a, :c, :b, :d]


This makes you really appreciate closure support in languages. Java 7’s support of closure should bring more sanity to the Java world.

I’m also learning Clojure right now, the solution is just as nice as Ruby.

user=> (def x {:a 1 :b 2 :c 1 :d 5})
user=> (sort-by #(x %) (keys x))
(:a :c :b :d)