Splitting Rails Routes

Problem: We have ever growing list of our rails routes. Coupled with those routes are some that we need only for our non-production environments. This results in a rails env checks all over the place; like this: Of course this is not an ideal situation to be. This leaking over to production can be a simple side effect of removing those conditions mistakenly.

Apart from reasons stated here and for better routes management I know there is a strong case to split routes often. Here is how you can achieve it:

Geek Crawlers isn't dead yet!

As promised in last post, I have started some work towards structuring the crawling script to support pluggable components. The basic design is far from elegant or complete, but at least there has been a start.  If you are interested, find new source code on my git-hub repository here.

More on this, soon!

GC: Geek Crawlers

As part of my job at ThoughtWorks, there are lot of things that I get to do in an interesting manner. We have been brainstorming on how to increase the effectiveness of some of the geek community events that we do. As far as I understand, there are three primary things that we need to ensure.

• Gather people passionate about technology.
• Gather people passionate about technology.
• Gather people passionate about technology.

And thats about it. 

However, it's easier said than done.  As much as we would like to have more and more passionate people joining the events; there is apparent lack of either visibility to events we are doing, to the right set of people or those community events are not compelling enough. (Which ties back to the lack of participation by people who bring in value in terms of passion and knowledge both)

We have tried leveraging the existing forums and meetup lists to ensure, it goes around by word of mouth, however I personally think; that this is something so vital to creating a right eco system that we need to work harder than just that.

So I decided that it's about time; I dug up some of those folks and invited them personally. The idea is to find people who are teeming with knowledge and are visible on different forums like StackOverflow, Twitter,  HackerNews, Quora etc.

In short, crawl for intelligence. Only if it was so easy.

In my effort to find the right set of people; I have started crawling some of these forums for their user base; starting with StackOverflow. 

I decided to take Mechanize (a nokogiri based html parser; which wraps nokogiri parsing capabilities in nice syntactically sugary objects) for a spin. Also along the way, I chose mongodb (mongoid) as the local data store for no apparent reason. 

My objectives were simple:

• Crawl user-base that StackOverflow has.
• Store all users in mongo with location, name, reputation and a url that points to resource.
• Using mongodb's regex capability filter out people based on different criteria like location.
• Manually google-stalk relevant users and make a list whom I should approach in a non intrusive way. (I consider invite emails as spam, and before I make any moves I want to give it a hard thought, as to what would be a good way to invite someone; or let them know of events happening)

Here is a small script that I wrote exactly to do the crawling bit. 

I am certain beyond doubt, that there are better ways to do it. I also think that there is SO API available to do this a little better; but I really wanted to explore mechanize a bit along the way.

Anyway, in couple of hours I had all the StackOverflow users residing in mongodb. Total number of users I could crawl is 663876. Which is a decent number to start off with. Although that does not reflect in anyway the people who are in same geographical location as I am. However that, was a job easily done by regex matches of mongodb.

Sorted by SO reputation; there is lot of data that I can explore on each user; but it's still a task that I have to do manually for now. Although I am inclined to hack together a quick component to call SocialNetwork's API to get more relevant information. Of course it isn't going to be definitive and would still require manual intervention; but if it could do 1/4th of what Rapportive does today (and must I say, it does it well); I would think of it as a successful experiment.

And yet, this is merely one site where I have had an opportunity to crawl with considerable success. I am looking forward to extend this script into a more structured components where in crawlers can be more generic however information extractors could be pluggable components tailored for each portal.

And then I am also tempted to throw a bit of map-reduce into the mix to speed it up. Considering that SO user base isn't as vast as it would be on other sites; I definitely need to structure that better; as I go along.

I'll come back soon and post more results about how this goes. If nothing, it is definitely a great learning experience as it involves playing with large datasets. 

My 45 minutes of romance with Android on HTC Elf/Elfin. (HTC Touch)

Demo Here

Context: I have a useless HTC phone; which thanks to it's hardware lacking doesn't respond well to the network signal of Vodafone and leaves me begging for some usability all the time. So I use a regular nokia stud for my daily conversations. However I haven't left my HTC completely craving for attention as it is my usual fav. subject to experiment. Sometime back I had tried installing Windows 6.5 release candidate and it worked out so well that I kept it as my OS for about an year or so. Finally with the kick off, of I-phone dev, I decided to give android a shot; in case I decide to learn a little bit of Android app-dev. So here are my 10 cents of my 45 mins of flirting with Android.

Yays :)

• Setup was super easy; probably not because Android people made it so; instead because some guys on XDA Developer forum really worked hard and left me a bunch of sooper easy cab files to just click and install with precise instructions. (As mentioned in their post, sound, Wifi, BlueTooth and some other stuff didn't work.. but whatever worked; worked well)
• Android made Stylus totally useless, as it is totally for finger operation and I loved that aspect of it.
• UI was not Apple-ish but it has nicely contrasting colors and animations were not overdone. I loved simple transitions effects.
• Desktop real estate was not cluttered instead was left clean for users. Didn't see an outright usage of google search bar there; but what the heck.. if I have internet I am sure I am going to use it.
• From what it seems like; Android has built in developer tools and instrumenting and profiling of code is possible too. This is really exciting as a developer if you have to work on it. Goes a long way in plucking out issues.
• Some of the neat tricks of pulling down the notification screen from nowhere was cool; I think it was a clean way of putting that together.

Nays :(

• UI Could be much much better still. I am sure, there is a theme support but as if now I didn't find it and that was a bit of disappointment. Especially that same look and feel tends to get boring soon; especially with cellphones as you use them so frequently.
• I didn't find anything outright new and never-seen-before type. I think maybe because I was just looking at core android and not the apps. I saw Nexus-One though and some of the stuff there looks promising and exciting both.
• Performance from my ELF standpoint was a let down. (Jittery screen transitions sometimes) This is understandable too as I realize it was officially release for the models with qualcomm processors in HTC; I am just running a tweaked version inside a Wing Linux Bootloader.

Apart from that I just felt it was a great OS and I'd replace my Windows Mobile anyday with Android, if it came with full/equal feature-set. I have also heard that testing and development framework for android is excellent and much much better than that of I-phone. Especially from unit testing point of view; which is something I'll be checking out soon.

Thanks for Guys on XDA Developer community for showing me the way to get android on HTC Elf. As if now; you don't have to flash it in your ROM; instead you just install the CABS and Wings Linux bootloader loads up your phone in Android; which means it's safe, non-disruptive and easy. You won't brick your phone and you won't loose your data. (Never hurts to make a backup though!)

Finally, will I choose to work on Android app-dev over I-phone?

My answer is "yes". For simple reason.

1. I am pro-google.

2. I feel Android has a brighter future considering that Apple is focused on making dead closed systems; from chips to OS coming in-house starting I-Pad. On the other hand; Android has not only created waves but also has mass adoptors ranging from cellphone to net-book manufacturers.

3. Better development framework and testing support.

4. I am guessing bigger community? (that shouldn't matter though!)

Hope to see you on the other side!!

Sneaky JSONP to do cross domain scripting

Last year I was working on a project which included an existing legacy web application communicating with an embedded iframe widget in it. Since we didn't want to go back to servers for calls; we *desperately* wanted to do cross site scripting between those two sites. Unfortunately; widget and host application were hosted separately and Javascript wouldn't work seamlessly due to cross domain security issues.

At that point the best I could do was a "hash fragment modifier" approach, which somehow did the trick and then using that token web servers synced up later between themselves. The whole solution worked quite well; and went into production and is running stably till today (hopefully).

But I had always wished there was a better way to communicate between those two. Now I am involved in developing public APIs for an e-commerce/social-networking site and I just stumbled upon JSONP. It's amusing how things just don't turn up with repeated Google searches but sneak in quietly; when you are searching for an unrelated problem.I wish I was wiser when I had to actually implement that solution. I spent several hours today understanding how Cross Scripting with JSON works and turns out it's such a neat way to do some things. Just wanted to share it with anyone; who may be hunting for a similar solution. There are two parts to the solution. One with JQuery (I gotta admit; I am just growing more and more fond of jquery; as it's absolutely brilliant) and one in simple HTML/Javascript language. I'll go with latter first.

• Rule of thumb: If you make a ajax call to get data from a domain which is not in same domain as the hosted application and nor is a sub domain; then Java-script raises security errors and doesn't let you communicate. This is due to cross domain scripting issues. 
• The Loophole: If you include a script tag for javascript on your page with src pointing to a javascript resource lying on a remote server it allows you to access and run that javascript. 

Using the second point what we do is; create script tag dynamically and assign it's source to a URL that returns data encapsulated in a method call. The condition is that method should exist in your web-page. This is JSON Data wrapped in a callback function. The limitation is that, both requesting domain and serving domain should know the name of the callback function. (That is if you are doing the plain HTML/Javascript way). Jquery provides seamless support for this in it's getJSON method. I am using twitter API's to fetch my Twitter stream; as twitter has a support for JSONP and while requesting for JSON, you can pass the callback function name. For those who are unaware; if you are working on Ruby on Rails; it's gonna be a piece of cake supporting JSONP in your APIs as if you look at the to_json method's documentation; it has an in-built support for callback method wrapping; all you need to do is pass additional parameter :callback. As long as the value for that is set; it'll wrap it in a req. callback method. Convention is, if no method name is specified; the default method name is "callback". Hope it has helped.

Reject/Ignore/Stop Rails Active Record Callbacks Selectively

Faced a small issue today. I had a model in rails; where Model had :before_save filter set for the model. To make things more complex before_save had not one, but several invocations.

Unfortunately one of those invocations went about fetching a service instance from injected spring context. This really pissed off a database migration script; because it didn't have spring context.

So effectively I needed to ignore that particular before_save callback. After a bit of googling I found the solution.

If you want to skip all the callbacks on a particular model; use following statement:

MyModel.before_save.clear

or to skip a particular method invocation following statement can be used:

MyModel.before_save.reject! {|callback| callback.method == :method_name}

This would skip the call to particular method invocation. It's particularly helpful in

1. Rake Tasks, where you may not want some of the filters/callbacks to run as they do in actual code.
2. Database Migration scripts

There can be other reasons; why you may want them to be skipped. Whatever works for you man..

Why RESTful over SOAP (almost always)?

In a bloated world of world wide web the biggest issue is "interoperability". Don't let anyone distract you from this point. It would be imprudent to assume that islands of isolation is what makes web a happening place.

SOAP has been around for a while and has proven it's grit with several enterprisy solutions; implemented successfully. And while SOAP has given it's run for money; it's time to catch some REST.

I am no expert on either subjects; but from my superficial understanding of both I'm going to delve into a bit of "What is REST?" and "What does REST brings to Table?" What are the primary differences between REST and SOAP. And while I am doing "all that"; I'll try to suggest that REST is more suitable than SOAP for most applications; but don't you go and buy my garbage; until you see for yourself.

So lets get rolling.

What's up with RESTing anyway?

RESTful design is not a standard. It's like a architectural style. Somewhat like Client-Server analogy; which is again not a standard but a design choice. REST leverages existing features/methods of HTTP as it works gracefully over it. That is one severe lacking of SOAP. It fails to harness the HTTP capabilities to it's optimal level. And thats what I think; is a big drawback in complex and "enterprisy" solution; as you often build those things ground up; which otherwise could have been used; if HTTP and it's features were used properly. More on that as we get our hands dirty with details. Oh and by the way, REST stands for Representational State Transfer. Thats so, because as you request resources from a system; it changes it's state to pass back the information and so it's basically passing you back the representational state. Which is why R.E.S.T

What does REST brings to Table

My one word answer to that is "Simplicity".

I wanted to go with "A lot", but figured; maybe that won't go down as 'one word answer'. Now to answer that question; as to what does really REST brings to us; think of yourself surfing a website. You see information and you see links. Each link (when clicked) transforms the page into a portal which provides you more detailed/relevant information. It is not so in web services between disparate systems. You need to be explicit, have pre-defined formats and make appropriate calls to get information which is limited to services conceived notion of "what is expected with a call to this service?"

I hope you are following.

Now think that you want to bring the browsing experience to web services. Can you do that? Of course you can do that. It just needs some smart programming with SOAP and voila. But what if you can escape a big chunk of that programming to inculcate browser like experience in your web services; instead you use existing capabilities of XML, HTML, URL, HTTP, Http header etc.

Here is a small logical example to clear things up in more detail. But before I do that; here are the REST rules:

1. Everything is represented as a resource and you perform operations on a resource. So for example if "post" (on a blog) is a resource then http POST means you are trying to create a new post; a GET operation means, you are trying to fetch a post, a PUT request means update and a DELETE request means you want to delete. (So now the http request TYPE defines the operation)
2. A base URL + a NOUN represents a resource and thats how operations are done on it. For example for a blog post a possible URL can be http://www.slymonkey.com/post/1
3. A GET request would always either provide a list of resource type specified or will provide a detailed info if requested with appropriate id.
4. Each response would contain a way to navigate to more details. This would be achieved by either including URLs with appropriate resource ids embedded or other standard formats. (this is req. only if more details can be possibly fetched. A small example to showcase in a bit)
5. Describe how services need to be invoked? WSDL or HTML?
6. Specify a response formate (DTD, W3C schema etc) for a POST/PUT response. (If a service accepts POST/PUT response)

A simple example now.

Assume a simple service with maintains electronic gadgets. It provides options to list the gadgets, get gadget details and option to view technical specifications. Also a new order can be placed and updates to order can be done.

• Getting listing of gadgets: http://dumb-gadget.com/gadgets
• A response to above GET request may look like this. If you'll notice the url for detail about each gadget is provided alongside. These urls can provide more information about each gadget and each of those request will be a GET request but for a specific resource rather than listing as it provides ID of the resource in the URL.

• The response for each gadget details could in turn contain URL for more detailed technical specifications in appropriate section.

As you can see that this approach actually simulates a browsing kind of experience in web-service. And only development of code that you have done is, handling web service requests in a REST format of resource and http Request TYPE. If you do that; rest should fall into place already.

Now if you wanted to create an order; you could fill the details in an XML and do a POST for the resource "Order" and system would know that you are trying to create an order. Another simple way would be, as displayed by following URL:

http://dumb-gadget.com/gadget/100/order

If the above mentioned URL is accessed in POST way, then it would explicitly mean that a request is being placed to place a new ORDER for GADGET with id 100. Order could contain all details like quantity and other miscellaneous attributes; however gadget id may not be required as that would be understood by service looking at the URL.

I presume that now you see; how we are leveraging HTTP request types and URLs to specify things explicitly. And this is not even the beginning. :)

Here are other things that we often forget about HTTP; when we talk in web service terms:

• Caching: Http supports caching so you can mark your responses as cachable if you want to. That would hold up repeated requests to server if the last provided data is not yet expired as set by you. This is especially good for system where data change is not very frequent; instead it's a ad-hoc lookups that system serves.
• Uniform interface: No new protocol, no new standards. HTTP, URL etc is know well globally.
• MIME Types to specify the response type. The client can now render based on HTTP headers the response appropriately.
• Proxy, Firewalls, Gateways and other security guidelines may be imposed without any issues. This leads to a layered architecture without any change in contract or any knowledge to client as to how it is dealing with the server. This is especially because everything rides on HTTP protocol.

Now that I have managed to blabber so much about REST; I must mention once more, "explicitly", that REST achieved everything that SOAP does and with much more simplicity and much more optimal usage of existing capabilities. However that does not mean that REST is the blind solution to everything and SOAP can he de-commissioned here on and considered dead.

I still personally feel; that SOAP is a good choice when you have complex types in contract. Complex types can be returned with SOAP and works well. Although it's always a good idea to break complex types into simpler primitives and use REST; but that may not be possible always and in those cases SOAP is a decent choice.

I am sure there are more intricacies on comparisons of SOAP and REST; but this was my general and a very high level take on it; especially for those, who are just getting started with REST or want to get started.

For a comprehensive text on REST please refer Roy Fielding's Dissertation on REST

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

Thunderbird Plugins

Finally dark ages are over and we have got rid of notes. Of all the desktop email clients I have had to use (corporate level); I have hated the Lotus Notes most. I will never understand how it managed to survive so long in market.

Thoughtworks is officially on cloud and we have migrated to robust Google solution of apps on cloud. Gmail, Google docs, Google Calendar.. the whole nine yards. And who knows; maybe in foreseeable future, google wave!

Funny part of this transition is that; while I was looking forward to move to gmail due to it's feature rich nature; as soon as I moved to gmail; I craved for a desktop client to manage emails. I know; I don't need one; but I guess it's just become second nature to classify corporate email accounts with thick desktop clients. So I downloaded Thunderbird.

It's an excellent desktop email client. It took a while to set it up with all the plugins; but once I was done; I was really happy with what I accomplished. Here is a comprehensive list of plugins that I used to set my thunderbird up; to enable personalization to a great extent and to peak my productivity:

• CS Lite: Cooke manager add on for using thunderbrowse successfully.
• Thunderbrowse: To enable a web browser within the email client. Saves me trouble.
• Google Contacts: To sync all contacts from my google address book.
• Lightning: To integrate SunBird Calendar with Thunderbird calendar. Sunbird is in turn synced with google calendar. Works flawlessly.
• Minimize to Tray: I like to see my email client in tray rather than in task bar, as it's open 24x7.
• mozpod: To sync my contacts to my i-pod when I connect my i-pod to laptop.
• Provider for Google Calendar: To integrate google calendar with Thunderbird's Lightning extension.
• SamePlace: IM add on for Thunderbird. Lets me go online on my personal gtalk account as well as thoughtwork's gtalk account. Increases my productivity; as I am saved from multiple windows.
• xmmp4moz: Req. for SamePlace to integrate with thunderbird.

I must say, that to setup filters and other stuff I do visit my gmail inbox now and then; but apart from that thunderbird has been amazing and I have really gotten used to it in last few days.

Windows 6.5 on HTC Touch (Elf)

I have been meaning to put some info somewhere about installing Windows Mobile 6.5 on HTC Touch (Elf). I did it few days back and results have been splendid for me. Works like a charms and I thoroughly enjoy the experience.

Although I must mention clearly; I am not expert and I am reading and learning too. This post provides only a outline of step by step process that I underwent to install it successfully. But it by no means in warrantable or guranteed. :)

To note; I chose ONYX Windows Rom; which includes several other apps pre-installed. Following are the things you'll need.

ONYX 5 - At the bottom of the screen we have few releases. Choose the latest one.

ROM Backup - Rom Backup software, which will help you revert back to old version if new install is a failure. It's critical, that you maintain a backup; as it may save a lot of trobile to you.

CID/Device Info - CID and Device info will be needed if your cellphone bricks

Hard-Spl Bootloader - Bootloader, req. to load new version of windows on your cellphone.

That is it! So once you have all of it present on your computer; we are all set to go. Few precautions that you must take though.

• Make sure that your computer is not liable to switch off during installation.
• Cellphone must have more than 50% battery at least.
• Cellphone should not have any screen savers, stand by modes enabled.
• Treasure link as this may help you unbrick if requiered.

To give little background about whole process; your cellphone comes with a Official Bootloader which basically installs only signed ROMs and since the ONYX Rom that we are going to burn is unwarrantable and unsigned; it would refuse to burn it. And that would be; once you have deleted everything from your cellphone. So it bricks, if that happens.

Take backup of all your data. My Preferred tool for backup is PIM Backup but of course you can use your fav. Once data has been backed up; follow the below given steps with some common sense and guidance provided on screen by various tools:

1. Figure out and save CID and Device info of your cellphone using the downloaded tool. (Mentioned above in step 3).
2. Take back up of your existing ROM. The instructions are given on the page where you download it from.
3. Install SPL Hard bootloader on your cellphone. It'll pretty much want to restart itself. You may be req. to try several times to ensure it is done successfully. If this step fails and you try to burn ONYX rom; your cellphone may brick.

Finally you are ready to burn the Windows 6.5 on your cellphone.

Once you are through; you can set up your screen and settings. PIM Backup comes preinstalled to reload all the data. And if you were as lucky as me; in as less as 45 minutes you will be done.

For more help; contact XDA developers forum; they have all your answers and tips/tricks there. Also for a more detailed process follow this link.

Hope it helped! So long.