Pushing to Amazon Elastic Beanstalk with Git

I’ve just deployed my first Ruby and Sinatra app to AWS Elastic Beanstalk and it was pretty simple. One catch I did have at first was using Bundler. Everything was setup on my Mac and working fine, however once I had pushed to AWS, Passenger was returning the error:

Could not locate Gemfile (Bundler::GemfileNotFound)

After trial and error I discovered that the gemfile on my mac was named ‘gemfile’ but on my EC2 instance needed to be called ‘Gemfile’ (with a capital G). A quick rename, commit and push and Bundler worked fine.

Fiddler tips for HTTP Debugging

Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP(S) traffic, set breakpoints, and “fiddle” with incoming or outgoing data.

Download Fiddler from http://www.fiddler2.com/, it’s freeware! It runs on Windows, but can debug traffic originating in any operating system (by making that OS point to Fiddler on Windows as a proxy). Before reading this you should read these articles which provide an overview of Fiddler.

Stubbing network responses

During development with a third party it’s often handy to insulate yourself from any downtime/network problems that might affect your testing. Quite often this involves writing a piece of code to simulate network responses and pointing your app to that. Instead of doing this, turn to Fiddler.

Record and replay

Configure your application to use Fiddler as a proxy (see this for .NET apps, use localhost:8888), then hit your third party endpoint with your application. Fiddler will capture the traffic in the session list. Now, click on the Auto Responder tab and enable Automatic Responses. Drag each row from the session list into the Auto Responder list. Now re-run your app, and instead of connecting to the remote machine, Fiddler will auto-respond for you. (If you are using SSL, read how to decrypt SSL traffic and also in .NET you’ll need to suppress the invalid man-in-the-middle cert that Fiddler uses by returning true in the ServerCertificateValidationCallback

From an interface spec

If you have an interface spec but no endpoint to hit, create a file matching the content you expect to be returned, define a match for the URI, and use your sample file as the response content. See AutoResponder reference for more information.

You can use regex pattern matching for the URI, and you can either respond with a local file, or captured session. With a regex to match the entire host you can make all calls to your network resource respond with a HTTP 403 Denied and ensure your app behaves as expected.

Custom rules to show Akamai cached pages

I’ve used Akamai edge caching on a number of sites over the past few years to improve site performance, and it’s always useful to see which pages are being served from cache, and which aren’t. The easiest way I’ve found to do this is to add a custom rule to Fiddler to highlight requests for me. From Fiddler, choose Rules, Customize Rules. In the Javascript that opens, enter the following code:

With the other field definitions…

	public static RulesOption("Highlight Akamai cache Hits")
	var m_HighlightAkamaiHits: boolean = false;

In the “OnBeforeRequest” method…

	if (m_HighlightAkamaiHits) {	
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-remote-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-true-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-check-cacheable");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-extracted-values");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-nonces");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-ssl-client-session-id");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-serial-no");

In the “OnBeforeResponse” method…

	if (m_HighlightAkamaiHits) {
		if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_MEM_HIT")) {
			oSession["ui-customcolumn"] = "HIT";
		} else if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_IMS_HIT")) {
			oSession["ui-customcolumn"] = "HIT";

Now close the Javascript file, and go back to Fiddler. If you made any mistakes in the Javascript, Fiddler will tell you immediately. From the Rules menu you now have a new option – “Highlight Akamai cache Hits”. Enable this, and visit http://www.facebook.com/ in your browser. In Fiddler, you should see the word “HIT” for several of the requests in the “custom” column. You can rearrange the column order to move the custom column if you like.

Add request time

This is a simple new rule but surprisingly handy.

With the other field definitions…

	public static RulesOption("Show response time")
	var m_ShowResponseTime: boolean = false;

Add to either “OnBeforeRequest” or “OnBeforeResponse” method…

	if (m_ShowResponseTime) {
		oSession["ui-customcolumn"] = DateTime.Now.ToString();

Remember when using these rules that when you save the Javascript file, the Rules menu will be reset so any previously enabled rules will need re-enabling.

Fiddler also has a nice set of C# APIs which allow you to embed the fiddler engine directly into your test suite, which makes for a really nice set of integration tests (using the AutoResponder) with only a few lines of code. I’ll go into this in a future post.


Mozilla has launched an interesting labs add-on for Firefox called Ubiquity. It’s a productivity tool for passionate users, as the Scobleizer writes “It’s not for non-passionate Internet users. They won’t get it.“.

Download it, read the tutorial, have a play, extend it, and increase your productivity :o)

UPDATE 8/Sep/2008: I’ve creaed a ubiquity command for my Pub Finder, to install it, visit my install page. To activate it open up Ubiquity and type pubfinder and hit enter to see the pubs.

Xml Feed from the London Underground Real-time status

I’ve written a script to parse the London Underground real-time tube status page and convert it into an Xml feed. It caches the output from TfL for 1 minute. If their page structure changes i’ll endeavour to update my parser.

You can get the feed as Xml or HTML. I’ve also started the ball rolling with a Mac OS X Widget (15kb) (download, unzip and install), and a Google Gadget (add http://www.socialanimal.com/tube/status.php?format=google as a custom URL in iGoogle.

Enjoy and let me know if you find any problems with the feed.

UPDATE 13/May/08: TfL are now offering a selection of iGoogle gadgets. At the moment I can’t find an Xml feed, and the XHTML gadget just states ‘loading’ though which is a shame.

UPDATE 8/June/08: I’ve released the source code for this under GPL license so you can use it in your programs. Please read the GPL license to ensure your usage complies. I didn’t write the output caching files i’ve included and I can’t find the original author to check licensing terms.

UPDATE 24/Dec/08: I was notified recently that the feed had broken since TfL updated their site to add their Travel Tools selection, but still no Xml feed! Anyway, i’ve updated my parser code so it’s all working again. I’ve also added another last updated time attribute to the Xml version, it’s expressed in seconds since 1970 (ala Unix).

UPDATE 2/Feb/09: I’ve added an RSS feed format to the status feed. Not sure how useful it’ll be, let me know.

Screengrab a website in Firefox

I’ve had a few occasions to print out a website exactly as the browser is displaying it. Usually it involves a screen grab, scroll down, screen grab, scroll down etc. and then stitching the images back together. I’ve now found a much simpler solution – ScreenGrab:

Screengrab! is an extension for Firefox that makes it easy to save a web-page as an image. With it, you can save anything that you can see in a browser window – from a small selection, to a complete page.