Travel times of the UK compared


from BoingBoing.net… “The MySociety project has produced an incredible set of travel-time maps of the UK, showing the voyage-time using color shading (red for close, blue for far) and contour lines to indicate each hour’s travel — they compare the overall travel time for going from A to B by rail and car and cab. They’re laying an open geodata-bank for use in correlating house prices to travel times, cost-to-time, and generating realtime web-services.”

Advertisements

Debugging C# memory consumption

The website I work on recently received a large increase in visitors due to a marketing drive. This caused IIS to start recycling the worker process very often as memory consumption was increasing out of control. The user experience wasn’t affected by this problem as we run a load-balanced system, however we had to fix it of course.

2 weeks ago, I knew nothing about debugging a live server. Now I know a fraction more. Here’s a quick guide to what we did to identify the source of the problem, and then fix it.

Using perfmon we monitored various counters on the production servers. The useful ones were:

.NET CLR Memory
Gen Heap 0 Size
Gen Heap 1 Size
Gen Heap 2 Size

This showed us that over a period of time, lots of things were getting put into the 2nd generation heap and were never being garbage collected. But how do we see what objects are actually in that heap? Enter WinDbg – the Windows Debugger (obvious huh?) This app feels very intimidating with it’s hex output and cryptic commands, but once you get the hang of a few commands you’re away.

So, to debug our app I needed a memory dump. I copied (note: not installed, xcopy is fine) the Windows Debugging Tools onto the production server. I could then run “cscript.exe adplus.vbs -hang -iis” which creates a dump of the memory being used by IIS. This file was copied to my local machine for analysis.

I loaded up the WinDbg tool and opened the memory dump. Type “.load clr10sos” in order to load the .NET debugger extension – son of strike. Then, enter “!dumpheap -stat -gen 2” to get a list, by statistical popularity of the objects in the 2nd generation heap. I was able to see a large number of SqlString objects belonging to NHibernate, an open-source ORM, still lingering about. Using the ants profiler and OpenSTA – a load test tool – I was able to view the object hierarchy, i.e. which objects are holding a reference to the SqlStrings.

A little bit of investigation further and I found that NHibernate QueryTranslators which translate HQL to SQL are cached based, in part, on their unique HQL string. This meant that every different HQL string passed to NHibernate would create another cached QueryTranslator. With no scavenging going on, this was surely the problem cause. I replaced the internal hashtable with a cache implementation from the NHibernate environment and all was well.

Having tested and deployed this fix, memory consumption was drastically reduced, by still rising. After some more perfmon monitoring I noted that the ASP.NET cache was now increasing at a similar rate to the gen 2 heap. A bit more ants profiling and it turned out that items are being put into the cache with an expiration policy but are never expiring.

Then I spotted it. In the SysCache code when items are added they are added with an expiration policy, but if they are updated, they are added using the Cache shorthand notation “Cache[key] = value” which does not set an expiration policy. This meant popular items would cached correctly initially, but then subsequently their expiration policy would be overridden and they would remain in memory until the worker process was recycled.

After a quick code change to the SysCache provider this functionality was corrected, and today the updated version has been released and is behaving properly.

Useful links:
http://blogs.msdn.com/tess/ – A blog written by one of the Microsoft team.
http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/

Programming through interfaces

Here’s a quick example of how we do layer seperation through interfaces to make each layer testable. Although the tests run, the UI section is untested. Download file.

[csharp]
using System;
using System.Web;
using System.Web.UI;
using NMock;
using NUnit.Framework;
using Socialanimal.Example.Business;
using Socialanimal.Example.Business.Implementation;
using Socialanimal.Example.Core.Entities;
using Socialanimal.Example.Core.Interfaces;
using Socialanimal.Example.SqlDatabase;

namespace Socialanimal.Example.Core.Entities
{
public class Employee
{
public string Forename;
public string Surname;

public Employee(string forename, string surname)
{
this.Forename = forename;
this.Surname = surname;
}
}
}

namespace Socialanimal.Example.Core.Interfaces
{
public interface IFacade
{
Employee GetEmployee(string surname);
}

public interface IBusinessManager
{
Employee GetEmployee(string surname);
}

public interface IDatabaseProvider
{
Employee GetEmployee(string forename, string surname);
}
}

namespace Socialanimal.Example.Business
{
public class FacadeFactory
{
public IFacade CreateFacade()
{
IDatabaseProvider databaseProvider = new SqlDatabaseProvider(“server=localhost;database=test”);
IBusinessManager businessManager = new BusinessManager(databaseProvider);
IFacade facade = new Facade(businessManager);
return facade;
}
}

public class Facade : IFacade
{
private IBusinessManager businessManager;

public Facade(IBusinessManager businessManager)
{
this.businessManager = businessManager;
}

#region IFacade Members

public Employee GetEmployee(string surname)
{
return businessManager.GetEmployee(surname);
}

#endregion
}
}

namespace Socialanimal.Example.Business.Implementation
{
public class BusinessManager : IBusinessManager
{
private IDatabaseProvider databaseProvider;

public BusinessManager(IDatabaseProvider databaseProvider)
{
this.databaseProvider = databaseProvider;
}

#region IBusinessManager Members

public Employee GetEmployee(string surname)
{
if (surname == null)
{
throw new ArgumentNullException(“surname”);
}
return databaseProvider.GetEmployee(null, surname);
}

#endregion
}
}

namespace Socialanimal.Example.SqlDatabase
{
public class SqlDatabaseProvider : IDatabaseProvider
{
public SqlDatabaseProvider(string connectionString)
{
}

#region IDatabaseProvider Members

public Employee GetEmployee(string forename, string surname)
{
// Get from database
return new Employee(“john”, “doe”);
}

#endregion
}
}

namespace Socialanimal.Example.ClientWebSite
{
public class BasePage : Page
{
private IFacade Facade
{
get { return HttpContext.Current.Items[“Facade”] as IFacade; }
}

protected override void OnLoad(EventArgs e)
{
Employee employee = this.Facade.GetEmployee(“john”);
Response.Write(employee.Forename + employee.Surname);
base.OnLoad(e);
}
}

// global_asax
public class Global : HttpApplication
{
protected void Application_BeginRequest(Object sender, EventArgs e)
{
HttpContext.Current.Items[“Facade”] = new FacadeFactory().CreateFacade();
}
}
}

namespace Socialanimal.Example.Business.Tests
{
[TestFixture]
public class FacadeTest
{
[Test]
public void TestFacade()
{
// Create a mock IBusinessManager
DynamicMock mock = new DynamicMock(typeof (IBusinessManager)); // Create a proper Facade
Facade f = new Facade((IBusinessManager) mock.MockInstance); // Set expectations for this request
Employee expectedResult = new Employee(“john”, “smith”);
mock.ExpectAndReturn(“GetEmployee”, expectedResult, null, “smith”);
Employee actualResult = f.GetEmployee(“smith”);
Assert.AreEqual(expectedResult.Forename, actualResult.Forename);
Assert.AreEqual(expectedResult.Surname, actualResult.Surname);
mock.Verify();
}
}

[TestFixture]
public class BusinessManagerTest
{
[Test]
public void TestBusinessManager()
{
// Create a mock IDatabaseProvider
DynamicMock mock = new DynamicMock(typeof (IDatabaseProvider));
// Create a proper BusinessManager
BusinessManager bm = new BusinessManager((IDatabaseProvider) mock.MockInstance);
// Set expectations for this request
Employee expectedResult = new Employee(“john”, “smith”);
mock.ExpectAndReturn(“GetEmployee”, expectedResult, null, null, “smith”);
Employee actualResult = bm.GetEmployee(“smith”);
Assert.AreEqual(expectedResult.Forename, actualResult.Forename);
Assert.AreEqual(expectedResult.Surname, actualResult.Surname);
mock.Verify();
}

[Test]
public void TestGetEmployeeWithNullSurname()
{
// Create a mock IDatabaseProvider
DynamicMock mock = new DynamicMock(typeof (IDatabaseProvider));
try
{
// Create a proper BusinessManager
BusinessManager bm = new BusinessManager((IDatabaseProvider) mock.MockInstance);
bm.GetEmployee(null);
}
catch (ArgumentNullException ex)
{
Assert.AreEqual(“surname”, ex.ParamName);
}
finally
{
mock.Verify();
}
}
}
}
[/csharp]

C# Tools and Frameworks

It’s been a while since I wrote about the C# tools / frameworks i’m currently using, so here we go.

  • log4net – for all our logging needs.
  • nunit – automated testing.
  • nmock – for mocking interfaces and writing more testable code. (see my blogpost)
  • nant – automating our build cycles (although we’re looking at msbuild to replace Cruise Control, as it may fit better with our skill set).
  • OpenSTA – for web-based load testing. Microsoft took a step back with ACT replacing WAST, and although the new version of ACT in vs2005 is supposed to be great, OpenSTA is providing an interim solution for us.
  • Selenium – for web-based smoke testing.
  • nHibernate – our chosen ORM. (see my blogpost)
  • nSpectre – provides entity validation within our systems. Written mainly by my colleague Mark with occasional input from myself and another colleague Ben.