Category Archives: Tools

Configuring multiple public DHCP IP addresses on a Linksys WRT54G with OpenWrt

I hit a problem the other day whilst trying to map a bunch of public IP addresses (provided by Virgin Business) to various services within the network.  Essentially I’m running a VMWare ESXi server with several web servers on, and I want to use the public IP addresses to expose these servers to the Internet through the business broadband connection.

Rather than splash out on some expensive networking kit, I decided to have a go at hacking with OpenWrt (an open-source mini-Linux router firmware which can run on number low-end network devices).  I decided to plumb for a new Linksys WRT54G, as these are renown for their support of firmware upgrades like OpenWrt (also dd-wrt and tomato to name a few).

The Challenge

When you buy public IP addresses from your ISP, most will hand you a static block which are assigned specifically to your account.  However, some ISPs (Virgin Business in my case), assign your public IPs by DHCP (I initially bought 5).  So every time you connect a different device to the cable modem their DHCP server hands out a different public IP address.  Whilst this seems all very nice, it makes life more difficult when configuring a router to listen on all of those addresses.   DHCP hands out a public IP and stores the hardware MAC address of the device requesting an address in it’s lease table.  The problem with trying to grab more than one address from a single router is that it only has one physical network port (for WAN), and therefore only one hardware MAC address.  The trick here is to create multiple virtual interfaces in the router, each with their own (made up) MAC address, so they can each make a DHCP request to the ISP.

The Hardware

It turns out that there are a plethora of models of Linksys WRT54G, some with different hardware and supporting different firmware features.  The way to check is to turn the router upside down:

Untitled

(N.B. I have highlighted the area to look at with the red box.)

In my case I had a WRT54GL v1.1, which I bought from eBuyer for £45.

The Firmware

I originally looked at tomato, but the project seems to be languishing and gathering Internet dust.  I settled on OpenWRT because I found the kmod-macvlan package which allows you to create virtual MAC interfaces on the router, which is exactly what I needed.  I followed the installation instructions here http://wiki.openwrt.org/toh/linksys/wrt54g (scroll down to the Installing OpenWRT section).

Now, this is important: you must use the bcrm47xx targeted build of OpenWRT to get access to the kmod-macvlan package (this took me a while to figure out).  The one I used came from here:

http://downloads.openwrt.org/backfire/10.03.1/brcm47xx/

After flashing your device (either via SSH if enabled, or via the web GUI), I suggest you enable the SSH daemon, life gets much easier that way.

Network

Here’s an overview of my network setup:

image

OpenWRT comes with a package manager called opkg which is incredibly useful for installing/managing additional packages.

opkg update
opkg install ip
opkg install kmod-macvlan

This will get the latest list of packages from OpenWRT and install ip and kmod-macvlan packages which we need to configure the virtual MAC interfaces.

Next I modified /etc/rc.local to create the virtual MAC interfaces:

# set up virtual mac addresses as aliases on the main WAN i/f eth0.1

ip link add link eth0.1 eth2 type macvlan
ifconfig eth2 hw ether 58:55:ca:23:32:e9

ip link add link eth0.1 eth3 type macvlan
ifconfig eth3 hw ether 5d:a4:02:04:24:0d

ip link add link eth0.1 eth4 type macvlan
ifconfig eth4 hw ether 8C-89-A5-57-80-E7

ip link add link eth0.1 eth5 type macvlan
ifconfig eth5 hw ether 58:4f:4a:df:40:03

ifup -a

# default route
route add default gw 82.7.16.1 dev eth0.1

exit 0

This script configures 4 additional virtual interfaces on top of the main WAN interface (eth0.1), each with it’s own unique MAC address (you can generate a random MAC address using the instructions here.  I’ve added a default route to Virgin’s router to make life easier when it comes to configuring the firewall (you can find out what your’s is by running ifconfig before making any of these changes).

For each new WAN interface I added a section to OpenWRT’s network config (in /etc/config/network):

config 'interface' 'wan2'
    option 'ifname' 'eth2'
    option 'proto' 'dhcp'
    option 'defaultroute' '0'
    option 'peerdns' '0'
    option 'gateway' '0.0.0.0

This maps the WAN2 interface onto the eth2 hardware device, and specifies that it should obtain an address using DHCP.  The route and gateway entries are to force all outgoing requests through the main WAN interface (eth0.1).

After saving these changes you’ll need to reboot your router (use reboot –f).  You can then check the status with ifconfig – look at each interface and check they all have public IP addresses.

Firewall

The next step was to configure some Network Address Translation (NAT) rules in the firewall to forward traffic coming in on certain public IPs to the relevant hosts on my internal network.  This was achieved relatively easily by adding the following sections to OpenWRT’s firewall config (in /etc/config/firewall):

config zone
    option name             wan1
    option network          'wan1'
    option input            REJECT
    option output           ACCEPT
    option forward          REJECT
    option masq             1
    option mtu_fix          1

# forwards from 1st WAN i/f to SP2010 Web01
config redirect
    option src              wan1
    option src_dport        3389
    option proto            tcp
    option dest_ip          192.168.180.94

The first block configures a firewall zone named wan1 which maps to the wan1 network, with some default rules (e.g. reject all input by default, accept all output by default).  The second block forwards tcp traffic on port 3389 (remote desktop protocol) from wan1 to a local IP of 192.168.180.94.  This happens to be a SharePoint 2010 web server sitting on the ESX host.

Tidying Up

I had one other problem which was that my Linksys box was sitting on the 192.168.0.x network but needed an additional interface to talk to the Virtual Machines on the ESX server.  This was simply achieved by adding an alias section to the OpenWRT network config (in /etc/config/network):

config 'alias'
    option 'interface' 'lan'
    option 'proto' 'static'
    option 'ipaddr' '192.168.180.1'
    option 'netmask' '255.255.255.0'

After another reboot of the router everything was looking good.

Next Steps

The next thing I want to get working is OpenVPN server running on the Linksys, so I can support remote VPNs into the local network.  Naturally there’s a package for this, but it looks like it needs a bit of configuration, as always.

Happy hacking!

Advertisements

Moving away from WordPress.com and 10 must-have WordPress plugins

Moving-Day-Pic-for-Blog-Post-2a0i04v

Last night, with a couple of hours to spare, I decided to get on and move my blog away from WordPress.com to my own hosting.  Why do this?

  • Access to my own Google Analytics data
  • Use of any plugin I like
  • Association my content with my domain name
  • Greater control over the site, to, for example, add Google Authorship links

I’ve been running the blog on WordPress.com since 6th December 2007, but time has come to move to new pastures.   The move process was very simple:

  1. Set up WordPress on my Linux server (5 minutes)
  2. Export existing blog content using WordPress XML export
  3. Import XML into new blog
  4. Pay $12 / year to 301 all pages / posts from the old blog to the new domain
This was I won’t lose any of my Google ranking weight, and users won’t get the dreaded 404 on that key post they were looking for.
To get my new blog up to scratch I did some research and loaded up the following 10 awesome WordPress plugins:
  1. Akismet – the de facto comment and trackback spam protection
  2. Digg Digg – add as many social media buttons as you can shake a stick at
  3. Google Analytics for WordPress – all in the name
  4. Head Space2 – manage SEO meta-data to a fine grained level
  5. Multi Social Favicon – auto-generate the blog favicon using a social media account
  6. Smart Video – simple YouTube, Vimeo video embeds
  7. SyntaxHighlighter Evolved – awesome syntax highlighting for code in posts
  8. Tweet old post – keep old posts alive by tweeting them every now and then
  9. W3 Total Cache – speed up your blog with caching
  10. WP-DBManager – database management, backups – essential
WordPress is an awesome blogging platform; it moves out of the way so you can focus on writing content.  And with the large plugin and theme offerings out there, there’s no excuse to share your knowledge with the wider community.
Get on and write something!

Synchronising Highrise, Basecamp and Google Shared Contacts–Part 2

Background: In Part 1 of this series I wrote about a recent project that I worked on to synchronise contact data between a number of web applications, and how I had to work around the (current) shortcomings of their APIs.

As Scott Hanselman says, “talk is cheap – show me the code”.  So here we go…

Basecamp & Highrise APIs

Implementing an interface to the Basecamp & Highrise APIs is relatively straightforward – I did it using PHP’s curl().  My implementation doesn’t support all of the API’s functions, but you should be able to easily add any additional calls that you need.

The functions centre around a single function which does the hard work on making the request:

private function _getData($uriExtension) {
  $base_url = $this->url.$uriExtension;
  $session = curl_init();
  curl_setopt($session, CURLOPT_URL, $base_url);
  curl_setopt($session, CURLOPT_USERPWD, $this->apiKey.':X');
  curl_setopt($session, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
  curl_setopt($session, CURLOPT_HEADER, false);
  curl_setopt($session, CURLOPT_HTTPGET, 1);
  curl_setopt($session, CURLOPT_HTTPHEADER, array('Accept: application/xml', 'Content-Type: application/xml'));
  curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($session, CURLOPT_SSL_VERIFYPEER,false);
  curl_setopt($session, CURLOPT_FRESH_CONNECT, true);
  $response = curl_exec($session);
  curl_close($session);
  $xmlObj = new SimpleXMLElement($response);
  return ($xmlObj);
}

Google Shared Contacts

This was simply done using the Zend GData classes which you can grab here.  Here’s some sample code for updating a shared contact in Google:

$client=$this->googleClientAuth('cp');
$client->setHeaders('If-Match: *');
$gdata=new Zend_Gdata($client);
$gdata->setMajorProtocolVersion(3);
// get the existing contact.
$query = new Zend_Gdata_Query($id);
$entry = $gdata->getEntry($query);
$xml = simplexml_load_string($entry->getXML());
// update title.
$xnl->title = $title;           
// update the fullname.
$xml->name->fullName = $title;
// update entry.
$entryResult = $gdata->updateEntry($xml->saveXML(), $entry->getEditLink()->href);

Get more details over at the Google Apps Shared Contacts API page.

Basecamp & Highrise – Filling in the API gaps

So the big problem with the Basecamp & Highrise APIs is that they have bits missing – rather significant bits it turns out.  In particular the ability to create and update companies.

To get around this I wrote a couple of classes that use a combination of curl() and the awesome phpQuery to walk through the front-end pages and execute URLs just as a real user would.

Whilst this is not impervious to UI changes by 37Signals, it’s a fairly generic implementation and it’s a best endeavours approach given the current lack in the APIs.

Here’s some sample code:

public function createCompany($companyName) {

    $success = true;

    // go to the company page first, to get the token.
    $resp = $this->_fetchUrl($this->_urls['base'].$this->_urls['companies'], 'GET', array(), false, $this->_ch);

    phpQuery::newDocument($resp['data']);
    $authToken = pq('input[name=authenticity_token]')->val();

    $formFields = array(
        'authenticity_token' => $authToken,
        'company[name]' => $companyName,
        'commit' => 'Create Company'
    );

    $resp = $this->_fetchUrl($this->_urls['base'].$this->_urls['companies'], 'POST', $formFields, false, $this->_ch);

    // check - has this company already been created?
    phpQuery::newDocument($resp['data']);
    if(pq('div.flash_alert')->text() != '') {
        $success = false;
        $this->_errMsg = pq('div.flash_alert')->text();
    }

    return $success;
}

Show me the code!

So, in the hope of helping others to get synchronisation working (and work around the lack in the APIs) you can download the code at GitHub for your own use.

Enjoy!

Synchronising Highrise, Basecamp and Google Shared Contacts–Part 2

Background: In Part 1 of this series I wrote about a recent project that I worked on to synchronise contact data between a number of web applications, and how I had to work around the (current) shortcomings of their APIs.

As Scott Hanselman says, “talk is cheap – show me the code”.  So here we go…

Basecamp & Highrise APIs

Implementing an interface to the Basecamp & Highrise APIs is relatively straightforward – I did it using PHP’s curl().  My implementation doesn’t support all of the API’s functions, but you should be able to easily add any additional calls that you need.

The functions centre around a single function which does the hard work on making the request:

private function _getData($uriExtension) {
$base_url = $this->url.$uriExtension;
$session = curl_init();
curl_setopt($session, CURLOPT_URL, $base_url);
curl_setopt($session, CURLOPT_USERPWD, $this->apiKey.':X');
curl_setopt($session, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_HTTPGET, 1);
curl_setopt($session, CURLOPT_HTTPHEADER, array('Accept: application/xml', 'Content-Type: application/xml'));
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
curl_setopt($session, CURLOPT_SSL_VERIFYPEER,false);
curl_setopt($session, CURLOPT_FRESH_CONNECT, true);
$response = curl_exec($session);
curl_close($session);
$xmlObj = new SimpleXMLElement($response);
return ($xmlObj);
}

Google Shared Contacts

This was simply done using the Zend GData classes which you can grab here.  Here’s some sample code for updating a shared contact in Google:

$client=$this->googleClientAuth('cp');
$client->setHeaders('If-Match: *');
$gdata=new Zend_Gdata($client);
$gdata->setMajorProtocolVersion(3);
// get the existing contact.
$query = new Zend_Gdata_Query($id);
$entry = $gdata->getEntry($query);
$xml = simplexml_load_string($entry->getXML());
// update title.
$xnl->title = $title;
// update the fullname.
$xml->name->fullName = $title;
// update entry.
$entryResult = $gdata->updateEntry($xml->saveXML(), $entry->getEditLink()->href);

Get more details over at the Google Apps Shared Contacts API page.

Basecamp & Highrise – Filling in the API gaps

So the big problem with the Basecamp & Highrise APIs is that they have bits missing – rather significant bits it turns out.  In particular the ability to create and update companies.

To get around this I wrote a couple of classes that use a combination of curl() and the awesome phpQuery to walk through the front-end pages and execute URLs just as a real user would.

Whilst this is not impervious to UI changes by 37Signals, it’s a fairly generic implementation and it’s a best endeavours approach given the current lack in the APIs.

Here’s some sample code:

public function createCompany($companyName) {

$success = true;

// go to the company page first, to get the token.
$resp = $this->_fetchUrl($this->_urls['base'].$this->_urls['companies'], 'GET', array(), false, $this->_ch);

phpQuery::newDocument($resp['data']);
$authToken = pq('input[name=authenticity_token]')->val();

$formFields = array(
'authenticity_token' => $authToken,
'company[name]' => $companyName,
'commit' => 'Create Company'
);

$resp = $this->_fetchUrl($this->_urls['base'].$this->_urls['companies'], 'POST', $formFields, false, $this->_ch);

// check - has this company already been created?
phpQuery::newDocument($resp['data']);
if(pq('div.flash_alert')->text() != '') {
$success = false;
$this->_errMsg = pq('div.flash_alert')->text();
}

return $success;
}

Show me the code!

So, in the hope of helping others to get synchronisation working (and work around the lack in the APIs) you can download the code at GitHub for your own use.

Enjoy!

Synchronising Highrise, Basecamp and Google Shared Contacts–Part 1

NOTE: You can read part 2 here or just simply get the code.

I recently worked on a project where we needed to synchronise contact data between Highrise CRM (the source), Basecamp (for project management) and Google Shared Contacts (for email).  This is a common issue for small businesses – how to keep disparate systems in sync.  There are already some proprietary systems out there which help with this but they didn’t fulfil the needs of this client so we opted for a custom tool.

I’ll write up the steps I took to get all of this working in a series of posts, but the overall architecture is thus:

image

Feasibility

At the time of writing I determined the feasibility of synchronising all of these three systems by looking in detail at the APIs provided.  I’ve summarised my findings here:

Google Shared Contacts

It’s worth pointing out that we needed to load up NOT Google’s personal contacts, but the domain shared contacts.  This is a repository of contacts that Google provides on its Google Apps product, and works like a Global Address List, providing a single store of contact data to all domain users.  However, for some puzzling reason Google does not provide any user interface to manage this database.  Instead you must use the Google Shared Contacts API.

IMPORTANT POINT: I got very confused as to which versions of Google Apps supports the Google Shared Contacts API. Even though the documentation pages state that it is only supported by Apps for Business (Premium) and Education, the confusing point is that YOU CAN EXECUTE THE API CALLS SUCCESSFULLY and the data IS PERSISTED.

What I later discovered was that although all of the API calls work, the data is not exposed in Google Apps.  This is very frustrating and I wish Google would make this clearer in their documentation.

Also don’t get confused between the Google Contacts Data API and the Google Shared Contacts API.

Highrise & Basecamp

Both of these products are made by 37Signals, and if you’ve ever read their books, you’ll know that they are keen proponents of open systems.  They have well documented APIs which are very easy to get up and working with.

However, when you start to use them, you discover that some key elements are missing. The APIs they have provided allow you to get all information out, but when it comes to creating data, the calls just aren’t there. Despite numerous forum posts, 37Signals have not yet given good reasons for the lack of these API calls.

To get around this, I wrote a cheeky HTTP GET, Parse and Re-POST class which effectively walks through the front-end of the site programmatically, as if it were a standard user. Whilst this has been very stable so far, if either of the products publish changes to their UI then this code may need modification.

Coding Approach

I wrote all of the code in PHP 5, so all the samples here will be in PHP, but you could easily apply the concepts to different programming languages.

I built singelton classes for each of the products, and then a controlling class to tie them up all to achieve the synchronisation. I used PHP curl() to do the web requests, and for the Google API stuff I used Zend’s GData classes to speed up the development.

Synchronisation Approach

In order to synchronise these systems I took the following approach:

  • Get all Highrise Contacts
  • Add to Basecamp, storing the IDs of the two data objects in a synchronisation file
  • Add to Google Contacts, storing the ID of the Highrise Contact in the Notes field

Next Steps

In the next post I’ll start introducing the code I wrote to hook up to the Basecamp and Highrise APIs.

Synchronising Highrise, Basecamp and Google Shared Contacts–Part 1

NOTE: You can read part 2 here or just simply get the code.

I recently worked on a project where we needed to synchronise contact data between Highrise CRM (the source), Basecamp (for project management) and Google Shared Contacts (for email).  This is a common issue for small businesses – how to keep disparate systems in sync.  There are already some proprietary systems out there which help with this but they didn’t fulfil the needs of this client so we opted for a custom tool.

I’ll write up the steps I took to get all of this working in a series of posts, but the overall architecture is thus:

image

Feasibility

At the time of writing I determined the feasibility of synchronising all of these three systems by looking in detail at the APIs provided.  I’ve summarised my findings here:

Google Shared Contacts

It’s worth pointing out that we needed to load up NOT Google’s personal contacts, but the domain shared contacts.  This is a repository of contacts that Google provides on its Google Apps product, and works like a Global Address List, providing a single store of contact data to all domain users.  However, for some puzzling reason Google does not provide any user interface to manage this database.  Instead you must use the Google Shared Contacts API.

IMPORTANT POINT: I got very confused as to which versions of Google Apps supports the Google Shared Contacts API. Even though the documentation pages state that it is only supported by Apps for Business (Premium) and Education, the confusing point is that YOU CAN EXECUTE THE API CALLS SUCCESSFULLY and the data IS PERSISTED.

What I later discovered was that although all of the API calls work, the data is not exposed in Google Apps.  This is very frustrating and I wish Google would make this clearer in their documentation.

Also don’t get confused between the Google Contacts Data API and the Google Shared Contacts API.

Highrise & Basecamp

Both of these products are made by 37Signals, and if you’ve ever read their books, you’ll know that they are keen proponents of open systems.  They have well documented APIs which are very easy to get up and working with.

However, when you start to use them, you discover that some key elements are missing. The APIs they have provided allow you to get all information out, but when it comes to creating data, the calls just aren’t there. Despite numerous forum posts, 37Signals have not yet given good reasons for the lack of these API calls.

To get around this, I wrote a cheeky HTTP GET, Parse and Re-POST class which effectively walks through the front-end of the site programmatically, as if it were a standard user. Whilst this has been very stable so far, if either of the products publish changes to their UI then this code may need modification.

Coding Approach

I wrote all of the code in PHP 5, so all the samples here will be in PHP, but you could easily apply the concepts to different programming languages.

I built singelton classes for each of the products, and then a controlling class to tie them up all to achieve the synchronisation. I used PHP curl() to do the web requests, and for the Google API stuff I used Zend’s GData classes to speed up the development.

Synchronisation Approach

In order to synchronise these systems I took the following approach:

  • Get all Highrise Contacts
  • Add to Basecamp, storing the IDs of the two data objects in a synchronisation file
  • Add to Google Contacts, storing the ID of the Highrise Contact in the Notes field

Next Steps

In the next post I’ll start introducing the code I wrote to hook up to the Basecamp and Highrise APIs.

Web Development Toolsets – PHP vs. .NET IDEs

[tweetmeme source=”shuggill” only_single=false]

I’ve been writing code for the web for 10 years now, and in a (full-time) professional capacity for almost three and a half.  I’ve worked extensively on the Linux Apache MySQL and PHP (LAMP) stack as well as in Microsoft’s ASP.NET.  This post is a helpful summary of my experiences developing in both worlds, and what you need to focus on your job.

Important information on operating systems: I carry out all of my development on Windows, so you’ll not find any suggestions or reviews of software for Macs or Linux here.  If you’re one of these people, why not write up your own review?

Integrated Development Environments (IDEs)

Let’s be honest, we all started out in Notepad, writing angle brackets in long hand and making extensive use of Ctrl+C.  However once we graduated from the single scripted site, managing multiple files and object oriented links between them soon left us lost in a sea of notepad.exe windows.  To curb this problem, and many others, software vendors developed IDEs, to incorporate a number of development tools into one application.

My favourite PHP IDE – Netbeans

If you write any amount of PHP code then you’ll want to get comfortable in one of the many PHP IDEs out there.  In the last two years I’ve spent many hours bashing keys in Netbeans IDE.  Whilst this application is written in Java (which undoubtedly makes it dog slow at times), Netbeans have dramatically improved the performance in the last few releases to the point where most features are finally usable!

Debugging

Remote team debugging does require screwing around with XDebug and Komodo Python Debug Proxy, which I did manage to get working on a Linux development VM.   However it has proved to be a little temperamental, requiring frequent restarts.

Source Control

Netbeans includes integration for SVN Source Control and this works pretty seamlessly – you can commit and revert from within the application, as well as see which files are under source control and their status.

Code Navigation

With the right hints in your code, you can jump between classes with Ctrl+Click pretty quickly.

Summary

I have toyed with other IDEs, notably phpDesigner, however Netbeans has been my favourite (free) IDE, and has served me very well.

My favourite .NET IDE – Visual Studio 2010

Much has been written about Visual Studio 2010, so I’m hesitant to try and add to the endless blog posts detailing every feature of this application.  All I will say is that it is an incredible piece of software.  It runs like grease lightening, it has a database explorer, browser and designer built in, as well as an extension model for a variety of useful add-ons.

Debugging

Even since my days of programming in Visual Basic 6, step-by-step debugging has been the most productive way to debug, rather than the old “Made it to here” debugging output.  You can attach to remote servers, or simply fire up the built in development web server to step through your code, with watch windows and full stack traces to hand.  This debugging experience is head and shoulders above that of Netbeans.

Source Control

Whilst you can get some source control add-ins for Visual Studio, my impression is that use of the Team Foundation Server is quite heavily pushed.  I have yet to find a good HG (Mercurial) add-in, so I hack away on the command line to feel like a real programmer.  On this item, Netbeans is my winner.

Code Navigation

When combined with IntelliSense, it beats Netbeans hands down, purely due to the speed and variety of options.  You get hinting at method parameters, namespaces to include as well as class properties, methods and events.

Summary

No serious .NET developer would use anything else, and as Microsoft make their express editions freely available, there’s really no excuse not to be using Visual Studio if you’re writing .NET code.

The Final Word

My overall development experience is much slicker in Visual Studio 2010; it feels like the right tool for the job, and it is so quick that I never feel like it gets in the way.  Netbeans is more of a necessary evil for development, and it often feels like I’m having to kill time whilst waiting for things to happen, or have those moments of keystroke regret when you see the “scanning for indexes” message in the status bar.  But compared to Notepad, it is so many steps ahead that I’ll just have to keep putting up with the performance issues.