JSON Web Service for Weather Display Live

Posted October 26, 2014 by waynedgrant
Categories: Meteorology, Weather Station, Code Projects

Tags: , , , ,

I’ve been busy implementing a new website for my weather station over the last few weeks at Wayne’s Weather.

There are a number of new technologies being employed. I’ve ditched my bespoke (and ugly) HTML and instead use Bootstrap for a client responsive UI. Chart and image viewing is now more user-friendly thanks to Lightbox. I have also made extensive use of JQuery to pull in information from various sources and to provide a more dynamic experience.

The functionality of the site has also been extended. There are now more in-depth current readings supporting both metric and imperial measurements, forecasts and weather maps from Wunderground.com for the local area have been integrated and I have added a weather webcam and a weather almanac. I intend to write posts to cover the wunderground and webcam build outs soon.

For now this post will concentrate on an unexpected off-shoot of my efforts: a Web Service that exposes Weather Display Live data.

Weather Display Live

Weather Display (and its equivalents such as MeteoHub) can generate clientraw.txt files containing a weather station’s readings and can upload them to a web server periodically and frequently via FTP (in some cases every minute). The primary purpose of this is to drive a Weather Display Live (WDL) dashboard:

Click to Enlarge

The WDL clientraw.txt format is well documented and I have previously used the HTTP addressable clientraw.txt to drive a custom Android widget and my PiFaceCAD Weather Display Console. The disappointing aspect is WDL itself. Don’t get me wrong it’s an excellent weather dashboard and I still have it available in my new website. The problem with it is that is Flash based which is not as widely supported as it used to be. It’s pretty much not an option for mobile device clients running IOS or Android, for example.

To counter this I wanted my new website to make far greater use of the information contained in the clientraw.txt files outside of WDL dashboard itself. My old weather website made use of PHP to grab weather readings from clientraw.txt and place them into web pages before serving them up to the client. This was an approach I wanted to move away from.

I’d just cut my teeth on using JSON formatted web services with my experience as a client to the Wunderground API. I liked the idea of having a page load its own data from the client side where it could conceivably be mix and match data from various services. I figured it wouldn’t be too difficult to create my own web services to expose WDL data and call them from my own web site.

json-webservice-wdlive

I settled on using PHP for the implementation for a couple of reasons. Firstly because I’ve used it for server-side processing on-and-off for a few years and know if fairly well. Secondly because I intended to write a general solution which I could open source for others to use and PHP is ubiquitous in the hosting world.

The result was json-webservice-wdlive, a JSON formatted web service API.

The API exposes two URLs. The first URL returns current weather conditions including Temperature, Pressure, Rainfall, Wind, Humidity, Dew Point, Wind Chill, Humidex, Heat Index and UV. The second URL returns a weather almanac for Month-to-Date, Year-to-Date and All Time records. Both JSON and JSONP (enabled with the addition of a callback attribute to the URL) are supported.

Besides exposing the data in the default units found in clientraw the responses also contain many alternative units. For example, clientraw files store wind speeds in knots. The Web Service responses, on the other hand, respond with Bft, knots, km/h, mph, and m/s.

json-webservice-wdlive-output

Click to Enlarge

Note that I only expose the data I need for my own purposes (i.e. what my weather station setup supports). However, it would be a simply matter to expand the service calls to add, say, solar measurements or extend the selection of almanac measurements. Anything else WDL clientraw.txt files provide can be exposed, if required, with minor code additions.

I have made the json-webservice-wdlive source code available as a GitHub project. Have a look in the project’s README for installation instructions. json-webservice-wdlive is also running live on my own weather website. You can try it at by clicking on the links below:

Refer to the API page on my website for more details on the Web Service’s response fields including the different measurement units and number field formatting details.

I also make use of the service in three places on my own website: the Current Conditions, Weather Almanac (pictured below) and the Forecast page where I mix my service’s results with those from the Wunderground web service.

almanac

Click to Enlarge

If you have a WDL enabled website feel free to install json-webservice-wdlive to expose your weather data to others and/or to include your information in your own pages.

PiFaceCAD Weather Display Console Build Out

Posted July 19, 2014 by waynedgrant
Categories: Code Projects, Meteorology, Weather Station

Tags: , , , , ,

I bought a Raspberry Pi back when they were first released in February 2012. It was purchased with the intention of setting it up as data logger for my WMR88 weather station. I got it up and running for that purpose using wview. However, I soon found a better solution for my data logging requirements in the Meteo Sheeva.

The Pi then found its way into a box and remained there unpowered for over two years. I wanted to put it to good use but no ideas were forthcoming to utilise it. Then inspiration struck.

I had just completed an Android application widget that could poll the WD Live files hosted on a website and display the current weather conditions therein. I use it on my smart phone to see the current conditions at home. Could I use the same principle to create a custom weather display console? It seemed like a cool little project and would put the Pi to good use at last.

The first thing I needed was some kind of simple, dedicated display for the Pi. Trawling the web I found a few products on offer that did just that. Unfortunately they all required soldering which I’m just not brave enough to try. Then I discovered the PiFace Control and Display (PiFaceCAD).

The PiFaceCAD is a neat bit of kit. It attaches to a Pi via its GPIO ports and provides a small backlight 16×2 character screen, five buttons, a three position navigation switch and an IR receiver. All of these features are programmable via Python. I ordered it and the compatible Camden Boss case from MCM and they arrived a few days later.

Assembly

The assembly was a little fiddly as the Camden Boss case is a very tight fit around the Pi and PiFaceCAD. I’ve illustrated the process in the pictures below.

The unboxed PiFaceCAD:

20140421_192221

Click to Enlarge

The unboxed case with the included rubber feet. I prefer clear cases if I can get them as they are more interesting to look at than opaque cases:

20140421_191636

Click to Enlarge

My Pi ( a Model B Rev 1) maneuvered into the bottom half of the case. It’s a tight fit to get it in place, especially the composite video port:

20140421_192746

Click to Enlarge

The PiFaceCAD attached to the Pi via its GPIO pins. Getting the PiFaceCAD into the bottom half of the case is difficult as the five buttons have to fit into small cut outs. I was sure I was going to break something but fortunately I didn’t:

20140421_192916

Click to Enlarge

The top of the case snapped into place. Again it’s tricky to fit it around the Pi’s USB and Ethernet ports:

20140421_193119

Click to Enlarge

Finally the assembled unit powered up, with the PiFaceCAD software installed and running the supplied SysInfo demo program:

20140421_194321

Click to Enlarge

Coding in Python

With the hardware up and running the next step was to code the weather display application itself. The PiFaceCAD provides a functional, easy to use Python API. The API exposes the ability to write to the display, control its backlight and add event handlers to the various buttons and the IR receiver.

I had never coded in Python before (Java has been my mainstay for last few years with the occasional foray into PHP and C#) but found it to be a straightforward language to pick up with a gentle learning curve and excellent documentation. The language’s dynamic typing took a little getting used to given my background but I started the exercise prepared to learn new ways of doing things (I did not want to “code Java in Python”). Coming from Java I really appreciated the lack of boilerplate code required by Python and just how little code it took to get things done without sacrificing readability. I expect to be doing a lot more coding in Python in the future.

Python’s library support is also excellent. I was pleased to find a JUnit like capability in the “unittest” Module and made use of decimal, urllib, threading and time modules in my solution (as well as creating a few modules of my own to handle, transform and format weather data).

What I did not do was code on the Pi itself. I instead coded the project on my Macbook using the excellent PyCharm Community Edition IDE from JetBrains. I periodically uploaded my unit tested solution to my Pi via FTP for integration testing. The code targets Python 3.2.3 which was the version available on my current version of Raspbian (released Jan 9th 2014).

pifacecad-wdlive on GitHub

The result of my endeavours was the pifacecad-wdlive application which I have hosted on GitHub. If you are interested in installing pifacecad-wdlive then head to the README for setup instructions. If you want to fork pifacecad-wdlive to add extra capabilities then all of the source code and unit tests are available in the GitHub project.

The current version of pifacecad-wdlive supports 16 different weather displays which are selectable using the PiFaceCAD’s navigation switch (this switch also controls the backlight) and 16 different measurement units toggled using the PiFaceCAD’s buttons. Better yet all of these functions can be controlled via a normal IR remote control (I have mine working via my XBox 360 remote). The application can be configured on the command-line to point at any WD Live enabled website and polls for updated weather conditions every 60 seconds.

Finishing Touches

If my Pi ever lost power I did not want to have to start pifacecad-wdlive manually every time the Pi restarted. To execute pifacecad-wdlive on startup I added the following line to the Pi’s /etc/rc.local file:

su - pi -c "python3 /home/pi/pifacecad-wdlive/pifacecad-wdlive.py
http://www.waynedgrant.com/weather/clientraw.txt" &

Those wanting to do the same will want to adapt these lines to match their particular pifacecad-wdlive installation and clientraw.txt URL.

A normal weather display unit such as the WMR88 or WMR200 can display weather anywhere it can receive mains power and wireless sensor data. My PiFaceCAD, on the other hand, was shackled to an Ethernet cable. I decided to make my setup a little more portable by adding a wireless adapter to the Pi. I picked up an Edimax Wifi USB from Amazon to accomplish this.

A wasted hour of following various (conflicting) manual instructions on the web ensued as I tried and failed to set the wireless adapter up on the Pi. I then stumbled on and installed wicd-curses and promptly got the Wifi working.

My one concern in using a Wifi USB adapter was that of power. I feared that having the Wifi USB adapter and the PiFaceCAD connected to the Pi at the same time may require me to use a powered USB hub. Fortunately my 0.7A Samsung phone charger was up to the job so a powered hub was not required.

Here is the finished build out complete with Wifi adapter and displaying pifacecad-wdlive’s summary weather display screen for my own weather station:

Click to Enlarge

Click to Enlarge

I’m quite pleased with how the build out has turned out. While it would probably have cost around the same money to buy a new weather display console from Oregon creating this was a lot more fun and satisfying.

Accessing UK TV from the USA

Posted May 31, 2014 by waynedgrant
Categories: Expat

Tags: , , ,

I have been living in the USA for nine months now. While I have settled comfortably one thing I have missed from back home in Scotland is UK Television. Specifically I have been missing UK-centric news and documentaries. There are currently two BBC channels available on US cable. However, compared to the UK-based BBC offering they’re light on decent content. Not having the likes of BBC News 24 , Horizon, Panorama, etc. available finally spurred me to start exploring options to access UK-based streaming and catch up services from the US.

In the course of my research I have read many ill-informed forum posts saying certain approaches to accessing UK content are not possible. This post details my experiences so others can see what is possible and not be put off by such incorrect information.

Accessing Geoblocked Content

The first thing that is required to access UK catchup services is the ability to trick the geoblocks that such services employ to restrict their content to the UK. I investigated various VPN providers before settling on OverPlay’s Smart DNS service. This is not a full-blown VPN service and the low price of $4.95/month reflects this. Instead it is a simpler offering which utilises OverPlay’s own DNS to get around geoblocks for select services automatically. Specifically it supports all of the UK catchup services: BBC iPlayer, 4oD, Demand 5 and ITV Player.

Setup is straight forward and requires only a change to your device’s DNS configuration to point at OverPlay’s DNS Servers. The Overplay website contains detailed setup instructions for common devices such as Windows and Mac computers. Alternatively you can simply change the DNS setup on your router to cut over all of your home based devices to Smart DNS in one fell swoop. Many devices do not support individual DNS settings (e.g. Roku boxes) so this was my preferred option.

I was concerned that using Overplay’s DNS would lead to a slow down of my Internet speed. However, speed tests show no degradation of my internet speed when using OverPlay’s DNS Servers. Streaming content from the UK via Smart DNS is also very fast and reliable even in HD.

Besides making UK catchup services available Smart DNS has the ability to switch your Netflix region on-the-fly. By making a menu selection on the Smart DNS website your Netflix region can be changed to any one of the 15 available regions. The change takes effect immediately although you will need to restart running instances of Netflix on devices for them to update to the new region. Switching Netflix region is useful because different countries can have very different content. For example, I am currently enjoying the US TV Series “The Shield” on Canadian Netflix. Surprisingly the show is not available on US or UK Netflix. Helpfully you can find out if certain content is available on a particular Netflix region by using the (currently beta) Moreflicks website.

There are other options available to access geoblocked UK content but I cannot comment on those. I have found OverPlay’s Smart DNS service to be very reliable with only one or two short outages while I have used it. Their support is also very responsive.

Whatever service you use to defeat geoblocks you will require devices to access the content. I have explored a few of the options available detailed my experiences below.

Option 1 – Laptop or Tablet

Initially I connected to BBC iPlayer, et al using a Macbook and the relevant catchup service’s website. This was okay for short periods of viewing but using a laptop to watch TV seems a little restricted when I have a large screen TV available. For longer viewing sessions I purchased an extra long HDMI cable and connected it to my Samsung TV to mirror the display. I also have an Apple TV which can be used to mirror my Macbook’s display wirelessly via Airplay. However, I find this approach can be a little laggy compared to the cable option.

The other option is to use a tablet and the relevant catchup service’s app. Beware that you will need a UK based App Store or Play Store account to access the iPlayer, 4oD, etc apps.

One advantage of using a laptop or tablet application to access catchup content is that it provides the option to stream live television and (for iPlayer at least) download programs for later viewing. The other options below do not provide this option and instead only cater for catchup TV.

Option 2 – Now TV Box

Accessing iPlayer, etc via a laptop works well but plugging in cables or setting up AirPlay every time I want to watch UK TV is not convenient. I therefore started looking into set-top box options that would be more integrated with my TV. Factory resetting my Apple TV back into the UK region was not an option as it does not support BBC iPlayer. I would need to buy a new set-top box of some kind. I heard that Sky had released their own little streaming box for their Now TV service that also supported the various free to air UK catch up services and decided to explore that as an option.

The Now TV box is very cheap at £10 but is only available in the UK. Fortunately I maintain a UK address and was able to order one using a UK credit card and could get family to forward it onto my address in the States. The only problem was that it shipped with a UK power adapter. This problem was easily solved as it turns out that the Now TV box is simply a rebadged Roku box. I was able to order a replacement US power adapter directly from Roku themselves. The cost of the adapter and postage to the States inflated the final price of the box but it still looked like a bargain.

When the Now TV box arrived it was easy to set up. Surprisingly the Now TV box came complete with a HDMI cable. Sky has obviously heavily subsidised the device to get it into as many households as easily possible. The Now TV box does not support Ethernet only wireless. To my surprise it also does not support dual band wireless only 2.4 GHz. When I discovered this during setup I had my concerns. The 2.4 GHz band is very congested where I live as it is a heavily populated area. In fact it is so slow that I operate all of my devices on the 5GHz band which is almost unpopulated.

I gave up on the mandatory sign up for Now TV via the device’s setup process as it insisted that I purchase a Now TV pass which I was not interested in. I instead linked the device with my existing Sky ID which I had from my time as a Sky subscriber in the UK and skipped this requirement.

Note that Now TV boxes do not allow the configuration of DNS so you will have to set up your Smart DNS or VPN centrally on your router to get it to work in the US.

All seemed well. I could access iPlayer, 4oD and Demand 5 directly on my television as if I was in the UK. The interface was a little slow but usable which is understandable given the low-cost of the device. However, upon accessing actual content the problems started.

Video playback was terrible with frequent buffering interrupting viewing. Worse still apps such as 4oD and Demand 5 are prone to crash out completely when buffering occurs (iPlayer tends to be more resilient in the face of a slow connection). As I suspected the 2.4GHz band was far too slow and congested to stream content reliably. I experimented with all of the available channels on the 2.4GHz band to no avail. Using a scanning app on my smart phone showed every channel was packed with competing WiFi networks.

At this point I gave up and put the Now TV box on eBay where it promptly sold. If you live in an area with uncongested 2.4GHz wireless and can source a Now TV box from the UK then I recommend it if you want to access catchup services on your TV without a computer. Otherwise steer clear and look at another option.

Option 3 – Samsung Smart Hub

When I moved to the States I bought a new TV. Like most new TVs it’s a “Smart TV” and comes complete with app support. Another bonus was my TV’s support for dual band wi-fi. After the failure of my Now TV experiment I looked into the possibility of resetting the firmware on my TV to access UK apps such as iPlayer. Expecting a complex, potentially device destroying process I was surprised to find that five button presses on my remote allowed me to switch the TV’s region to any country I wanted. Upon making the switch to the UK region all of the US-based apps disappeared to be replaced by iPlayer, 4oD, Demand 5 and itv player. In combination with my Smart DNS configured router I finally had access to catchup services without requiring a computer. Samsung’s Smart Hub interface used to access the catchup apps is a little clunky and slow but is convenient enough for everyday use.

This setup worked great until a couple of weeks ago. Suddenly all video playback via the TV for iPlayer and 4oD became scrambled although sound was still playing correctly. Playback via a laptop or tablet worked fine as did Netflix playback via Samsung’s app. I could only conclude Samsung has (perhaps temporarily) broken their UK catch up apps. While I expect the apps will be fixed at some point I decided not to wait and sought out yet another solution.

Option 4 – Roku Box

The Now TV solution had come close to working with only the 2.4GHz wireless limitation scuppering the solution for me. Given that a Now TV box is simply an old rebadged Roku 1 box I reasoned that I may have more success with an up-to-date Roku 2 or 3 box. I checked the specs and found that both models did indeed support dual band wireless and, in the UK, supported iPlayer, 4oD and Demand 5. I could pick up either model in Best Buy for $70 and $100 respectively and forum posts suggested that, with certain prerequisites, a US Roku box could be reset to the UK region.

The first thing to do was to set up a UK-based Roku account. This is necessary because the act of linking a Roku box to a UK account causes it to install the UK-based catchup applications. To do this successfully you will need to access the Roku sign-up site via a UK-based VPN. I used UK Free VPN for this purpose. You will know you have been successful in setting up a UK-based account if the account balance is listed on the Roku website in UK Pounds rather than dollars.

I plumped for the Roku 3 as it is much faster than the Roku 2 and I was tiring of the slow user interfaces I had found on the likes of the Now TV box and my Samsung TVs. It also has an Ethernet port for extra flexibility although I am using 5GHz wireless successfully. I thought that Roku not including an HDMI cable in the box was a little cheap and highlighted just what a bargain the Now TV box is. Setup was simple. Connecting to the Internet via my Smart DNS enabled Router (like Now TV the Roku box has no DNS settings available) and linking to my UK Roku account the box immediately installed the relevant UK-based catch up apps and everything just worked.

While the Roku solution was not the cheapest it has worked out the best. The box is fast and can be controlled by a smart phone or tablet. It also works reliably which is more than can said for some of the other options I tried. The only thing it can’t do is stream live TV as, like Now TV and Samsung Smart Hub, the various catchup apps do not support this feature. However, when I want to stream live TV there is alway the option of using a Macbook or tablet.

Summary

Whatever option you go for you will need a service to get around geo blocking. I recommend Over Play’s Smart DNS because its fast, reliable and cheap.

A laptop or tablet is a must for streaming live UK TV. For a more integrated experience a set-top box or Smart TV can be persuaded into the UK region as described above.

A Now TV box is alow cost option if you can source one from the UK, have uncongested 2.4GHz wireless and can put up with the slow interface.

A Smart TV is an even cheaper option if you have one already and can change its region. However, my experience in the UK and US is that TV manufacturers are terrible at supporting their applications and their interfaces tend to be slow.

My best experience was with the Roku 3 box because it has a fast interface and has a wide array of networking options.

UVN800 Wireless Remote Sensor Review

Posted April 13, 2014 by waynedgrant
Categories: Meteorology, Weather Station

Tags: , , , ,

Until recently I lived in Scotland where UV is hardly ever an issue given the hideous weather we tend to get there. However, I now live in New Jersey in the USA where the summers can be very hot and the risks of UV exposure are more of a concern for me. Given that I took my weather station with me when I relocated (which comprises of both the WMR88 and WMR200 base stations) it made sense to consider expanding its capability to record UV.

One of the advantages of Oregon Scientific’s range of wireless weather stations is the ability to add extra sensors. I have, for example, previously taken advantage of this by adding an additional temperature and humidity sensor in the form of a THGR810 unit.

Both of my base stations also support a UV sensor in the form of the UVN800. This sensor is not normally bundled with the WMR88 or WMR200 but I purchased one recently as an add-on. The unit normally retails for $59.99 but I was able to pick it up for $43.79 from Amazon (not including sales tax). Both the full and discounted prices are on par with what I would expect to pay for other additional sensors such as the THGR810 mentioned above. It arrived undamaged and well packaged in a relatively small box:

UVN800 boxed

Click to Enlarge

Upon unboxing I was presented with the UVN800 unit itself, a wall mount with two screws, a ground spike, AA batteries and instructions.

UVN800 unboxed

Click to Enlarge

This provided me with two different installation options. Either use the ground spike to insert the sensor in the ground or the wall mount and screws to attach it to a wall or pole. I chose the wall mount option which I attached to my existing sensor pole. However, I appreciate the flexibility of the ground spike option. The trick with the UVN800 is to orient it such that the UV sensor on top of it has a constant, uninterrupted view of the sky which I could more easily achieve with the pole mounting.

UVN800 mounted

Click to Enlarge

Installation of the UVN800 is fairly straight forward if a little more awkward than it could be due to some weird choices made by the unit’s designers. First of all accessing the battery compartment requires the removal of four small screws from the base of the unit to access the battery compartment. Why the compartment is not accessed by a sliding mechanism like most of Oregon’s sensors is a mystery.

Secondly the wall mount is screwed into place at the bottom of the unit obscuring the battery compartment and reset button. Given my pole mounted configuration changing batteries will be far more time-consuming than it should be. I will have to unscrew the wall mount from the pole, detach the sensor from the wall mount, remove the screws from the battery compartment. Only then can I change the batteries and will then have to reverse the procedure to reinstall the unit.

So far this is my only gripe with the UVN800 and it is not a deal breaker by any means. One the batteries were installed pairing it with my base stations was as simple as hitting the sensor’s reset button and initiating a search from each base station. They both started displaying UV Index readings straight away. On both the WMR88 and WMR200 this takes the form of a live UV Index display and a graph of the last 10 hours of values.

Having UV Index values displayed on my base stations was just the start, however. I publish weather data to my own website and wanted to add UV Index information to it. I use a Meteo Sheeva connected to my WMR200 as a data logger and to do uploads of weather readings and graphs to my website. As expected it was a snap to get it to start logging data from the UVN800. As an example here is one of several graphs I have configured on the Meteo Sheeva which display the last 7 days of maximum UV Index values:

7day_uv

Click to Enlarge

My Meteo Sheeva also uploads data in WD Live’s clientraw.txt format which includes UV Index readings. I have rearranged my existing WD Live console on my website to incorporate a UV Index bar:

wdlive

Click to Enlarge

Finally I wanted to be able to see up-to-date UV readings when I am outside. To do this I ideally want to be able to see the current UV Index on my phone. One of my winter projects was to develop an Android widget which displays the values found in online WD Live clientraw.txt files. The data the widget displays is user configurable and one of the options is for colour coded UV Index values:

Cirrus_ori_portrait

Click to Enlarge

The widget is pretty much production ready and supports 18 different weather data points and 16 different measurement units. I use the widget extensively for both my own weather station and others close by locally or when I travel. However, at this time I am not sure whether or not I will publish it on the Google Play Store given how many similar apps are already available.

Returning to the UVN800 itself there is one more thing to note. The reviews on Amazon for the UVN800 indicate that many units permanently fail just after a year of operation. At the time of writing I have only been operating the sensor for a few days but will add a note to this review if and when it fails.

Should the sensor last at least a couple of years (as all my other sensors have already done) then I would not hesitate to recommend the UVN800 as a useful, easy to use addition to an existing Oregon Scientific wireless station.

Sprint Retrospective Techniques Roundup

Posted March 28, 2014 by waynedgrant
Categories: Agile, Scrum

Tags: , ,

Over the last three years I have written a number of posts detailing various Sprint Retrospective techniques. Each technique’s writeup describes what it is good for, the steps to run it and includes pictures of real sessions. This post represents a roundup of all of the techniques I have written about with links to the original posts.

retros roundup

 

Enjoy.

Sprint Retrospective Techniques 3

Posted February 9, 2014 by waynedgrant
Categories: Agile, Scrum

Tags: , ,

This post presents three more sprint retrospective techniques to add to the six I have already detailed in my previous posts Sprint Retrospective Techniques 1 and Sprint Retrospectives 2. Why present three more techniques? Surely six is enough to drive continuous improvement in any team? First of all I have found these techniques to be useful additions to my arsenal for reasons I outline below. Secondly variation is one of the keys to maintaining the effectiveness of a team’s retrospectives and more techniques makes for more variety.

Technique 1 – 4Ls

The 4Ls is shorthand for the following:

  • What was Liked? What were the things that the team really appreciated about the sprint?
  • What was Learned? What were things that the team learned that they did not know before the sprint?
  • What Lacked? What were the things that the team think could have done better in the sprint?
  • What was Longed For? What were the things the team desired or wished for but were not present during the sprint?

A 4Ls retrospective can be run by following these steps:

  • Create a poster for each of the Ls and stick them up on the wall (easel paper is good for this purpose but drawing sections on a whiteboard is a good alternative).
  • Explain the meaning of each of the Ls to the team.
Click to Enlarge

Click to Enlarge

  • Hand out sticky notes and markers to the team.
  • Encourage the team to place stickies with ideas onto each relevant poster and wait until everyone has posted all of their ideas.
Click to Enlarge

Click to Enlarge

  • Have the team group similar ideas together on each poster.
Click to Enlarge

Click to Enlarge

  • Discuss each grouping as a team and note any corrective actions.
Click to Enlarge

Click to Enlarge

The reason I like using the 4Ls is that it has the potential to cover a wide range of topics in a compact session. It addresses both the positive and negative aspects of the sprint (Liked and Lacked) but also specifically calls out the teams growing experience (Learned) and problematic gaps that can filled (Longed For).

Technique 2 – Satisfaction Histograms

There are many variations of the Satisfaction Histogram. I came across the version outlined here when a team member on a project I was a Scrum Master on offered to run a couple of retrospective sessions. This is how he ran one of the sessions and it turned out to be very effective.

  • As preparation pick around four topics that you want to gauge the teams satisfaction of. These can be practices, behaviors or anything else you can think of.
  • For example: Testing practices, Keeping a clean build, Standup effectiveness, Accuracy of estimates, etc.
  • Draw a satisfaction histogram for each of the topics on a whiteboard. Label the x-axis 1-5 for each and add the topic name as a heading.
Click to Enlarge

Click to Enlarge

  • Explain the meaning behind each of the topics  to the team.
  • Explain the meanings of the 1-5 scale to the team. For example, for “Our Team Communication is…”
    • 1 – “…disastrous, is actively impeding the team”
    • 2 – “…bad, not being done effectively”
    • 3 – “…satisfactory, requires improvement”
    • 4 – “…mostly very good, could still be improved further in small ways”
    • 5 – “…awesome, little or no room for improvement”
  • Distribute sticky notes to the team, one per topic.
  • Invite the team to place one sticky note in each topic’s histogram to grade how satisfied they are with the team’s performance for that topic. Sticky notes placed on the same topic and number are stacked.
  • Wait until everyone has placed their stickies.
satisfaction histogram 2

Click to Enlarge

  • Discuss the results for each topic in turn. Where there is low satisfaction or a wide-spread of satisfaction grades dig into why this is.
  • As potential corrective actions are identified by the team, especially for topics with mostly low numbers, note them down.
satisfaction histogram 3

Click to Enlarge

This version of Satisfaction Histogram has a number of advantages. First of all the selection of topics means that it can be targeted to certain problem areas. Note that the technique can also be varied to allow the team to suggest topics. This can be done by leaving one histogram blank for the team to suggest the topic during the session. As the team becomes more familiar with the technique you can allow them to suggest all of the topics to be scored for satisfaction.

Secondly it is very visual. At a glance everyone can see the topics where the team is satisfied, dissatisfied or in disagreement. This allows the team to focus on the topics where they belive they are most lacking or conflicted about.

Technique 3 – Circles

Circles is more commonly known as Circles and Soup. I dislike the “Soup” metaphor so I refer to the technique simply as Circles. In addition when I run this technique I replace “Soup” with “Concern”. I understand that the technique is based on Stephen Covey’s book Seven Habits of Highly Effective People.

The idea behind circles is to get the team to focus their energies on what they can change and not to waste time worrying about what they cannot affect.

A Circles retrospective can be run by following these steps:

  • Define what an impediment is to the team:
    • “An impediment is anything that prevents you or the team from delivering work as efficiently as possible. An impediment is anything that blocks you working or slows you down.”
  • Distribute sticky notes and markers to the team.
  • Ask the team to write down all of the impediments encountered in the sprint, one per sticky note, and have them post them onto a whiteboard.
  • Wait until everyone has posted all of their impediment ideas.
circles 1

Click to Enlarge

  • Ask the team to identify and remove any duplicate impediment stickies.
Click to Enlarge

Click to Enlarge

  • Draw three concentric circles on the whiteboard and label them, from the inside out, “Control”, “Influence” and “Concern”.
Click to Enlarge

Click to Enlarge

  • Define what each of the circles means to the team:
    • Control - Impediments for which the team can take action to remediate.
    • Influence - Impediments for which the team can collaborate with or make a recommendation to an outside entity to remediate. For example, another team, group or line management.
    • Concern - Impediments over which the team has no ability to Control or Influence.
  • Invite the team to collaboratively place each of the impediments in the appropriate circle.
  • Encourage and guide any debate as to what should go where. 
  • Wait until all of the impediments have been placed in one of the circles.
Click to Enlarge

Click to Enlarge

  • Go through each Control impediment one at a time and have the team:
    • Gain an understanding of what each impediment is.
    • Identify remedial actions that are within the team’s control and write these on the whiteboard.
  • Go through each Influence impediment one at a time and have the team:
    • Gain an understanding of what each impediment is.
    • Identify contacts and recommendations to influence their remediation and write these on the whiteboard.
  • Review the Concern impediments to gain an understanding of what they are.
  • Note that as the impediments are discussed the team may identify actions that had not previously occurred to them.  This can cause the impediments to move inward. For example an impediment that was initially placed in Concern may move to Influence or Control.
Click to Enlarge

Click to Enlarge

Circles is my new favourite technique but I use it sparingly. The reason I try not to overuse it is because it focuses very much on impediments and does not have the “celebration of success” aspect that is built into most other retrospective techniques. However, it is a powerful technique for sprint retrospectives as well as for other purposes. It is, for example, ideally suited to holding retrospectives into releases especially those that proved to be problematic.

Definition of Done Workshop

Posted December 11, 2013 by waynedgrant
Categories: Agile, Scrum

Tags: , ,

A few months ago I underwent a career change by converting from being a Scrum Master to an Agile Coach. I am still adjusting to my new role but am, by and large, enjoying the challenge and change of pace that it entails. I work with a number of internal teams which represent a fairly broad spectrum of Agile experience from novice to intermediate.

When I engage with a team for the first time I ask them a number of initial questions to help me to gauge their Agile maturity. Some of these concern the Agile ceremonies they practice and the particular Agile artifacts they make use of. What has surprised me so far is that an overwhelming number of these Agile teams did not have a Definition of Done.

For those not familiar with what a Definition of Done (DoD) is I will attempt to summarise the concept here. A DoD is a checklist of useful activities that are carried out by a software team every time they implement a user story. The DoD can, for example, include things like ‘Code’, ‘Unit Test’,  ‘Integration Test’, ‘Peer Review’, etc. The idea is that the culmination of all of these activities against a given set of requirements will result in potentially releasable software. That is quality software that satisfies the requirements. Until all of these activities are completed successfully for a given user story it cannot therefore be considered ‘Done’.

It is important to note that every team’s DoD will almost certainly contain different activities because they do different types of work and have different means of ensuring quality. Therefore every team has to spend some time coming up with their particular DoD.

For Agile teams an agreed and published DoD is essential as it brings transparency to a team’s way of working. Firstly it helps to ensure that everyone on the team is on the same page as to what “Done” means and how they should get there. Secondly as the DoD is written down it can be discussed and subsequently altered as the team agree to make changes to how they work to improve quality and efficiency. For example, via their retrospectives.

Some of the teams I coach had not heard of a DoD. Some had but had not written theirs down. However, they insisted that they knew what it was. As an experiment I asked several members of the same team what they thought their team’s DoD was. I got back as many different answers as the number of team members I asked. Although the teams thought they had a shared idea of their DoD they did not.

As a result of these findings I have been on a DoD offensive of late. This has involved explaining the benefits of having a DoD and helping teams to get their initial DoD published. I have helped so many teams define these recently that I have come up with a workshop format to help drive them out. I present the outline of this DoD Workshop format in this post.

The Workshop Format

The workshop is designed to be run in-person and ideally the entire team should be present. When not all of the team members can attend and the team is not cross-functional then all functions should be represented. No specific preparation is required on the part of the team. I allow an hour for the workshop and hold it in a room with a white board or similar usable wall space.

I start by explaining what the session is about, i.e. creating the team’s DoD which will subsequently be published on their wiki. This of course involves explaining what a DoD is and how it can help the team. This normally raises a few questions from the team and I address these before moving onto the next stage of the workshop.

Next I hand out sticky notes and sharpies. I request that the team think of all of the high-level activities that they normally engage in to get a story done. I ask them to write each of these items onto the sticky notes with one activity featured on each sticky note. I insist that everyone writes up at least a few sticky notes and that they only include the activities that they do now not things they would ideally do in the future.

Once everyone has finished writing I get the team to post all of their items, in no particular arrangement, onto the white board.

Click to Enlarge

Click to Enlarge

Next I ask the team to collaboratively group similar and identical items together. Team members may have used different terminology to describe an otherwise identical activity but by working together they can identify which stickies are actually the same thing.

Click to Enlarge

Click to Enlarge

With the groupings complete we go through each one and validate all of the items within it are truly the same activity. I also check that the team actually does the particular activity now and that it is not simply an aspiration. Any such aspirational items are removed.

Next I work with the team to place each activity group into a timeline. The placement of the activities reflects the order in which each takes place within the team’s development process.

Click to Enlarge

Click to Enlarge

With the timeline in place we now turn attention to creating a headline name for each activity. This is necessary because individual team members may refer to the same activity by a different name. For example, “Implementation”, “Coding” and “Development” are all different terms for the same things. Having the team agree on a single name for each activity introduces a consistent terminology that the team can use to enhance their communication when discussing their DoD. In addition if a team struggle to create a headline name for one of their activity groups then it suggests that it may need to be broken up into several more nameable activities.

The agreed headline names are written against each activity.

Click to Enlarge

Click to Enlarge

With the headline names agreed I go through each activity again. This time I ask the team to define what is involved in accomplishing each of them focusing on the deliverables, outcomes and conditions that represent success. These descriptions are written against each activity beside the headline name.

Click to Enlarge

Click to Enlarge

The combination of the ordered activities and the more detailed descriptions represent the team’s DoD.

Post Workshop

Finally I snap a picture of the board and collaborate with the Scrum Master to have it documented in the team’s wiki and communicated out to the team. If possible I also recommend that a paper copy be posted somewhere in the team’s physical work area.

Note my insistence throughout the process of only including the things the team does now. This may seem overly constraining when the team could be suggesting good adaptions. However, the main purpose of the exercise is to capture what the team does now. Immediately after the DoD is agreed and published the team is free to discuss it and make positive changes. This may even happen in a separate, later section of the workshop. The key is to write it down first, warts and all, so that the team’s improvements can be made against a known state.

When to Run the Workshop

So far I have only used this workshop format to help document the DoD for teams that are already sprinting. However, new teams should not wait until they are partway through a project before they create their DoD. They should instead put it in writing at the very beginning.

I envisage that this workshop format could be used equally as effectively to create the initial DoD for a newly formed team. The only difference would be to have the team write-up stickies for the activities they think they will need based on their previous experience. Otherwise the session could be run in much the same way.


Follow

Get every new post delivered to your Inbox.

Join 79 other followers