FT Technology Internal Conference: Engine Room 2015

The FT technology team held their inaugural (and pleasingly successful) internal conference at the start of February.   How we came to put the event together, and how it all went so right we will put (irritatingly smugly, I expect) in a later post.  First an account of the day itself, the benefits of holding such an event and some of the ‘takeaways’ we gleaned from it. We named the conference after this blog, calling it ‘Engine room 2015’ and we hoped it would be an opportunity for people to exchange ideas about how we work and what we work with/on.  The organisers sold it to the rest of the department as a chance to ‘lift your heads up from the coal face for a day and maybe even choose a better pickaxe’.

The format for the day was shamelessly stolen from the successful Edge Conf which has been running a couple of times a year since 2013.  It was based around five panel and audience discussion sessions on subjects interesting to the good people of the FT technology department. Questions were  submitted beforehand by people from across the department and put to the panel in a style strongly reminiscent of Question Time.  Each panel started with a ten minute presentation to introduce the topic. We had an average attendance of over a hundred people at each session across three countries (London and, via video conference, Romania and Manila). We also held a couple of small break out discussions over pizza during lunch.

The day kicked off with a short introductory talk on how much the FT technology department had changed in the last few years and how a conference fitted in perfectly with the ‘always learning’ ethos of the FT’s parent company Pearson.


New ways of working as a delivery team

The first panel of the day was kicked off by a talk from Matt Chadburn about how we need to consider building the right ‘it’, rather than simply building ‘it’ right. We need early and regular feedback loops, achieved by adopting a healthy Build/Measure/Learn cycle in order to check whether what we’re doing actually works.

There was much talk about QuickStarts and variations thereof. Where a team really understand their goal, and put their heads together to “have a go” at an idea to test it out they have had some real success. Much discussion was then had around the lifecycle of a product. We need to test the value and validity of an idea before building anything – our Discovery is often too much around the How and not enough around the Why / If. Also, at the other end of the lifecycle – we do not tend to retire things. Coming up soon – new steps at the start and end of our lifecycles, with possible measures in place to help us remember them!

Automated Testing

Jay Sethi kicked this session off with an introduction to automated testing, introducing the concept of the testing pyramid.

The panel talked about how BDD and specification by example can help with making sure tests reflect customer behaviour. Testers should be focusing on exploratory testing and defining behaviour while developers should be writing test fixtures since this stops test writing being a separate task, and encourages a focus on testable code.

There was some discussion around whether it’s OK to break things in production if we react to that break quickly: paying customers have an expectation of quality and failures can impact the company’s reputation. With good communication there’s more tolerance of failure. Every product needs to assess risk (some things should never fail). Small releases make failures less likely.

UI testing is a problem because it tends to be slow, and tests can be flaky so it’s hard to know if there’s a ‘real’ problem. Metrics may be better: pick up bugs by looking at how customer interactions change. However, we do need to test the UI in a range of browsers: UI tests frameworks should be built for scalability to minimise run times.

Is Continuous Delivery at the FT “Good Enough”?

Continuous delivery (CD) is about frequent releases to production via a process that is heavily automated. Deploying to production is the only real measure of project progress: “done” means in production. Frequent small releases gives quick feedback and each release is lower risk. CD means optimising for mean time to fix rather than mean time between failures.

Several panelists are already using CD at the FT (and one team are continuously deploying). However, are we doing CD well, or just “good enough“?

Generally we’re convinced of the benefits of CD, but we do still need to improve our processes and tools. No-one is happy with Jenkins as a CD tool and alternatives don’t seem much better. Maybe the answer is to form a team focusing on CD improvements?

There’s a sense that development teams need to work differently for CD to work: more of the team need to have skills we traditionally left to integration engineers. But do we need to be ‘full stack engineers’ or is it better to have a mix of generalists and specialists? Also, how do we know where the boundaries are between the dev team and infrastructure operations: for example, if something like Heartbleed happens, who should be fixing it? We need to be absolutely clear so that things don’t fall through the gaps.

What’s next for platforms?

During the penultimate session of the day, a couple of themes surfaced in pretty much each question:

  • how we are and will be using AWS and Heroku
  • and what the plans are for the internally developed FTPlatform tool suite.

Frankly the organising team were expecting (read: hoping) much of this would be contentious (read: lead to fist fights).  It turned out there was a lot of agreement in the room that people responsible for the tools the delivery teams use should listen to their customers and ensure the benefits and functionality of the tools is understood.  Toolsmiths need to ‘make the right thing easy’ in terms of enabling operational requirements, such as monitoring and patching, to be seamless and simple so that delivery teams can just get on with making nice shiny new things.  However if delivery teams wanted more autonomy to create their own tooling/platforms they should be able to do so and assume responsibility for security/maintenance/monitoring etc.   We also spoke about the need to be cognisant of heavy reliance on vendors and internal tooling when it came to business critical services and future agility.

The Future of FT Tech

The tone of the final session was set by John O’Donovan’s talk about the culture of a forward-looking technology department. Management cannot “empower” a  team, the team must empower themselves by looking up from their screens and challenging themselves and the business. Ask questions, push boundaries, challenge constraints and don’t be scared to open your minds and put forward your ideas. Some of the concepts were taken from a fascinating “inno-mation” based on an account from a submarine captain: Embed the Captain. Rob Shilston then highlighted what a different world we are operating in now compared to 10 years ago, and how the “now” requires us to work in very different ways.

Throughout the panel discussion, the message around finding the balance between the rights of teams to make their own choices, and the responsibilities that come with that came up again and again. The introduction of microservices opens up many possibilities for a wealth of new technologies to play with (or drown in). We want people to get excited about the new things, and to be bold and brave, but we also need to remember what we have out there today. We need to support our current solutions, and we also need to remember that there ought to come a time to switch them off (something that was echoed from the first session). These points also raised a number of questions for our Product Management team – how to ensure there are time and funds available to look after existing solutions, and also how to know when they have reached a natural end of life and should be switched off. (Matt Larsson has written an interesting piece capturing these questions – link TBC).


The feedback for the event was surprisingly positive. Attendees enjoyed the format, the time out to meet each other, to learn what’s going on in different teams, and to offer up challenges on how things could get better. They feel they got a lot out of it, and they want more events like this to keep them thinking.

There are some real, tangible outcomes where people have taken specific actions away to try and change something:

  • Teams are saying they will try new ways of working.

  • A monitoring “action group” has been formed from the breakout session, setting themselves some specific changes to take away and own.

  • The internal tools team committed publicly to making small individual tools for dev teams to use

Just as importantly (if not more) there is a real sense that this has been a springboard, and that people have started to think about how they can change things.

  • People want to see change – not in a grumpy “everything’s pants” kind of way, but in a “we want to be even better” kind of way. That feels really positive, and we’re hoping that this will help people to bring about their own changes…

  • The conversations are still going – for example in the “techchat” internal mailing group, Product Management discussions, and new suggestions for the FT’s agile bookclub

  • People now know each other, know what they’re doing, and know who to go and talk to if they want to hear more about a particular idea

Also, we got lots of good ideas on how we can make this better next time – and there will be a next time 🙂