Caveat Emptor SaaS

The Financial Times began its journey into cloud computing a number of years ago and recent years and months have seen an acceleration in the use of the cloud beyond just email and CRM and into core systems of record.

Cloud computing describes any kind of computing hosted online, via the internet.  Software as a Service (SaaS) is one of the many cloud computing terms, describing a software delivery model in which software and associated data are hosted centrally on the cloud.  Other cloud terms include Platform as a Service (PaaS) and Infrastructure as a Service (IaaS).  In a SaaS instance, a single version of the hosted application is used for all customers in a so-called multi-tenant architecture, in which the application scales horizontally to serve multiple customers.

SaaS promises a number of benefits over traditional software, including:

  • reduced costs through outsourcing hardware and software maintenance and support to the SaaS provider

  • usage scaling to meet the demands of the customer, supported by pay-as-you-use models
  • faster feature release cycle and automatic upgrades

I jumped on the FT cloud wagon 2 years or so ago and have been actively involved in selecting, integrating with and managing services with a number of SaaS vendors during that period.  The journey hasn’t always been easy and for some of our requirements, SaaS may not have been the right model.  This post details some of my thoughts, considerations and challenges with SaaS, including non-functional considerations such as performance, latency and availability, the challenges of fitting SaaS around your application development lifecycle and coping with operational restrictions.

Solution Architecture

Make sure you have a clear policy about where in the organisation you’re prepared to invest in SaaS.  Some companies perceive the security or service availability risks (see later for details) as being too great for core brand revenue systems.

Invest in up-front architecture if you need to integrate with on-premise solutions.  Understand as part of the evaluation how you’ll get data out of the SaaS solution for reporting or analytics.  I’ve been surprised by some SaaS vendors’ API immaturity – sparse or poorly written with SOAP only, or RESTful interfaces that score low on Richardson’s Maturity Model.

If possible during the evaluation, bring the vendors’ professional services teams on site to deliver meaningful proof of concepts.  You’ll learn huge amounts about both the product and the willingness of that company to invest in you.  Actively invest time in the user community, where you’ll gain insights into real customer scenarios.

Custom Features

In contrast to many on-premise hosted applications, SaaS applications may only be customised for an individual customer in limited ways.  This has important implications for responding to customers’ changing business needs.  Whilst customising some traditional software products may still require expensive vendor’s consultants, this is absolutely the case for any feature development in the SaaS world.  It’s vital then that during product evaluation, any identified gaps in short and medium term feature requirements either exclude the vendor from the evaluation process or result in the vendor’s written agreement of their ability to ship this feature within an appropriate timescale.

When comparing vendors, always ensure there is a mix of SaaS and on-premise in your matrix, so that the different options can be adequately compared.


When integrating with any vendor, it’s important to consider how the solution will integrate within your software development lifecycle.  Many teams within the FT have complex development lifecycles with several pre-production environments to test different degrees of integration and functionality, including a test environment which usually has a large dataset, often an anonymised copy of live data.

Yet the model with some SaaS providers is to provide a single pre-production sandbox and a single production environment.  Some providers may provide additional pre-production environments but at an additional cost.  In general consider the following:

  • Include your pre-production SaaS tenant requirements in your vendor selection criteria.  Ensure costs to create the required tenants are understood or preferably included in your Master Services Agreement (MSA).

  • Determine whether you can self-serve on creation or refresh of your pre-production tenants and if not, determine the impact of having to request your vendor’s professional services or operations teams to do this for you.

  • Determine how you intend to load test your pre-production tenants and whether these tenants can scale to your demands.  One common scenario is for vendors to arrange for a time-slice of a shared sandbox configured for performance or bulk testing.  If, like the FT, you need to be able to load test your continuously enhanced application at frequent  but unscheduled intervals, this will not be appropriate for your load test needs but might work out for the occasional stress test.

  • Pre-production tenants are frequently multi-tenanted but reduced in performance.  Unlike on-premise software, where you may know how your pre-production environments are scaled compared to live, you’re unlikely to gain this information from a SaaS vendor.  Try anyway to get absolute sizing or multi-tenant limits information on your sandbox and your live environment.  If you’re unable to adequately test your latency or scaling needs, then you’ll have to try to simulate your expected load (plus headroom) in live as soon as you can.

  • Understand how you can manage configuration and common static data between pre-production and live SaaS tenants.  One particular SaaS product at the FT provides only a user interface for managing much of the configuration of any tenant, rendering configuration management between environments a manual process which is impossible to adequately source control or automate.


Whilst some SaaS vendors are solidly mature in the enterprise space, others have grown from a more local customer base and struggle to meet internationalisation requirements such as multi currency, multi time zones or international character sets.  One of the largest challenges we’ve so far encountered at the FT is a particular SaaS solution that processes client data in local US Pacific time (including daylight savings) rather than UTC, causing a number of serious complications which require manual code and process workarounds.

Global Performance

As a company with an international customer base, the FT would like to be able to offer the same experience to users in all our identified global Tier 1 locales.  We’ve invested significantly in Content Delivery Networks (CDNs) to enable greater local performance for much of our estate and where possible designed our back-end systems for distributability to global data centres – an area we’re investing in at present.

SaaS can represent a threat to this model.  Many of the SaaS solutions I’ve been exposed to are only hosted in the US.  Whilst some make use of CDNs, some do not or require calls to pass through to the SaaS origin for processing.  Many of these solutions are currently integrated with our data centres in UK, so typically every call to the SaaS solution incurs a trans-Atlantic delay of around 100 milliseconds.


A typically quoted benefit of the cloud is uptime, where the logic has it that the demands of multi-tenancy drive incredible operational availability metrics.  Your MSA will define the availability SLAs (measured per minute monthly).  Even if it sounds high, this can give long outages, even within the SLA. For example, 99.9% monthly uptime equates to more than 30 mins outage.  Additionally you may have to commit to a proportion of scheduled downtime, which can add hours to your system unavailability.  Add to that the availability multipliers when you integrate with your internal systems and if you have a business continuity plan, you could well exceed your Recovery Time Objective (RTO).

After initially jumping on the SaaS bandwagon directly for a number of our core operational systems, we’re now investing in architecting away from a synchronous use of SaaS solutions where possible to that of buffering and eventual consistency with the SaaS system of record.  It adds considerable complexity to our solutions and dilutes some of the benefits.

Security, Compliance & Data Backup

Data security is often touted as an advantage of the cloud.  Once on the cloud your data is safe from fire and flood – if replicated to multiple sites – something you should of course check.  Also check you have adequate legal protection for your data if hosted away from your territory.  For European customers hosting data in the US, a certified US-EU Safe Harbour agreement is a must.

Ensure all data transmission is made securely through Transport Layer Security (TLS – previously SSL).  You should also consider encryption both for transmission and storage of all your customer and other business critical data on the cloud.  If you’re using the SaaS solution to transmit, process or store credit card data, then ensure PCI-DSS Level 1 compliance.

Also ensure you have adequate processes for data backups outside of the SaaS vendor.  Should the vendor go bust or be acquired, what will happen to your vital customer or other business critical data and what impact would this have on your business?

The recent spate of attacks on media companies, demonstrates that putting your business assets into the hands of others’ carries significant risks.  Attacks on the New York Times and Washington Post were both carried out through social attacks on third-party services used by these companies.  Since SaaS services are administered through public websites, they’re naturally vulnerable to attack.  You really do need to ensure that your customer data and services are protected by more than a username and password – preferably by two or multi-factor authentication.

You may also want to ensure that any suppliers or other trusted users of the systems use only your company email addresses for usernames to reduce the attack on compromised accounts being vulnerable to password reset attacks – at least you have control of your own domain if an account is compromised.

Operations & Support

As for any product or system, ensure you have adequate support both during your development phases and for the evolving live service.  Due to the immaturity of some vendors, we’ve encountered a number of support challenges including US West coast-only support that introduced considerable latency into the resolution of both development and live support tickets, poorly defined methods of escalating critical tickets quickly and systems which do not provide a single view of all open tickets within your company.  Whilst these problems aren’t limited to SaaS, the temptation to use less mature companies in the SaaS world (because they may have grown quickly and cornered a key market) makes support challenges greater.

The FT is maturing quickly in how we monitor our systems, with enhanced monitoring now written into our Technology department objectives.  We’ve invested in Splunk and Nagios for application monitoring and we’re rapidly developing a great operational picture of our systems.  Unfortunately SaaS systems rarely provide any monitoring capability beyond a ‘Trust’ page displaying tenant-level availability and the ability to periodically ping a representative end-point to confirm the service is still available.

We’ve become accustomed to logging everything a user does on our site, with the ability to correlate calls throughout the architectural stack and to aggregate data with powerful queries or to accurately trace individual problems.  This is all at risk with SaaS where the system is typically a black box with no visibility of any application logging.

A 16 hour partial outage earlier this summer of a SaaS product vital to the FT’s core services, proved impossible for us to debug and left us at the mercy of an inadequate vendor’s Operations department. It cost us dearly – Customer Services, FT Operations and Development teams all devoted huge efforts and we lost significant revenue and probably brand loyalty on the day.  It’s resulted in some soul searching and has made us seriously question the use of SaaS for such service critical systems.

Charles Eschinger, research vice president at Gartner stated in 2012: “The decision to deploy SaaS based applications within an enterprise is dependent on the business criticality of the solution, as well as geography, business agility, usage scenario and IT architecture”. He added “Few organisations will completely migrate to SaaS.”    I would advise anyone thinking of investing in SaaS to heed these words and to consider carefully the decision to adopt SaaS in their enterprise.