Last Thursday I did a talk at the Ticketing Professionals Conference 2018 with Caroline Aston, Audience Insight Manager at Chichester Festival Theatre.

Our talk was about lessons learned the hard way when managing a (new) website during a big online on-sale. Here're our notes:

===

JAMES: 3 years ago we undertook a project to redevelop Chichester Festival Theatre's website, to coincide with the launch of both a new CRM system (Spektrix) and their 2015 Winter Season.

We’ll take you through that process – the on-sale, what we learnt from it, and what’s been put in place since.

CAROLINE: Chichester Festival Theatre is one of the UK’s flagship regional theatres.

Capacity: 1,621

Performances in 2016/17 seasons: 508

Number of tickets available in 2016/17: 405,820

Annual turnover: £14.2m

We have two venues (Chichester Festival Theatre and the Minerva), and two distinct seasons per year – the spring/summer Festival Season, during which our own productions are staged, and the shorter Winter Season, primarily featuring touring productions and one-nighters.

Each season has two big on-sale dates – first for Friends, then a general on-sale. Over those four on-sale days we take around 29% of our box office for the year. And 42% of box office revenue is taken over the first two weeks of each on-sale.

Our project to change to new a CRM (Spektrix) started in April 2015, and it quickly became apparent we’d need a new website before the Winter Season launch in September, when Spektrix was due to go live. (We opted to launch at this point rather than the Spring season as web traffic is lower, plus there was a reduced Winter programme in 2015 due to refurbishment of the Minerva.) We began working with Supercool in June 2015.

We had a very small window – 4 days, during which there were no performances – to migrate all data from the old CRM into Spektrix, and make the website live. With 4 shows left in the Festival season, downtime had to be kept to a minimum.

JAMES: By that point we already had a decent amount of experience designing and developing these types of sites – but not dealing with huge traffic spikes.

Our primary concerns:

• Limited time to design and build the site*
• No soft launch
• Unknown demand – other than we knew it’d be high
• Load testing
• Season launch – so load testing was difficult
• Using Amazon AWS (then new for us)

*So, we decided on a 2-phase approach – loosely meaning we’d build a ‘minimum viable product’ ready for launch, with secondary development planned-in after the initial launch.

We put together a (very) comprehensive risk assessment – aka The Disaster Plan – listing everything that could possibly go wrong, with back-up plans for various scenarios:

CAROLINE: We meticulously plan activity around our on-sales; from when flags will be hoisted on site, to when we contact the audience through various channels.

Tickets are available online only for the first few days, so we provide additional support for customers. Before any on-sale we encourage customers to go to the website to check their login details.

In 2015, for the first on-sale with the new systems, we increased this effort and contacted all customers who’d booked online over the previous 3 years, and encouraged them to create a password (their details weren’t migrated to the new system). It paid off with 27% of all those contacted visiting the login page.

We also produce user-guides in PDF format to take people through the booking process. Links to these were circulated in the week between go live and on-sale. Preparing the audience on these straightforward things means we can focus on any issues that come up during the on-sale itself.

Frontline staff are also well-prepared – they’re taken through the common issues that customers may ask about (password resets, cookies, out of date browser), and have an ever-expanding list of FAQs to refer to. This pools knowledge from across the organisation, so we’re all able to answer questions.

JAMES: Design and development went smoothly, and the new website went live as scheduled on 9 September 2015, along with Spektrix; with the Winter season announced the next day.

The initial launch and season announcement went pretty smoothly and, based on traffic from the announcement day, we had a good idea of the traffic expected for the big on-sale.

The Friends on-sale happened at 10am on 14 September 2015, and was the website’s first big test. The site lasted for about 30 minutes … before it came crashing down. Worst. Nightmare.

We scaled-up the server and rebooted it – but it crashed again. We scaled-up even more – but another crash. Third time lucky? No. After about 30 minutes we reverted to the back-up plan, pointing the load-balancer directly at Spektrix’s iframes, so people could at least still buy tickets.

But we still had to diagnose the problem, fix it, and get the website back up and running.

After a *very* long-seeming 2 hours, we’d fixed the issue, and the website was working fine again. (For anyone interested, the issue had been with the Varnish config; we weren’t caching query parameters from marketing email links.)

CAROLINE: During the downtime we couldn’t inform customers what the issue was, as our main tool for communication – the website – was down. To manage complaints and questions, different staff were assigned different platforms.

Communication between us and Supercool was over the phone which wasn’t ideal as it took James away from working on the problem with the developers.



JAMES: We learnt a lot that day; once the site was stable again we immediately started the post-mortem.

The specific critical issue fell into the category of ‘unknown unknowns’ as Donald Rumsfeld might say. It would've taken a lot of testing and blue sky thinking for us to spot this before it happened. Having said that, there were certainly areas of process we could improve for the future, including:

• Using instant chat between teams at Supercool, CFT and Spektrix – we use Slack
• More testing of the season programme before on-sale
• Monitoring of social media to highlight as-yet-unreported issue

We’ve since added a number of features to the site to cope well with sharp spikes in traffic.

One of my favourites is the ability to update certain elements of the page without breaking page caches. Sitewide-notifications and ticket buttons update through javascript, without pages even needing to be refreshed, which means pages update instantly – even under heavy load.

And each subsequent on-sale has gone smoothly; in fact we’ve made improvements to the process season-on-season.

CFT like to keep us on our toes, changing something for each on-sale e.g. advertising the time of announcements/on-sales, ticket purchase queuing, adding a new ‘advance priority’ on-sale level.

And we’re always looking at ways to make the booking process quicker and easier for people …

CAROLINE: Because it was a new website we’d installed a Feedback button – and had some choice responses once the website was back online! But also some useful points which we actioned:

User guides – we developed more user-guides, including a video. The more we prepare people in advance, the easier it is for us to address concerns voiced on the day.

Notifications – we have worst-case-scenario messaging planned in advance and ready to go live, on Spektrix, the website, and other platforms; it’s easier to construct the wording when you’re not under pressure.

Customer response – staff are assigned specific platforms (social, email) to monitor during the peak period, so responses are quick.

Marketing/social – Details of planned emails and social posts during the on-sale are shared with Supercool beforehand, so they can predict and prepare for traffic spikes.

The benefits of a well planned on-sale are:

• Calmer, more effective staff
• Happier customers
• Higher sales
• More donations (253% increase in donations on Friends first day of booking since our first Festival on-sale with the new systems)
• Positive social responses …

JAMES: In summary, for a successful big online on-sale you need:

• Thorough planning (including disaster planning)
• Good communication (internally, between all external teams, and with your audience/customers)
• No complacency
• Carbs and caffeine
• Debrief / lessons learned … ready for next time

Here’s a link to the full set of slides from the talk, which might not make much sense by themselves – if you want more detail, pop me an email: james@supercooldesign.co.uk

===

As an aside; one of the main reasons this became a useful, rather than disastrous, experience is the good relationship we have with the team at Chichester.

We’d built a strong and trusting relationship during the design and build process so, despite the crash, how we then handled the situation (plus our stellar and extensive Disaster Plan) gave the CFT team confidence in our ability to cope under pressure. And it's quite possibly made for an even stronger working relationship now.

Not that it’s an experience we ever intend to repeat with either CFT or anyone else.

We’ve since put what we learnt to good use for other clients too, managing huge – and hugely successful – on-sales for Kraftwerk, London Grammar, Gary Barlow, and Sir Cliff for THSH, and the monumentally popular ticket ballot and on-sale for RADA’s production of Hamlet, directed by Kenneth Branagh and starring Tom Hiddleston.