Table of contents

This documentation is intended for internal use by the RE team.

Performance Platform

The Performance Platform is a set of web applications which power gov.uk/performance. Reporting data is point 17 on the Digital Service Standard.

Hosting

The Performance Platform consists of a number of microservices:

  • backdrop
  • stagecraft
  • spotlight
  • performance-platform-admin

They are all hosted on PaaS, apart from the production backdrop which is still running in GOV.UK’s Carrenza infrastructure.

Services

Backdrop

Backdrop is a datastore with two separately deployable HTTP APIs for reading and writing data.

Backdrop can run either with MongoDB (legacy) or using PostgreSQL.

Stagecraft

Stagecraft is the internal management Django application for managing the Performance Platform. It also is the canonical source for dashboard configuration which is available publicly. It’s also known as the “admin API”.

Stagecraft has a number of collectors which are asynchronous tasks coordinated by Celery.

Performance Platform Admin

Performance Platform Admin is a web frontend for dashboard owners to upload XLSX sheets to the Performance Platform, and for “power” users to edit/manage dashboards.

Spotlight

Spotlight is a Node.JS frontend for rendering Performance Platform dashboard. Spotlight sits behind Fastly, which is used as a CDN for improved performance.

Spotlight is the only app where we really care about downtime, everything else is fairly resilient due to the CDN. If Spotlight is down for a long time then users notice and the CDN may also cache a bad page.

Users

Citizens

Anyone can view statistics hosted on gov.uk/performance, everything is public, including dashboard metadata hosted on stagecraft.

Service owners & developers

There are three ways of putting data into the Performance Platform, two of which are self-service:

  • API
  • XLSX upload

Service owners can upload data via the performance-platform-admin tool

Developers can implement functionality in their service to automatically push statistics to the Performance Platform, more information is here, hosted on readthedocs.io.

Internal management users

Clifford Sheppard and Daisy Wain are the two people at GDS who manage the Performance Platform. They liaise with services for the quarterly transaction explorer import.

Support, monitoring, outages

There is no longer a team who work on this, and there is not much observability of the apps. We did a small amount of work to add some monitoring. The things that we care about principally are:

  • Backdrop (read) 200s
  • Backdrop (write) 200s
  • Backdrop (worker) task successes
  • Stagecraft task successes

We only really care about production, staging is used just as a demo environment for transaction explorer data, and ensuring that code changes work.

Stagecraft uses django-celery which allows you to view and create tasks through a web-ui.

We have added Flower, a Celery monitoring tool, to monitor both backdrop and stagecraft:

App overview

Backdrop

Has two functions: read and write. The read API receives a request and serves it. The write API receives data from a request and enqueues it.

Backdrop has Celery workers which take enqueued data blobs and writes them to the backdrop database.

The database when running in GOV.UK’s infrastructure is MongoDB (v2.4). The database when running on the PaaS is PostgreSQL.

Stagecraft

Is a big Django app that also uses Celery. The Django app is for management of dashboards (either via an API), or manually using the admin panel.

Stagecraft also has collectors which connect to services such as:

  • Pingdom
  • Google Analytics
  • GOV.UK

And retrieve data about various things, e.g. session length, website uptime, etc

In order to access these services, it uses credentials stored in the Stagecraft (PostgreSQL) database.

Transaction explorer

Every quarter GDS sends out to each department a subset of a large spreadsheet which they get each service to fill in. The results are collated in this spreadsheet and then ready for being uploaded to the Performance Platform.

When uploading the data, different pieces need to go to different places. The two destinations are backdrop and stagecraft. As above, the data needs to go into backdrop, and the dashboard metadata needs to go into stagecraft.

The process for importing data into backdrop is described in backdrop-transactions-explorer-collector.

The process for importing data into stagecraft is described in this document, but can be roughly summarised as:

  • Using credentials in paas-pass:
  • Staging: cf target -o gds-performance-platform -s staging cf ssh "performance-platform-stagecraft-web" -t -c "/tmp/lifecycle/launcher /home/vcap/app 'bash' ''" export GOOGLE_APPLICATION_CREDENTIALS=credentials.json export SUMMARIES_URL=https://performance-platform-backdrop-read-staging.cloudapps.digital/data/transactional-services/summaries rm -rf *.pickle # in case any files have been cached by this insane process python -m stagecraft.tools.import_dashboards --update --commit python -m stagecraft.tools.import_organisations # this failed for us, and we don't care because the organisations haven't changed
  • Notify Clifford that staging has been done
  • Wait 1 week, get new data
  • Production: cf target -o gds-performance-platform -s staging cf ssh "performance-platform-stagecraft-web" -t -c "/tmp/lifecycle/launcher /home/vcap/app 'bash' ''" export GOOGLE_APPLICATION_CREDENTIALS=credentials.json export SUMMARIES_URL=https://www.performance.service.gov.uk/data/transactional-services/summaries rm -rf *.pickle # in case any files have been cached by this insane process python -m stagecraft.tools.import_dashboards --update --commit python -m stagecraft.tools.import_organisations # this failed for us, and we don't care because the organisations haven't changed

What we did in Q2 2018

This work was conducted by the PaaS team over a period of months between July and September 2018.

Code

  • Rewrote backdrop to be able to use PostgreSQL instead of MongoDB, upgrade version of Redis, etc
  • Use new redis transport in performanceplatform-admin and vendor the css because it was not building
  • Add support for multiuser pingdom, use new redis and celery versions (Stagecraft)
  • Added support for the rediss transport used by the PaaS’s redis service
  • Added some basic monitoring for stagecraft and backdrop using Flower
  • Wrote some scripts to help migrate from staging MongoDB to PostgreSQL

Data

  • Migrated the staging MongoDB to PostgreSQL:
  • Supported transaction explorer quarterly import into staging/integration/Production sccessfully with the change.

Transaction Explorer data

To do

  • Migrate backdrop in production from Carrenza to PaaS
  • Remove MongoDB database interaction code
  • Implement “collection” capping (if necessary)
  • Address any performance problems from PostgreSQL migration

References