The Performance Platform is a set of web applications which power gov.uk/performance. Reporting data is point 17 on the Digital Service Standard.
The Performance Platform consists of a number of microservices:
They are all hosted on PaaS, apart from the production
backdrop which is still running in GOV.UK’s Carrenza infrastructure.
Backdrop is a datastore with two separately deployable HTTP APIs for reading and writing data.
Backdrop can run either with MongoDB (legacy) or using PostgreSQL.
- Source code
- alphagov/pay-performance-reporter In the wild example of how backdrop is written to
- GOV.UK Puppet manifests
cmd + f "backdrop"
Stagecraft is the internal management Django application for managing the Performance Platform. It also is the canonical source for dashboard configuration which is available publicly. It’s also known as the “admin API”.
Stagecraft has a number of
collectors which are asynchronous tasks coordinated by Celery.
Performance Platform Admin
Performance Platform Admin is a web frontend for dashboard owners to upload XLSX sheets to the Performance Platform, and for “power” users to edit/manage dashboards.
Spotlight is a Node.JS frontend for rendering Performance Platform dashboard. Spotlight sits behind Fastly, which is used as a CDN for improved performance.
Spotlight is the only app where we really care about downtime, everything else is fairly resilient due to the CDN. If Spotlight is down for a long time then users notice and the CDN may also cache a bad page.
Anyone can view statistics hosted on gov.uk/performance, everything is public, including dashboard metadata hosted on stagecraft.
Service owners & developers
There are three ways of putting data into the Performance Platform, two of which are self-service:
- XLSX upload
Service owners can upload data via the performance-platform-admin tool
Developers can implement functionality in their service to automatically push statistics to the Performance Platform, more information is here, hosted on readthedocs.io.
Internal management users
Clifford Sheppard and Daisy Wain are the two people at GDS who manage the Performance Platform. They liaise with services for the quarterly transaction explorer import.
Support, monitoring, outages
There is no longer a team who work on this, and there is not much observability of the apps. We did a small amount of work to add some monitoring. The things that we care about principally are:
- Backdrop (read) 200s
- Backdrop (write) 200s
- Backdrop (worker) task successes
- Stagecraft task successes
We only really care about production, staging is used just as a demo environment for transaction explorer data, and ensuring that code changes work.
Stagecraft uses django-celery which allows you to view and create tasks through a web-ui.
We have added Flower, a Celery monitoring tool, to monitor both backdrop and stagecraft:
- Stagecraft (Prod)
- Stagecraft (Staging)
- Backdrop production is still running in Carrenza so there is only monitoring in staging.
Sometimes we get reports of “Big Edit” being broken (users see a
500 error after clicking the button). This is specifically in reference to the dashboards page on Performance Platform Admin.
We’ve not worked out why this happens yet but previously we’ve resolved this problem by restaging stagecraft-web (
cf restage performance-platform-stagecraft-web).
Has two functions: read and write. The read API receives a request and serves it. The write API receives data from a request and enqueues it.
Backdrop has Celery workers which take enqueued data blobs and writes them to the backdrop database.
The database when running in GOV.UK’s infrastructure is MongoDB (v2.4). The database when running on the PaaS is PostgreSQL.
Is a big Django app that also uses Celery. The Django app is for management of dashboards (either via an API), or manually using the admin panel.
Stagecraft also has
collectors which connect to services such as:
- Google Analytics
And retrieve data about various things, e.g. session length, website uptime, etc
In order to access these services, it uses credentials stored in the Stagecraft (PostgreSQL) database.
Every quarter GDS sends out to each department a subset of a large spreadsheet which they get each service to fill in. The results are collated in this spreadsheet and then ready for being uploaded to the Performance Platform.
When uploading the data, different pieces need to go to different places. The two destinations are
stagecraft. As above, the data needs to go into
backdrop, and the dashboard metadata needs to go into
The process for importing data into backdrop is described in backdrop-transactions-explorer-collector.
The process for importing data into stagecraft is described in this document, but can be roughly summarised as:
- Using credentials in paas-pass:
cf target -o gds-performance-platform -s staging cf ssh "performance-platform-stagecraft-web" -t -c "/tmp/lifecycle/launcher /home/vcap/app 'bash' ''" export GOOGLE_APPLICATION_CREDENTIALS=credentials.json export SUMMARIES_URL=https://performance-platform-backdrop-read-staging.cloudapps.digital/data/transactional-services/summaries rm -rf *.pickle # in case any files have been cached by this insane process python -m stagecraft.tools.import_dashboards --update --commit python -m stagecraft.tools.import_organisations # this failed for us, and we don't care because the organisations haven't changed
- Notify Clifford that staging has been done
- Wait 1 week, get new data
cf target -o gds-performance-platform -s staging cf ssh "performance-platform-stagecraft-web" -t -c "/tmp/lifecycle/launcher /home/vcap/app 'bash' ''" export GOOGLE_APPLICATION_CREDENTIALS=credentials.json export SUMMARIES_URL=https://www.performance.service.gov.uk/data/transactional-services/summaries rm -rf *.pickle # in case any files have been cached by this insane process python -m stagecraft.tools.import_dashboards --update --commit python -m stagecraft.tools.import_organisations # this failed for us, and we don't care because the organisations haven't changed
What we did in Q2 2018
This work was conducted by the PaaS team over a period of months between July and September 2018.
- Rewrote backdrop to be able to use PostgreSQL instead of MongoDB, upgrade version of Redis, etc
- Use new redis transport in performanceplatform-admin and vendor the css because it was not building
- Add support for multiuser pingdom, use new redis and celery versions (Stagecraft)
- Added support for the
redisstransport used by the PaaS’s redis service
- Added some basic monitoring for stagecraft and backdrop using Flower
- Wrote some scripts to help migrate from staging MongoDB to PostgreSQL
- Migrated the staging MongoDB to PostgreSQL:
- Git repo
xargs cf run-task
- Supported transaction explorer quarterly import into staging/integration/Production sccessfully with the change.
Transaction Explorer data
- Migrate backdrop in production from Carrenza to PaaS
- Remove MongoDB database interaction code
- Implement “collection” capping (if necessary)
- Address any performance problems from PostgreSQL migration