Skip to main content

This documentation is intended for internal use by the RE team.

GDS AWS Account Management Service


Creating new AWS accounts

Reliability Engineering are responsible for creating and setting up new AWS accounts for the rest of CO. These should be joined to the CO organisation so the account appears on the consolidated billing. The root credentials should also be secured in a standard way.

Requests for new AWS accounts should be made via the gds request an aws account app. When someone submits a request the app generates new terraform config, which needs to be applied.

The steps are:

Basic account creation

Read the “New AWS account request” email.

The ‘gds request an aws account app’ sends an email to the address. There are basic instructons in the email to get you started.

Review and merge the pull request

The app generates terraform config in JSON format and opens a pull request with the changes against the aws-billing-account git repo. You need to review the PR, and if it looks sensible, approve and merge it.

You will have noticed that the terraform always includes a "role_name": "bootstrap", line - AWS Organizations will have created a bootstrap role, which we will use, and then delete it later. (Deleting this role does not cause any problems with terraform or AWS Organizations.)

Run the terraform

You will need to be listed as an org admin (billing-org profile) in account_terraform/ to perform this (technically you could be a full billing account admin, but please don’t use that for this if you have it).

The aws-billing-account terraform is applied using the gds-cli billing-org profile.

Change to the aws-billing-account‘s terraform directory and apply the terraform:

cd ~/git/gds/aws-billing-account/terraform && git checkout master && git pull
gds aws billing-org -- sh -c 'terraform init && terraform apply'

This will show a plan. Review the output and if you’re happy, continue with the apply.

You might encounter issues with punctuation in the tags provided by the requestor at this point. Some characters are not permitted in Amazon tags. If this happens you will need to edit the tag text in the file and commit and push the changes once the account creation has succeeded.

The terraform will have created the new account and joined it to the organisation. However, it is not able to complete any other steps required.

Keep a note of the new account ID. (If you lost the scrollback, you can find the account ID via aws organizations list-accounts.)

Configuring IAM access to the account

Edit the trust relationships on the bootstrap role

We have a small script that is used to assume the bootstrap role and alter it to trust the bootstrapping user. The script still lives in the old re-infra-release-automation repository and should be given the account ID number you found in the terraform output above.

It assumes you have an aws-vault profile named gds-users, which most gds-cli users probably do.

~/git/gds/re-infra-release-automation/ 012345678912
gds-cli bootstrap namedRole

Since we’re all gds-cli users now, this process will assume you want to bootstrap the account using gds-cli, and possibly hand it over to users for them to just use through gds-cli conveniently.

Open gds-cli.git and go to pkg/gds_aws/aws-roles.yaml. You will need to add a new entry to one of the accounts lists in a group (you may also invent a new group if it doesn’t fit into the existing ones). Each account needs the AWS account ID (which you will have from the terraform output above) and a list of roles.

Here’s an example group with a new account:

- name: Reliability Engineering
  - id: '012345678901'
    - type: namedRole
      name: re-example
      desc: admin access to re-example
      role_name: bootstrap

Run go build and we’ll use our newly built gds-cli tool in the next step.

Account terraform

Instead of handing access to a bootstrap role to the requesting users and telling them to get on with it, we create admin roles for the requested initial admin users. The role names should be compatible with gds-cli’s adminRole naming convention, i.e. they should be suffixed with -admin.

We do this through some terraform that we store in the tech-ops-private repo.

  • Decide whether this will be kept in reliability-engineering, cyber-security, or cabinet-office.
  • Run git log on the directory to find the latest created account there.
  • You should see files like reliability-engineering/terraform/deployments/re-example/account/, .terraform-version and possibly .terraform.lock.hcl.
  • Copy just the and .terraform-version files to $new_account/account.
  • Update your .terraform-version to the current version of Terraform, if it isn’t already.
  • Modify
    • Comment out the terraform s3 backend section at the top.
    • Change the allowed account ID to the ID of the new account.
    • Change the state bucket name.
    • Change the admin user modules such that there is one for every requested initial admin user, and yourself (you will need it to test everything is working correctly, and the requesting users may come back to you for help and with questions - they may remove the role later themselves)
    • At time of writing, restrict_to_gds_ips should be fine for GDS and CO users, it might need to be disabled for contractors.
    • If you intend to change iam_policy_arns to something other than full admin, change role_suffix, and consider providing a userRole (with a format field of e.g. %s-newsuffixhere) instead of adminRole below.
    • It is expected that all accounts have at least two full admins other than the bootstrapping user.
    • Remove the gds_security_audit_role module at the bottom. This is no longer necessary.

~/git/gds/gds-cli/gds-cli aws $(basename $(dirname "$PWD"))-admin -s
terraform init &&
    terraform providers lock \
        -platform darwin_amd64 \
        -platform darwin_arm64 \
        -platform linux_amd64 \
        -platform linux_arm64 &&
    terraform apply -auto-approve

If Terraform takes a long time trying to create a user role, it’s probably because the user doesn’t exist. Check that the email address is correct and if necessary, create the user via the request-an-aws-account app.

Now return to and uncomment the s3 backend section. Re-run terraform init and let it upload the statefile (which so far is only on your local machine) to S3.

terraform init -migrate-state -force-copy

Commit your, .terraform-version and .terraform.lock.hcl files, raise a PR and get it reviewed and merged.

gds-cli adminRole

Now it’s time to configure gds-cli how we actually tend to use it, in adminRole mode (as programmes become large and we began to hit the size limits on assume role trust policies, we moved away from a single role trusting everyone, known in gds-cli as namedRole, to per-user roles, known in gds-cli as adminRole). Remove the role_name: bootstrap field and change type to adminRole. You should now have something more like this:

- name: Reliability Engineering
  - id: '012345678901'
    - type: adminRole
      name: re-example
      desc: admin access to re-example
Test gds-cli and remove bootstrap role

Rebuild gds-cli.

go build

Use the new gds-cli binary to log into the new account and remove the bootstrap role.

./gds-cli aws re-example -- bash -xc '
aws iam detach-role-policy \
    --role-name bootstrap \
    --policy-arn arn:aws:iam::aws:policy/AdministratorAccess &&
aws iam delete-role --role-name bootstrap'
Raise a PR for the gds-cli changes

Remember to get this reviewed and merged.

Configure root access to the account

Trigger a password reset for the root user

As the following process involves logging in as the root user for this account, the Cyber Security team will receive an alert via the CloudTrail logs. You should notify them before doing the following steps by using the Action Notification action in the #cyber-security-notifications slack channel. Be sure to include the account ID you’re working on.

In a browser open the AWS console login page. Use the root user email address used when the account was created. This will be in the terraform output. Select the “forgot my password” link.

Generate a long, random password:

$ pwgen -sy 64

You will have received a password reset email to the aws-root-accounts@ email address. Click the reset link in the email and use the password generated above to reset the password and log into the account.

You do not need to store the new password anywhere and should be forgotten. We will use the password reset process if we ever need to access the account using the root user.

Set up MFA for the root user

MFA must be enabled for the root user on all accounts. We do not store the MFA tokens anywhere, but instead rely on Amazon’s MFA reset process to access an account using the root user.

Note: previously we used to store the root MFA tokens on pairs of Yubikeys in a safe in the Whitechapel building, but we no longer do this.

First, check the phone number in the account Contact Information is set correctly. This phone number is used during the MFA reset process and must be the number from the Google Voice hunt group. The contact information can be found by clicking the account name in the top right and then selecting Account.

Next, in the AWS console, click on the account name in the top right, then select My Security Credentials. Then select MFA, then Activate MFA, then Virtual MFA device. This will display a QR code.

Click the “Show secret key” link under the QR code, which will display the key as a string which you can seed the yubikeys with.

Make sure your Yubikey is connected to your laptop, then do:

ykman oath accounts add '[account name]' '[MFA token string from above]'

The AWS console will request 2 consecutive codes, which will be generated based on the clock of whichever device you have inserted the Yubikey into, every 30 seconds. The following command will show the current code:

ykman oath accounts code '[account name]'

Once you have inserted the two consecutive TOTP codes AWS will check they are correct before saving.

As we do not store the MFA token you should now delete it from your yubikey:

ykman oath accounts delete '[account name]'

You can now sign out of the console.

Inform the requestor the account is ready to use.

Tell the requestor the account is ready to use. You might find the following template email useful for this:

Hi [Recipient],

I have created your new Amazon account as requested. The details are:

re-example (Account ID)

You and the other people listed in your request should now be able to access these accounts using your gds-users identities by assuming your user-specific admin roles, which follow the naming format firstname.surname-admin. There is guidance on how to do this here.

I have raised a pull request to add these accounts to the gds-cli. The pull request is here.

If you have any further questions, ask on the #reliability-eng slack channel.