AWS Secrets Manager storage
With storage the encryption envelope of every new secret lives
in AWS Secrets Manager instead of in the TYPO3 database. The decision and its
trade-offs are in ADR-054: AWS Secrets Manager storage adapter.
Table of contents
What lives where
| Data | TYPO3 database | AWS Secrets Manager |
|---|---|---|
| Identifier, description, context, metadata | yes | no |
| Owner, group tiers, expiry, counters, availability | yes | no |
Pointer to the AWS secret (external_, written
{region}:) | yes | no |
| Encrypted value, wrapped data key, nonces, checksum | no (columns are empty) | yes |
| Master key | no | no (see Key custody) |
The value is encrypted by nr-vault before it is sent. AWS stores ciphertext it cannot read, and the master key never reaches AWS.
Existing secrets are not migrated. A secret stored locally stays local and keeps
working after you switch; only new secrets go to AWS. Roll back by switching the
setting back: secrets already in AWS become unreadable until you switch to
aws again, so do not remove them from AWS. A secret stored in HashiCorp Vault
is not usable under aws and the other way round; nr-vault refuses it with a
message naming the adapter that owns it.
Install the AWS SDK
The SDK is optional. It is not installed with nr-vault; add it where you select
aws:
composer require aws/aws-sdk-php
Without it, the first operation that needs AWS fails with a configuration error that names this command. It never falls back to local storage, and secrets that are stored locally stay readable.
Prepare IAM
Give the adapter its own IAM user or role with only the actions it calls. Every
secret is named {prefix}/, with typo3/ as the default
prefix:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "secretsmanager:CreateSecret",
"Resource": "*",
"Condition": {
"StringLike": {"secretsmanager:Name": "typo3/*"}
}
},
{
"Effect": "Allow",
"Action": [
"secretsmanager:PutSecretValue",
"secretsmanager:GetSecretValue",
"secretsmanager:DeleteSecret"
],
"Resource": "arn:aws:secretsmanager:eu-central-1:123456789012:secret:typo3/*"
}
]
}
Create acts on a secret that does not exist yet, so AWS cannot scope it
to the prefix by ARN; the secretsmanager: condition limits it to names
under the prefix. The pattern typo3/* matches the default prefix; change it
if aws. differs. The condition is not exercised by the tests,
because the emulation does not evaluate policies like AWS does. The adapter calls no other action: it does not list, describe, tag or
rotate secrets. That the four actions are enough is tested; the ARN pattern is
not, because the emulation the tests run against does not evaluate it (see
ADR-054: AWS Secrets Manager storage adapter).
Provide credentials
Credentials are explicit. nr-vault never uses the SDK's default credential
chain: no environment variable named AWS_*, no ~/. file, no
instance metadata service. It reads three variables of its own, whose names are
configurable:
NR_VAULT_ AWS_ ACCESS_ KEY_ ID - The access key id. May instead be stored in
aws.; it is an identifier, not a secret.access Key Id NR_VAULT_ AWS_ SECRET_ ACCESS_ KEY - The secret access key. Read from the environment only; there is no setting that stores it.
NR_VAULT_ AWS_ SESSION_ TOKEN - Optional. Set it when the credentials are temporary, for example from a role your deployment assumes outside PHP. nr-vault does not assume roles itself.
Export them to the PHP process (the web server's pool or the CLI user), and name
other variables with aws., aws. and
aws. if you must. The defaults are not the SDK's, so
credentials that belong to something else on the host are not used by accident.
Configure TYPO3
$GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['nr_vault']['storageAdapter'] = 'aws';
$GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['nr_vault']['aws'] = [
'region' => 'eu-central-1',
'secretPrefix' => 'typo3/',
];
Every setting is described under AWS Secrets Manager. The adapter reads its configuration on each request; nothing is cached across requests.
The secret name is the prefix, a slash and the identifier. AWS does not accept a
colon in a name, so a : in an identifier (the site:<namespace>:<name>
form) is written as +. Changing aws. or aws.
changes where the adapter looks, see below.
Network and TLS
- Without
aws.the adapter talks to the regional AWS endpoint over https.endpoint - A custom
aws.(a VPC endpoint, a compatible service) must be a bare origin: no credentials, path, query or fragment. It must useendpoint httpsunlessaws.is set, which is for local development: the signed requests and every envelope would cross the network unencrypted.allow Insecure Http AWS_andENDPOINT_ URL AWS_are ignored.ENDPOINT_ URL_ SECRETS_ MANAGER -
Requests use the same client as every other outbound call of the extension. It refuses loopback, private, link-local and metadata addresses, follows no redirects and logs no requests. An endpoint on a private network, such as a VPC endpoint, therefore needs a literal entry (a wildcard does not count) in
$GLOBALS:['TYPO3_ CONF_ VARS'] ['HTTP'] ['allowed_ hosts'] config/system/additional.php$GLOBALS['TYPO3_CONF_VARS']['HTTP']['allowed_hosts'][] = 'secretsmanager.vpce.example.com';Copied! - A whole transfer is bounded to ten seconds, and a failed request is retried once.
Behaviour you should know
- Failure
- An AWS outage or rejected credentials make the secrets stored in AWS unreadable and new secrets unwritable; secrets stored locally are unaffected. On store, AWS is written first: a failure leaves no row behind.
- Names that already exist
- A new secret is created, never written over an existing AWS secret. If the
name is taken by something that is not an nr-vault secret for that
identifier, the store is refused (code
1794500011), the AWS secret is left unchanged and no row is written. Use a dedicatedaws.so names cannot collide with other users of the account, and give each installation its own. An unreferenced secret that an interrupted nr-vault write of this installation left behind is recognised and reused; one written by another installation is not. A row whose stored location differs from the one the current region and prefix resolve to is refused (codesecret Prefix 1794500014). If the audit write of a create then fails, a reused secret is put back as it was; a secret the call created is removed. A rollback that is skipped because somebody wrote to the secret since is logged as a warning. - Half-finished writes
- A crash between the AWS write and the row write leaves an unreferenced secret
in AWS, which the next store of that identifier overwrites. A row whose AWS
secret is missing is reported as an error when the value is read (code
1794500009), never returned as an empty secret.vault:anddelete vault:still reach the row, so you can remove it or write a new value.rotate - Delete
- Deleting removes the AWS secret with
Force, so there is no recovery window and no version survives, then the row. RunDelete Without Recovery vault:again after an interrupted delete. The flip side: nothing is left in AWS to recover a deleted secret from. A crash, or a failed row delete whose restore is also refused, after the AWS secret was removed leaves a row whose envelope is gone; that value is lost and has to be restored from a backup, or the row removed.delete - Versions
- Every rotation adds a version in AWS. The previous one stays staged as
AWSPREVIOUSuntil the secret is deleted. It is ciphertext and needs the master key, like the current one. - Master-key rotation
vault:refuses while any secret is stored outside the local database. It names the first one it meets and changes nothing. Data keys held in AWS are not re-wrapped by this release.rotate- master- key - Audit
- Every mutation is written to the nr-vault audit log, and reads are written
when
auditis enabled. When the audit write of a mutation fails, the previous state is restored on a best-effort, conditional basis: the rollback is skipped when somebody wrote to the AWS secret since, or when AWS refuses it. Either case is logged as a warning, and the secret and its row then need a manual check. Unlike local secrets, the AWS write and the database write are not one transaction: a crash after the AWS write and before the audit write persists the change without an audit entry. CloudTrail records every call to Secrets Manager, so compare writes and deletes there with the nr-vault audit log.Reads - Changing
aws.orregion aws.secret Prefix - The secret of a row is looked up under the current setting, and a row whose recorded location differs from it is refused. This release has no migration for the recorded location, so keep the original region and prefix for as long as AWS-backed secrets exist.
- Another external adapter
- A row bound to
hashicorpis refused underaws, and the other way round. Store, read and delete all fail with a message naming the adapter that owns the row; list and the backend module still show it. - Backup
- The database no longer contains the envelopes. A restore needs the database dump, the master key and the Secrets Manager secrets; back all three up, and keep them apart (Backup and restore).
Troubleshooting
| Message | Likely cause |
|---|---|
the AWS SDK for PHP is not installed | Run composer require aws/ in the project that
selects aws. |
aws. or is not an AWS region name | Set aws. to a region such as eu-. |
AWS credential is missing | The environment variable named in the message is not visible to the
PHP process. The SDK's own AWS_* variables are not read. |
endpoint must use https | The endpoint is http. Use https, or set
aws. on a development machine. |
could not be sent with a blocked-host message | The endpoint is private or loopback and not listed in
allowed_. |
was rejected with HTTP 403 | The key is unknown or revoked, the secret access key is wrong, the
session token expired, or the policy does not allow the action.
With the AWS error code in brackets (for example
Access) the cause is in your IAM policy. |
was rejected with HTTP 400 or
(Invalid | The name is taken, or the secret is scheduled for deletion by something other than nr-vault. nr-vault always deletes without a recovery window. |
already exists under the name nr- | Another user of the AWS account owns that name. Choose another
aws. or remove the secret. |
holds no value for it | The AWS secret was deleted outside nr-vault, or the region or
prefix changed. Restore the secret, or write a new value with
vault:. |
is stored in the "hashicorp" adapter | The row belongs to the other external adapter. Select that adapter again to work with it. |