HashiCorp Vault plugins
HashiCorp Vault plugins are standalone applications that Vault server executes to enable third-party secret engines and auth methods. After their enablement during Vault server initialization, the plugins can be used as a regular auth or secrets backends. This project uses secrets plugins to retrieve dataset credentials by the running modules. The plugins retrieve the credentials from where they are stored, for example, data catalog or in kubernetes secret. Vault-plugin-secrets-kubernetes-reader plugin is an example of Vault custom secret plugin which retrieves dataset credentials stored in a kubernetes secret.
Additional secret plugins can be developed to retrieve credentials additional location. This tutorial can serve as a good starting point to learn about Vault plugin development.
Before you begin
Ensure that you have the Vault v1.9.x to execute Vault CLI commands.
The following steps are for configuring a new secret plug-in for Fybrik:
-
Enable the plugin during Vault server initialization in a specific path.
An example of that can be found in helm chart values.yaml file in the project where Vault-plugin-secrets-kubernetes-reader plugin is enabled inkubernetes-secrets
path:vault secrets enable -path=kubernetes-secrets vault-plugin-secrets-kubernetes-reader
- Add Vault policy to allow the modules to access secrets using the plugin.
Following is an example of a policy which gives permission to read secrets in Vault pathkubernetes-secrets
:vault policy write "allow-all-dataset-creds" - <<EOF path "kubernetes-secrets/*" { capabilities = ["read"] } EOF
- Have the
CatalogDatasetInfo
structure from the data catalog response contain the Vault secret path which should be used to retrieve the credentials for a given asset. When the Vault plugin is used to retrieve the credentials the parameters to the plugin should follow the plugin usage instructions. This path will later be passed on to the modules. For example, when the credentials are stored in kubernetes secret as is done in the Katalog built-in data catalog; the Vault-plugin-secrets-kubernetes-reader plugin can be used to retrieve the credentials. In this case two parameters should be passed:paysim-csv
which is the kubernetes secret name that holds the credentials andfybrik-notebook-sample
is the secret namespace, both are known to the katalog when constructing the path.
The following snippet showsCatalogDatasetInfo
structure with Vault secret path inCredentialsInfo
field.connectors.CatalogDatasetInfo{ DatasetId: fybrik-notebook-sample/paysim-csv, Details: &connectors.DatasetDetails{ Name: fybrik-notebook-sample/paysim-csv, Geo: theshire, DataStore: fybrik-notebook-sample/paysim-csv, CredentialsInfo: &connectors.CredentialsInfo{ VaultSecretPath: "/v1/kubernetes-secrets/paysim-csv?namespace=fybrik-notebook-sample" }, }, }
- Update the modules to use the Vault related values to retrieve dataset credentias during their runtime execution. The values contain
secretPath
field with the plugin path as described in the previous step. The following snippet, taken from hello-world-module values.yaml file, contains an example of such values.
vault:
# Address is Vault address
address: http://vault.fybrik-system:8200
# AuthPath is the path to auth method used to login to Vault
authPath: /v1/auth/kubernetes/login
# Role is the Vault role used for retrieving the credentials
role: module
# SecretPath is the path of the secret holding the Credentials in Vault
secretPath: /v1/kubernetes-secrets/paysim-csv?namespace=fybrik-notebook-sample