At playpass we use jenkins for managing our servers. We had a simple system for secret management called json files. These files were never committed to our git repo and shared between the people that needed them. I think it's obvious that this process is quite error prone. If one person changes the secrets but forget to share his changes (read: USB stick) then the next deploy will blow up.
Enter the Vault
Recently we hired a new dev-ops guy and he looked at a better way to manage our secrets. After some experiments with different tools he picked Vault. The cool part with vault is that you can plug in different backends (AWS/Postgres/SSH). At the moment we just use the default backend.
Vault can be seen as a basic folder/file structure. You write a secret (which can contain multiple key/values) to a specific path. After that you can retrieve the secret using the path.
fritz@mac:~$ vault write secret/production/database user=test password=secret fritz@mac:~$ vault read secret/production/database
But you have to specify the exact path. Doing a
read on a path that does not contain any secrets will return nothing. Surprisingly for me the following command results in nothing:
fritz@mac:~$ vault read secret/production No value found at secret/production
I would expect that it would return 0 secrets / 1 subfolder
database. For reading folders I need to use the
list command. Funny enough a
list command will only show folders and no secrets... It's like a Finder/Explorer where you only see folders and a FinderFile/ExplorerFile where you only see files. You're constantly juggling the
list commands to get the job done.
Where the Dev steps in
I wrote a new service and teamed up with dev-ops to get it deployed. It was quite difficult since multiple services and dependencies were needed. I was sitting next to the dev-ops guy and was just horrified by the vault CLI tool. We wasted days getting the secrets in shape since the process is so error prone.
So I created my own CLI tool for easily managing vault. It uses the Ruby client for Vault and has some pretty cool commands. The CLI is made with Thor and Hirb (for table/tree printing).
It displays the given path as a tree representation, showing all folders/secrets recursively. To achieve this I have to propagate down the "folder" structure and read/list every path (since a path can only contain folders but also hold secrets). For large tree's the performance is quite bad, but nothing I can do about that yet.
fritz@mac:~$ pp_vault tree secret 📂 secret |-- 📂 edge | |-- 🔑 key:values | `-- 📂 playpass | |-- 🔑 all:buOTHi0d9tP-4XgN2MpV1-Jo88tzgFvo | `-- 🔑 all-data:curQZb16B88Pz3qNWReZ0nJ_IcbW_LXj |-- 📂 production | `-- 📂 playpass | `-- 🔑 all-data:btUzvTd308P1jBMii2xoqFVGjPdKJFD- `-- 📂 staging `-- 📂 playpass |-- 🔑 all:eJxtCk29osHMv5LSw19Pk2Mwho_c0fOv `-- 🔑 all-data:qNLg9ACdGr_o6k2_IrrayH1dGeOKPfj6
You would think this is supported by default but ... no. If you want to add a new key/value to an existing path (secret) you have to repeat all existing key/values. So you end up with first reading/listing and then copy pasting existing keys/values and then add the new key/value. Wow thanks Vault that really helps...
fritz@mac:~$ pp_vault add secret/edge key:new-value Current value: | Key | Value | |------ | --------| | key | values | | key2 | values | 2 rows in set Existing secret found, overwrite? y Successfully written to secret/edge Current value: | Key | Value | |------ | -----------| | key | new-value | | key2 | values | 2 rows in set
So I start with first displaying the original values at the path. If any of the new keys already exist in the current secret then we display a confirmation. After our write operation we display the result.
99% of the time you store a secret in Vault but there is no way to easily generate secrets. So ruby to the rescue and let's add a
generate command. It allows to easily generate a password for a given key at a specific path. We merge existing values/keys with our new key/generated value and ask for confirmation if there are conflicts in the keys.
fritz@mac:~$ pp_vault generate secret/edge/playpass key Current value: | Key | Value | |--------------- | ----------------------------------| | all | buOTHi0d9tP-4XgN2MpV1-Jo88tzgFvo | | all-data | curQZb16B88Pz3qNWReZ0nJ_IcbW_LXj | | key:new-value | tcrGYPN6Cu7PQKgdAy1LV3gHOT0_gA8- | 3 rows in set Existing secret found, overwrite? y Successfully written to secret/edge/playpass Current value: | Key | Value | |--------------- | ----------------------------------| | all | buOTHi0d9tP-4XgN2MpV1-Jo88tzgFvo | | all-data | curQZb16B88Pz3qNWReZ0nJ_IcbW_LXj | | key:new-value | NptQ7Q0y8msgCSRwKX4-Roz1e5unTumu | 3 rows in set
I also included options for generating typical password styles, for example HMAC keys (ex:
--hmac). Or if you want to change the password length a simple
--value 23 will do.
When I was creating the CLI I wondered why nobody else had the same struggles. But reading through multiple github issues people already asked for it. For example the
list command comes from the following issue after half a year it's finally implemented but is barelly usable and does not solve the original issue.
I do understand that Hashicorp wants to keep the API backend easy so a lot of different backends can easily be supported. But when you provide a CLI tool then this tool should be user friendly and not just implement the API and be done with it. Closing issues because they are outside the scope of your CRUD API is really shitty. I guess Vault isn't really meant to be used as a single tool but rather as an enterprise packed with Consul/Terraform. A good example of a CLI tool done right is Docker, it has an API backend with a CLI (also written in go) that extends the API and makes it more user friendly.
Finally, I'm not able to open source it at this time. I included specific commands for our own infrastructure and they are not usable / sensitive for anybody else. I'll update when I have something ready.