Agree the people and process side is very difficult to do well. Familiar with all those and more -- we have extremely good, dedicated employees who care deeply about doing those things right.
We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)
I have few more stories like the time when I closed the HSM rack door a bit too energetically and caused outage to entire company as we had to bring in third security officer to re-initialize it.
We also had special screens created for all cameras in the datacenter to block view on the HSM racks.
The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)
We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)
(If anyone reading is looking for a job in security engineering, we're hiring! https://www.clover.com/careers/engineering)