loudwhisper

When they need, they'll learn.

100% agree. But. If you are a principal engineer claiming to have experience hardening the thing, you would expect that learning to have already happened. Also, I would be absolutely fine with "I never had a chance to dig into this specifically, I just know it at a high level" answer. Why coming up with bs?

Maybe those engineers were like that too.

I mean, we are talking about people whose whole career was around Kubernetes, so I don't think so?

I partially agree, but not only we are looking for experts of that thing, we are also looking for security experts, and security knowledge is very much meta-knowledge. A software developer might not care at all about - say - how the CI/CD works, because all they care is that the thing builds the code. A security expert generally has a broader scope, and their job is not functional, which means their job is exactly understanding the thing to be able to model the risks around it. So they might not care of all the tools used in that CI/CD or the exact details of the steps, but they should understand the execution flow, the way third party dependencies are pulled, verified, consumed, the authorization model etc.

There is no such thing of security professional who doesn't understand - at least from an academic point of view - the overall setup of a thing they worked with.

If I take the image attestation example I made in the post, I consider the "inner workings" to be the cryptographic details, such as ciphers and their working mechanisms, or the exact details of the way that attestation can be verified offline, or what exactly is computed and how. I am OK with someone not knowing this. But not understanding the whole flow? Well, without this what's left? Copying the 3 lines of code that do something from the Github documentation? Any software engineer can very much do that, what is your contribution as a security specialist?

…..people lie on CVs and cover letters. If your ad has buzzwords and technology X, Y, and Z

Totally agree. It is very likely, although the more people I interview, the more I think that they are not lying from their perspective. It's that people can legitimately make a career today by stitching together stuff with scotch tape, spending years by just by doing that and effectively have little to show for those years. But from their perspective, they might be experienced in that stuff, maybe?

I wouldn't say it's a large expansion of skillset, meaning it's not massive. But yes, indeed it is problematic to find people. It is because this is a vicious circle in which companies are digging their own graves by eliminating a market for those people, which in turn means that those who would want to hire some can't find them easily, leading to outsourcing instead. Do this for 15 years across the whole industry and it stops being an option, which is pretty much where we are today. That said, training and upskilling is always a possibility for companies who invest on their own employees and are playing the long game...

Well, for the relatively small sample of Kubernetes experts I interviewed, basically any topic beyond "you use this tool" was a disaster, including Kubernetes knowledge. I am not selective, it's not like I expect a specific skillset, but what would you think if someone with a decade of platform security doesn't understand cryptography and supply chain, Linux permissions, Kubernetes foundational concepts, container isolation or networking? At some point the question is legitimate, what are you expert in? The answer I have been able to give myself so far is "stitching together services that do stuff" and "recommend what the documentation/standard recommends". I consider myself satisfied to have somewhat decent knowledge in some of those areas, I am not expecting someone understanding all of that, but none of them? Maybe from someone who just joined the industry.

You say "incompetent" and "less skilled" as general statements on senior engineers. Those statements are false.

I am saying that the competencies of people who grew up (professionally) with outsourced services are more superficial and give them way less understanding (and agency) on the systems they oversee. I make the opinionated argument that knowing which service to use in a cloud provider is not just a different skill from implementing that functionality "manually", but is hierarchical inferior, easier to acquire and less useful in general.

A weird parallel would be someone who hikes 100% of the time with a guide who takes care of orientation, camp setting etc., and someone who goes alone. If I am simply comparing the pictures they are showing me, I might not appreciate the difference, but if you asked me who I would trust to come hiking with me, I wouldn't have doubt, because I consider the skill "finding, choosing and listening to the guide" to be hierarchial inferior to "orient, set camp etc. by yourself".

So it's not just a matter of matching the skills I need, is actually a much broader argument about deskilling engineers.

It depends. An EKS cluster can cost easily 20x what an equivalent cluster costs with same resources. The amount of people necessary to manage it is very close compared to a bare cluster, which depending on the scale can save hundreds of thousands or millions per year, therefore allowing extra headcount.

For example, a company I worked for had a team of 6 managing all their kubernetes cluster on rented dediservers. The infra costed around 50k/year. The same clusters on EKS could be managed by 4 people (maybe?), but would have costed easily 5-600k, especially since they were beefy machines, possibly even more. That amount of money would pay for 7-8 additional headcount in local hires.

Considering that in those clusters there were 40-50 postgres clusters, if moving those to RDS they would have probably looked at millions in cloud bills per year, and the effort to run those dB's once the manifests were developed was negligible (same team was managing them). This was a tiny startup, with limited resources for internal tools and automation development.

So it's not like managing everything can save headcount, it's that not outsourcing everything can save so much money that largely compensates for more headcount, plus you are giving money to real people, who spend local and pay taxes.

But you know what the kernel is. You know that syscalls are a thing, you know what role the kernel performs, you know that different filesystems have different properties (and pros and cons), etc..

You don't need to know the details, perhaps, but you can't ignore the fundamental theoretical concepts of kernel and OS. You might not know the whole detail of the boot procedure, but if your machines are stuck on boot, you know at least what to look for.

Here I was talking about equally foundational topics. There is nothing "above" - say - producing attestations and then verifying them. That's literally all there is to it, but if you don't understand the theory behind it, what exactly are you doing? As as I said, I don't care about the details, I didn't expect someone mentioning ciphers or timestamp authorities, transparency logs etc. All I would expect is "we produce a signature with a bunch of metadata and we verify it where we consume the artifact, so that we are sure that the artifact has the properties attested by the signature".

Not knowing this is like someone claiming that they administer Linux machines but can't explain what network interfaces are or how routing is determined. This is not a question of being expert on different layers, this is just being oblivious to those other layers completely.

A cloud VM, just shut it down and you're done.

If this flexibility is needed, and it's an "if", a dedicated server does the same. But even a cloudVM is already lower level compared to other services (which are even more abstract) - like EKS, SQS, etc.

The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.

In my experience this often translates in values that flows to AWS, while the company giving value to customers is stuck with millions of cloud bills each month, and a large engineering footprint that eventually needs to cut, leaving fewer and fewer people working on the product.

That said, I acknowledge that cloud has business reasons to exist, I wrote an entire other post about my hate for it, but I still acknowledge that. However there are some myths that finally are getting dispelled (outsource infra and focus on your product).

I mean, the person in question had "hardening EKS" on their CV. EKS still means that the whole data plane is your responsibility. How can you harden a cluster without understanding the foundation of container security (isolation primitives, capabilities, etc.)? Workload security is very much part of the job.

I mean the moment some pod will need to run with some privilege (say, a log forwarder which gets host logs), and you need to "harden" the cluster, what do you do if you don't understand the concept of capabilities? I will tell you what, because I asked this very question, and the answer was "copy the logs elsewhere", which is the "make it work with the hammer solution" that again shows the damage of not understanding.

I am with you about different scopes, skillsets etc. But here we were interviewing people with a completely matching skillset on paper.

Thanks, indeed I think there are many parallels with other areas. I will check it out.

I totally agree with you, but I don't think this is the specific case. Most of the rejections in our case (which I can see) on the preliminary screening were based on lacking CV skills. Which is stupid in its own way, but at least makes sense assuming we are looking for those skills specifically.

For the rest, the company is a remote company paying good salaries for the European market, I would say slightly above market average in many metrics.

I will sift more into the rejections, but from what I have seen, almost all those who had the screening phone call made it to the interview (I.e., rejections were mostly cv-based).

But those are absolutely not the only 2 levels. Server rental can be managed easily by the same infra team who manages the cloud, for a fraction of cost.

I will say more, the same exact team that spends time managing EKS clusters could manage self-managed clusters and have money to spare for additional hires.