--- title: "Another Day, Another Lesson in Securing Your Infrastructure" source: "https://wordpress-1075849-4005834.cloudwaysapps.com/blog/another-lesson-in-securing-your-infrastructure/" --- [Technical Blog](https://wordpress-1075849-4005834.cloudwaysapps.com/category/technical-blog/) # Another Day, Another Lesson in Securing Your Infrastructure By [Christopher Liljenstolpe](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/author/liljenstolpe/) on Mar 22, 2018 • 3 min read Last week I wrote a [blog post](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/tesla-compromise-network-policy/) that was a response to an unfortunate event at Tesla.  In it, I detailed some techniques one could use to secure the infrastructure to reduce the risk of suffering a similar security ‘event.’ One of the points I made in that post was that security isn’t “just for production” as production is increasingly a “fuzzy” concept between PoC, PoC with trial customer(s), first application deployment, and so forth. Right on the heels of that blog post, security researcher [Giovanni Collazo](http://elweb.co/author/gcollazo/) published [this blog post](http://elweb.co/the-security-footgun-in-etcd/) where he details a fairly simple search for publicly accessible [etcd](http://coreos.com/etcd/) servers on the Internet.  etcd does not ship with user authentication turned on by default.  If you can find an etcd server on the network with a default configuration, you can interrogate it and potentially modify its contents. Ok, I imagine many of you are wondering “Why this is important, and what is etcd?”  This is where this gets interesting – etcd is a database that was designed to hold configuration data for other servers and services.  The folks at [CoreOS](http://coreos.com/) (soon to be part of Red Hat) initially wrote etcd to help automate the (re)-configuration of servers and services to ensure that they are kept up-to-date.  This is an admirable, security-driven initiative. It was so admirable that many projects now include etcd as part of their infrastructure, not the least of which is [Kubernetes](http://kubernetes.io/), the very popular container orchestration system that came out of Google and is now the darling of the cloud-native world. Kubernetes stores lots of “really interesting” data in etcd, as do lots of other projects and users.  In the 2000+ public etcd servers that Giovanni found, he located almost 9000 keys labeled “password” and over 650 keys labeled “aws_secret_access_key.”  I’ll pause here for a minute to allow you to go run and check your etcd servers…. If you install etcd in a default manner, you are basically raining your newly compromised keys on an appreciative, if somewhat unscrupulous, audience. Let me reiterate the point I made earlier here, and in the previous blog.  **Security is not just for production!**  If you don’t PoC what you plan to deploy, it is most likely that you will deploy what you PoC’d. A best practice is to evaluate the components that make up your infrastructure and your application environment and make sure they have rational security settings.  Enabling user authentication is good, enabling mutual TLS (mTLS) authentication for RESTful and similar services (such as etcd) is even better. With [Project Calico](https://wordpress-1075849-4005834.cloudwaysapps.com/project-calico/)’s [network policy](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/micro-segmentation-in-the-cloud-native-world-355d2591cb6d/), and specifically [host endpoints](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/securing-host-endpoints-with-calico-part-3/), as I discussed in the [previous post](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/tesla-compromise-network-policy/), you could isolate what endpoints could even connect to your etcd servers.  With [Tigera’s CNX](https://wordpress-1075849-4005834.cloudwaysapps.com/tigera-products/compare-products/), you could go further and control who within your organization can change the policies that control etcd access. Security starts with some basic fundamental hygiene, like understanding what it is that you are using, how it deploys by default, and what tools are inherent in the component that can improve your security posture.  From there you can build on those basic protections with defense in depth including network policy. Doing this at the beginning of a deployment is easier, and certainly less disruptive than trying to patch after the fact.  Unfortunately, patching after someone has pulled your “aws_secret_access_key” from your open etcd server is not only disruptive, but also ineffective. The re-keying necessary to recover from that sort of incident (assuming you detect it at all) is more disruptive still. [Best Practices](https://wordpress-1075849-4005834.cloudwaysapps.com/tags/best-practices/)[Open Source](https://wordpress-1075849-4005834.cloudwaysapps.com/tags/open-source/)[Project Calico](https://wordpress-1075849-4005834.cloudwaysapps.com/tags/project-calico/) Add Tigera as a preferred source on Google to see more of our content in your search results. [![Add as a preferred source on Google](https://wordpress-1075849-4005834.cloudwaysapps.com/app/themes/tigera2021/dist/img/google-preferred-source-light.png)](https://www.google.com/preferences/source?q=tigera.io) ## Related posts [![Calico Load Balancer: Simplifying Network Traffic Management with eBPF](https://wordpress-1075849-4005834.cloudwaysapps.com/app/uploads/2026/03/Calico-Load-Balancer-Simplifying-Network-Traffic-Management-with-eBPF.png)](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/calico-load-balancer-simplifying-network-traffic-management-with-ebpf/) #### [Calico Load Balancer: Simplifying Network Traffic Management with eBPF](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/calico-load-balancer-simplifying-network-traffic-management-with-ebpf/) By [Aadhil Abdul Majeed](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/author/aadhil-abdul-majeed/) on Mar 21, 2026 Authors: Alex O’Regan, Aadhil Abdul Majeed Ever had a load balancer become the bottleneck in an on-prem Kubernetes cluster? You are not alone. Traditional hardware load balancers add cost, create coordination overhead, and can make... [Read more](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/calico-load-balancer-simplifying-network-traffic-management-with-ebpf/) [![Lift-and-Shift VMs to Kubernetes with Calico L2 Bridge Networks](https://wordpress-1075849-4005834.cloudwaysapps.com/app/uploads/2026/03/Mar-20-Lift-and-Shift-VMs-to-Kubernetes-with-Calico-L2-Bridge-Networks.png)](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/lift-and-shift-vms-to-kubernetes-with-calico-l2-bridge-networks/) #### [Lift-and-Shift VMs to Kubernetes with Calico L2 Bridge Networks](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/lift-and-shift-vms-to-kubernetes-with-calico-l2-bridge-networks/) By [Aadhil Abdul Majeed](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/author/aadhil-abdul-majeed/) on Mar 20, 2026 On paper, lift-and-shift VM migration to Kubernetes sounds simple. Compute can be moved. Storage can be remapped. But many migration projects stall at the network boundary. VM workloads are often tied to IP addresses, network... [Read more](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/lift-and-shift-vms-to-kubernetes-with-calico-l2-bridge-networks/) [![AI Assistant for Calico: Troubleshooting at the Speed of Thought](https://wordpress-1075849-4005834.cloudwaysapps.com/app/uploads/2026/03/Blog-Mar-20-AI-Assistant-for-Calico.png)](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/ai-assistant-for-calico-troubleshooting-at-the-speed-of-thought/) #### [AI Assistant for Calico: Troubleshooting at the Speed of Thought](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/ai-assistant-for-calico-troubleshooting-at-the-speed-of-thought/) By [Veronika Smolik](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/author/veronika-smolik/) on Mar 19, 2026 Despite the wealth of data available, distilling a coherent narrative from a Kubernetes cluster remains a challenge for modern infrastructure teams. Even with powerful visualization tools like the Policy Board, Service Graph, and specialized dashboards,... [Read more](https://wordpress-1075849-4005834.cloudwaysapps.com/blog/ai-assistant-for-calico-troubleshooting-at-the-speed-of-thought/)