Miguel Carneiro

How I manage the infrastructure of skeeled alone

2025-12-05T00:00:00Z

How I manage the infrastructure of skeeled alone? Plot twist: I'm not really alone!

Over the past 5 years, I built and managed what is today skeeled's infrastructure. I worked alone, but with the trust and guidance of Rui, my manager, who knows infrastructure well and trusted me completely with this task. All while staying confident in our stack and sleeping well at night.

This was accomplished through two main approaches:

Keeping it simple with standards and "boring" technology
Standing on the shoulders of giants (mentors and open source)

Keeping it simple

We follow industry standards and use well-tested boring technology. Our infrastructure is coded simply and clearly.

This approach let us migrate environments, join clusters, and adapt when tools like Rancher Fleet added features we needed. We could revert changes and try again seamlessly.

Everything declared as code gives us an always-updated recipe for our infrastructure. This is like having the blueprint of the Disaster Recovery Plan (DRP) defined as code.

When problems arise, we move quickly because we have a solid foundation, not a castle of sand. We have proper disaster recovery and data backups.

Standing on the shoulders of giants

I stand and work on the shoulders of giants: mentors and the open source community.

Mentors

Having mentors accelerates your career and knowledge. When I joined Farfetch, my team ~~forced~~ inspired me to read two technical books:

The second book is special and biased because it was written by two team members who became important mentors.

My daily routine started with reading, taking notes, doing exercises, and asking questions. They were kind yet demanding in their explanations and expectations.

Before that, during my interview, they recommended The Phoenix Project, which I bought and read before starting.

They taught me to read open source code and understand how it works. When issues appear, check the source code—you might find a bug and contribute a fix.

When I joined skeeled knowing little about Kubernetes, the team again provided courses, books, and most importantly, time to properly learn.

Even after changing companies, I contact mentors for implementation advice. I pay it forward by helping friends or even strangers that ask for help. These helping moments matter more than all the solo coding hours. I still remember the feeling of helping the wife of a friend that works at TripAdvisor with some Git issue, or helping a chilhood friend with his company GitHub Actions.

There's joy in getting help and learning from others, and even more joy in teaching and helping others.

Open source community

My job would be impossible without volunteer developers making the world better, usually for free. The world runs on software written by developers who provide it freely with source code we can learn from and improve.

This creates both a learning source and high-quality industry standard tools.

Our infrastructure is setup and runs almost exclusively on open source tools: Kubernetes, Helm, Prometheus, Grafana, Thanos, OpenTelemetry, SigNoz, Rancher Fleet, and Terraform^[1].

Rather than diving into technical details, I'll focus on the important decisions these tools enabled.

Kubernetes is complex, but it's learnable complexity. Once you understand it, it saves significant work and provides sensible defaults.

Running Prometheus and Grafana in each cluster is essential (thanks to mentor advice). You can have all the global monitoring you want, but when network issues hit, you'll thank yourself for having local tools to understand the problem.

Trust me, it's always DNS.

Source: https://www.auvik.com/franklyit/blog/securing-your-dns/

Thanos provides global metric storage, letting us compare application behavior across years. This helped us understand strange behavior from a deployment months earlier.

We use Dead Man's Snitch^[2] for watchdog alerts. Prometheus should be silent to avoid interrupting work, but you need to know your monitoring works. A watchdog alert provides peace of mind.

These industry standards gave us flexibility to:

Migrate from Datadog to SigNoz in days using OpenTelemetry (saving money)
Move Thanos data from AWS S3 to OVH Object Storage (saving money)
Migrate clusters from Scaleway to OVH easily (saving money)
Switch Kafka from Confluent to OVH (saving a lot of money, reduced costs to 20% of original bill)
Migrate from Mongo Atlas
Move freely between cloud providers (AWS → OVH → Scaleway → Azure → GCP)
Migrate regions after a datacenter fire
Recover from a drop database in production.
And more...

Key takeaways

You're never truly alone
Keep it simple—use boring, well-tested technologies and standards
Get a mentor and mentor others
Read technical books and manuals (RTFM)
Invest in education; managers should provide learning time and resources
Use and contribute to open source (start small even with documentation typos or examples)
Define everything as code, including disaster recovery and backups
Choose options that avoid vendor lock-in and enable movement
Accept challenges even when you're not ready yet and do your best to learn

Terraform now uses Business Source License rather than being fully open source. ↩︎
Dead Man's Snitch is not open source. Let me know if you know a good open alternatives. ↩︎

Cloudflare outage and infrastructure as code

2025-11-20T00:00:00Z

Last Tuesday (Nov 18, 2025), 20% of the internet was "down" due to some proxy issue on Cloudflare.

skeeled was affected by this. João (our Quality Control Analyst) asked me if I broke our non-production environment (again). Soon after, we received a message from Eric (our Head of Customer Success) that some clients were experiencing issues.

Quickly we noticed something that had never happened to us, Cloudflare was down! The Cloudflare dashboard was also down, but we disabled the proxy feature in our config as code and the API was working, mitigating the issue.

Within minutes, our clients were able to use our application without Cloudflare as a middleman, which increased our load a bit. Sorry for not giving our clients some slack!

Things like these happen to all companies: Cloudflare, AWS, GCP, OVH, Scaleway... How it's handled and communicated shows their true value.

Nix - using multiple channels in a shell.nix file

2021-09-07T00:00:00Z

This post is a public archival of a self-contained shell.nix file to use multiple channels of Nix. In this case the default and unstable channels.

Template:

# shell.nix
let
  unstable = import (fetchTarball https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) { };
in
{ nixpkgs ? import <nixpkgs> {} }:
with nixpkgs; mkShell {
  buildInputs = [
    packageFromTheDefaultChannelHere
    unstable.packageFromTheUnstableChannelHere
  ];
}

E.g. I needed kubectl 1.21.1 (from 21.05) and Terraform 1.0.6 (from unstable):

let
  unstable = import (fetchTarball https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) { };
in
{ nixpkgs ? import <nixpkgs> {} }:
with nixpkgs; mkShell {
  buildInputs = [
    kubectl
    unstable.terraform
  ];
  shellHook = ''
    export KUBECONFIG="`echo $(pwd)`/kubeconfig"
  '';
}

The shellHook is an extra to have kubectl ready using the kubeconfig file that is generated by the Terraform code.

Result:

[nix-shell:~/tmp/nix-example]$ which terraform
/nix/store/ya9w0sdja8d3d7vkjv2cbj6qjb70gdzs-terraform-1.0.6/bin/terraform

[nix-shell:~/tmp/nix-example]$ terraform version
Terraform v1.0.6
on linux_amd64

[nix-shell:~/tmp/nix-example]$ which kubectl
/nix/store/hjvsr2zgiz11n7dc3g7yp04qcxad9jrd-kubectl-1.21.1/bin/kubectl

[nix-shell:~/tmp/nix-example]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Thank you Rui (our Head of Engineering at skeeled) for sharing this with me.

P.S. This page could be a TIL (Today I Learned) bit in a category like this one https://mrkaran.dev/tils/. When I do mine I will probably move it there.

Originally published at https://mig4ng.hashnode.dev/nix-using-multiple-channels-in-a-shellnix-file.

Hello World

2019-05-09T00:00:00Z

Hello, world! My name is Miguel Carneiro but you can call me Mig. I am also known as mig4ng. More about me here.

This is my very first and awkward hello world article. Breaking the ice is never easy and this article's purpose is to break it. Therefore, I will try to make it as short and embarrassing as possible so all articles that come next should feel easier to publish than this one.

You are here either because you know me personally or because you read something I wrote and wanted to find out if there is more. Either way, I think we have wasted enough of our time.

The only lesson I have to give in this article is that you start. Whatever it is that you are trying to do (a blog, a company, exercising, meditating, reading, etc.) just do it! Nike knew it all along. Baby steps, break your goal into smaller and more easily achievable goals and pick the first one on the list. Make the first item on the list very easy and achievable (e.g. read 5 minutes, write 100 words or 10 push-ups). Upon starting, it is way easier to keep going.

Now I have no alternative than to follow my own advice and go publish my very first article.

Originally published at https://mig4ng.com/articles/hello-world/.