So for stuff like secrets management, buckets, api gateways and such, you deploy those services to k8s? And if you don’t mind, is maintaining those services cost effective? I’m asking because I’m always looking to do the trade off of money per time
Kubernetes already has simple secrets, good enough for me.
I would provision buckets with Terraform/tofu, we just use ingress so idk about API gateways.
The eye opener for me was "I can just do this in Kubernetes", which is pretty much always true (though not always right).
Kubernetes + Prometheus + Grafana (with friends), cert-manager, CSI, LB and some CNI you have something resembling what I'd use from $cloud provider.
Deploying K3s is really easy, it can definitely be a time-sink when you're learning but the knowledge transfers really well.
You also don't really need all Kubernetes features to use it, you can deploy K3s on a single VM and run your pods with hostnetworking and local path mounts, essentially turning it into a fancy docker-compose which you can grow with instead of throw out.
I value FOSS and being able to run "anywhere" with the same tools. K8s and Postgres gets me there, I haven't worked on any "web scale" projects though but I know both can scale pretty high.
