Logging is definitely not an unsolved problem with K8s. It's trivial to set up Fluentd to snarf all container logs into some destination, such as Graylog, ElasticSearch or a simple centralized file system location.
The Github issue talks a lot about "out of the box" setup via kube-up. If you're not using kube-up (and I wouldn't recommend using it for production setups), the problem is rather simple.
The logging story could be better, but it's not "unsolved".
Thanks for that clarification. We are trying to get our feet wet by building a small 2 node setup.
We don't want to stream our logs or setup fluent, etc. I just want to make sure my logs are captured and periodically rotated. Now, the newer docker allows me to use journald as logger (which means all logs are sent to journald on the HOST machine)... But I can't seem to figure out how to do this in k8s.
Also as an aside, for production deployment on aws.. What would you suggest. I was thinking that kube-up is what i should use.
Yes, journald would definitely be superior, conceptually, to just writing to plain files. I can't help you there, though, as I've never tried setting it up.
(As an aside, I don't understand why more Unix software doesn't consistently use syslog() — we already have a good, standard, abstract logging interface which can be implemented however you want on the host. The syslog protocol is awful, but the syslog() call should be reliable.)
As for kube-up: It's a nice way to bootstrap, but it's an opaque setup that doesn't give you full control over your boxes, or of upgrades. I believe it's idempotent, but I wouldn't trust it to my production environment. Personally, I set up K8s from scratch on AWS via Salt + Debian/Ubuntu packages from Kismatic, and I recommend this approach. I have a Github repo that I'm planning to make semi-generic and public. Email me if you're interested.
Whats the k8's limitation here? You can do it however you like. e.g. make a fluentd daemonset, deploy on both nodes have them slurp up to s3. There are tons of ways to handle the logs and k8s can either help or stay out of the way.
logging of application logs generally in the world is an unsolved problem, but I feel like k8s does better than most; fluentd is the default, and then logs are collected by whatever end system you like the best that's appropriate for your needs (e.g. GKE -> fluentd -> Cloud Logging, and I've heard of K8S -> fluentd -> ELK).
since most people use k8s in the context of GKE.. there seems to be some magic that Google has put in.
But if you read the bug, this doesnt work quite well in the wild. In fact, an entire spec is being written on it and right now the debate is whether to forward to journald and have it proxy further or reimplement a new logging format.
well, there's definitely systemd people pushing for journald, that's for sure. The first comment talks about how it doesn't fit, though. I wonder what in particular about the solution doesn't work for those people.
Take for example k8s - I just started exploring it as something we could move to. https://github.com/kubernetes/kubernetes/issues/24677 - logging of application logs is an unsolved problem.
And most of the proposals talk about creating yet another logger...rather than patching journald or whatever exists out there.
For those of you who are running k8s in production - how are you doing logging ? does everyone roll their own ?