Thanks, Karsten, for writing up what we discussed recently with
OpenTelemetry! Operate First can provide valuable information to the
OpenTelemetry and K8s communities. Experimental tracing has been recently
added to the kubernetes APIServer and etcd. Also, instrumentation is in the
works with CRI-O and the kubelet. It will be necessary to turn on the
tracing feature in each component in order to export and collect traces
(and in OpenShift, to figure out the best way to do this). This is new to
kube and etcd only in the current release so we'd be one of the first if
not the first to try out the feature and report back. We will be able to
measure performance overhead from trace collection, and determine from
which code paths are the most useful places to export and visualize trace
One thing to note, in the apiserver and etcd, as well as upcoming in CRI-O
and kubelet, they are instrumented to export Otel trace data rather than
logs or metrics (logs and metrics are collected also, but not with OTel).
The OpenTelemetry trace APIs are stable and have been for some time,
whereas logging is in the works and metrics very recently declared stable.
Once exported to the OpenTelemetry collector, traces can then be exported
to a variety of backends, such as zipkin and jaeger.
I've been experimenting with a kubeadm cluster in the last few weeks, and
here is a view of some trace data collected from etcd, apiserver, kubelet,
and CRI-O. I am far from an expert with any of this so I look forward to
working with the community to get everything running correctly!
Senior Software Engineer | Emerging Technology