10:08 a.m.
Hello all Op1st,
Let me open this thread by welcoming Sally O'Malley, who is interested in
bringing OpenTelemetry to the Operate First community cloud.
Welcome, Sally! Great to have you here. This forum/mailing list is the new
central communication hub for community discussions, as the open access and
archives counterpart to the discussions happening around artifacts in the
git repositories and in unstructured chat
Speaking of which, here is the issue I opened on the Community Board
<https://github.com/orgs/operate-first/projects/16> to help keep track of
all the steps and movement involved:
https://github.com/operate-first/community/issues/83
I haven't been involved in onboarding a project yet, so I'm excited to work
with Sally on my first one. Sally and I have mostly worked together around
DevConf.US, where Sally is event Co-Chair and a great leader to follow. I'm
confident inviting her to jump right in and start engaging with everyone,
and I'll follow around and keep notes.
For simplicity of keeping all aware of the conversation, let's use this
thread and start discussing what and how to bring OpenTelemetry
experimental features to Operate First users, and how to begin delivering
Op1st environment value back to OTel as an environment user/contributor.
I'll reply to this message to take a part of the thread into the "what does
this look like for everyone" discussion.
Kind regards,
- Karsten
--
Karsten Wade [he/him/his] | Senior Community Architect | @quaid
Red Hat Open Source Program Office (OSPO) : @redhatopen
The Open Source Way : https://theopensourceway.org
Operate First : https://operate-first.cloud
10:38 a.m.
For simplicity of keeping all aware of the conversation, let's
use this
thread and start discussing what and how to bring OpenTelemetry
experimental features to Operate First users, and how to begin delivering
Op1st environment value back to OTel as an environment user/contributor.
I'll reply to this message to take a part of the thread into the "what does
this look like for everyone" discussion.
Sally should hop in anytime and correct or expand on anything here, most of which came out
of our discussion.
From the tracking issue https://github.com/operate-first/community/issues/83:
Vision includes:
- The project will run latest upstream code with experimental features enabled.
- Open source devs can tap into the experimental features, e.g. OpenTelemetry Tracing
enabled etcd, then use the API server to tie that to their own code so they can get an
end-to-end trace for the time of a request (for example.)
- The Op1st community cloud SRE team also benefits from having the latest in metrics,
logs, and tracing capabilities. OTel Collector is able to take exported data and export to
different platforms, such as Prometheus.
- Roadmap should include having a way to use GitOps to provide granular control over which
experimental feature is enabled for which environment. Then people can test against a
subset or all of the experimental features.
- What can we expose to SRE/Data Scientist users? Can we let them have some level of
access to what OTel is providing to services running on Op1st CC?
- What value is the OpenTelemetry project able to get?
- Demo environment so show people how it works in a running Kubernetes environment.
- Whatever operational data it can collect with the various workloads it tracks in the
project
4:14 p.m.
Hello,
Thanks, Karsten, for writing up what we discussed recently with
OpenTelemetry! Operate First can provide valuable information to the
OpenTelemetry and K8s communities. Experimental tracing has been recently
added to the kubernetes APIServer and etcd. Also, instrumentation is in the
works with CRI-O and the kubelet. It will be necessary to turn on the
tracing feature in each component in order to export and collect traces
(and in OpenShift, to figure out the best way to do this). This is new to
kube and etcd only in the current release so we'd be one of the first if
not the first to try out the feature and report back. We will be able to
measure performance overhead from trace collection, and determine from
which code paths are the most useful places to export and visualize trace
information.
One thing to note, in the apiserver and etcd, as well as upcoming in CRI-O
and kubelet, they are instrumented to export Otel trace data rather than
logs or metrics (logs and metrics are collected also, but not with OTel).
The OpenTelemetry trace APIs are stable and have been for some time,
whereas logging is in the works and metrics very recently declared stable.
Once exported to the OpenTelemetry collector, traces can then be exported
to a variety of backends, such as zipkin and jaeger.
I've been experimenting with a kubeadm cluster in the last few weeks, and
here is a view of some trace data collected from etcd, apiserver, kubelet,
and CRI-O. I am far from an expert with any of this so I look forward to
working with the community to get everything running correctly!
Talk soon,
Sally
--
Sally O'Malley
Senior Software Engineer | Emerging Technology
<https://www.redhat.com/>
942
days inactive
944
days old
community@lists.operate-first.cloud
2 comments
3 participants
participants (3)
-
Karsten Wade -
kwade＠redhat.com
-
Sally O'Malley