Perfessor - Continuous Profiling Without Instrumentation

Today we're introducing an open source continuous profiling utility called Perfessor!

Perfessor is a sidecar utililty that uses perf to continuously profile applications, converts them into pprof format, and ships them to a storage backend.

How it works

It uses perf to sample CPU functions of the given process, which generates a perf.data report. Using perf data converter and pprof we convert the perf data into a pprof profile. Then we can ship that profile to conprof or to any other supported storage system like Polar Signals. Perfessor has actually very little code, and it leverages these powerful utilities.

How to get started

Next we'll walk through how to setup profiling a live Kubernetes application with Perfessor.

First we'll start by creating an API token that we can use to have Perfessor ship it's profiles to Polar Signals.

Navigate to the settings page of your project, name a token, and click create. Copy that token out to be used later.

Now that we have an API token, it's time to add Perfessor as a Kubernetes sidecar. Below you'll see the relevant section of a Kubernetes manifest that adds the sidecar container.

There are a few interesting things you can do with perfessor. You can add custom labels to profiles shipped by the sidecar. Below you'll see the `--ship.label=app=perfessor-demo` which will ship profiles with the label pair of `app=perfessor-demo`

You can set the duration of the profiles using the `--perf.duration` flag. You will need to tell it the process name to take profiles by using the `--perf.pname` flag. In our example we are targeting the `api` process in this manifest.

Now from the step before we can copy in our API token into the `--ship.bearer-token` flag.

To allow perfessor to see the processes in the other container in the pod we will need the `shareProcessNamespace: true` setting in our manifest. The last thing of note in this sidecar is the `securityContext`. Because Perfessor uses the perf utility, which makes low-level kernel calls, we'll need to add the SYS_ADMIN capability and run the pod as root.

spec:
  shareProcessNamespace: true
  securityContext:
    capabilities:
      add:
        - SYS_ADMIM
    runAsUser: 0
  containers:
    - command:
        - /bin/perfessor
        - --log.level=debug
        - --ship.store=grpc.polarsignals.com:443
        - --ship.bearer-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwcm9qZWN0SWQiOiIzODI4ODdjNy02YjYxLTRiNDUtODYxNS03ZTUwOTFkMjgyMzMiLCJ3cml0ZVByb2ZpbGVzIjp0cnVlLCJqdGkiOiI4ZWE4MjM3ZC05NzNjLTRiNDUtYWE2ZC03ZGM4ZjM0YjY4ZDQiLCJpc3MiOiJodHRwczovL2FwaS5wb2xhcnNpZ25hbHMuY29tLyJ9.hIRy8K4NiTQnxC3mbkAyXeiI95pxJc-uw2TCRrOJ990
        - --ship.label=app=perfessor-demo
        - --ship.label=duration=10s
        - --perf.pname=api
        - --perf.duration=10s
      image: quay.io/conprof/perfessor:d561d3ba-linux-amd64
      name: perfessor

Once we've applied our manifest Perfessor should start taking profiles and shipping them to storage! If you're using Polar Signals you can navigate to the project and start browsing the profiles.

You can see in the video below, we find our profiles in Polar Signals, with the labels specified in our manifest, as well as the process name as a label. You can see the perf command that was used to generate the profile in the graph view.

With Perfessor you can now start profiling your applications even if they don't support pprof.

What's next?

Perfessor is being open sourced under the Conprof Github organization. We invite the community to help contribute to this product to bring continuous profiling to eveyrone.

If you'd like to try this out for yourself, sign up for Polar Signals private beta.