Create throughput same as APM throughput

[walterandriesse]

Hello,

Maybe somebody can help me?

I have different values in throughput.

I have created een throughput overview in a dashboard.
I have tried to make it the same as the APM throughput.
The same variabeles.
I also used all the data as suggested in Display latency, throughput of APM instrumented apps in custom Kibana dashboard - Elastic Stack / Kibana - Discuss the Elastic Stack.

Everything is the same, but still mine data is approximately half.

I use version 8.18.
Does somebody perhaps knows why i have this difference.

thanks
Walter Andriesse

[stephenb (Stephen Brown)]

Hi @walterandriesse Welcome to the community.

Yes that post used the 7.X schema so that is probably why it did not work as the 8.x underlying schema / data model is a bit different. In addition perhaps you are using OTEL?

Perhaps if you Provide a bit more information we can help.

I see you are on 8.18 stack

Are you using the Elastic APM Agent or OTEL Agent...
Which One, Which Version? Which Language?
Are you using sampling? If so what is the value?

[walterandriesse]

Hello Stephen,
Thanks for the welcom, and for your answer.

I will try to give some more info.
I use Elastic and Observability since a few months and i want to create a customized dashboard.
One of the parts I want to include is the Throughput of a service.
This is allready working in de APM. (standard feature i guess)
So i actually just want to copy/paste this data part.
I included this line : service.name: "......" and processor.event: "transaction".
The data is half of the value of the troughput at the APM at the same service.

(I'am not using OTEL Agent.) I'am using APM Server binary.
It also states : This is being replaced by a managed APM Server in Elastic Agent from version 8.0.
The Language is English.
And I don't think i'am using sampling.

I Hope this addition will help?

Walter

[stephenb (Stephen Brown)]

Thanks for the details

@walterandriesse

(I'am not using OTEL Agent.) I'am using APM Server binary.

APM Agent and APM Server are 2 different things but I will assume you are using the Elastic APM agents

@walterandriesse

The Language is English.

Apologies I meant Client Language, example Java, .NET Python etc

@walterandriesse

So i actually just want to copy/paste this data part.
I included this line : service.name: "......" and processor.event: "transaction".
The data is half of the value of the troughput at the APM at the same service.

You are going to have to show me exactly what you are doing and how you are trying to create

I went to Lens and I can duplicate the TPM

This assumes no Sampling

Important Normalize to Per Minute << THIS is probably what you are missing

Break down by Service name

I get pretty much the same visualization

[walterandriesse]

Hello Stephen,
Thanks for your help.
I have created everything the same as you did.
It was very helpfull for me to see that I created the right settings.

Anyway I still got the differences.
I think that the sampling maybe the cause of my difference.
I am going to look at this tomorrow.

Thanks again for you help,
Walter Andriesse

[stephenb (Stephen Brown)]

@walterandriesse

Anyway I still got the differences.
I think that the sampling maybe the cause of my difference.
I am going to look at this tomorrow.

If you are using sampling you will need to use a different approach you have to use the summary metrics...

Metrics | Elastic Docs

I think this is it. I might need to check a little more

EDIT : CORRECTED

Lens
Dataview: APM data View
KQL Bar : metricset.name : "service_transaction" and agent.name : * << CORRECTED
X Axis : Date Histogram on @timestamp
Y Axis : count. <<< CORRECTED...
Breakdown : Top Values service.name

[walterandriesse]

Hello Stephen,

Your help is more then welcome.

About the settings you asked:

Our Elastic apm-agents have the ELASTIC_APM_TRANSACTION_SAMPLE_RATE set to the value: '0.1'
Using the metricset.name : service_summary as a filter gives the following result (picture, last orange screenshot).
When I look at the instruction about and the documents with the metricset.name : service_summary I do not see direct clues on how to duplicate the existing throughput graph available in the APM app.

i tryed to embed screenshots. But i'am not allowed to sent uploads
Perhaps you can activate my permission to share data.

hopefully this helps you to understand the differences,

Walter

[stephenb (Stephen Brown)]

You should be able to add an image.

Apologies.... I Realized I had an issue / mistake above there is some "magic" here

So go to Lens

Try this.....

Lens
Dataview: APM data View
KQL Bar : metricset.name : "service_transaction" << CORRECTED
X Axis : Date Histogram on @timestamp minimum interval : 1m <<<< VERY IMPORTANT
Y Axis : count. <<< CORRECTED...
Normalize by 1 minute. << IMPORTANT
Breakdown : Top Values service.name

The magic is if you go to Discover and

KQL Bar : metricset.name : "service_transaction"

If you look at a doc you will see this... this is how the metrics work ....
This can be a little confusing there is only actually 1 document but it represents aggregate count of 6 ...

[walterandriesse]

Hello Stephen,

White smoke.
I did it.(kidding) ----> You did it.
Your help was super clear and specific.
The screenshots helped me perfect.

I have a 100% match with the APM.

.

Thanks and have a nice day.

With friendly greetings,
Walter Andriesse