Configuration

The data can be automatically exported to as user-defined endpoint. Each export can be configured separately and if no configuration is given for an export it will be simply disabled.

📘

Multiple endpoints

In general you will have a single endpoint for all the export. However, if you use AWS, each export will most likely use a different endpoint because the hosted Prometheus does not support logs and traces.

⚠️

Best Practices - API Key

Headers are optional but in the case where you require API key we highly suggest using environment variables or argument to pass the headers

Monitoring properties

Example

{
  "monitoring": {
    "metrics": {
      "endpoint": "http://127.0.0.1/otel/v1/metrics",
      "headers": {
      	"Authentication": "Basic xxxxxx"
      }
    },
    "logs": {
      "endpoint": "http://127.0.0.1/otel/v1/logs",
      "headers": {
      	"Authentication": "Basic xxxxxx"
      }
    },
		"traces": {
      "endpoint": "http://127.0.0.1/otel/v1/traces",
      "headers": {
      	"Authentication": "Basic xxxxxx"
      }
    },
 	}
}

How to

Local Prometheus

Requirements

• Docker

Start a local instance of Prometheus using the following Docker command. You will need to create a empty YAML file named prometheus.yml in the same directory from where you run the command.

docker run -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.enable-otlp-receiver

Once Prometheus is running, set the Paymaster monitoring as follow

{
  "monitoring": {
    "metrics": {
      "endpoint": "http://localhost:9090/api/v1/otlp",
    },
	}
}

Once your configuration is set, start the Paymaster service and open the Prometheus frontend by accessing the following URL http://localhost:9090. If everything is set properly you should be able to see your metrics.

📘

Export Frequency

Metrics are automatically exported every 60 seconds so if you don't see them immediately, try to wait at least 1-2 minutes

AWS using OTEL Collector

On AWS you need to use different endpoint to collect the different data. Additionally, managing Amazon authentication natively can be very cumbersome and we didn't build the support inside the Paymaster for it. The reason is very simple, in general people use a special side-car that they deploy along with their containers to export the monitoring data.

This side-car is running the Opentelemetry Collector (or OTEL collector) which is a parametrizable exporter maintained by several key-player of the industry. Basically, customers define pipelines which are composed or a receiver, a processor and an exporter, making the process of exporting data to different platforms extremely easy.

In our case, we can use the OTEL collector to define a pipeline that receives OTEL data from the Paymaster and send them to the relevant AWS endpoint while managing the authentication for us.

Inside the Paymaster repository, we provide an example of a configuration that we use in our own production environment. We let you refer to the original documentation if you want to understand exactly how the configuration works.

extensions:
  health_check:
  sigv4auth:
    region: ${env:AWS_REGION}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch/metrics:
    timeout: 5s
  batch/traces:
    timeout: 5s
  batch/logs:
    timeout: 5s

exporters:
  prometheusremotewrite:
    endpoint: ${env:AWS_PROMETHEUS_ENDPOINT}
    namespace: ${env:AWS_METRIC_NAMESPACE}
    auth:
      authenticator: sigv4auth
    resource_to_telemetry_conversion:
      enabled: true

  otlphttp/logs:
    compression: gzip
    logs_endpoint: "https://logs.${env:AWS_REGION}.amazonaws.com/v1/logs"
    headers: 
      x-aws-log-group: ${env:APP_AWS_NAMESPACE}
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth
      
  otlphttp/traces:
    compression: gzip
    traces_endpoint: "https://xray.${env:AWS_REGION}.amazonaws.com/v1/traces"
    auth:
      authenticator: sigv4auth

service:
  extensions: [health_check, sigv4auth]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch/logs]
      exporters: [otlphttp/logs]
    traces:
      receivers: [otlp]
      processors: [batch/traces]
      exporters: [otlphttp/traces]