Grafana Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.

Compared to other log aggregation systems, Loki:

• Indexes a set of labels for each log stream, instead of indexing the context of logs.
• Indexes and groups log streams using the same labels as Prometheus.
• Automatically scrapes and indexes Kubernetes](https://kubernetes.io/) metadata, such as pod logs.
• Has native support in Grafana v6.0.

A Loki-based logging stack consists of 3 components:

• Promtail is the agent, responsible for gathering logs and sending them to Loki.
• Loki is the main server, responsible for storing logs and processing queries.
• Grafana is platform, responsible for querying and displaying the logs.

Notes:

• This stack requires a minimum configuration of two nodes at the $10/month plan (2GB memory / 1 vCPU).
• The Loki stack 1-Click App also includes a $1/month block storage for Loki time series database (starting at PVs of 5GB).

Software Included

Creating an App using the Control Panel

Click the Deploy to DigitalOcean button to install a Kubernetes 1-Click Application. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

Creating an App using the API

In addition to creating Grafana Loki using the control panel, you can also use the DigitalOcean API. As an example, to create a 3 node DigitalOcean Kubernetes cluster made up of Basic Droplets in the SFO2 region, you can use the following doctl command. You need to authenticate with doctl with your API access token) and replace the $CLUSTER_NAME variable with the chosen name for your cluster in the command below.

doctl kubernetes clusters create --size s-4vcpu-8gb $CLUSTER_NAME --1-clicks loki

Getting Started After Deploying Grafana Loki

Getting Started with DigitalOcean Kubernetes

You can connect to your DigitalOcean Kubernetes cluster by following our how-to guide.

For additional instructions on configuring a DigitalOcean Kubernetes cluster, see the following guides:

• How to Set Up a DigitalOcean Managed Kubernetes Cluster (DOKS)
• How to Set up DigitalOcean Container Registry

Confirming that Loki Stack is Running

First, verify that the Helm installation was successful by running following command:

helm ls -n loki-stack

If the installation was successful, the STATUS column value in the output reads deployed:

NAME NAMESPACE  REVISION UPDATED                              STATUS   CHART            APP VERSION
loki loki-stack 1        2022-02-16 14:47:29.497728 +0200 EET deployed loki-stack-2.5.1 v2.1.0

Next, verify that the Loki pods are up and running with the following command:

kubectl get pods -n loki-stack

If they’re running, all pods listed in the output are in a READY state and the STATUS for each reads Running:

NAME                           READY   STATUS    RESTARTS   AGE
loki-0                         1/1     Running   0          20h
loki-promtail-kvjxr            1/1     Running   0          20h
loki-promtail-nc7zg            1/1     Running   0          20h
loki-promtail-strvq            1/1     Running   0          20h

Configuring Grafana with Loki

First, expose the Grafana web interface on your local machine:

Note:

Grafana isn’t installed by default when the Loki Stack 1-Click App is installed and needs to be installed. We recommend installing the Kubernetes Monitoring Stack 1-Click App, which includes Grafana and its monitoring components.

To access the Grafana Web Panel, run the following command using the default credentials admin/prom-operator:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n kube-prometheus-stack

Navigate to http://localhost:80/ and login with admin and the password (default credentials: admin/prom-operator). Then, follow the instructions for adding the Loki datasource, using the URL http://loki.loki-stack:3100.

For more details about deployment status and functionality, see the Loki tutorial.

Tweaking Helm Values

The loki-stack has custom default Helm values. See the values file from the main GitHub repository.

To inspect the stack’s current values, run the following command:

helm show values grafana/loki-stack --version 2.5.1

To change these values, open the Helm values file values.yml, change whatever values you want, save and exit the file, and apply the changes by running helm upgrade command:

helm upgrade loki grafana/loki-stack --version 2.5.1 \
  --namespace loki-stack \
  --values values.yml

Upgrading the Loki Stack Chart

YYou can check what versions are available to upgrade by navigating to the loki-stack official releases page from GitHub. Alternatively, you can use ArtifactHUB.

To upgrade the stack to a newer version, run the following command, replacing the < > placeholders with their corresponding information:

helm upgrade loki grafana/loki-stack \
  --version <KUBE_Loki_STACK_NEW_VERSION> \
  --namespace loki-stack \
  --values <YOUR_HELM_VALUES_FILE>

See helm upgrade for command documentation.

Uninstalling

To uninstall Loki, you need to have Helm 3 installed. Once installed, run the following uninstall command:

helm uninstall loki -n loki-stack

And then the following delete command:

kubectl delete ns loki-stack