Prometheus Memory Issue Handling

Overview

We use Elastio for Prometheus management. Sometimes, the Prometheus instance may run out of memory.
When this happens:

• Prometheus stops working.
• A batch of alerts is sent to Slack.

To restore Prometheus, you need to clean up memory on the instance.

How to Validate the Issue

1. Navigate to Elastio → Services.
2. Choose the Prometheus instance
   (currently dev-prometheus, used for production).
3. Open the Logs tab.
   • If you see Out of memory messages, the instance is affected.

Steps to Clean Up Memory

1. On the Prometheus service page, click Open Terminal (in the header).
   A terminal window will open in a new page.

2. Check memory usage:

   df -h
   
   • If most of the memory is taken by Docker, proceed to the next step.

3. Clean up unused Docker volumes:

    docker volume prune
   
   • This command will remove all unused Docker volumes, freeing up memory.

4. After this cleanup, Prometheus should return to normal operation.