# WellD challenge report

## What I did

- **Analyzed the existing Spring Boot/Thymeleaf order management application** for missing metrics.
- **Added and exposed additional custom metrics** using Micrometer and Spring Boot Actuator, making them available at `/actuator/prometheus` for Prometheus scraping.
- **Implemented the following custom metrics:**
  - `orders_deleted_total`: Total orders deleted.
  - `orders_created_per_product_total` (with `product` tag): Orders created per product.
  - `order_quantity_average`: Distribution summary for order quantities (enables average, max, count, sum such as `order_quantity_average_sum` and `order_quantity_average_total`).
 - `log_events_total` (with `level` tag): Counts of INFO and ERROR log events.
- **Ensured JVM and HTTP metrics are available** (latency, memory, CPU, threads, GC, etc.) via Actuator.
- **Created a Grafana dashboard** (see `grafana/monitoring/`) with panels for all key metrics and business KPIs.

---

## How to use the dashboard

1. Build the source code with `mvn clean package`

2. Start the stack: `docker compose up -d`

3. Access Prometheus at http://localhost:9090

4. Access Grafana at http://localhost:3000 (default login: `admin:admin`)

5. Import the JSON dashboard provided in this repository under `grafana/monitoring/`

6. Interact with the application at http://localhost:8080/web/orders , metrics will update in realtime on the dashboard.


---

## Which panels were created and their purpose

- **Total Orders Created:** 
  Visualizes the cumulative number of orders created (`orders_created_total`).
- **Total Orders Deleted:** 
  Shows the number of orders deleted (`orders_deleted_total`).
- **Orders per Product:** 
  Bar chart/table using `orders_created_per_product_total{product="..."}` for business insight.
- **Average Quantity per Order:** 
  Displays the average order quantity using `order_quantity_average_sum / order_quantity_average_count`.
- **HTTP Request Latency (p95, p99):** 
  Shows high-percentile request durations per endpoint using:
  `histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, uri))`
  and
  `histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, uri))`
- **JVM Memory & CPU Usage:** 
  Monitors resource usage with `jvm_memory_used_bytes`, `process_cpu_usage`, etc.
- **Log Event Counters:** 
  Visualizes counts of INFO and ERROR logs (`log_events_total{level="INFO"}` and `log_events_total{level="ERROR"}`).
- **Other Stability Metrics:** 
  Panels for thread count, GC pause time, and queue size.


## Proposed alerts and thresholds

1. **High HTTP Latency**, threshold of `p95 > 1s for 5m`, to detect slow endpoints

2. **High ERROR log rate**, threshold of `>5 errors/min`, might indicate bugs or failures

3. **High JVM memory usage**, threshold of `>80% for 5m`, might help to prevent OOM errors

4. **High CPU usage**, threshold of `>80% for 5m`, might help to detect resource exhaustion

---

## Custom metrics and their purpose

- **orders_deleted_total:** Tracks deletions for auditing and anomaly detection.
- **orders_created_per_product_total (with optional `product` tag):** Enables product-level business insights and anomaly detection.
- **order_quantity_average:** Monitors average order size, useful for business KPIs and detecting outliers.
- **log_events_total (with `level` tag):** Provides visibility into application health and error rates.