config/kibana.md

# Purpose
Create a small **fake application log environment** with realistic log entries, use Kibana's Discover interface to practise KQL (Kibana Query Language). By the end, you'll have:
1. A test Elasticsearch index containing application logs.
2. Different log levels: INFO, WARN, ERROR, DEBUG.
3. Different application services.
4. HTTP request and response logs.
5. User login and authentication logs.
6. Sample KQL queries to practise searching, filtering, and troubleshooting.

We'll use a fictional application called `myapp`.

Folow the following steps,

# Step 1. Create test log data

You can insert test data directly into Elasticsearch using the Kibana Dev Tools console.

Open Kibana:

`http://localhost:5601`

Navigate to:

**Management → Dev Tools → Console**

Paste the following command.

### Create test logs

```http
POST /_bulk
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:00:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User john successfully logged in","user":{"name":"john","id":"1001"},"source_ip":"192.168.1.10","event_type":"login","status":"success"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:01:00Z","level":"INFO","service":"web-service","host":"server02","message":"HTTP request received","http":{"method":"GET","status_code":200,"url":"/api/products"},"response_time":120,"source_ip":"192.168.1.10"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:02:00Z","level":"WARN","service":"payment-service","host":"server03","message":"Payment processing took longer than expected","response_time":3500,"transaction_id":"TX10001","amount":150.50,"currency":"USD"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:03:00Z","level":"ERROR","service":"payment-service","host":"server03","message":"Payment transaction failed","error":{"type":"PaymentGatewayException","message":"Connection timeout"},"transaction_id":"TX10002","amount":250.00}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:04:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User alice successfully logged in","user":{"name":"alice","id":"1002"},"source_ip":"192.168.1.20","event_type":"login","status":"success"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:05:00Z","level":"ERROR","service":"auth-service","host":"server01","message":"Failed login attempt","user":{"name":"admin","id":"0"},"source_ip":"203.0.113.50","event_type":"login","status":"failed"}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:06:00Z","level":"DEBUG","service":"web-service","host":"server02","message":"Database connection pool initialized","connection_pool_size":20}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:07:00Z","level":"WARN","service":"web-service","host":"server02","message":"High response time detected","http":{"method":"GET","status_code":504,"url":"/api/orders"},"response_time":5000}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:08:00Z","level":"INFO","service":"inventory-service","host":"server04","message":"Product inventory updated","product_id":"P1001","quantity":150}
{"index":{"_index":"logs-lab"}}
{"@timestamp":"2026-11-09T01:09:00Z","level":"ERROR","service":"inventory-service","host":"server04","message":"Database connection failed","error":{"type":"DatabaseException","message":"Connection refused"}}
```

Click **Run**.

You should receive a response indicating that 11 documents were indexed successfully.

Note: If your Elasticsearch container requires authentication, Kibana Dev Tools normally handles the connection automatically.

---

# Step 2. Create a Data View in Kibana

Elasticsearch stores the logs, but Kibana needs a Data View to display them in Discover.

1. Open Kibana.
2. Navigate to **Stack Management**.
3. Select **Data Views**.
4. Click **Create data view**.
5. Enter:

| Setting | Value |
|---|---|
| Name | My Application Logs |
| Index pattern | `logs-lab*` |
| Timestamp field | `@timestamp` |

6. Click **Save data view**.

Now navigate to:

**Analytics → Discover**

Select `My Application Logs`.

You should see your 11 test log entries.

**Important:** Your sample timestamps are in UTC. In Discover, change the time picker (top-right) to **Last 24 hours** if you don't immediately see the logs.

# Step 3. Practise KQL searches

Now comes the interesting part!

In Discover, locate the KQL search bar at the top. Try these queries one by one.

### Exercise 1: Search by log level

Find all ERROR logs:

```kql
level: "ERROR"
```

Find all INFO logs:

```kql
level: "INFO"
```

Find all WARNING logs:

```kql
level: "WARN"
```

### Exercise 2: Search by application service

Find logs generated by the payment service:

```kql
service: "payment-service"
```

Find logs generated by the authentication service:

```kql
service: "auth-service"
```

Find logs generated by either payment or inventory services:

```kql
service: ("payment-service" OR "inventory-service")
```

### Exercise 3: Combine multiple conditions

Find ERROR logs generated by the payment service:

```kql
level: "ERROR" AND service: "payment-service"
```

Find all ERROR or WARN logs:

```kql
level: ("ERROR" OR "WARN")
```

Find failed login attempts:

```kql
event_type: "login" AND status: "failed"
```

Find logs from a particular IP address:

```kql
source_ip: "203.0.113.50"
```

### Exercise 4: Search using numeric conditions

Find requests taking longer than 1 second:

```kql
response_time > 1000
```

Find requests returning HTTP 504:

```kql
http.status_code: 504
```

Find requests taking longer than 1 second and returning HTTP errors:

```kql
response_time > 1000 AND http.status_code >= 400
```

Find transactions with amounts greater than 200:

```kql
amount > 200
```

### Exercise 5: Search for specific text

Find logs containing the word `timeout`:

```kql
message: *timeout*
```

Find logs containing `database`:

```kql
message: *database*
```

Find logs containing either `failed` or `failure`:

```kql
message: (*failed* OR *failure*)
```

### Exercise 6: Search for missing fields

Find logs without a response time:

```kql
NOT response_time: *
```

Find logs containing an error field:

```kql
error.type: *
```

Find logs containing a transaction ID:

```kql
transaction_id: *
```

---

# Step 4. Practise more realistic troubleshooting scenarios

Now let's simulate some real-world scenarios that you might encounter when monitoring production systems.

| Scenario | KQL Query |
|---|---|
| Find all application errors | `level: "ERROR"` |
| Find payment failures | `service: "payment-service" AND level: "ERROR"` |
| Find failed user authentication | `event_type: "login" AND status: "failed"` |
| Find slow HTTP requests | `response_time >= 3000` |
| Find HTTP server errors | `http.status_code >= 500` |
| Find database problems | `error.type: "DatabaseException"` |
| Find logs from server03 | `host: "server03"` |
| Find all logs except DEBUG | `NOT level: "DEBUG"` |
| Find payment errors or warnings | `service: "payment-service" AND level: ("ERROR" OR "WARN")` |

These are useful patterns to understand because production troubleshooting often involves combining multiple conditions.

---

# Step 5. Practise using the Discover interface

Besides writing KQL, I recommend practising these Kibana features:

**1. Add fields to the table**

In Discover, expand a log entry and add these fields to your displayed columns:

- `@timestamp`
- `level`
- `service`
- `host`
- `message`
- `response_time`

**2. Sort logs**

Click the `@timestamp` column to sort logs by time.

**3. Filter using field values**

Expand a document and click a field value to add it as a filter.

For example, clicking `level: ERROR` will automatically create a filter.

**4. Save searches**

Save your KQL searches so you can reuse them later.

For example:
- All Application Errors
- Payment Service Errors
- Failed Login Attempts
- Slow HTTP Requests

# Step 6. Understand KQL vs Elasticsearch SQL

One important thing to remember:

**KQL is not SQL.**

KQL is primarily a filtering language. It helps you find documents matching particular conditions.

For example:

```kql
level: "ERROR" AND service: "payment-service"
```

If you want to perform SQL-like operations such as:
- GROUP BY
- COUNT
- AVG
- SUM
- ORDER BY

You should explore **ES|QL (Elasticsearch Query Language)** as well.

For example, ES|QL can perform aggregations:

```sql
FROM logs-lab
| WHERE level == "ERROR"
| STATS error_count = COUNT(*) BY service
| SORT error_count DESC
```

This counts errors grouped by application service.

---

## Advanced topics

I suggest learning Elasticsearch and Kibana in this order:

| Level | Topic | What to learn |
|---|---|---|
| 1 | Discover | Search and explore logs |
| 2 | KQL | Filtering and searching |
| 3 | ES\|QL | Aggregation and data analysis |
| 4 | Visualize | Create charts and dashboards |
| 5 | Dashboard | Build monitoring dashboards |
| 6 | Alerts | Configure alerts for ERROR logs |
| 7 | Ingestion | Use Logstash, Beats or Elastic Agent |
| 8 | Advanced | Index lifecycle management and Elasticsearch architecture |

**One particularly useful next exercise:** Create a dashboard showing:
- Total log count
- ERROR count
- ERROR count grouped by service
- HTTP response time
- Failed login attempts over time

This will give you a practical introduction to how Elasticsearch and Kibana are used in real production environments.