OpenSearch Enhanced Stack Features

🚀 New Services Added

1. OpenTelemetry Collector

Purpose: Collects traces, metrics, and logs from applications
Access: http://localhost:8080 (via Traefik)
Endpoints:
OTLP gRPC: :4317
OTLP HTTP: :4318
Jaeger gRPC: :14250
Prometheus metrics: :8889

2. Jaeger Tracing

Purpose: Distributed tracing UI and storage
Access: http://localhost:16686
Features:
Service dependency graphs
Trace timeline visualization
Performance bottleneck identification
Error rate tracking

3. Fluent Bit

Purpose: Advanced log processing and forwarding
Features:
Container log collection
Host system logs
Log parsing and filtering
Direct OpenSearch integration

🔍 OpenSearch Enhanced Features

Performance Optimizations

# Enhanced indexing performance
indices.memory.index_buffer_size: 20%
thread_pool.write.queue_size: 1000
action.auto_create_index: true

# Search performance
indices.fielddata.cache.size: 40%
indices.queries.cache.size: 20%

Index Templates Created

Application Logs (app-logs-*)
Optimized for log searching
Trace ID correlation
Service-based filtering
Jaeger Traces (jaeger-span-*)
Span and trace indexing
Performance analytics
Service topology
Metrics (metrics-*)
Time-series data
Label-based queries
Aggregation support

📊 Use Cases & Capabilities

1. Application Performance Monitoring (APM)

Trace Analysis: Follow requests across microservices
Error Tracking: Identify and debug failures
Performance Insights: Find slow operations
Service Maps: Visualize service dependencies

2. Advanced Log Analytics

Structured Logging: JSON log parsing
Log Correlation: Link logs to traces
Real-time Search: Sub-second log queries
Pattern Detection: Identify anomalies

3. Metrics & Monitoring

Custom Metrics: Application-specific data
Infrastructure Metrics: System performance
Business Metrics: KPI tracking
Alerting: Threshold-based notifications

4. Security & Compliance

Audit Trails: Track user actions
Security Events: Monitor for threats
Compliance Reporting: Regulatory requirements
Data Retention: Automated lifecycle management

🛠 Getting Started

1. Start the Stack

docker-compose up -d

2. Setup Index Templates

# Wait for OpenSearch to be ready, then run:
./opensearch/setup-indices.sh

3. Access the UIs

OpenSearch Dashboards: http://localhost:5601
Jaeger Tracing: http://localhost:16686
Grafana Monitoring: http://localhost:3000

4. Send Sample Data

Traces via OpenTelemetry

# Example: Send trace data
curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [...]
  }'

Logs via Fluent Bit

Fluent Bit automatically collects logs from: - Container logs (/var/log/containers/) - Host system logs (/var/log/)

Search in OpenSearch

# Search application logs
curl "http://localhost:9200/app-logs-*/_search?q=level:ERROR"

# Search traces
curl "http://localhost:9200/jaeger-span-*/_search?q=operationName:api-call"

🔧 Configuration Files

OpenTelemetry Collector (`otel-collector-config.yaml`)

Receives traces, metrics, logs
Processes and enriches data
Exports to OpenSearch and Prometheus

Fluent Bit (`fluent-bit.conf`)

Collects container and host logs
Parses and filters log data
Sends to OpenSearch with proper indexing

OpenSearch (`opensearch/config/opensearch.yml`)

Performance optimizations
Index management settings
Cluster configuration

📈 Monitoring & Alerting

Built-in Dashboards

Service Performance: Response times, error rates
Infrastructure: CPU, memory, disk usage
Application Logs: Error trends, log volumes
Trace Analytics: Service dependencies, bottlenecks

Custom Queries

# Find slow traces (>1 second)
{
  "query": {
    "range": {
      "duration": {
        "gte": 1000000
      }
    }
  }
}

# Correlate logs with traces
{
  "query": {
    "bool": {
      "must": [
        {"term": {"level": "ERROR"}},
        {"exists": {"field": "trace_id"}}
      ]
    }
  }
}

🚀 Next Steps

Instrument Applications: Add OpenTelemetry SDKs
Create Dashboards: Build custom visualizations
Setup Alerts: Define monitoring rules
Optimize Indices: Tune for your data patterns
Scale Horizontally: Add more OpenSearch nodes

🔗 Integration Examples

Java Spring Boot Application

Spring Boot Auto-Configuration (Recommended)

// build.gradle - Simple Spring Boot integration
dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
    // One dependency for automatic tracing
    implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter:1.29.0-alpha'
}

# application.yml - Simple configuration
otel:
  service:
    name: spring-app
    version: 1.0.0
  exporter:
    otlp:
      endpoint: http://localhost:4318
  traces:
    exporter: otlp
  # All instrumentation automatically enabled by default!
  # Including: HTTP, JDBC, JPA, Feign, RestTemplate, WebClient, etc.

  # To exclude database tracing, add:
  # instrumentation:
  #   jdbc:
  #     enabled: false

What Gets Automatically Traced (Zero Config Required): - ✅ All HTTP requests/responses (controllers, filters, interceptors) - ✅ All database calls (JPA, JDBC, Hibernate, MyBatis) - ✅ All Feign client calls (service-to-service communication) - ✅ All RestTemplate/WebClient calls (external APIs) - ✅ All JVM metrics (memory, GC, threads) - ✅ All exceptions and errors (stack traces linked to traces)

Why Spring Boot Auto-Configuration is Better: - ✅ Simpler setup - just one dependency + config - ✅ Better Spring integration - works with Spring profiles, properties - ✅ Type-safe configuration - IDE autocomplete and validation - ✅ No external files - everything in your application - ✅ Easier deployment - no need to manage separate agent files - ✅ Better for containers - smaller Docker images - ✅ Spring Boot best practices - follows Spring conventions

Simple Controller - No Manual Tracing Code Needed!

// Main Application - Enable Feign clients
@SpringBootApplication
@EnableFeignClients
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

@RestController
@RequestMapping("/api/users")
public class UserController {

    @Autowired
    private UserService userService;

    // Automatically traced - request/response captured
    @GetMapping("/{id}")
    public ResponseEntity<User> getUser(@PathVariable Long id) {
        User user = userService.getUserById(id);
        return ResponseEntity.ok(user);
    }

    // Automatically traced - all HTTP methods captured
    @PostMapping
    public ResponseEntity<User> createUser(@RequestBody User user) {
        User savedUser = userService.saveUser(user);
        return ResponseEntity.status(HttpStatus.CREATED).body(savedUser);
    }

    // Automatically traced - exceptions captured too
    @PutMapping("/{id}")
    public ResponseEntity<User> updateUser(@PathVariable Long id, @RequestBody User user) {
        User updatedUser = userService.updateUser(id, user);
        return ResponseEntity.ok(updatedUser);
    }
}

@Service
public class UserService {

    @Autowired
    private UserRepository userRepository; // JPA Repository

    @Autowired
    private JdbcTemplate jdbcTemplate; // Raw JDBC

    // JPA calls automatically traced with query details
    public User getUserById(Long id) {
        return userRepository.findById(id) // Traced: SELECT * FROM users WHERE id = ?
            .orElseThrow(() -> new UserNotFoundException("User not found: " + id));
    }

    // Custom repository methods automatically traced
    public List<User> findActiveUsers() {
        return userRepository.findByActiveTrue(); // Traced: SELECT * FROM users WHERE active = true
    }

    // JDBC Template calls automatically traced
    public int getUserCount() {
        return jdbcTemplate.queryForObject( // Traced: SELECT COUNT(*) FROM users
            "SELECT COUNT(*) FROM users", Integer.class);
    }

    // Save operations automatically traced
    public User saveUser(User user) {
        return userRepository.save(user); // Traced: INSERT INTO users (...) VALUES (...)
    }

    // Transaction boundaries automatically traced
    @Transactional
    public void updateUserStatus(Long id, boolean active) {
        User user = userRepository.findById(id).orElseThrow();
        user.setActive(active);
        userRepository.save(user); // All SQL within transaction traced together
    }

    // HTTP client calls automatically traced
    @Autowired
    private RestTemplate restTemplate;

    @Autowired
    private UserServiceClient userServiceClient; // Feign client

    public String callExternalService() {
        return restTemplate.getForObject("http://external-api/data", String.class);
    }

    // Feign client calls automatically traced
    public User getUserFromExternalService(Long id) {
        return userServiceClient.getUser(id); // Automatically traced!
    }
}

// Feign Client - Automatically traced with zero configuration
@FeignClient(name = "user-service", url = "http://user-service")
public interface UserServiceClient {

    @GetMapping("/users/{id}")
    User getUser(@PathVariable("id") Long id);

    @PostMapping("/users")
    User createUser(@RequestBody User user);

    @PutMapping("/users/{id}")
    User updateUser(@PathVariable("id") Long id, @RequestBody User user);
}
}

Alternative: Java Agent (For Legacy Apps)

# Use this approach for non-Spring Boot apps or when you can't modify dependencies
java -javaagent:./opentelemetry-javaagent.jar \
     -Dotel.service.name=my-app \
     -Dotel.exporter.otlp.endpoint=http://localhost:4318 \
     -jar your-app.jar

# Download agent: wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

When to use Java Agent: - 🔧 Legacy applications without Spring Boot - 🔧 Cannot modify dependencies (strict dependency management) - 🔧 Multiple app types (not just Spring Boot) - 🔧 Runtime configuration needed

What Gets Automatically Captured

HTTP Requests & Responses

Request method, URL, headers, query parameters
Response status codes, headers, body size
Request/response timing and duration
User agent, IP addresses
Correlation IDs and trace propagation

Database Operations

✅ All SQL queries with parameters (sanitized for security)
✅ JPA/Hibernate operations (find, save, update, delete)
✅ Connection pool metrics (active connections, wait times)
✅ Query execution time and performance
✅ Database connection details (driver, URL, user)
✅ Transaction boundaries (begin, commit, rollback)
✅ Repository method calls (Spring Data JPA)
✅ MyBatis mapper calls (if using MyBatis)

External Service Calls

RestTemplate, WebClient, OkHttp calls
Feign client requests/responses
HTTP client request/response details
Service-to-service communication
Circuit breaker status
Retry attempts

Supported HTTP Clients (All Automatic)

Spring RestTemplate - Traditional Spring HTTP client
Spring WebClient - Reactive HTTP client
OpenFeign - Declarative REST client
OkHttp - Square's HTTP client
Apache HttpClient - Apache HTTP components
Netty HTTP Client - High-performance async client
Java 11+ HttpClient - Built-in Java HTTP client

JVM & Application Metrics

Memory usage (heap, non-heap)
Garbage collection metrics
Thread pool statistics
CPU utilization
Custom application metrics

Error Tracking

Exception stack traces
Error rates per endpoint
Failed database queries
HTTP error responses
Business logic errors

Zero Configuration Required!

# application.yml - Optional configuration for fine-tuning
otel:
  service:
    name: ${spring.application.name:my-app}
    version: @project.version@
  exporter:
    otlp:
      endpoint: http://localhost:4318
  # All instrumentation enabled by default
  instrumentation:
    http:
      client:
        enabled: true
        capture-headers: true
      server:
        enabled: true
        capture-headers: true
    # DISABLE database tracing
    jdbc:
      enabled: false        # Disables all JDBC/database tracing
    jpa:
      enabled: false        # Disables JPA-specific tracing
    hibernate:
      enabled: false        # Disables Hibernate tracing
    resttemplate:
      enabled: true
    webmvc:
      enabled: true
      capture-request-headers: true
    openfeign:
      enabled: true

Selective Database Tracing Control

# Fine-grained database tracing control
otel:
  instrumentation:
    jdbc:
      enabled: true
      # Only capture slow queries (over 100ms)
      statement-sanitizer: enabled
      # Exclude specific databases or tables
      exclude-statements:
        - "SELECT * FROM health_check"
        - "SELECT 1"
    jpa:
      enabled: false        # Disable JPA tracing but keep raw JDBC
    hibernate:
      enabled: false        # Disable Hibernate-specific tracing

Complete Exclusion Options

otel:
  instrumentation:
    # Database & Persistence
    jdbc: { enabled: false }           # All database calls
    jpa: { enabled: false }            # JPA operations
    hibernate: { enabled: false }      # Hibernate operations
    r2dbc: { enabled: false }          # Reactive database calls

    # HTTP Clients (keep these for API tracing)
    http: { client: { enabled: true }, server: { enabled: true } }
    resttemplate: { enabled: true }
    webmvc: { enabled: true }
    openfeign: { enabled: true }
    webclient: { enabled: true }

    # Other services you might want to exclude
    redis: { enabled: false }          # Redis operations
    mongodb: { enabled: false }        # MongoDB operations
    cassandra: { enabled: false }      # Cassandra operations

# application.yml
otel:
  exporter:
    otlp:
      endpoint: http://localhost:4318
  traces:
    exporter: otlp
  metrics:
    exporter: otlp
  logs:
    exporter: otlp

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus,metrics
  endpoint:
    health:
      show-details: always
    prometheus:
      enabled: true
  metrics:
    export:
      prometheus:
        enabled: true

logging:
  pattern:
    console: "%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level [%X{traceId},%X{spanId}] %logger{36} - %msg%n"
  level:
    io.opentelemetry: DEBUG
    org.springframework.web: DEBUG

Complete build.gradle Configuration

plugins {
    id 'org.springframework.boot' version '3.2.0'
    id 'io.spring.dependency-management' version '1.1.4'
    id 'java'
}

group = 'com.example'
version = '1.0.0'

java {
    sourceCompatibility = '17'
}

repositories {
    mavenCentral()
}

dependencies {
    // Spring Boot Starters
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'

    // Feign Client (automatically traced)
    implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'

    // OpenTelemetry - ONE dependency for everything!
    implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter:1.29.0-alpha'

    // OpenSearch Client (optional)
    implementation 'org.opensearch.client:opensearch-java:2.11.1'

    // Vault Integration (optional)
    implementation 'org.springframework.vault:spring-vault-core:3.0.4'

    // Testing
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
}

tasks.named('test') {
    useJUnitPlatform()
}

Key Benefits of Spring Boot Approach: - 🚀 One dependency instead of multiple OpenTelemetry JARs - 🎯 Simple configuration via application.yml - 🔧 Environment-specific configs using Spring profiles - 📦 Smaller deployments - no external agent files - 🛠️ Better IDE support with autocomplete and validation

This enhanced setup provides enterprise-grade observability with OpenSearch as the central data store for logs, traces, and metrics!