Observability is a crucial part of modern backend systems, yet many solutions feel bloated, expensive, or overly complex. I wanted to build something lightweight, self-hosted, and extensible—a distributed logging and metrics system that developers can use on their own infrastructure. That’s how Go-Insight was born.
Today, I spent the entire day laying the foundation for this project, focusing on database design, log ingestion, and setting up the API. While there's still a long way to go, the progress made today has given me a clear direction for where this system is headed.
📌 What is Go-Insight?
Go-Insight is a lightweight, distributed observability platform designed to collect and store logs, metrics, and traces from backend applications. The goal is to provide a self-hosted alternative to enterprise observability tools like Datadog and New Relic, but with a focus on developer flexibility and ease of deployment.
🚀 Log Collection - Ingest structured logs from any backend service.
📊 Metrics Tracking - Capture API response times, error rates, and custom performance metrics.
🌐 Self-Hosted - Deploy on your own infrastructure using Docker/Kubernetes.
🔌 Middleware for Any Language - Plan to support integrations with C#, Python, Go, and more.
🎯 Why Build This?
I've always been interested in backend systems and observability, but most existing tools are either proprietary, expensive, or come with too much overhead for smaller applications. Go-Insight is meant to be lightweight and developer-friendly, something you can spin up quickly and start collecting logs and metrics without complex configurations.
🛠️ Setting Up the Core
The first step was structuring the project properly. I designed Go-Insight to follow a modular architecture, separating concerns for API handlers, database logic, and observability utilities.
PostgreSQL as the Database - Chosen for its performance and strong ACID compliance.
Gorilla Mux for Routing - A lightweight yet powerful router for handling API endpoints.
Environment Variables for Configuration - Avoids hardcoded credentials and allows flexibility in deployment.
Docker for Local Development - Makes it easy to spin up a database and test API endpoints in an isolated environment.
📊 Logs and Metrics: Database Schema
I started by designing the database schema for logs and metrics. These tables serve as the foundation for all observability data collected by Go-Insight.
Logs Table: Stores structured log entries, including timestamps, service names, log levels, and messages.
Metrics Table: Tracks API response times, error rates, and request metadata.
🔗 API Endpoints Implemented
To interact with the system, I built the first API endpoints for log and metrics ingestion.
POST /logs → Accepts structured log data from any backend service.
POST /metrics → Stores API performance data like response times and status codes.
GET /logs → Fetches stored logs for debugging and analysis.
🌍 Expanding the Ecosystem
One of the key ideas behind Go-Insight is that it will be language-agnostic. This means developers should be able to integrate it into their backend applications without friction.
To achieve this, I plan to develop middleware for multiple languages. Each middleware package will act as a lightweight client that formats and sends logs/metrics to Go-Insight.
dotnet-go-insight-mw → A C# NuGet package for .NET developers.
python-go-insight-mw → A Python package for Flask/Django apps.
go-insight-mw → Native Go client for direct integration.
🛠️ What’s Next?
✅ Finalize database migrations and data retention policies.
✅ Add a UI for visualizing logs and metrics.
✅ Implement real-time log streaming with WebSockets.
✅ Explore integrations with OpenTelemetry & Fluent Bit.
🚀 Final Thoughts
This project is already proving to be a fantastic learning experience. I’m refining my Go skills, getting deeper into distributed logging, and gaining hands-on experience with observability—something that directly aligns with my career goals.
If you’re interested in observability, logging, or backend engineering, follow along as I continue building Go-Insight. I’ll be sharing progress updates, challenges, and lessons learned.