Service Health Monitor
So I find out before the client does.
The problem
The production server started failing and going down with no warning: we had no idea when it happened because no notification ever reached us. We found out from the clients, who started calling us constantly about the outages. Always being a step behind, putting out fires that had already escalated, was unsustainable.
The solution
First I needed to stop finding out from the client: I set up a monitor that checks each service on its own and pings me instantly over Telegram the moment something changes state. But the real problem showed up right away: many outages were blips of a few seconds, and alerting on each one turned the channel into pure noise that I'd stop watching. So before shouting I added a re-check: if the outage doesn't hold, it ignores it. And I kept it extensible —console, Telegram, a Slack/Discord webhook— behind a one-method interface, so I can add a channel without touching the monitor. The underlying idea: report what's real, fast, and on the channel where I'll actually see it.
Technical decisions
One goroutine per service
Each service is checked in its own goroutine, with its own ticker and interval; a slow or down one never blocks the rest of the monitoring.
Alert only on state change
No noise while healthy; it pings when it goes down and when it recovers, not on every check.
Flap-filter before alerting
Re-checks N times before shouting; a few-second blip never pages anyone at 3am.
Notifiers behind a one-method interface
Adding a new channel never touches the monitor: console, Telegram and webhook are each their own file.
Live
The monitor in action: startup, a forced outage and recovery, with its Telegram alerts.