Sentinel — outside monitoring for your stack

Sentinel watches your machines and your public endpoints from outside, every minute, and mails you when something breaks. It is on for every stack and needs no setup.

What Sentinel is

Sentinel is a monitor that runs inside Skalpai Cloud, outside your nodes. It looks at your stack the way your users do: it opens your sites over HTTPS, connects to your public ports, and checks that each machine still sends its host metrics. When a check fails for real, it mails you one message that says what is down, since when, what still works, and the most likely cause.

Sentinel only observes. It never logs into your machines, never runs a command on them and never restarts anything. Every check is read-only: a connection, a TLS handshake, an HTTP GET, a DNS lookup, or a read of the metrics your nodes already send.

Because it runs away from your nodes, it still works when a node is completely down, which is exactly the case where the node cannot report anything itself.

            Skalpai Cloud                              your nodes
  ┌─────────────────────────────┐
  │ discovery (every 5 min)     │   reads what Skalpai already knows:
  │   nodes, spaces, routes,    │   registered nodes, applied manifests,
  │   hosts sending metrics     │   gateway routes, host metrics
  │            │                │
  │            ▼                │
  │ checks (every minute) ──────┼──► https://app.example.org   (status + certificate)
  │            │                │──► mx.example.org:25         (SMTP banner)
  │            │                │──► mx.example.org:993        (TLS + certificate)
  │            │                │    host metrics of node-1    (silence)
  │            ▼                │
  │ false-alert guards          │
  │            │                │
  │            ▼                │
  │ mail: DOWN / STILL DOWN /   │──► your recipients
  │       RECOVERED / daily     │
  └─────────────────────────────┘

Zero setup

Sentinel is on for every stack. You do not declare checks: Sentinel derives them from what Skalpai already knows about your stack, and keeps them up to date.

What it discovers

For each machine of your stack, discovery looks at four sources: the nodes registered for the stack, the spaces your Git repository declares on them together with the manifest each node actually applied, the gateway routes, and the hosts that sent host metrics in the last 7 days. From these it creates:

Ports are probed by name, as your clients reach them. If a space routes mx.example.org and one of its services publishes port 25, Sentinel dials mx.example.org:25, never the node's private address. Port 443 is not checked separately: the website check already covers it.

Some things are left out on purpose:

• Relayed routes — a domain a space routes on to another space only gets the HTTPS check, because its ports belong to the other space.
• UDP ports and ports bound to loopback (127.0.0.1:…) are not reachable from outside and are skipped.
• Internal names are skipped: bare IP addresses, names without a dot, wildcard names (*.example.org), and names ending in .internal, .local or .localhost.
• A website check does not follow redirects: a 301 or 302 counts as an answer below 500, so it passes.

How often discovery runs

Discovery runs every 5 minutes. To pick up a new space or domain right away, press Discover now in the Sentinel tab.

When something disappears from your stack, its discovered checks are retired silently: they stop running and send no mail, but their incident history is kept. If the same thing comes back, the check is revived.

Discovery is careful when it cannot see everything. If one source of a machine cannot be read (for example its manifest), that machine may gain checks but none of its checks are retired. If the list of machines itself cannot be read, nothing is retired at all. A missing source never removes the check that would report the problem.

How a check runs

Every minute, Sentinel runs every active check of every stack, in parallel.

• Each probe has a 10 second timeout.
• A probe that times out is retried once in the same round before it counts as a failure. A refused connection, a bad certificate or a wrong answer is not retried: it is already a clear answer.
• A machine-online check does not connect anywhere: it reads the timestamp of the last host metrics your node sent.

States

How Sentinel avoids false alerts

A false alert at 3 a.m. is almost as bad as a missed outage, so an alert has to pass several guards.

1. A failure that lasts. A check must keep failing for 5 minutes before it alerts, and on at least 3 consecutive runs. A blip shorter than that sends nothing: no DOWN mail, and therefore no RECOVERED mail either. The rule in the code is 2 consecutive failures confirmed by a second vantage point outside both Skalpai Cloud and your node; no second vantage point is in service yet, so Sentinel asks for one more failure instead. The 5-minute floor applies in both cases. A single pass resets the count.

2. Reference hosts. Before it trusts a failing network check, Sentinel connects to a set of reference hosts it does not own. If none of them answers, the fault is Sentinel's own network: every network result of that round is thrown away and nothing is mailed about your machines.

3. Metrics ingest. A machine-online check only counts if some host, anywhere, sent metrics within the silence window. If no host at all did, the problem is Skalpai's metrics ingest, not your machines: the silence results of that round are thrown away.

4. Unverified ports. A port a space publishes may be firewalled or reachable only on the private mesh. A discovered port check stays unverified, and its failures are ignored, until it passes once. After that it is a normal check.

5. Network-wide faults become one mail. If every network check of your stack fails at the network level (unreachable or timeout) at once, and those checks span at least two machines, the cause is almost certainly the path between Skalpai Cloud and your nodes, not each node. You get a single mail for all nodes with that diagnosis instead of one alert per machine.

What you receive

DOWN mail

When a check becomes failing, Sentinel sends one mail per machine. The subject names the machine and the checks that changed:

[sentinel] DOWN mail-1: smtp_banner mx.example.org:25 timeout

The body contains:

• Node and Failing since — the time of the first failed run and how long ago it was.
• Diagnosis — one sentence from a fixed set of written rules (see below).
• Failing checks — each with its address, the exact error and the failure type (unreachable, timeout, protocol, certificate, cert_expiring, silent).
• Passing checks — what still works on that machine, which is often the best clue.
• Host metrics — the last report time, 1-minute CPU load, memory used and disk used of the machine, when it sends host metrics.

The diagnosis is plain code, never a language model, so an alert never waits on one. The rules, in order:

Reminders

While an incident lasts, Sentinel sends a STILL DOWN reminder at most once an hour, not a mail every minute.

RECOVERED mail

When a failing check passes again, you get one RECOVERED mail with the time it was failing since and the current list of failing and passing checks. A check that was only suspect never alerted, so its recovery sends nothing.

Delivery is retried

A mail that cannot be delivered to every recipient stays owed and is sent again on the next round, every minute, until it goes through. The same holds when no recipient address is available at all.

Daily report

The daily report is off by default. When on, it is mailed once a day after the hour you choose (UTC, 07:00 by default). It lists every active check, machine by machine, with its current state and the incidents of the last 24 hours with their durations. Its subject says all N checks OK or X of N checks failing.

Who receives the mail

Mail goes to the recipients configured for the stack. When none is configured, it goes to the stack owner. The Sentinel tab says which of the two applies.

The Sentinel tab

Open your stack, then Infrastructure → Sentinel.

• Verdict banner — at the top. All good with the number of machines, sites and the time of the last check; Checking… while failures are being confirmed; or N problems, each in plain words (Site app.example.org is down, Mail server not answering (SMTP), Certificate of … expires in 5 days) with the machine and since when. If the stack has nothing to watch, the banner says why.
• Machines — one row per machine with a status dot and a summary (Online · 4 sites OK, Checking…, 2 problems). A machine with a problem opens by itself.
• Groups — inside a machine, checks are grouped: Machine online, Websites, Mail, Secure ports, Other ports, Manual checks. Each group shows a count such as 3/3 OK; only checks with a problem are listed until you ask for all of them. Awaiting first verification and Turned off are folded below.
• Details — every check has a details link with the last result, the exact target, the check kind and its source (auto or manual), and when it last ran.
• Turn off / Turn on — turning a check off stops it at once, closes its open incident and cancels any mail it still owed. Discovery never turns it back on. A discovered check cannot be deleted, only turned off; a manual check can be deleted.
• Add a manual check — on a machine, pick a kind and an address. Use it for what discovery cannot know: an oidc_discovery check on an identity provider (https://auth.example.org, which fetches the OpenID configuration and the JWKS), an smtp_egress_source check (forward-confirmed reverse DNS of your outgoing mail address), or a site with a specific status. A manual https_status check expects status 200.
• Discover now — runs discovery for this stack immediately.
• Alerts, incidents & daily report — the recipients list (add or remove addresses), the daily report switch and hour, and the incidents of the last 30 days.

Limits

• Email only. There is no SMS, push or chat channel.
• UDP is not probed. Only TCP ports are checked.
• Some checks are manual. Identity providers (oidc_discovery) and the outgoing mail address (smtp_egress_source) are not discovered; add them by hand.
• Ports are dialled by domain. A port is only checked on the public domains a space routes itself. A port with no public domain in front of it is not checked.
• No repair. Sentinel diagnoses; it never acts on your machines.

FAQ

My stack shows "Nothing to watch yet". Sentinel knows no machine for the stack: no registered node, no host sending metrics, and no manual check. Deploy a space with a public domain or port on a node, or make a node ship host metrics, then press Discover now. If every check exists but is turned off, the banner says so instead.

A port of mine stays in "Awaiting first verification". Sentinel has never managed to reach it from outside. Either it is firewalled or mesh-only on purpose, and then it will simply never alert, or it should be public and is not: test it from outside your network.

How fast will I know? About two minutes after the first failed run, plus up to one minute until that run happens.

I got a mail for "all nodes". Every network check of several machines failed at once. That points to the network between Skalpai Cloud and your nodes, not to each machine. Check your provider's status page before looking at each node.

What should I do when I get an alert? Read the diagnosis and the passing checks first: they tell you which layer is broken. If the machine-online check is the only one failing, your services still answer and the node agent or its metrics shipper is the problem. If everything on the machine fails at the network level, look at the host itself, its firewall and its routing. You will get a RECOVERED mail when it is fixed; you do not need to acknowledge anything.

I don't want to be alerted for a check. Turn it off in the Sentinel tab. It stays off, even after discovery runs again.