Beyond Manual SSH Troubleshooting: Lightweight Executor Cluster Monitoring with Pulse

In executor, build, or test machine clusters, invisible machine status is a typical operations pain point. Pulse is a lightweight monitoring solution made of two single-file Go binaries. It helps you achieve real-time cluster visibility within 5 minutes without introducing a heavy monitoring stack. This article introduces Pulse's core features, engineering details, and deployment steps.

1. The 'Observability Vacuum' of Executor Clusters

Traditional troubleshooting of executor clusters relies on SSH login and manually running commands such as top, free, df -h, and iostat. This approach has obvious lag, is inefficient, and makes it difficult to detect 'hung' states. Common monitoring solutions such as Prometheus + Grafana are powerful, but for intranet, no-external-network, or large-machine-count executor scenarios, deployment and maintenance costs are often too high.

Pulse's approach is: use as little resource consumption as possible to solve the core problem of 'visibility.'

2. Pulse Architecture: Agent + Server

Pulse consists of two independent single-file Go binaries:

• Agent: Runs on each executor machine, collects metrics, and reports them.
• Server: Runs on a central node, receives reported data, and provides the Dashboard.

The data flow is one-way (executor machines → Server). The Server does not actively collect data, nor does it reversely control machines.

Agent Features

• Cross-compiled single-file Go binary, only a few MB in size, with no runtime dependencies.
• Collects metrics every 10 seconds and reports them via HTTP POST.
• The reporting address is specified by the PULSE_SERVER environment variable.
• Managed by systemd for boot autostart and automatic restart on crash.

Server Features

• Implemented purely with the Go standard library, with zero external dependencies.
• Maintains an in-memory mapping from hostname:ip to the latest metric.
• Provides four endpoints:
  • /: Web monitoring Dashboard (the Dashboard is embedded into the binary via go:embed)
  • /report: Agent reporting entry point
  • /status: Full status JSON (for Dashboard polling)
  • /health: Health check

Port note: The Dashboard runs on port 8080 by default; the installation script and Agent reporting address examples use port 8123. In actual deployment, use your own configuration as the standard.

3. Comprehensive Metric Collection

Pulse collects metrics that cover the key dimensions of daily executor machine inspection:

Collection Robustness

If collection of any single metric fails, the Agent only logs it and keeps a zero value, while other metrics are reported as usual. This means that even if a machine lacks a data source (such as /proc/loadavg), it will not cause the entire machine to become 'lost.'

For network and disk I/O rates, the Agent saves the cumulative value and timestamp from the previous sampling round, and calculates bytes/s by dividing the difference by the interval. On the first sampling round or when a counter resets or wraps around, the rate is recorded as 0 to avoid negative or abnormally large values.

4. Smart Alerting: Notify Only on State Transitions

Offline Determination

If no report is received for more than 30 seconds, the host is marked as offline.

Threshold Alerts

The default thresholds are as follows. All thresholds can be overridden via environment variables:

• CPU usage: 90%
• Memory usage: 90%
• Disk usage: 85%
• Swap usage: 50%
• Process count: 2000

Alert Deduplication (State Machine)

Pulse maintains a state machine keyed by 'host + alert type.' Only when the state transitions (for example, from normal to abnormal, or from abnormal back to normal) does it output ALERT or RECOVER logs. This avoids the noise problem of traditional solutions, where logs are printed every round after a threshold is exceeded.

5. Key Engineering Details

1. Reported IP is based on the TCP source: The Server parses the true source address from r.RemoteAddr and does not trust the IP self-reported by the Agent. This is especially accurate in network address translation (NAT) or container environments, and it is also more secure.
2. Failed-report retry with exponential backoff: When reporting fails, the Agent retries at intervals of 1s → 2s → 4s, up to 3 attempts, to avoid data loss caused by transient network jitter.
3. Single implementation of threshold judgment: The /status query and the alerting goroutine share the same threshold judgment function, eliminating the inconsistency where the Dashboard shows normal while alerts are frantically firing.
4. Per-request panic isolation: Each Server handler is wrapped with recover, so a panic in a single request will not bring down the entire service.

6. Deployment Steps

Prerequisites

• The Server side needs a Go environment installed to compile the Server and Agent.
• The executor machine side does not need any runtime installed; use the precompiled binary directly.

1. Compile and Start the Server

Run the following on the machine where the Server resides:

# Install Go (if not already installed)
# For example on Ubuntu: sudo apt install golang-go

# After cloning or downloading the source, compile the Server
go build -ldflags "-s -w" -o pulse-server ./server

# Compile the Agent (optional, for cross-compilation)
# CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-s -w" -o pulse-agent ./agent

# Start the Server
./pulse-server

The Server listens on port 8080 by default, providing the Dashboard and API. The installation script and Agent reporting address examples use port 8123. In actual deployment, use your own configuration as the standard.

2. Install the Agent (on Each Executor Machine)

On each executor machine, use a one-line command to complete installation, configuration, and startup:

curl -fsSL http://<server-ip>:8123/resources/install-pulse.sh | bash -s -- "http://<server-ip>:8123/report"

The script automatically performs the following operations:

• Downloads the Pulse Agent binary.
• Writes a systemd service file, setting boot autostart and automatic restart on crash (5-second self-healing).
• Starts the service and verifies its running status.

After installation is complete, the Agent reports metrics to the Server every 10 seconds.

3. Access the Dashboard

Open http://<server-ip>:8080/ in a browser to see the cluster overview panel. The panel supports real-time search by hostname or IP, header sorting, and pagination. Click any host to view detailed metric cards.

7. Applicable Scenarios and Boundaries

Suitable Scenarios

• Daily health inspection of executor, build, and test machine clusters.
• Intranet, no external network, or environments where a heavy monitoring stack cannot be introduced.
• Teams that only need to solve machine status visibility and do not want to bear the monitoring operations burden.

Unsuitable Scenarios

• Long-term metric storage and complex queries are required (Pulse is in-memory and clears on restart).
• Multi-tenancy, RBAC, and other permission systems are required.
• Business metrics are required (such as QPS, latency distribution, and error rate) — Pulse only manages machines, not business.

If your needs fall into the above 'unsuitable' categories, Prometheus + Grafana is still a more comprehensive choice.

8. Summary

With two single-file Go binaries, 10-second reporting, smart alert deduplication, and an embedded Dashboard, Pulse provides a lightweight, zero-dependency monitoring solution for executor clusters. It does not pursue being big and complete; instead, it focuses on the core problem of 'machine status visibility,' freeing operations teams from manual SSH troubleshooting.

You can verify whether deployment succeeded by accessing the Dashboard: if all hosts are reporting status normally, the system is running well. If you encounter problems, check the Agent's systemd logs (journalctl -u pulse-agent) or the Server's console output.