New Cluster DX

Table of contents

Intro
Objectives
Arguments in favor of a **new service**
- Decentralization and scalability
- Real-time data distribution with low overhead
- Improved filtering & personalization
- Integration with modern systems (IoT & Web)
- Security & Authentication
- Propagation tools and AI-driven insights
- Reduced infrastructure costs
- Mobile and offline operation
Potential challenges
High level architecture proposal
Next steps

1. Intro

The idea of using MQTT protocol for a modern ham radio cluster system is compelling and tries to resolve endemic problems related to current DX Cluster technologies and habits in use. Traditional DX clusters, such as DX Spider, CC Cluster, and AR Cluster, rely on aging telnet-based architectures that suffer from inefficiencies, high latency, lack of scalability, and limited filtering capabilities. These systems require users to manually configure text-based filters, leading to an overload of irrelevant spots, while also being prone to connection instability and centralized points of failure. Additionally, integration with modern applications, mobile devices, and web platforms is cumbersome due to the legacy nature of these protocols.

By leveraging MQTT’s lightweight, real-time publish-subscribe model, a next-generation cluster can offer a more efficient, scalable, and personalized experience. Hams could subscribe only to the bands, modes, or regions of interest, significantly reducing noise and improving usability. The inherent low-bandwidth nature of MQTT makes it ideal for ham radio environments, where connectivity might be intermittent. Furthermore, integrating real-time propagation analysis from sources like WSPR, PSK Reporter, and Reverse Beacon Network (RBN) would allow the system to automatically provide actionable insights, alerting operators to band openings and rare DX opportunities without the need for constant manual checking.

Security and authentication are also major areas of improvement. Current DX clusters are often open to abuse, as they rely on weak or non-existent authentication mechanisms. An MQTT-based system could enforce TLS encryption, OAuth2 authentication, and access control lists (ACLs), ensuring a more secure and trusted environment for licensed operators.

Lastly, modernization through MQTT would enable seamless integration with web apps, mobile applications, SDR receivers, and even IoT-based remote stations, paving the way for a truly interconnected, data-driven ham radio ecosystem that enhances the DXing experience while maintaining backward compatibility with legacy systems through protocol bridges.

2. Objectives

Low Latency: Natural travel from telnet polling -> real-time updates.
Scalable & resilient: Enable multi-broker setups.
Fine-grained filtering: Enable subscribing only to relevant spots/bands.
Mobile-friendly: Able to works smoothly on smartphones and tablets.
Security: Addition of TLS encryption, AuthN & AuthZ with OAuth2, JWT, etc…
Easier Integration: Able to allow connecting from tools such us Grafana, InfluxDB, Node-RED, Home Assistant, etc…

3. Arguments in favor of a new service

Following are some of the multiple key arguments and improvements the community should work on to resolve current issues with the cluster service:

a. Decentralization and scalability

Traditional clusters like DX Spider rely on a network of connected servers with telnet or IRC-like interfaces, which can be cumbersome and not fully decentralized.
MQTT’s publish-subscribe model allows for a more distributed architecture, where clients subscribe to specific topics and receive real-time updates without direct server dependencies.
This makes it easier to scale by adding more nodes without needing a predefined topology.

b. Real-time data distribution with low overhead

MQTT is designed for low-bandwidth, real-time messaging, making it more efficient than traditional telnet-based clusters.
Ham radio propagation and DX spots can be instantly disseminated with minimal latency.
Unlike traditional systems that use full message retransmission, MQTT allows clients to subscribe only to relevant information, reducing unnecessary data load.

c. Improved filtering & personalization

Current cluster systems require text-based commands for filters (e.g., sh/dx 10m).
With MQTT, clients can subscribe only to the frequencies, locations, or call signs they are interested in, reducing spam and making the system more personalized.
Topic-based filtering (e.g., dx/10m, dx/20m/Europe, dx/KP4) ensures that operators receive relevant information.

d. Integration with modern systems (IoT & Web)

Traditional clusters are not designed for API access, automation, or integration with modern software.
MQTT can easily integrate with dashboards, web applications, mobile apps, and IoT devices.
DX spots and propagation data could be visualized in Grafana, Home Assistant, or custom web dashboards without needing clunky software.

e. Security & authentication

Older clusters have limited security mechanisms and often use open telnet ports, making them vulnerable to abuse.
MQTT supports TLS encryption, authentication via JWT or OAuth, and access control lists (ACLs) to prevent unauthorized use.
A federated authentication system could allow trusted hams to log in via callsign authentication (LoTW, QRZ, etc.).

f. Propagation tools & AI-driven insights

Traditional clusters mostly focus on DX spotting but lack analytical tools.
MQTT could support real-time propagation models, automatically publishing alerts when certain bands open up based on AI models and real-time reports.
Integration with WSPR, PSK Reporter, and Reverse Beacon Network (RBN) could enhance DX spotting.

g. Reduced infrastructure costs

Current cluster systems require dedicated servers with persistent TCP connections.
An MQTT-based system could run on lightweight cloud instances or even on Raspberry Pi nodes, reducing operational costs.

h. Mobile & offline operation

Many ham radio operators use mobile devices in remote areas where connectivity is unstable.
MQTT’s “last will and testament” and retained messages can ensure they receive missed spots when they reconnect.
A progressive web app (PWA) or mobile app could provide a seamless DX cluster experience.

4. Potential challenges

Adoption: Many hams are accustomed to telnet-based clusters and might resist switching.
Legacy Integration: Some old logging software may not support MQTT.
Server Federation: Current systems rely on a network of interconnected servers—how would a decentralized MQTT cluster manage redundancy?

5. High level architecture proposal

a. Core Components

MQTT Broker (central messaging hub)
- Manages real-time communication between all nodes.
- Handles topics like dx/spots, propagation, alerts, commands.
- Software options: Eclipse Mosquitto, EMQX, HiveMQ, VerneMQ.
DX spotting clients
- Hams can post and receive real-time DX spots.
- Clients publish to dx/spots/{band}/{callsign} (e.g., dx/spots/20m/KP4).
- Subscribers receive only the spots they are interested in.
Propagation monitoring
- Integrates with WSPR, PSK Reporter, Reverse Beacon Network (RBN).
- Publishes real-time propagation data (e.g., propagation/20m).
APIs & Web interfaces
- REST API & WebSockets for non-MQTT clients (old loggers).
- Progressive Web App (PWA) or mobile app for hams.
Bridges for legacy cluster support
- Connects to DX Spider, CC Cluster, AR Cluster via telnet.
- Translates old cluster messages into MQTT topics.

b. MQTT Topic Structure

Topic	Description
`dx/spots/{band}/{callsign}`	DX spots (e.g., `dx/spots/20m/KP4`)
`propagation/{band}`	Real-time propagation updates
`alerts/{region}/{band}`	Critical alerts (e.g., `alerts/EU/10m`)
`commands/{callsign}`	User commands (e.g., filtering, subscriptions)
`logs/{callsign}`	Activity logs for monitoring

c) How Components Interact

Ham operator posts a DX spot
- Publishes: dx/spots/20m/KP4 → "CQ DX de KP4, 14.200 MHz"
- Subscribed stations receive real-time updates.
Propagation data updates
- External sources (WSPR, PSK Reporter) publish:
- propagation/20m → "Strong opening from EU to NA"
- Operators subscribe based on interest.
Legacy cluster bridge
- Connects to other cluster solutions via telnet.
- Converts telnet messages to MQTT topics.
- Older software can still interact with the system.
Web & mobile dashboard
- Displays live DX spots, propagation, and alerts.
- Uses WebSockets + MQTT.js for real-time data.

6. Next steps

v.0.1 @ 2025-01-13

Prototype a small MQTT-based DX Cluster with basic topics (dx/20m, dx/15m).
Create a bridge to existing cluster systems (e.g., telnet to MQTT adapter).
Test message latency and scalability with multiple users.
Draft a modern web/mobile UI to make adoption easier.