Telemetry Collector and GPU Utilization Dashboard

Design a data collector and analytics dashboard for GPU-cluster telemetry. The system must ingest both 30-second and 1-second resolution metrics, show 3-month utilization trends, and allow drill-down to per-GPU per-second data. Requirements Design a collector and dashboard for hardware / GPU telemetry. Functional requirements: Collect telemetry from GPU-cluster hosts at both 30-second and 1-second resolution. Track longer-range utilization trends over roughly three months. Support dashboard drill-down from aggregate trend views to per-GPU, per-second data. Use workflow analysis to calculate GPU utilization and compute-resource needs for hardware design teams. Expose enough metadata for operators to understand cluster, host, GPU, workload, timestamp, and metric type. Constraints and design points: The hot path must handle high-cardinality time-series writes. Storage should separate low-resolution historical aggregates from high-resolution drill-down data. Dashboard queries need predictable latency despite multi-month retention. OpenTelemetry-style instrumentation and DCGM-style GPU metrics are relevant interfaces to discuss. Notes A strong design usually separates collection, ingestion, storage tiers, and query serving: Host agents collect local GPU and workflow telemetry, buffer briefly, and push or stream to a central ingestion layer. The ingestion layer validates schema, deduplicates retry traffic, and writes raw high-resolution data plus downsampled aggregates. Low-resolution trend data can live in a time-series or analytics store optimized for range scans; high-resolution per-GPU data can use shorter retention or colder partitions. Dashboard queries should hit pre-aggregated views first, then fetch narrow raw slices for drill-down. Key trade-offs are resolution vs retention cost, push vs pull collection, hot-partition avoidance, and how much aggregation belongs in the collector versus the backend. Preparation Draw the path from host agent to metrics store to dashboard, with a separate downsampling path. Prepare schema fields for GPU telemetry: host, GPU id, job / workflow id, timestamp, metric name, value, and tags. Practice explaining why per-second data for three months is expensive and how retention tiers preserve usability without storing every query path hot.