The observability space continues to evolve as enterprises face the growing challenge of handling massive volumes of log and event data. Storing data cost-effectively while still keeping it immediately available for analytics has long been the “holy grail” for operations, security, and business intelligence teams. Today, Hydrolix, a company specializing in high-density storage and real-time queries, has announced a new Apache Spark connector for its streaming data lake today. Set to work seamlessly with Databricks, the connector promises “split-second” queries of full-fidelity event data that spans both real-time and historical logs.

This release is indicative of an approach to how organizations want to process and learn from logs, metrics, and other time-series data while not having to move such massive volumes of data across data lakes. Rather than limiting data ingestion or relying on short retention windows, businesses increasingly demand platforms that can economically store and swiftly analyze the entirety of their event data. Here is Stratola’s quick take on this announcement and what it means for both the companies and their customers.

The Evolving Role of Observability Data

Modern organizations produce logs and metrics at an extraordinary pace – we are talking about north of 5-10 TBs per day! This “observability data” is more critical than ever for guiding operational decisions, troubleshooting performance issues, and even feeding machine learning models that anticipate outages or detect anomalies. However, several longstanding challenges persist –

  1. High Costs: Retaining logs at scale can become exorbitantly expensive, especially when data needs to remain “hot” (readily accessible).
  2. Siloed Architectures: Data often gets copied or moved from one platform to another for advanced analytics, risking duplication and stale datasets. Not to forget, there is also the challenge of data immediacy.
  3. Downsampling: Faced with storage constraints, teams frequently downsample data, potentially omitting outliers that are crucial for diagnosing issues.

Hydrolix has always been a leader in this space with a model that promises to keep petabytes of data “hot” in object storage. They have already empowered many of their customers to monitor multiple Content Delivery Networks (CDNs) for performance, outages, and issues. The new connector extends this proposition by letting users directly query that data via Spark—most notably in Databricks—bypassing complex ETL workflows.

Understanding Hydrolix’s Approach

Hydrolix emphasizes high-density compression, often citing compression ratios well above 90%. In principle, this allows enterprises to store far more data without incurring typical cost penalties. Instead of archiving older logs to cold storage, companies can keep everything immediately available for any analysis that arises, whether it’s a routine dashboard query or an ad hoc deep dive into historical trends.

The streaming architecture also supports indexing and partitioning strategies that expedite query performance, especially for time-series data. According to Hydrolix, it’s not just about compressing data on disk but also about retrieving exactly the slices of data required for each query.

With the release of the Spark connector, Hydrolix is effectively merging its storage and query optimization techniques with Spark’s well-known parallel execution capabilities. This may be particularly interesting to users of Databricks, where Spark is the foundational technology.

Key Elements of the Spark Connector

From a high-level viewpoint, some of the key highlights of the new connector are –

  • Leverages Spark’s Parallelism: Queries against Hydrolix data use Spark executors for parallel operations, reducing the risk of resource-intensive analytics interfering with real-time dashboards or ingestion tasks within Hydrolix.
  • Pushdown Optimizations: The connector takes advantage of Spark’s Data Source V2 API to prune columns, push down predicates, and optimize partition scanning. Essentially, it avoids reading unnecessary data from object storage whenever possible.
  • Seamless Access Across Clouds: Because Hydrolix can store data in multiple clouds or buckets, the connector provides a unified way to query data regardless of whether it’s in AWS, Azure, or elsewhere.

For those who already rely on Databricks for data science and machine learning workflows, this integration aims to remove barriers to exploring large volumes of operational data. The proposed benefit is minimized data movement. Rather than duplicating logs into a separate data lake for Spark-based processing, teams can query directly from Hydrolix.

Comparing Hydrolix to Delta Lake

Well, the obvious question in the room will be – “Why wouldn’t I just ingest logs/metrics data directly into Delta lake and then process it using Spark?”. A notable aspect of this announcement is the benchmarking that positions Hydrolix alongside Delta Lake, as written by Hydrolix. As you may know, Databricks popularized the Delta Lake format to bring ACID transactions, schema enforcement, and improved performance to data lake storage.

From the published results, Hydrolix claims stronger compression and faster query times across several typical log analytics scenarios, such as grouping by string fields or generating histograms. While these early metrics are promising, several caveats remain:

  1. Photon: Databricks Photon, an engine designed for superior performance on Delta Lake, was not enabled during testing. Photon can accelerate certain workloads but adds cost, so there’s a trade-off.
  2. Cluster Sizing: Benchmark results can vary significantly depending on the size and configuration of the Spark cluster. Smaller clusters sometimes favor Delta Lake, while larger clusters may highlight Hydrolix’s advantages.
  3. Summary Tables vs. Raw Data: Hydrolix’s speed gains in certain percentile queries rely on summary tables. While it is a valid approach that precomputation can vastly accelerate queries, it’s still not a strict apples-to-apples comparison with raw Delta Lake queries.

Regardless, these benchmarks suggest that Hydrolix is well-optimized for log analytics, where high-cardinality string fields and long retention windows are common. Organizations with particularly large volumes of event data might see meaningful performance or cost benefits, especially if they can exploit Hydrolix’s compression and indexing capabilities.

Observed Use Cases

Beyond the technical benefits, the connector could influence how customers approach several high-value problems –

  1. Extended Retention for Compliance: Regulated industries often need to retain logs for months or years. Hydrolix claims to make this feasible without ballooning costs, and Spark then provides the toolset to run forensic analyses or advanced ML.
  2. Anomaly Detection and Security: Outlier detection models frequently benefit from more data. By retaining full-fidelity logs, security analysts and data scientists can detect and investigate anomalies that a typical downsampling approach might miss.
  3. Predictive Maintenance: Internet of Things (IoT) deployments can produce staggering volumes of machine-generated data. With this connector, sensor logs could remain accessible for iterative model training, troubleshooting, and performance tuning.
  4. Fraud Detection: Subtle patterns of fraudulent behavior may only emerge when analyzing long-term transaction logs. A high-compression log lake can help keep these data sets accessible.

In each scenario, Databricks serves as the analytical workbench. Data is queried in place on Hydrolix, with Spark’s parallelism ensuring that even complex JOINs and aggregations can be completed in a predictable timeframe.

Stratola’s take and guidance

Hydrolix’s Spark connector arrives at a time when many organizations are reassessing their observability strategies, balancing real-time visibility with long-term analysis. In offering a specialized approach to full-fidelity log retention, Hydrolix challenges the industry norm of partial log data retention. Meanwhile, Databricks offers an established environment for big data. data intelligence, and machine learning, representing an appealing platform on which to run large-scale log analytics. The combined offering could significantly reshape cost models and workflows for teams dealing with massive log volumes.

That said, real-world results will vary based on each organization’s infrastructure, data volumes, and query patterns. Enterprises evaluating this solution should run proof-of-concept tests, comparing total costs and performance for their specific workloads. They should also monitor ongoing enhancements, like potential pushdowns for percentile approximations and columnar execution features, to see if Hydrolix can maintain or extend performance gains over time.

In sum, Hydrolix’s new Spark connector underscores a broader industry priority about ensuring all event data is readily accessible for both operational troubleshooting and advanced analytics. If the platform delivers on its promise, it may become a compelling option for Databricks users who want to store, query, and analyze log data at scale without sacrificing depth, speed, or cost-effectiveness.