The JSON file format has become a standard for logging, and it’s common for data engineers to have heavily nested JSON within custom logs like CloudTrail and Sidecar. However, the JSON format requires significant preparation and transformation to present it for analysis – and the flattening of JSON can result in exploding data volume growth. These issues often prohibit companies from making JSON files available for on-demand analytics. Data engineers have limited options: they can either exclude and/or transform JSON via complicated data pipelines where such pipelines need to be revisited if requirements change; or at worst, fully expand all the JSON permutations upfront with the drawback of data explosion. These options create challenges for the business analysts who want access to all of the data and the ability to experiment with various searches as part of their analysis.
To solve these challenges, ChaosSearch has created JSON Flex, which allows organizations to store all their JSON content and analyze it, as if structured, at different permutations and nested levels. ChaosSearch enables organizations to both keep and analyze all data efficiently and compactly without losing fidelity of insights. Customers can use the patented Chaos Refinery to expand and explore all JSON data virtually and instantly on the fly, regardless of size or query structure. JSON Flex eliminates the time sensitive process of data preparation, and individuals outside of IT can access critical insights that drive business decisions.
With JSON Flex, users can:
- Store “all logs” in full native format
- Index “all content” of a JSON file at once
- Create “index views” dynamically to explore JSON without limitations
“Until the introduction of JSON Flex, there hasn’t been a scalable analytics solution for complex, nested JSON files,” said Thomas Hazel, CTO, Founder and Chief Scientist, ChaosSearch. “Organizations have been forced to rely on static, limited views of their data that ultimately lead to less valuable insights. We’re unlocking that data and democratizing it for the masses by delivering a platform that can automatically index and present all the data at once -- making it easier to search, understand and leverage for business insights.”
ChaosSearch also announced a new capability to perform logical “relational inner joins” of log sources within the ELK stack. Until now, correlating log sources required a pipeline to aggregate, de-normalize and ship logs from multiple sources into a single object store prior to ingestion. With the ChaosSearch inner join capability, customers can ingest raw data from multiple log sources into multiple object stores and then perform relational inner joins in the Chaos Refinery where they can perform dynamic, conditional transforms and join multiple data sources into a single view (e.g., Index patterns, relational tables). This further reduces the need for data prep and pipelines and lets users experiment with log sources dynamically so they can create insights faster.
About the ChaosSearch Data Lake Platform
ChaosSearch is an ideal replacement for the commonly deployed Elasticsearch (ELK) stack. With ChaosSearch, customers can perform scalable log analytics on AWS S3 or GCS, using the familiar Elasticsearch API for queries, and Kibana for log analytics and visualizations, while reducing costs and improving analytical capabilities.
ChaosSearch helps modern organizations Know Better™ by activating the data lake for analytics. The ChaosSearch Data Lake Platform indexes customers’ cloud data, rendering it fully searchable and enabling analytics at scale with massive reductions of time, cost and complexity.
ChaosSearch was purpose-built for cost-effective, highly scalable analytics encompassing full text search, SQL and machine learning capabilities in one unified offering. The patented ChaosSearch technology instantly transforms your cloud object storage (Amazon S3, Google Cloud Storage) into a hot, analytical data lake.