Architecture

Integrations: To connect data sources to the data lake, we have over 250 integrations that span major cloud services like AWS, Azure, GCP and application infrastructure such as Kubernetes, Jenkins and OpenTelemetry.

Datastreams are a way to organize and manage the flow of data into Observe, making it easy to track and analyze different sources. They’re managed with unique revokable tokens and allow you to control retention periods, permissions, and token rotation.

Data Lake is a single destination for all your machine data whether it’s logs, metrics, traces or anything else. Data is brought into the Lake via open-source collectors and pre-built integrations. All data is compressed an average of 10x and retained for 13 months by default.

Datasets: We curate and transform machine data in the lake to create datasets. They’re structured, optimized for query, always kept up to date and by default hold 30 days worth of data. Datasets are the foundation for exploring and analyzing data, creating dashboards, and setting up monitors for alerting.

Data Graph, which is unique to Observe, transforms your machine data into relatable everyday “things” (called Datasets) that you want to ask questions about, whether it’s pods, shopping carts, customers, or load balancer logs. The Data Graph allows you to visually explore Links between Datasets, enabling seamless navigation to relevant context for the problem you’re investigating.

Links are connections between datasets, similar to the way a foreign key links tables in a database. For example, a container dataset is related to a pod dataset, which may in turn be related to a customer dataset. This allows for quick navigation to relevant context during investigations.

Data Freshness refers to how up-to-date a dataset is. As new data arrives, Observe updates the dataset incrementally. Freshness goals define how frequently incremental updates happen. In Live Mode for example, your Freshness goal would be 1 second (but incurs higher costs).

Data Acceleration refers to the process of updating datasets in the background. Acceleration eliminates the need to query raw data, making queries faster and more efficient, resulting in faster troubleshooting and cost savings.

Data Apps consist of pre-built content such as Dashboards, Monitors, and curated Datasets. Apps make it easy to get up and running with some of the more common scenarios you might encounter in your environment.

Dashboards visually track, analyze, and display key performance metrics. They are endlessly customizable, and because they’re backed by the Data Graph, you can use them as a starting point for investigation to drill into the underlying data.

Monitors watch Datasets for particular conditions and send an alert when it occurs. You can configure monitors to respond to specific threshold metrics and have an alert trigger when the metric meets defined conditions.