Insights

Insights shows what your pipelines and syncs are doing, without any setup. Open Insights in the left rail. It covers the environment picked in the top bar, like the rest of the app.

Inside, there are five sections:

• Overview: a ready-made dashboard of how the environment's syncs are doing.
• Queries: SQL, or the Builder, against Pipeloom's own data or one of your warehouses.
• Dashboards: saved queries laid out as tiles, with filters on top.
• Monitors: checks that run on a schedule and raise an anomaly when something is off.
• Anomalies: what the monitors found, to acknowledge, mark as not an anomaly, or mute.

The Overview

The Overview needs no setup. It shows records loaded, sync success, failures, freshness, the slowest syncs and rejected records, for the period you pick. Filter it by pipeline or connection.

Each tile's menu has Open as query, which opens the SQL behind the tile in the editor so you can change it and save your own version, and Download CSV. To make the whole Overview your own, duplicate it: you get a dashboard of saved queries you can edit.

Insights elsewhere in Pipeloom

• A pipeline's editor has an Insights tab: the Overview, for that pipeline only.
• A run's page and a sync's page have Open in Insights, which opens the Overview filtered to it.
• A Data check step runs a query inside a pipeline and fails the step unless the result is as expected. See Orchestration.
• The Copilot can answer questions with a chart, write and save queries, build dashboards and set up monitors. It shows you what it would save and asks first.

Two kinds of data

Results are cut at 10,000 rows, and the editor says when that happens.

A result is kept for 5 minutes. Anyone opening the same query or dashboard in that time gets that result without running the query again. Use Refresh to run it now.

What warehouse queries cost

Every warehouse query that runs is charged: runs in the editor, dashboard tiles, auto-refresh, monitors, the schema browser and Data check steps. A query costs the time it ran, in the same credits as everything else, and at least 1 credit. A result served from the 5-minute cache costs nothing. Insights queries are shown separately on the Usage page.

When your organization is out of credits, warehouse queries stop like pipeline runs do. Monitors on a warehouse show Paused and say why. Queries on Pipeloom data and the built-in monitors are never charged.

Who can do what

Insights has five actions in Settings → Access. Like every other action, you can change the role each one needs, per environment.

A dashboard's tiles and the Overview run for anyone who can see Insights: their SQL is saved, and seeing it run is the point of a dashboard.

Environments

Each environment has its own queries, dashboards and monitors, and sees only its own data. A query in one environment can't read another's syncs, even with hand-written SQL: the database itself enforces it.

To bring work from one environment to another, use Copy from environment on the Queries, Dashboards or Monitors page. Pick the environment and what to copy, and review what would happen first:

• Queries and dashboards are matched by folder and name. Copying again updates the one already there instead of adding another.
• A dashboard brings the queries its tiles show, and a monitor brings the query it watches.
• A query on a warehouse runs on the target's warehouse of the same name. If the target has no warehouse with that name, the query is not copied, and neither is anything that uses it.
• Where a monitor's alerts go stays with each environment. An existing monitor keeps its channels, and a new one starts with none.
• A monitor that starts a pipeline starts the same pipeline in the target, as promoted there. If the target doesn't have that pipeline, or you can't start pipelines there, the monitor is not copied.
• A variable's default that points at something in one environment (a pipeline, a connection, an environment or a warehouse) is left empty in the copy.

A release between environments can carry queries, dashboards and monitors too, so they move with the pipelines they belong to. On the release's page, under Queries, dashboards and monitors, choose Pick. They're copied the same way, after the release's pipelines land, so a monitor can start a pipeline the same release brings. The review shows what copying would do. Anything that can't be copied counts as blocked, like a pipeline. A release can carry only these, with no pipelines. Picking them needs the same rights as Copy from environment. Rolling the release back doesn't undo them.

With Git Sync on, saved queries, dashboards and custom monitors are also written to the repository, under insights/.