How to Create a Connection in Precog

In Precog, connections control what data is loaded and when it's loaded into your destinations. Each connection defines the datasets to include, the destination to use, and whether the process runs automatically or manually.

Steps

1. Open Your Workspace

From the Organization Dashboard, select the workspace where you want to create the connection.

2. Go to Connections

In the workspace menu (left-side navigation), select Connections. This opens a view of all configured connections in your workspace. Select Add Connection in the top-left corner.

3. Select a Source

On the Create New Connection page, select your data source from the list. Then select the datasets you want to include in this connection.

You can include datasets from more than one source — simply select another source and then select datasets from that source as well.

For best performance, start with a few key datasets before adding more.

4. Select a Destination

Select the destination where you want Precog to load the data. Then select Continue (bottom-right corner).

5. Enter Connection Details

Add a Connection Name — this will also be used as part of the schema name in your destination database.

Optionally, add a Description to help your team understand what this connection does.

Select Continue (bottom-right corner).

6. Configure Connection Triggers

To run loads automatically, select Add Trigger and set your desired interval (for example, every 6 hours).

To run loads manually, do not add a trigger. You can start a data load at any time after the connection is created.

When finished, select Create Connection (bottom-right corner).

7. Run the Connection

Once your connection is created:

For a manual load, select Run Now (top-right corner) to load your data immediately.

If you configured a trigger, your data will load automatically at the scheduled time.

The page updates automatically when the load completes, and any issues will be shown in the Connection Status panel.

Result

Your new connection appears in the Connections list within your workspace. Precog runs it automatically based on your trigger settings, or manually whenever you select Run Now.

Why Connections Matter for Data Loads

Performance: Run data loads during off-peak hours (for example, overnight) to avoid using system resources during business hours.

Freshness: Control how often your data is updated — hourly, nightly, or weekly — depending on your business needs.

Flexibility: Different sources can run on different schedules. For example, daily updates for active sales data and weekly updates for archived or reference data.

Practical Advice

Refine your connections: Refine your connections until you find the right balance between data freshness and system performance.

Monitor long-running datasets: If a dataset takes longer to process than expected, it can delay subsequent loads in the same connection.

Prioritize critical data: Create multiple connections if needed — one for high-priority data that updates frequently, and another for lower-priority or slower datasets.