Incremental Loading in Precog

Overview

Incremental loading, also known as delta loading, is a method of keeping your destination data current by loading only what has changed since the last data load. Instead of reloading an entire dataset each time, Precog identifies and brings in only the new or updated records.

This approach enables faster, more efficient, and less resource-intensive data loading — especially when working with large or frequently updated data sources.

Why It Matters

Reloading all data every time is often unnecessary and expensive. Most of your data stays the same from one load to the next. Incremental loading focuses on what's changed, saving time, bandwidth, and processing costs.

It also helps ensure data freshness and consistency. Each load captures only new activity while preserving historical data already stored in your destination. That means your data warehouse or analytics platform is always up to date without redundant processing.

For organizations managing large datasets or frequent updates, incremental loading transforms what could be a heavy full load into a smooth, ongoing process.

How It Works Conceptually

Incremental loading relies on a simple idea: track what's new.

Each source contains fields that can tell when a record was created or modified. Precog uses these fields to determine which records need to be loaded during each run. Conceptually, this happens in two steps:

  1. Identify changes — Precog looks for records where a tracked field (often a date or timestamp) shows that something new or updated has occurred since the last successful load.

  2. Apply updates — Only those records are loaded into the destination, leaving existing data untouched unless there's a change.

Two key types of fields make this possible:

  • An incrementing field, such as a timestamp or numeric counter, indicates when new data appears or existing data changes.
  • A unique identifier, such as a record ID, distinguishes each record so changes can be matched correctly.

By combining these, Precog can detect changes automatically and maintain a complete, accurate dataset in your destination.

Example Scenario

Suppose your Shopify data includes a field called Updated At and each record has a unique Id.

During the first load, Precog imports all data. After that, it loads only records where Updated At is more recent than the last load time.

This means when a customer updates an order or a new one is created, only those changed or new records are sent to your destination. The result is a fast, efficient data loading process that minimizes redundant data movement.

Practical Insight

Incremental loading is ideal for ongoing operations — especially when data volumes are large or updates occur regularly.

  • Use it when performance and timeliness matter.
  • Combine it with a well-tuned schedule to control how often updates occur.
  • Rely on full (historic) loads only when resetting or rebuilding data for new destinations.

By focusing only on what's changed, incremental loading ensures your data stays up to date while minimizing unnecessary processing.