Replication Scheduling for Sources Using the API

Stitch supports three replication scheduling methods: Replication Frequency, Anchor Scheduling, and Advanced Scheduling. Learn about each scheduling type and how to use them in the Connect API.

What is replication scheduling?

Replication scheduling tells Stitch when and how often data extraction should occur.

Note: All replication scheduling methods (Replication Frequency, Anchor Scheduling, and Advanced Scheduling) define when data extractions begin. They do not control how long a replication job runs or when data is loaded into a destination.

For a more in-depth look at replication scheduling, refer to the Replication Scheduling overview.

Replication scheduling API properties

With a few exceptions, every source form property available in the API contains the following properties:

These properties are used to create a source’s replication schedule. Depending on the replication scheduling type you want to use, you will need to define values for one or several of these properties.

Scheduling property hierarchy

When determining which replication scheduling type to use, Stitch will consider the cron_expression property above the frequency_in_minutes and anchor_time properties.

The table below demonstrates which replication scheduling type will be used for the various combinations of scheduling properties:

• indicates that the property has been defined with a valid value
• indicates that the property has not been defined

Note: If you choose not to manually set the frequency_in_minutes value, Stitch will use the source’s default value. This means frequency_in_minutes will never be null. The default value varies by source.

Define a replication schedule for a source

In this section, we’ll cover which scheduling properties need to be set to use each replication scheduling type:

• Replication Frequency

• Anchor Scheduling

• Advanced Scheduling using cron

Replication Frequency

To create an interval schedule for a source, the frequency_in_minutes property must be set. The frequency_in_minutes property defines how often, in minutes, Stitch should attempt to replicate data from the source.

For example: The request below updates the frequency_in_minutes property to 1440, or 24 hours:

curl -X PUT https://api.stitchdata.com/v4/sources/86741
     -H "Authorization: Bearer <ACCESS_TOKEN>" 
     -H "Content-Type: application/json"
     -d "{
           "display_name":"Shopify",
           "properties":{
              "start_date":"2017-01-01T00:00:00Z",
              "frequency_in_minutes":"1440"
           }
          }"

This means that Stitch will attempt to replicate data from this source every 1440 minutes, or every 24 hours.

For more info, refer to the Replication Frequency documentation.

Anchor Scheduling

To create an anchored schedule for a source, the following properties must be set:

• anchor_time

• frequency_in_minutes

When both properties are defined, the anchor_time value will define the time that the frequency_in_minutes value is “anchored” to. For example:

curl -X PUT https://api.stitchdata.com/v4/sources/77234
     -H "Authorization: Bearer <ACCESS_TOKEN>" 
     -H "Content-Type: application/json"
     -d "{
           "display_name":"Marketo",
           "properties":{
              "anchor_time":"2018-04-30T03:30:00Z",
              "frequency_in_minutes":"360"
           }
          }"

In this case, Stitch will run a replication job for the source every 360 minutes, starting at 03:30:00. This means a job would run at 09:30:00, 15:30:00, 21:30:00, etc.

For more info, refer to the Anchor Scheduling documentation.

Advanced Scheduling using cron

To create an advanced (cron) schedule for a source, the cron_expression property must be set. The cron_expression value must be a valid Quartz cron expression representing the replication schedule for the source. For example:

curl -X PUT https://api.stitchdata.com/v4/sources/12345
     -H "Authorization: Bearer <ACCESS_TOKEN>" 
     -H "Content-Type: application/json"
     -d "{
           "display_name":"MySQL",
           "properties":{
              "cron_expression":"0 0 12 ? * MON-FRI *"
           }
          }"

In this case, Stitch will run a replication job for the source at 12:00PM every day between Monday and Friday.

For more info, refer to the Advanced Scheduling documentation.