Data snapshot settings

Platform discussions
,
1

Good morning,
I bumped into a problem with data snapshots and I have problem with it. Can you help me with proper configuration of this setting? How does it work and what is the best setup.
I am using the trial version and try to set up and fine-tune some integrations between our data sources.

thanks.

2

Hello, Marcel,

what kind of problem are you having?

In general data snapshot saves processed data in a database and can be displayed right in the user interface for each task run.

This setup can be set on each task step with these options:

  • “Never” - data snapshot is never saved for particular task step
  • “On error” - data snapshot is saved only when task step ends up in failure
  • “Everytime” - data snapshot is saved in every case scenario, hence with each task run

Data snapshot is saved sequentially, that means if you had a task with 2 task steps with setup:

  1. Read emails from mailbox (Exchange connector)

    • Data snapshot: “Never”

  2. Process emails (JS mapper)

    • Data snapshot: “Everytime”

You would be able to see data snapshot for both steps (output data from reading emails and also output data from processed emails with JS mapper).

But with this setup, if a task run ended up in failure right on the first step (reading emails), you would not be able to see a data snapshot. It would be probably better to set a data snapshot on the first task step to “On error”, so you would be able to see the DS when task fails right on the first step.

It really depends on a case and what kind of data structure you are storing (string, integer, Base64, etc.) and also on the environment. Depending on whether you are running the tasks on DEV/TEST or PROD environment, you can probably set DS to “Everytime” on DEV/TEST, but we do not recommend doing that on PROD environment in order not to overfill database.

Important is to say that there is some kind of data retention which should be set (also depending on environment), which basically, according to setup, deletes data from database (records older than “7 days” or “1 month” etc.).

1 Like
3

Just small addition: data snapshot (famous never, on error, everytime) can be set on each step as well as on endpoint.

In case of endpoint the behaviour is slightly different then on task. In case you select “never” or “on error” data snapshot we keep the data on endpoint anyway and it is removed based on setup or underlying task during nightly maintenance. So don’t be surprised that you may see data on endpoint even you select option “never” - next morning are deleted.

Be careful with option “everytime” on production system as database can grow up quite quickly. As Petr wrote data snapshot is cumulative so if you choose to save data on last step then the last step snapshot will contain data from all previous steps.

It is important to set-up proper data retention on company level:

image
image1354Ă—1058 61.6 KB

Note that we apply fair user policy and in case the database exceed 10GB we contact customer to adjust the setup or our sales will negotiate with you extra fee for larger database.

Tomas,

1 Like