Data storage column - example of JS mapping and output schema?

[Andrea (Pospichalova)]

Hi,

Could I please get an example of a practical use of a data storage column? I’m not sure what exactly it is used for.

Thanks,

A.

[tomas.kocyan (Tomáš Kocyan)]

The “Data Storage Column” (or data checkpoints in general) is a simple method for retaining user data at a step level between task executions. Some connectors natively utilize this feature to store the last processed item, such as an email or a file.

In JS Mapper, you have the flexibility to store any desired data. When you specify a “Data Storage Column,” you are essentially indicating which data should be retained (always the last value from the output), and it is subsequently assigned to the “DataCheckpoint” variable in the next run.

Let’s illustrate this with the following example:

Suppose we want to generate users with auto-incremented “ID” and “Name” properties. However, in each run, we should only generate 10 users. The challenge is to generate users with IDs ranging from 1 to 10 in the first run, from 11 to 20 in the second run, and from 21 to 30 in the third run, and so on. This is precisely the situation where DataCheckpoint can be effectively utilized.

Explanation: We will generate users in a cycle and remember the last generated ID. In the next task run, we will begin from the remembered value. Let’s see how straightforward it is:

1. First and foremost, we need to define an output User schema.

2. Specify in the configuration that the last ID will be remembered.

3. Enter the following JavaScript code into the JS Mapping section:

// if the data checkpoint is not definet, set initial value to 0
if (!DataCheckpoint) {
  DataCheckpoint = 0;
}

// number of generated users
let count = 10;

// number of the first user
let start = DataCheckpoint + 1;

// number of the last user
let end = start + count - 1;

// write into log info about generated sequence 
// (make sure you have turned on logging of info level)
log.info(`Generating users from ${start} to ${end}.`);

// empty list of users
const users = [];

// filling empty list
for (let i = start; i <= end; i++) {
  // add new user into list
  users.push({
    ID: i,
    Name: `User ${i}`
  });
}

// return filled list of users
return users;

4. On the step, set creating “Data snapshot” for “Everytime” to see the result:

5. That’s it! Publish the task, run it several times, and observe the data snapshot. The data is generated as expected.

First run:

Second run:

[Andrea (Pospichalova)]

@tomas.kocyan

Data Storage Column”

Hello,
Thank you for explaining, I understand now. I tried the example for myself and it helped a lot.