Integration processes often combine data from multiple sources. Typically, these are external web services, and the very nature of their operation is associated with a certain latency. For each response, we could be talking about several to tens of seconds, and if we need to process multiple responses, the total time starts to become a real issue, especially if we need to query some information repeatedly.
With the advent of the ability to add more inputs (multi-input step), there came an elegant way to easily scale up such integrations that consume data from multiple independent sources.
Imagine we need to import two diverse user collections - meaning different sources, different formats. However, what they have in common is a long response time (each around 5 seconds). We would need to unify these users and fetch some information for each of them.
How would we proceed conventionally?
We would download one data collection (5s), parse it, and “put it aside” – for example, into an SQL database. Then we would download the second collection (5s), parse it again, and add it to the first one. At this point, we have them in one place and can treat them as a whole. Already at this point, our integration takes ten seconds, has 7 steps (2x download, parsing, and storing + 1x loading), and starts to be quite cluttered. If we then wanted to download some additional information for each record from this newly created data collection, the processing time would start to skyrocket.
Let’s see how to do it much more elegantly…
If we start from the end and think about the address service, we can consider it a separate data collection. There’s no need to query it separately for each record, but we can download it all at once. We then implement the matching of the corresponding records ourselves. So, we end up with three independent data collections that we can download simultaneously. The solution looks like this in the following illustration:
At this point, we have all the necessary data downloaded, and all that remains is to perform the actual aggregation of the data. For any of these steps, we now add a new step in which we will carry out this aggregation. We will select the JS Mapper connector, version higher than 3.3.0.
We used a connector that supports so-called Multi Input, so a small plus appeared next to the connector’s input. When we click on it, we can select additional data sources. Thus, we will add the second set of users as well:
And also the entire collection of addresses:
Then, all that’s left is to link the users using a simple script and write a short function that will supplement their addresses.
And what’s the result? With this simple trick, we’ve not only accelerated our integration by approximately 300% but also made it more organized. This will make it much easier to maintain and less prone to errors. Remember the paragraphs above, when after more than 15 seconds, we only had the data prepared and our integration had 7 steps. Now, in seven steps, everything is done in 5 seconds.
This post showcases how the Multi-input step feature can streamline and simplify data integration processes, showcasing its practical application through a real-world example. This approach not only significantly speeds up the entire process but also leads to better organization and easier maintenance of your solutions.
If this method has piqued your interest and you’d like to learn more, or if you have your own experiences you’d like to share, please don’t hesitate to post your examples. We’d love to hear how you’ve applied this approach in your projects or answer any questions that might help you better understand this feature.
Your feedback and real-world examples are valuable to us and help build a community where we can learn from each other and grow together.