General processing question

1

Just looking for the most efficient way to handle this (if you have any ideas).

Currently within an action, I need to call out to another api to do a value lookup for each line of data. There may be 200+ lines i need to do the lookup on BUT there may be only 3 or 4 distinct values. Instead of doing the same lookup multiple times for no real gain, I would rather unique the values, do the lookup for each unique, then loop through the original lines and apply the data.

example data:

<line>
	<value>1</value>
</line>
<line>
	<value>1</value>
</line>
<line>
	<value>1</value>
</line>
<line>
	<value>2</value>
</line>
<line>
	<value>3</value>
</line>
<line>
	<value>1</value>
</line>

so from that i would only want to do the lookup 3 times then loop through and apply the correct value to each.

any ideas on an easy way to do that?

2

The general technique is to leverage the Lookup Tables actions to use the database as a cache. You would first check to see if a value is in the database, if not then fetch from the API and save.

You can drag Lookup Tables actions into your assemblies:

image
image1700×2372 214 KB

Within the conditional I put the Save action, but that would really be a subassembly to fetch the value via the API and then save.

As an easy way to get the same XML schema as output from the “cache lookup” subassembly, I would just do another “Get value” action, so that the final schema emitted is always “Get value” so the calling assembly can easily consume the result.

At the end of processing in the calling assembly, you can decide to either purge the database of the cached values if they will change in the future and need to be refetched everytime the calling assembly executes, or if they remain fixed values then just keep them cached forever and don’t delete.

If you ever want lookup table data to eventually be auto-purged by the system, you can precede the keygroup name with “temp_” and the system will auto-purge those entries after 60 days.

Caching logic like this is also possible at the automation level, we do this a lot in our integrations.

Hope this helps!

3

i considered building a separate automation to add all the values to the lookup tables but there are a ton of values it could be needing to sync. BUT I didn’t consider doing it within the automation itself. i will take a look. thanks

4

We often use “setup automations” that run once and load data into the database for large static tables of lookup values.

So that indeed is another option.

5

this is one of those cases where 99% of the values are only used 1% of the time and vice versa. i think doing the lookup within the action might be the right way to go. thanks