Reason to avoid appendToDocumentList

Whenever I see usage of appendToDocumentList in any flow service for creating output document list, I wonder if we can achieve it through some other way e.g. Output array option of loop, java service loop or putting index of incrementing variable inside loop.Also the question used to arise what are the benefits of using other methods over appendToDocumentList. There is a set of reaction which you get when one talks about the usage of appendToDocumentList e.g.

• Performance is badly hit with appendToDocumentList.

• Stay away from appendToDocumentList and use PSUtilities service addToList.

To find out the reason to avoid a particular service which is provided by SAG, I did a little brain storming session with the geniuses out there.

First let me put the alternates of appendToDocList.

• Explicit Loop: If the size of output document list to be created is same as input document list use output array option in loop.

• Sometime there are conditional mappings and output document list can be different from the input one. In that case try and use PSUtilities.list:addToList. Create a copy of this service in your package and use it.

• Implicit Loop: for simple lists of the same size, you may want to link them directly in a MAP step and let the IS handle the looping implicitly.

• Java Loop: Write a java service for looping logic.

Question Arises Why are we trying to avoid appendToDocumentList at all. It’s SAG provided utility and there should be some solid reason behind this. I got a very informative answer from Mr Percio (castropb) in wmusers.

Reason to avoid appendToDocumentList
Percio said – For large lists, appendToDocumentList performs poorly because of the way it is implemented. Every time you call appendToDocumentList, it basically creates a brand new list with size equal to plus 1 (assuming you’re appending one item). It then copies all the items from the original list to the new list and puts the appended item at the end of the new list. This frequent memory reallocation and copying of data is what gives you the performance hit.

When you use an output array, output array is allocated once with the same size as the input array so you don’t run into this problem.

I ran a test for a customer a few years ago to compare different methods of mapping a source list to a target list involving large lists (up to 100,000 items), and at the time, they ranked as follows from fastest to slowest:

1. Java Loop: looping done through a Java service.

2. Implicit Loop: for simple lists of the same size, you may want to link them directly in a MAP step and let the IS handle looping implicitly. Works when the variable names inside a doc list are identical.

3. Explicit Loop: using a LOOP step and its Output Array.

4. Append to Array List: similar to append to document list except that an array list is used in the background so there’s no reallocation and copying of data. It is important to set an appropriate initial size to maximize performance.

5. Append to Document List: using the WmPublic service.

6. Dynamic Index: using a LOOP step without specifying Output Array and mapping to a specific item in the output list using a variable index

Note: Time taken to copy the lists grew linearly as the size of the list grew for method 1 to 4 and grew exponentially for method 5,6.

One new question arises after reading “appendToDocumentList performs poorly for larger lists” is what is a large list? It depends on the physical resources available to the IS and complexity of your mappings, etc.

From the analysis given by Percio, it is very clear that one should avoid appendToDocumentList.

Another Perspective
I also got an interesting answer from reamon. He mentioned

“Key for all of the options listed in Percio’s excellent summary–test which approach works for your situation. Do *not* assume one approach will be faster than another.

In recent tests of my own with appendToDocumentList (though not with the number of elements Percio used) I found that performance had improved dramatically from the same tests I had done a few years ago. The JVM version and settings being used undoubtedly will have a big impact on this performance.

As Percio noted, there is no one answer. So the key is to try out different approaches until you get the performance your integration needs.”

Thank you Percio and Rob Eamon(reamon) for your valuable inputs.

Conclusion
I personally avoid appendToDocumentList because of performance and to shorten the code size.

As already said in post, before coming to conclusion of using or not using append service please make sure that you have tested all your scenarios and use the one which is best suitable for your integration.

Hope this post will help you guys. Comments/discussions are most welcome.

Appreciate if you can provide feedback in comment section or by clicking on reaction check box. Thanks