This is a complex issue. While we can detect single add/remove item from collection scenarios, and would likely be able to add/remove the affected items from the UI, the main problem are the parent collection property and the siblings of the item added/removed.
For instance the parent collection property often shows a count of child items in its value. This comes from a .NET property descriptor. Likewise, the items generally show an index (e.g. "") as their property name, which also comes from property descriptors. As sibling items are added/removed, the existing property descriptors aren't refreshed by .NET. New ones have to be retrieved from .NET for the properties, and that mainly requires a call to fully refresh the parent collection property.
This refresh process is what clears out all the child nodes and adds new ones, as if it were the first time being displayed. That's why you see what you see, because the data factory's GetDataModels method is being called for the parent collection property and is effectively rebuilding the expandable nodes underneath it.
We've looked into this for a while and there isn't really any easy way to do a simple add/remove while also updating the parent/siblings appropriately, without a full refresh like that. The downside of course is you do lose expansion states with the refresh.