For every release of LemonTree, we run a set of tests with different test-models to examine the change of both the memory consumption and the total diff/merge times.
This page shows the results of these tests.
As shown by the test results, our goal is to continuously both improve the performance and decrease the memory consumption of LemonTree.
Nevertheless, changes to the supported feature-set may impact both performance and memory consumption negatively. The Remarks section under the test results notes such changes.
Test Case Description
The test results provided on this page are based on a fixed set of test models, which we diff with every released version of LemonTree.
Details on the used test models can be found here:
Measurement Methods
To understand the measurement methods we have used to compile this data or conduct measurements on your own model for a comparison, please refer to this page:
By applying our measurement methods to your own models, you can calculate comparison values to approximate how LemonTree will perform for your models based on the data below.
Environment
Our test machine has the following setup:
- Operating System: Microsoft Windows 10 Enterprise N
- Processor: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 3504 Mhz, 4 Core(s), 8 Logical Processor(s)
- Memory: 64 GB
Test results
| Small Model | ||||
| LT Version | 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage |
|---|---|---|---|---|
| 3.3.1 | 3s | 0s | 0s | 901 MB |
| 3.3.0 | 3s | 0s | 0s | 740 MB |
| 3.2.1 | 4s | 0s | 0s | 651 MB |
| 3.2.0 | 3s | 0s | 0s | 890 MB |
| 3.1.4 | 3s | 0s | 0s | 947 MB |
| 3.1.3 | 4s | 0s | 0s | 881 MB |
| Medium Model | ||||
| LT Version | 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage |
| 3.3.1 | 1m0s | 7s | 13s | 6.867 MB |
| 3.3.0 | 1m4s | 8s | 12s | 6.364 MB |
| 3.2.1 | 54s | 7s | 11s | 6.846 MB |
| 3.2.0 | 1m2s | 7s | 11s | 6.374 MB |
| 3.1.4 | 58s | 7s | 12s | 6.831 MB |
| 3.1.3 | 1m1s | 8s | 10s | 6.250 MB |
| Large Model | ||||
| 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage | |
| 3.3.1 | 2m14s | 3m38s | 17m31s | 14.924 MB |
| 3.3.0 | 2m14s | 3m47s | 17m41s | 14.827 MB |
| 3.2.1 | 2m6s | 3m46s | 16m52s | 15.254 MB |
| 3.2.0 | 2m9s | 3m35s | 15m58s | 15.041 MB |
| 3.1.4 | 2m10s | 3m32s | 15m50s | 14.986 MB |
| 3.1.3 | 1m56s | 3m48s | 14m50s | 13.013 MB |
| EAExample | ||||
| 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage | |
| 3.3.1 | 9s | 8s | 2m19s | 1.886 MB |
| 3.3.0 | 8s | 9s | 2m23s | 1.563 MB |
| 3.2.1 | 8s | 7s | 2m19s | 1.573 MB |
| 3.2.0 | 8s | 8s | 2m6s | 1.288 MB |
| 3.1.4 | 7s | 8s | 2m6s | 1.748 MB |
| 3.1.3 | 8s | 8s | 2m6ss | 1.336 MB |
| Tagged Value Model | ||||
| 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage | |
| 3.3.1 | 1m7s | 18s | 2m53s | 7.810 MB |
| 3.3.0 | 1m11s | 14s | 3m5s | 7.799 MB |
| 3.2.1 | 1m10s | 13s | 3m12s | 7.271 MB |
| 3.2.0 | 1m6s | 18s | 2m44s | 7.133 MB |
| 3.1.4 | 1m5s | 13s | 2m47s | 6.974 MB |
| 3.1.3 | 1m6s | 13s | 2m46s | 6.764 MB |
| TV Model (Triple Size) | ||||
| 1) Loading models | 2) Premerge calculation | 3) Writing merge file | Max RAM usage | |
| 3.3.1 | 3m28s | 41s | 11m53s | 18.702 MB |
| 3.3.0 | 3m13s | 41s | 11m43s | 19.500 MB |
| 3.2.1 | 3m23s | 42s | 11m52s | 18.720 MB |
| 3.2.0 | 3m12s | 49s | 10m15s | 16.919 MB |
| 3.1.4 | 3m8s | 39s | 11m21s | 18.636 MB |
| 3.1.3 | 3m12s | 39s | 10m40s | 17.743 MB |
Remarks
The new test machine consumes significantly more memory for the "Large Model" and the "TV Model (Triple Size)" - even with LemonTree versions which have completed the same merges on the old test hardware with significantly less memory usage.
We currently assume that this is because the garbage collector isn't triggered as often on this machine because of excess available RAM - or, in layman's terms - if the RAM is available, LemonTree is going to use it. This speeds up the merge in total. However, we don't have any insight yet how much of the performance gain is due to the change in processing power, and how much is due to the additional RAM. This circumstance is still under investigation.