feat: benchmark against Luerl and C Lua

[davydog187]

This PR adds some benchmarks for getting an initial performance baseline. We have no implemented garbage collection or any performance optimizations yet, so there should be a lot of low-hanging fruit for improvement. We are slower than Luerl across the board. Unclear if its related to our choice to build a register-based vm vs a stack-based vm yet.

Environment

Operating System: macOS
CPU Information: Apple M1 Pro
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.19.4
Erlang 27.3.4.6
JIT enabled: true

Closures

Strange that eval would be signficantly faster here, it also seems that our closures are keeping around a lot more data

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       52.10 K       19.19 μs    ±76.36%       15.50 μs       53.63 μs
luerl                  1.62 K      615.41 μs    ±34.72%      545.50 μs     1197.02 μs
lua (eval)             1.18 K      850.62 μs    ±11.68%      815.13 μs     1100.65 μs
lua (chunk)            0.84 K     1187.88 μs    ±98.92%     1120.75 μs     1922.02 μs

Comparison: 
C Lua (luaport)       52.10 K
luerl                  1.62 K - 32.06x slower +596.22 μs
lua (eval)             1.18 K - 44.32x slower +831.43 μs
lua (chunk)            0.84 K - 61.89x slower +1168.68 μs

Memory usage statistics:

Name                    average  deviation         median         99th %
C Lua (luaport)      0.00016 MB     ±0.00%     0.00016 MB     0.00016 MB
luerl                   1.91 MB     ±0.00%        1.91 MB        1.91 MB
lua (eval)              8.50 MB     ±0.08%        8.50 MB        8.52 MB
lua (chunk)             8.50 MB     ±0.15%        8.50 MB        8.53 MB

Comparison: 
C Lua (luaport)      0.00016 MB
luerl                   1.91 MB - 11898.10x memory usage +1.91 MB
lua (eval)              8.50 MB - 53039.94x memory usage +8.50 MB
lua (chunk)             8.50 MB - 53077.07x memory usage +8.50 MB

Fibonacci

Crazy amount of memory usage here, clearly recursion needs some optimization. Surprising that it is twice as slow

Name                      ips        average  deviation         median         99th %
C Lua (luaport)        139.51      0.00717 s     ±2.86%      0.00711 s      0.00776 s
luerl                    0.94         1.06 s     ±2.91%         1.05 s         1.15 s
lua (eval)               0.49         2.04 s     ±4.62%         2.00 s         2.20 s
lua (chunk)              0.36         2.74 s     ±5.66%         2.80 s         2.84 s

Comparison: 
C Lua (luaport)        139.51
luerl                    0.94 - 148.49x slower +1.06 s
lua (eval)               0.49 - 284.63x slower +2.03 s
lua (chunk)              0.36 - 382.23x slower +2.73 s

Memory usage statistics:

Name               Memory usage
C Lua (luaport)      0.00000 GB
luerl                   2.45 GB - 15689106.52x memory usage +2.45 GB
lua (eval)             15.20 GB - 97157200.71x memory usage +15.20 GB
lua (chunk)            15.20 GB - 97165186.19x memory usage +15.20 GB

OOP

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       82.16 K       12.17 μs   ±235.61%       10.17 μs       39.29 μs
luerl                  5.81 K      172.07 μs   ±124.03%      151.04 μs      357.18 μs
lua (eval)             4.87 K      205.48 μs    ±77.63%      170.29 μs      515.09 μs
lua (chunk)            4.09 K      244.26 μs   ±112.67%      208.63 μs      581.54 μs

Comparison: 
C Lua (luaport)       82.16 K
luerl                  5.81 K - 14.14x slower +159.90 μs
lua (eval)             4.87 K - 16.88x slower +193.31 μs
lua (chunk)            4.09 K - 20.07x slower +232.09 μs

Memory usage statistics:

Name                    average  deviation         median         99th %
C Lua (luaport)      0.00020 MB     ±0.00%     0.00020 MB     0.00020 MB
luerl                   0.37 MB     ±0.00%        0.37 MB        0.37 MB
lua (eval)              1.32 MB     ±0.02%        1.32 MB        1.32 MB
lua (chunk)             1.32 MB     ±0.00%        1.32 MB        1.32 MB

Comparison: 
C Lua (luaport)      0.00020 MB
luerl                   0.37 MB - 1843.58x memory usage +0.37 MB
lua (eval)              1.32 MB - 6643.18x memory usage +1.32 MB
lua (chunk)             1.32 MB - 6645.65x memory usage +1.32 MB

Strings

String Concatenation via table.concat (n=100)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       76.16 K       13.13 μs   ±667.66%       10.46 μs       44.63 μs
lua (chunk)           21.83 K       45.82 μs    ±71.35%       44.08 μs       69.46 μs
lua (eval)            19.64 K       50.92 μs    ±75.82%       47.38 μs       86.92 μs
luerl                 16.31 K       61.30 μs   ±165.73%       55.04 μs      104.13 μs

Comparison: 
C Lua (luaport)       76.16 K
lua (chunk)           21.83 K - 3.49x slower +32.69 μs
lua (eval)            19.64 K - 3.88x slower +37.79 μs
luerl                 16.31 K - 4.67x slower +48.17 μs

Memory usage statistics:

Name               Memory usage
C Lua (luaport)         0.23 KB
lua (chunk)           362.68 KB - 1600.79x memory usage +362.45 KB
lua (eval)            368.18 KB - 1625.07x memory usage +367.95 KB
luerl                 173.28 KB - 764.83x memory usage +173.05 KB

String Formatting via string.format (n=100)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       65.41 K       15.29 μs   ±251.99%       13.08 μs       44.08 μs
luerl                  6.43 K      155.60 μs    ±15.63%      149.00 μs      242.68 μs
lua (chunk)            4.14 K      241.63 μs    ±77.24%      212.75 μs      526.65 μs
lua (eval)             4.12 K      242.52 μs   ±109.29%      212.46 μs      544.14 μs

Comparison: 
C Lua (luaport)       65.41 K
luerl                  6.43 K - 10.18x slower +140.32 μs
lua (chunk)            4.14 K - 15.81x slower +226.35 μs
lua (eval)             4.12 K - 15.86x slower +227.23 μs

Memory usage statistics:

Name               Memory usage
C Lua (luaport)      0.00022 MB
luerl                   0.59 MB - 2647.41x memory usage +0.59 MB
lua (chunk)             1.01 MB - 4577.52x memory usage +1.01 MB
lua (eval)              1.02 MB - 4607.03x memory usage +1.02 MB

Tables

Table Build (n=500)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       93.49 K       10.70 μs   ±151.15%        9.13 μs       34.17 μs
luerl                  7.72 K      129.49 μs   ±119.06%      118.63 μs      227.83 μs
lua (chunk)            7.39 K      135.28 μs    ±23.26%      127.21 μs      241.92 μs
lua (eval)             6.44 K      155.26 μs    ±71.39%      143.46 μs      314.71 μs

Comparison: 
C Lua (luaport)       93.49 K
luerl                  7.72 K - 12.11x slower +118.79 μs
lua (chunk)            7.39 K - 12.65x slower +124.58 μs
lua (eval)             6.44 K - 14.51x slower +144.56 μs

Memory usage statistics:

Name                    average  deviation         median         99th %
C Lua (luaport)      0.00016 MB     ±0.00%     0.00016 MB     0.00016 MB
luerl                   0.53 MB     ±0.00%        0.53 MB        0.53 MB
lua (chunk)             1.10 MB     ±0.00%        1.10 MB        1.10 MB
lua (eval)              1.11 MB     ±0.00%        1.11 MB        1.11 MB

Comparison: 
C Lua (luaport)      0.00016 MB
luerl                   0.53 MB - 3309.29x memory usage +0.53 MB
lua (chunk)             1.10 MB - 6871.84x memory usage +1.10 MB
lua (eval)              1.11 MB - 6932.10x memory usage +1.11 MB

Table Sort (n=500)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       10.52 K       95.07 μs   ±267.44%       82.46 μs      187.33 μs
luerl                  6.28 K      159.36 μs   ±150.83%      138.75 μs      333.43 μs
lua (eval)             2.71 K      368.51 μs    ±37.70%      343.31 μs      689.87 μs
lua (chunk)            2.18 K      458.56 μs   ±137.77%      403.31 μs     1014.46 μs

Comparison: 
C Lua (luaport)       10.52 K
luerl                  6.28 K - 1.68x slower +64.29 μs
lua (eval)             2.71 K - 3.88x slower +273.44 μs
lua (chunk)            2.18 K - 4.82x slower +363.49 μs

Memory usage statistics:

Name               Memory usage
C Lua (luaport)      0.00016 MB
luerl                   0.63 MB - 3957.29x memory usage +0.63 MB
lua (eval)              1.93 MB - 12019.86x memory usage +1.93 MB
lua (chunk)             1.86 MB - 11637.00x memory usage +1.86 MB

Table Iterate/Sum (n=500)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       89.41 K       11.19 μs   ±284.32%        9.25 μs       40.21 μs
luerl                  4.88 K      204.84 μs   ±110.31%      185.54 μs      384.03 μs
lua (chunk)            4.49 K      222.88 μs    ±76.20%      205.58 μs      543.46 μs
lua (eval)             4.28 K      233.63 μs    ±27.68%      209.08 μs      484.56 μs

Comparison: 
C Lua (luaport)       89.41 K
luerl                  4.88 K - 18.31x slower +193.66 μs
lua (chunk)            4.49 K - 19.93x slower +211.70 μs
lua (eval)             4.28 K - 20.89x slower +222.44 μs

Memory usage statistics:

Name               Memory usage
C Lua (luaport)      0.00016 MB
luerl                   0.72 MB - 4479.38x memory usage +0.72 MB
lua (chunk)             2.19 MB - 13666.52x memory usage +2.19 MB
lua (eval)              2.20 MB - 13714.14x memory usage +2.20 MB

Table Map + Reduce (n=500)

Name                      ips        average  deviation         median         99th %
C Lua (luaport)       82.44 K       12.13 μs    ±97.46%       10.58 μs       36.29 μs
luerl                  2.41 K      415.35 μs    ±61.61%      357.33 μs      713.10 μs
lua (eval)             2.24 K      445.63 μs    ±14.90%      420.08 μs      702.37 μs
lua (chunk)            2.16 K      462.46 μs   ±140.85%      426.58 μs      758.93 μs

Comparison: 
C Lua (luaport)       82.44 K
luerl                  2.41 K - 34.24x slower +403.22 μs
lua (eval)             2.24 K - 36.74x slower +433.50 μs
lua (chunk)            2.16 K - 38.13x slower +450.33 μs

Memory usage statistics:

Name               Memory usage
C Lua (luaport)      0.00016 MB
luerl                   1.31 MB - 8145.43x memory usage +1.30 MB
lua (eval)              3.96 MB - 24710.76x memory usage +3.96 MB
lua (chunk)             3.95 MB - 24650.76x memory usage +3.95 MB