Synchronize ID generation to guarantee uniqueness and K-ordering

.github/workflows/ci.yml

@@ -16,7 +16,7 @@ jobs:
     - name: Environment Setup
       uses: actions/setup-dotnet@v4
       with:
-        dotnet-version: 8.0.x
+        dotnet-version: 9.0.x
     - name: Build
       run: dotnet build ./src/FlakeId.sln
     - name: Test

README.md

@@ -4,7 +4,7 @@
 
 Snowflake IDs were originally introduced by Twitter in 2010 as unique, decentralized IDs for Tweets. Their 8-byte size, ordered nature and guaranteed uniqueness make them ideal to use as resource identifiers. Since then, many applications at various scale have adopted Snowflake-esque identifiers.
 
-This repository contains an implementation of decentralized, K-ordered Snowflake IDs based on [the Discord Snowflake specification](https://discord.com/developers/docs/reference). The implementation heavily focuses on high-throughput, supporting upwards of 10.000 unique generations per second on commodity hardware.
+This repository contains an implementation of decentralized, K-ordered Snowflake IDs based on [the Discord Snowflake specification](https://discord.com/developers/docs/reference). The implementation heavily focuses on high-throughput, supporting upwards of 10.000 unique generations per second on commodity hardware (up to the theoretical limit of around 4 million per second).
 
 You can grab the latest stable version from NuGet:
 
@@ -40,13 +40,6 @@ Where the original Discord reference mentions worker ID and process ID, we subst
 thread and process ID respectively, as the combination of these two provide sufficient uniqueness, and they are
 the closest we can get to the original specification within the .NET ecosystem.
 
-The Increment component is a monotonically incrementing number, which is incremented every time a snowflake is generated.
-This is in contrast with some other flake-ish implementations, which only increment the counter any time a snowflake is 
-generated twice at the exact same instant in time. 
-
-We have opted to increment every time an ID is generated, rather than when two or more IDs are generated at the exact same millisecond. 
-The reasoning behind this is that it's vastly simpler - we can avoid locking altogether - and thus, more performant. It is also closer to Discord's implementation, which was referenced when designing this library.
-
 # Epoch
 
 The timestamp component is a delta from a predefined instant in time, this instant is known as the **epoch**.
@@ -79,9 +72,8 @@ When exposing your IDs to web clients, it is recommended to use the `ToBase64Str
 
 ## Performance
 
-We've benchmarked FlakeId on .NET 8 against [MassTransit's NewId](https://github.com/phatboyg/NewId) library, and [IdGen](https://github.com/RobThree/IdGen) both libraries are widely used. It is worth noting that NewId generates 128-bit integers.
+We've benchmarked FlakeId on .NET 8 against [IdGen](https://github.com/RobThree/IdGen), which is another implementation of Snowflake IDs in .NET. FlakeId performs significantly better.
 
-We've also included `Guid.NewGuid` as a baseline benchmark, as it is very well optimized, and arguably the most widely used identifier generator in .NET.
 
 ```
 BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
@@ -93,9 +85,7 @@ AMD Ryzen 5 5600X, 1 CPU, 12 logical and 6 physical cores
 
 | Method         | Mean        | Error     | StdDev     | Code Size |
 |--------------- |------------:|----------:|-----------:|----------:|
-| Single_FlakeId |    26.48 ns |  0.020 ns |   0.019 ns |     358 B |
-| Single_Guid    |    41.85 ns |  0.481 ns |   0.450 ns |     245 B |
-| Single_NewId   |    31.83 ns |  0.013 ns |   0.012 ns |     303 B |
+| Single_FlakeId |    349.2 ns | 6.58 ns   |  6.16 ns   |     358 B |
 | Single_IdGen   | 3,473.96 ns | 69.295 ns | 168.673 ns |     671 B |
 ```
 
@@ -104,22 +94,21 @@ In this benchmark, IdGen was configured to `SpinWait` in the event multiple IDs
 Below are the benchmark results for FlakeId running on multiple runtimes.
 
 ```
-BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
-AMD Ryzen 5 5600X, 1 CPU, 12 logical and 6 physical cores
-.NET SDK 8.0.201
-  [Host]   : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
-  .NET 5.0 : .NET 5.0.17 (5.0.1722.21314), X64 RyuJIT AVX2
-  .NET 6.0 : .NET 6.0.27 (6.0.2724.6912), X64 RyuJIT AVX2
-  .NET 7.0 : .NET 7.0.16 (7.0.1624.6629), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
-
-
-| Method         | Job      | Runtime  | Mean     | Error    | StdDev   | Code Size |
-|--------------- |--------- |--------- |---------:|---------:|---------:|----------:|
-| Single_FlakeId | .NET 5.0 | .NET 5.0 | 27.85 ns | 0.111 ns | 0.103 ns |     254 B |
-| Single_FlakeId | .NET 6.0 | .NET 6.0 | 26.37 ns | 0.056 ns | 0.053 ns |     215 B |
-| Single_FlakeId | .NET 7.0 | .NET 7.0 | 26.72 ns | 0.211 ns | 0.176 ns |     209 B |
-| Single_FlakeId | .NET 8.0 | .NET 8.0 | 26.56 ns | 0.085 ns | 0.071 ns |     358 B |
+BenchmarkDotNet v0.13.12, macOS 15.6.1 (24G90) [Darwin 24.6.0]
+Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
+.NET SDK 9.0.100
+  [Host]   : .NET 8.0.2 (8.0.224.6711), Arm64 RyuJIT AdvSIMD
+  .NET 7.0 : .NET 7.0.11 (7.0.1123.42427), Arm64 RyuJIT AdvSIMD
+  .NET 8.0 : .NET 8.0.2 (8.0.224.6711), Arm64 RyuJIT AdvSIMD
+  .NET 9.0 : .NET 9.0.0 (9.0.24.52809), Arm64 RyuJIT AdvSIMD
+
+
+| Method         | Job      | Runtime  | Mean     | Error   | StdDev   |
+|--------------- |--------- |--------- |---------:|--------:|---------:|
+| Single_FlakeId | .NET 7.0 | .NET 7.0 | 354.0 ns | 6.87 ns | 13.89 ns |
+| Single_FlakeId | .NET 8.0 | .NET 8.0 | 356.6 ns | 7.17 ns |  9.81 ns |
+| Single_FlakeId | .NET 9.0 | .NET 9.0 | 349.2 ns | 6.58 ns |  6.16 ns |
+
 ```
 
 ## Issues

src/FlakeId.Benchmarks/FlakeId.Benchmarks.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFrameworks>net8.0;net7.0;net6.0</TargetFrameworks>
+    <TargetFrameworks>net6.0;net7.0;net8.0;net9.0</TargetFrameworks>
   </PropertyGroup>
 
   <ItemGroup>

src/FlakeId.Benchmarks/FlakeIdMultipleRuntimesBenchmarks.cs

@@ -1,5 +1,6 @@
 using BenchmarkDotNet.Attributes;
 using BenchmarkDotNet.Diagnostics.Windows.Configs;
+using BenchmarkDotNet.Environments;
 using BenchmarkDotNet.Jobs;
 
 namespace FlakeId.Benchmarks
@@ -8,8 +9,11 @@ namespace FlakeId.Benchmarks
     [SimpleJob(RuntimeMoniker.Net60)]
     [SimpleJob(RuntimeMoniker.Net70)]
     [SimpleJob(RuntimeMoniker.Net80)]
+    [SimpleJob(RuntimeMoniker.Net90)]
+#if _WINDOWS
     [DisassemblyDiagnoser]
     [InliningDiagnoser(true, null)]
+#endif
     public class FlakeIdMultipleRuntimesBenchmarks
     {
         [Benchmark]

src/FlakeId.Benchmarks/IdCreationBenchmarks.cs

@@ -6,8 +6,10 @@
 
 namespace FlakeId.Benchmarks
 {
+#if _WINDOWS
     [DisassemblyDiagnoser]
     [InliningDiagnoser(true, null)]
+#endif
     public class IdCreationBenchmarks
     {
         private static readonly IdGenerator s_idGenerator = new IdGenerator(10,
@@ -30,7 +32,7 @@ public void Single_NewId()
         {
             NewId.Next();
         }
-        
+
         [Benchmark]
         public void Single_IdGen()
         {

src/FlakeId.Tests/ConcurrencyTests.cs

@@ -0,0 +1,125 @@
+using System;
+using System.Collections.Concurrent;
+using System.Collections.Generic;
+using System.Reflection;
+using System.Threading;
+using System.Threading.Tasks;
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+
+namespace FlakeId.Tests;
+
+[TestClass]
+public class ConcurrencyTests
+{
+    private const int ThreadIdBits = 5;
+    private const int ProcessIdBits = 5;
+    private const int IncrementBits = 12;
+    private const int TimestampShift = ThreadIdBits + ProcessIdBits + IncrementBits;
+    private const int IncrementMask = (1 << IncrementBits) - 1;
+
+    private static readonly FieldInfo s_prevIdField =
+        typeof(Id).GetField("s_prevId", BindingFlags.NonPublic | BindingFlags.Static);
+
+    [TestCleanup]
+    public void TestCleanup()
+    {
+        Assert.IsNotNull(s_prevIdField, "Could not find the private static field 's_prevId' for testing.");
+        s_prevIdField.SetValue(null, 0L);
+    }
+
+    [TestMethod]
+    public void Create_ShouldGenerateNonZeroId()
+    {
+        long id = Id.Create();
+
+        Assert.AreNotEqual(0L, id);
+    }
+
+    [TestMethod]
+    public void Create_ShouldGenerateMonotonicallyIncreasingIds()
+    {
+        const int idCount = 500_000;
+        long[] ids = new long[idCount];
+
+        for (int i = 0; i < idCount; i++)
+        {
+            ids[i] = Id.Create();
+        }
+
+        for (int i = 1; i < idCount; i++)
+        {
+            Assert.IsTrue(ids[i] > ids[i - 1], $"ID at index {i} was not greater than the previous one.");
+        }
+    }
+
+    [TestMethod]
+    public void Create_ShouldGenerateUniqueIds_WhenCalledConcurrently()
+    {
+        const int idsPerTask = 100_000;
+        int taskCount = Environment.ProcessorCount;
+        int totalIds = idsPerTask * taskCount;
+        var generatedIds = new ConcurrentBag<long>();
+
+        var tasks = new List<Task>();
+
+        for (int i = 0; i < taskCount; i++)
+        {
+            tasks.Add(Task.Run(() =>
+            {
+                for (int j = 0; j < idsPerTask; j++)
+                {
+                    generatedIds.Add(Id.Create());
+                }
+            }));
+        }
+
+        Task.WaitAll(tasks.ToArray());
+
+        Assert.AreEqual(totalIds, generatedIds.Count, "The number of generated IDs does not match the expected total.");
+
+        var distinctIds = new HashSet<long>(generatedIds);
+        Assert.AreEqual(totalIds, distinctIds.Count, "Duplicate IDs were found when generating concurrently.");
+    }
+
+    [TestMethod]
+    public void Create_ShouldThrow_WhenClockMovesBackwards()
+    {
+        Assert.IsNotNull(s_prevIdField, "Could not find the private static field 's_prevId' for testing.");
+
+        long baseId = Id.Create();
+        long baseTimestamp = baseId >> TimestampShift;
+
+        long futureTimestamp = baseTimestamp + 5000;
+        long futureId = futureTimestamp << TimestampShift;
+        s_prevIdField.SetValue(null, futureId);
+
+        var ex = Assert.ThrowsException<InvalidOperationException>(() => Id.Create());
+        StringAssert.Contains(ex.Message, "Clock shifted backwards");
+    }
+
+    [TestMethod]
+    public void Increment_ShouldResetToZero_OnNewMillisecond()
+    {Assert.IsNotNull(s_prevIdField, "Could not find the private static field 's_prevId' for testing.");
+
+        long baseId = Id.Create();
+        long baseTimestamp = baseId >> TimestampShift;
+
+        // create a fake "previous ID" that is at the same timestamp but has a high increment.
+        // we preserve the other parts of the ID (thread, process) to make the state realistic.
+        const int highIncrement = 4000;
+        long otherParts = baseId & ~((1L << 42) - 1); // masks out the timestamp
+        long prevId = otherParts | (baseTimestamp << TimestampShift) | highIncrement;
+
+        s_prevIdField.SetValue(null, prevId);
+
+        Thread.Sleep(5);
+
+        long newIdValue = Id.Create();
+
+        long newTimestamp = newIdValue >> TimestampShift;
+        int newIncrement = (int)(newIdValue & IncrementMask);
+
+        Assert.IsTrue(newTimestamp > baseTimestamp, "Timestamp should have advanced.");
+        Assert.AreEqual(0, newIncrement, "Increment should reset to 0 on a new millisecond.");
+    }
+}

src/FlakeId.Tests/FlakeId.Tests.csproj

@@ -1,9 +1,10 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFrameworks>net8.0</TargetFrameworks>
 
     <IsPackable>false</IsPackable>
+
+    <TargetFramework>net9.0</TargetFramework>
   </PropertyGroup>
 
   <ItemGroup>

src/FlakeId.Tests/IdTests.cs

@@ -90,14 +90,5 @@ public void Id_ToString()
 
             Assert.AreEqual(id.ToString(), id.ToString());
         }
-
-        [TestMethod]
-        public void Id_ContainsTimeZoneComponent()
-        {
-            DateTimeOffset timeStamp = new DateTimeOffset(2020, 1, 1, 0, 0, 0, TimeSpan.FromHours(7));
-            Id id = Id.Create(timeStamp.ToUnixTimeMilliseconds());
-
-            Assert.AreEqual(timeStamp.ToUnixTimeMilliseconds(), id.ToUnixTimeMilliseconds());
-        }
     }
 }

src/FlakeId/FlakeId.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFrameworks>net8.0;netstandard2.0</TargetFrameworks>
+    <TargetFrameworks>netstandard2.0;net9.0</TargetFrameworks>
     <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
     <Description>Discord inspired, Twitter like implementation of Snowflake IDs, focused on performance. Snowflakes are 64 bit, highly optimized, decentralized, K-ordered, unique identifiers.</Description>
     <RepositoryUrl>https://github.com/aevitas/flakeid</RepositoryUrl>
@@ -15,6 +15,12 @@
     <PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
   </PropertyGroup>
 
+  <ItemGroup>
+    <AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleToAttribute">
+      <_Parameter1>FlakeId.Tests</_Parameter1>
+    </AssemblyAttribute>
+  </ItemGroup>
+
   <ItemGroup>
     <None Include="..\..\assets\snowflake-96.png">
       <Pack>True</Pack>