Broken resizing with rectangle tool when scale ≈0

[0HyperCube]

Reproduction

• New document (with artboard)
• Draw rectangle from the centre of the artboard with good width and height
• Use the handles so the rectangle has approximately no height (using snapping)
• Drag the handles to give height again
• Observe strange jitter

Also occurs with the select tool, ellipse tool, etc.

Video

broken_resizing_rectangle_tool.mp4

Fix

The bound transform is set to some extremely small value e.g. 1e-15. This makes the floating point imprecision cause jittering during the transform calculations (which aren't particularly optimised to keep the precision). A fix is simply to scale up the bounding box transform so it has always a scale of 1. Stretch parallel to x or y axis leave the resulting box invariant. Also there was some thing to detect if the box was flattened. However this happened in the bounds space rather than the viewport space. In the diff switched to viewport space as it probably makes more sense?

diff --git a/editor/src/messages/tool/common_functionality/shapes/shape_utility.rs b/editor/src/messages/tool/common_functionality/shapes/shape_utility.rs
index 5f0d6da01..c66a7b1fc 100644
--- a/editor/src/messages/tool/common_functionality/shapes/shape_utility.rs
+++ b/editor/src/messages/tool/common_functionality/shapes/shape_utility.rs
@@ -266,6 +266,8 @@ pub fn transform_cage_overlays(document: &DocumentMessageHandler, tool_data: &mu
 		transform.matrix2 += DMat2::IDENTITY * 1e-4; // TODO: Is this the cleanest way to handle this?
 		transform_tampered = true;
 	}
+	// Ensure the transform scale is set to one (only view rotation/skew). This gets around floating point imprecision causing jumpiness.
+	transform = transform * DAffine2::from_scale(DVec2::new(transform.x_axis.x, transform.y_axis.y)).inverse();
 
 	let bounds = document
 		.network_interface
diff --git a/editor/src/messages/tool/common_functionality/transformation_cage.rs b/editor/src/messages/tool/common_functionality/transformation_cage.rs
index e6c265e3c..28c32983b 100644
--- a/editor/src/messages/tool/common_functionality/transformation_cage.rs
+++ b/editor/src/messages/tool/common_functionality/transformation_cage.rs
@@ -639,12 +639,14 @@ impl BoundingBoxManager {
 
 	/// Determine if these bounds are flat ([`TransformCageSizeCategory::Flat`]), which means that the width and/or height is essentially zero and the bounds are a line with effectively no area. This can happen on actual lines (axis-aligned, i.e. drawn horizontally or vertically) or when an element is scaled to zero in X or Y. A flat transform cage can still be rotated by a transformation, but its local space remains flat.
 	fn is_bounds_flat(&self) -> bool {
-		(self.bounds[0] - self.bounds[1]).abs().cmple(DVec2::splat(MAX_LENGTH_FOR_NO_WIDTH_OR_HEIGHT)).any()
+		let viewport_size = self.transform.transform_vector2(self.bounds[0] - self.bounds[1]);
+		viewport_size.abs().cmple(DVec2::splat(MAX_LENGTH_FOR_NO_WIDTH_OR_HEIGHT)).any()
 	}
 
 	/// Determine if these bounds are point ([`TransformCageSizeCategory::Point`]), which means that the width and height are essentially zero and the bounds are a point with no area. This can happen on points when an element is scaled to zero in both X and Y, or if an element is just a single anchor point. A point transform cage cannot be rotated by a transformation, and its local space remains a point.
 	fn is_bounds_point(&self) -> bool {
-		(self.bounds[0] - self.bounds[1]).abs().cmple(DVec2::splat(MAX_LENGTH_FOR_NO_WIDTH_OR_HEIGHT)).all()
+		let viewport_size = self.transform.transform_vector2(self.bounds[0] - self.bounds[1]);
+		viewport_size.abs().cmple(DVec2::splat(MAX_LENGTH_FOR_NO_WIDTH_OR_HEIGHT)).all()
 	}
 
 	/// Determine if the given point in viewport space falls within the bounds of `self`.
diff --git a/editor/src/messages/tool/tool_messages/select_tool.rs b/editor/src/messages/tool/tool_messages/select_tool.rs
index d559a595b..b81363b83 100644
--- a/editor/src/messages/tool/tool_messages/select_tool.rs
+++ b/editor/src/messages/tool/tool_messages/select_tool.rs
@@ -768,6 +768,9 @@ impl Fsm for SelectToolFsmState {
 					transform_tampered = true;
 				}
 
+				// Ensure the transform scale is set to one (only view rotation/skew). This gets around floating point imprecision causing jumpiness.
+				transform = transform * DAffine2::from_scale(DVec2::new(transform.x_axis.x, transform.y_axis.y)).inverse();
+
 				let bounds = document
 					.network_interface
 					.selected_nodes()