So there is a way to get the zoom to work:
The approach taken here is to create graphical objects (LineElement, RectangleElement, etc) that are scaled at the origin, but not zoomed or translated to the center of the draw area. These last two parts of the transform are then added to the graphics transform.
One downside of altering the transform in the graphics is that if we want to draw extra stuff on the panel (like the detail string "Zoom = x, Scale = y" in the picture above) the original transform has to be captured before drawing, then restored after.
For example, see this commit: