If I set 175% scaling in Gnome Settings, the value is saved as 1.7518248558044434 in ~/.config/monitors.xml:
<monitors version="2">
<configuration>
<logicalmonitor>...
TBF the error can become that big if you do a bunch of unstable operations (i.e. operations that continue to increase the relative error), though that’s probably not what is happening here.
To get to 0.01 error, you’d need to add up trillions of trillions of floating point errors. It will not happen solely because of floating point unless you’re doing such crazy math that you shouldn’t be using primitives in the first place.
As the answer in the link explains, it’s adjustment of your scaling factor to the nearest whole pixel, plus a loss of precision rounding to/from single/double floating point values.
So I’m not really sure of the point of this post. It’s not a question, as the link quite effectively answers it. It’s more just “here’s why your scaling factor looks weird in your gnome config file”, and it’s primarily the first reason - rounding to whole pixels.
Floating point error? Yeaahhh no. No. Just… no. That is NEVER as big as 0.01 unless the number is also insanely massive.
The error is relative in scale. It’s not magically significant fractions off.
TBF the error can become that big if you do a bunch of unstable operations (i.e. operations that continue to increase the relative error), though that’s probably not what is happening here.
To get to 0.01 error, you’d need to add up trillions of trillions of floating point errors. It will not happen solely because of floating point unless you’re doing such crazy math that you shouldn’t be using primitives in the first place.
That’s why I said unstable operations. Addition is considered a stable operation (for values with the same sign)
As the answer in the link explains, it’s adjustment of your scaling factor to the nearest whole pixel, plus a loss of precision rounding to/from single/double floating point values.
So I’m not really sure of the point of this post. It’s not a question, as the link quite effectively answers it. It’s more just “here’s why your scaling factor looks weird in your gnome config file”, and it’s primarily the first reason - rounding to whole pixels.
0.001, but still