Skip to content
Advertisement

How to force Postgres to calculate with numeric precision

I have a .net core application using dapper with Postgresql. Windows, PostgreSQL 12.0, compiled by Visual C++ build 1914, 64-bit

When i do simple calculation in the select-statement, then i get different results if i do not provide fractions:

select round(25358-25358/105*100,0);

1258

select round(25358.0-25358.0/105*100,0);

1208

What’s the reason and what’s the best way to get correct numeric result?

Advertisement

Answer

Postgres, as many other databases (SQL Server for example), does integer division: when both operands are integers, the result is an integer as well (the decimal part is just dropped).

Both operands to 25358/105 are integers, so the result is 241.

On the other hand, 25358.0/105 has a decimal operand, so the result is decimal: 241.504....

The rest of the computation is affected accordingly.

select 25358/105 integer_divison, 25358.0/105 decimal_division
integer_divison |     decimal_division
--------------: | -------------------:
            241 | 241.5047619047619048
User contributions licensed under: CC BY-SA
3 People found this is helpful
Advertisement