When you store certain types of fractions in a Decimal value, their value is approximated, because they can not be exactly represented in the binary format used by computers. While we humans have no problems with 2 / 3 * 3 equaling 2, ordinary programming languages can't express all numbers precisely.
Let's take a look at another example. If we take sqrt(4) * sqrt(4), we get the answer 4. That's because the square root of a number is the number that, when multiplied by itself, yields that number. However, if we now take a look at sqrt(2) * sqrt(2), we end up with a value like 2.0000000000000004, instead of the logical conclusion of 2. This is because irrational numbers can not be expressed in the binary format native to decimal; we end up with only a close approximation.
The general rule for performing math in a programming language is to perform division last whenever possible, even if it means doing some algebra to figure out how to do so. Whenever you perform division, you must assume that some loss of precision will occur. The more times you divide, the greater loss of precision may occur. Generally, this means you'll need to reset the precision of the end result using Decimal.setScale. Additional precision will be lost.
Finally, note that in Apex Code, writing (5 / 3) * 3
literally results in the value of 3, not 4.99999999999999999. This is because those values are interpreted as integers, so you get integer division. To yield a result closer to what you observed, you'd need to introduce a decimal for implicit upcasting:
System.debug((5 / 3) * 3); // 3
System.debug((5.0 / 3) * 3); // 5.00000000000000000000000000000001
This is the exact same behavior as observed in Java.