The "answer" you know is wrong. The oxygen atoms aren't even close to balanced.

Starting from the beginning:

C3H5N3O9 → CO2 + N2 + H2O + O2

Balance the carbon:

C3H5N3O9 → 3 CO2 + N2 + H2O + O2

Balance the nitrogen:

C3H5N3O9 → 3 CO2 + 1.5 N2 + H2O + O2

Balance the hydrogen:

C3H5N3O9 → 3 CO2 + 1.5 N2 + 2.5 H2O + O2

Multiply everything by 2 to get rid of the fractions:

2 C3H5N3O9 → 6 CO2 + 3 N2 + 5 H2O + 2 O2

Now you've got 2 x 9 = 18 oxygens on the left, and 6 x 2 + 5 + 2 x 2 = 21 oxygens on the right.

You can't change the left side or CO2 or H2O without messing up the balances for C or H, so you must adjust the O2 on the right, which has 21 - 18 = 3 too many oxygen atoms, so subtract 3 atoms of O on the right:

2 C3H5N3O9 → 6 CO2 + 3 N2 + 5 H2O + 0.5 O2

Now the equation is completely balanced, but you still have a forbidden fraction, so multiply everything by 2 again:

4 C3H5N3O9 → 12 CO2 + 6 N2 + 10 H2O + O2