Xtykutl

$$begin{cases}
2x_1+5x_2-8x_3=8\
4x_1+3x_2-9x_3=9\
2x_1+3x_2-5x_3=7\
x_1+8x_2-7x_3=12
end{cases}$$

From my elementary row operations, I get that it has no solution. (Row operations are to be read from top to bottom.)

$$left[begin{array}{ccc|c}
2 & 5 & -8 & 8 \
4 & 3 & -9 & 9 \
2 & 3 & -5 & 7 \
1 & 8 & -7 & 12
end{array}right]
overset{overset{large{R_1to R_1-R_3}}{{R_2to R_2-2R_3}}}{overset{R_3to R_3-2R_4}{largelongrightarrow}}
left[begin{array}{ccc|c}
0 & 2 & -3 & 1 \
0 & -3 & 1 & -5 \
0 & -13 & 9 & -17 \
1 & 8 & -7 & 12
end{array}right]
overset{overset{large{R_3,leftrightarrow, R_4}}{R_2,leftrightarrow, R_3}}{overset{R_1,leftrightarrow,R_2}{largelongrightarrow}}
left[begin{array}{ccc|c}
1 & 8 & -7 & 12 \
0 & 2 & -3 & 1 \
0 & -3 & 1 & -5 \
0 & -13 & 9 & -17
end{array}right]$$

$$overset{R_4to R_4-R_3}{largelongrightarrow}
left[begin{array}{ccc|c}
1 & 8 & -7 & 12 \
0 & 2 & -3 & 1 \
0 & -3 & 1 & -5 \
0 & 10 & 8 & -12
end{array}right]
overset{overset{large{R_3to R_3+R_2}}{R_4to R_4-5R_2}}{largelongrightarrow}
left[begin{array}{ccc|c}
1 & 8 & -7 & 12 \
0 & 2 & -3 & 1 \
0 & -1 & -2 & -4 \
0 & 0 & 23 & -17
end{array}right]
overset{overset{large{R_2to R_2+2R_3}}{R_3to-R_3}}{largelongrightarrow}$$

$$left[begin{array}{ccc|c}
1 & 8 & -7 & 12 \
0 & 0 & -7 & -7 \
0 & 1 & 2 & 4 \
0 & 0 & 23 & -17 \
end{array}right]
overset{R_2,leftrightarrow,R_3}{largelongrightarrow}
left[begin{array}{ccc|c}
1 & 8 & -7 & 12 \
0 & 1 & 2 & 4 \
0 & 0 & -7 & -7 \
0 & 0 & 23 & -17 \
end{array}right]$$

However, the answer in the book $(3, 2, 1)$ fits the system.

Was there an arithmetical mistake, or do I misunderstand something fundamentally?

Hint: Try inputing the solution $(3,2,1)$ into every step. That will allow you to identify the step where you went wrong.

You do (in the third matrix): $$L3-L4=(0, -3, 1 mid -5)-(0, -13, 9 mid -19)=(0, 10, -8 mid 12)$$ but you have $(0, 10, 8 mid -12)$ instead.

Note 1: Referring to 5xum's answer, in case you don't know the final answer, any solution will do, e.g.:
$$(x_1,x_2,x_3)=(0,0,-1);(0,0,-1);(1,0,-1);(12,0,0) text{(respectively)}$$
Note 2: In step $3$, you can reduce column $3$ instead of column $2$:
$$left[
begin{array}{ccc|c}
1&8&-7&12\
0&2&-3&1\
0&-3&1&-5\
0&-13&9&-17\
end{array}
right] Rightarrow left[
begin{array}{ccc|c}
1&8&-7&12\
0&-7&0&-14\
0&-3&1&-5\
0&14&0&28\
end{array}
right] stackrel{frac{-R_2}{7};\ frac{-3R_2}{7}+R_3}{=}Rightarrow left[
begin{array}{ccc|c}
1&8&-7&12\
0&1&0&2\
0&0&1&1\
0&14&0&28\
end{array}
right]$$
The second and fourth equations are dependent, therefore they produce the same solution $x_2=2$. You can finish the problem now.

Note 3. When you can not find your mistake (sometimes the brain gets blocked/accustommed and can not see obvious mistakes), leave it for some time (1 hour, 1 day) and return with fresh mind. You will be surprised to easily spot the error.

Usually you don't know the solution of your equation. So you have to use other methods for check your calculation. One way is to add checksums to your calculation.

Don't start your calculations with the original matrix

$$left[begin{array}{ccc|c}
2 & 5 & -8 & 8 \
4 & 3 & -9 & 9 \
2 & 3 & -5 & 7 \
1 & 8 & -7 & 12
end{array}right]
$$

but start with the matrix augmented by a column at the right. An element of this column is equal to the sum of the other elements of this row. So the augmented matrix is

$$left[begin{array}{ccc|c|c}
2 & 5 & -8 & 8 & 7 \
4 & 3 & -9 & 9 & 7 \
2 & 3 & -5 & 7 & 7 \
1 & 8 & -7 & 12 & 14
end{array}right]
$$

You do the same row operations as with the original matrices.

So instead of

$$left[begin{array}{ccc|c}
2 & 5 & -8 & 8 \
4 & 3 & -9 & 9 \
2 & 3 & -5 & 7 \
1 & 8 & -7 & 12
end{array}right]
overset{overset{large{R_1to R_1-R_3}}{{R_2to R_2-2R_3}}}{overset{R_3to R_3-2R_4}{largelongrightarrow}}
left[begin{array}{ccc|c}
0 & 2 & -3 & 1 \
0 & -3 & 1 & -5 \
0 & -13 & 9 & -17 \
1 & 8 & -7 & 12
end{array}right]$$

you do

$$left[begin{array}{ccc|c|c}
2 & 5 & -8 & 8 & 7 \
4 & 3 & -9 & 9 & 7 \
2 & 3 & -5 & 7 & 7 \
1 & 8 & -7 & 12 & 14
end{array}right]
overset{overset{large{R_1to R_1-R_3}}{{R_2to R_2-2R_3}}}{overset{R_3to R_3-2R_4}{largelongrightarrow}}
left[begin{array}{ccc|c}
0 & 2 & -3 & 1 & 0 \
0 & -3 & 1 & -5 & -7 \
0 & -13 & 9 & -17 & -21 \
1 & 8 & -7 & 12 & 14
end{array}right]$$

Then you check if for the matrix you calculated the checksum is still valid.
Here the checksum is correct.

Now let's investigate the step where the error occurs. We get

$$left[begin{array}{ccc|c|c}
1 & 8 & -7 & 12 & 14 \
0 & 2 & -3 & 1 & 0 \
0 & -3 & 1 & -5 & -7 \
0 & -13 & 9 & -17 & -21
end{array}right]
overset{R_4to R_4-R_3}{largelongrightarrow}
left[begin{array}{ccc|c | c}
1 & 8 & -7 & 12 & 14\
0 & 2 & -3 & 1 & 0\
0 & -3 & 1 & -5 & -7\
0 & 10 & 8 & -12 & -14
end{array}right]
$$

For this result matrix the checksum of row 4 is not valid, so an error must have occurred.

Actually the checksum property means $(1,1,1,1)^T$ is a solution of the following system of equations that corresponds to the augmented matrix:

$$begin{eqnarray}
2x_1&+&5x_2&-&8x_3&+&8x_4&=&7\
4x_1&+&3x_2&-&9x_3&+&9x_4&=&7\
2x_1&+&3x_2&-&5x_3&+&7x_4&=&7\
x_1&+&8x_2&-&7x_3&+&12x_4&=&14
end{eqnarray}$$

So this method is similar to the method in the accepted answer, where the solution given by the textbook is used.

What if this method does not uncover the errors. This can happen if you made more then one error in a step- Then you can use a weighted checksum to try to uncover the error.