There's probably a more efficient way to do it, but the following method works.
First, note that
x² - x - 2 = (x + 1) (x - 2)
Now use synthetic division to divide
a x⁴ + b x³ - x² + 2x + 3
by the two factors above.
Division by x + 1 :
-1 | a b -1 2 3
… | -a a - b -a + b + 1 a - b - 3
= = = = = = = = = = = = = = = = = = = = = = = = = = = = =
… | a b - a a - b - 1 -a + b + 3 a - b
which is to say,
(a x⁴ + b x³ - x² + 2x + 3) / (x + 1)
= a x³ + (b - a) x² + (a - b - 1) x + (-a + b + 3) + (a - b) / (x + 1)
Division by x - 2 :
2 | a b - a a - b - 1 -a + b + 3
… | 2a 2a + 2b 6a + 2b - 2
= = = = = = = = = = = = = = = = = = = = = = = = = = =
… | a a + b 3a + b - 1 5a + 3b + 1
meaning,
(a x³ + (b - a) x² + (a - b - 1) x + (-a + b + 3)) / (x - 2)
= a x² + (a + b) x + (3a + b - 1) + (5a + 3b + 1) / (x - 2)
Putting everything together, we have
(a x⁴ + b x³ - x² + 2x + 3) / (x² - x - 2)
= a x² + (a + b) x + (3a + b - 1) + (5a + 3b + 1) / (x - 2) + (a - b) / (x² - x - 2)
Combine the remainder terms:
(5a + 3b + 1) / (x - 2) + (a - b) / (x² - x - 2)
= ((5a + 3b + 1) x + (6a + 2b + 1)) / (x² - x - 2)
We're given that the remainder is 3x + 5, so
5a + 3b + 1 = 3
6a + 2b + 1 = 5
Solve this system for a and b :
5a + 3b = 2
6a + 2b = 4
5a + 3b = 2
3a + b = 2
5a + 3b = 2
-9a - 3b = -6
(5a + 3b) + (-9a - 3b) = 2 + (-6)
-4a = -4
a = 1
5 • 1 + 3b = 2
3b = -3
b = -1
