At $t_0=\pi /(2\omega )$, $\sin \omega t_0=1$, so particle $2$ is at extreme position $x_2=x_0-d-a$ with zero velocity; particle $1$ is at $x_0+d+a$, also at rest. The spring is at its maximum stretch (or compression) so both particles are momentarily stationary.

Particle $3$ collides elastically with particle $2$ (both mass $m$): velocities are exchanged. Particle $2$ acquires velocity $u_0=a\omega /2$ rightward; particle $1$ remains at rest.

Centre-of-mass velocity of $1$ and $2$ after collision:

$v_{cm}=\frac{0+a\omega /2}{2}=\frac{a\omega }{4}\text{(rightward)}.$

Working in the centre-of-mass frame, the relative coordinate is $\xi =x_1-x_2=2(d+a)$ with zero relative velocity initially; this distance corresponds to one particle at $+(d+a)$ and the other at $-(d+a)$ from the centre of mass. After the collision the relative velocity becomes $v_1-v_2=-a\omega /2$. In the centre-of-mass frame each particle is at distance $a$ from the centre of mass and moving with speed $a\omega /4$.

For SHM in the COM frame about the relaxed separation, the relation between amplitude $A$, instantaneous position $x$, and speed $v$ is $v=\omega \sqrt{A^2-x^2}$. Here each particle has $x=a$ and $v=a\omega /4$:

$\frac{a\omega }{4}=\omega \sqrt{A^2-a^2}\Rightarrow A^2=a^2+\frac{a^2}{16}=\frac{17a^2}{16}.$

The amplitude of each particle's oscillation about the new centre of mass is $b=A$, so

$\frac{4b^2}{a^2}=\frac{4\cdot 17}{16}=\frac{17}{4}=\mathrm{4.25.}$