Geometric setup. From $\vec{u}\times \vec{v}=\vec{w}$, $\vec{u}$ and $\vec{v}$ are perpendicular and $\vec{w}$ is perpendicular to both. From $\vec{v}\times \vec{w}=\vec{u}$, $\vec{v}$ is perpendicular to $\vec{w}$ (already known) and $\vec{u},\vec{v},\vec{w}$ form a right-handed orthogonal triple.

Using the vector identity, $\vec{v}\times (\vec{u}\times \vec{v})=(\vec{v}\cdot \vec{v})\vec{u}-(\vec{v}\cdot \vec{u})\vec{v}=|\vec{v}{|}^2\vec{u}$ (since $\vec{u}\perp \vec{v}$). But $\vec{v}\times \vec{w}=\vec{u}$, so $|\vec{v}{|}^2\vec{u}=\vec{u}$, giving $|\vec{v}{|}^2=1.$ (P) $\to$ (2).

Magnitude of $\vec{u}$. Taking magnitudes in $\vec{u}\times \vec{v}=\vec{w}$: $|\vec{u}|\cdot |\vec{v}|=|\vec{w}|$, so $|\vec{u}|=|\vec{w}|=\sqrt{1+1+4}=\sqrt{6}.$ Hence ${\alpha }^2+{\beta }^2+{\gamma }^2=6.$

Also $\vec{u}\perp \vec{w}$ gives $\alpha +\beta -2\gamma =0.$

The linear system. The three given equations have the determinant $\left|\begin{array}{c}-t11\\ 1-t1\\ 11-t\end{array}\right|=-(t-2)(t+1{)}^2.$ For a nontrivial solution we need this to vanish, so $t=2$ or $t=-1$.

Case $t=2$: The three equations all reduce to $\alpha =\beta =\gamma$. Combined with ${\alpha }^2+{\beta }^2+{\gamma }^2=6$, we get $\alpha =\pm \sqrt{2}$.

For $\alpha =\sqrt{2}$: $t+3=5.$ (S) $\to$ (5).

Case $t=-1$: The system reduces to $\alpha +\beta +\gamma =0$. Combined with $\alpha +\beta -2\gamma =0$ from $\vec{u}\perp \vec{w}$, subtracting yields $3\gamma =0$, so $\gamma =0$ and $\beta =-\alpha$.

If $\alpha =\sqrt{3}$: then ${\gamma }^2=0$ and $(\beta +\gamma {)}^2=(-\sqrt{3}{)}^2=3.$ (Q) $\to$ (1) and (R) $\to$ (4).

$\therefore$ The correct option is (A).