The action of $A$ on standard basis vectors gives us its columns. From $A(0,0,1{)}^T=(1,3,1{)}^T$, the third column is $(1,3,1{)}^T$. Subtracting from $A(1,0,1{)}^T=(3,4,4{)}^T$, the first column is $(2,1,3{)}^T$.

From $A^T(0,0,1{)}^T=(3,1,1{)}^T$, the third row is $(3,1,1)$. Subtracting from $A^T(1,0,1{)}^T=(5,2,2{)}^T$, the first row is $(2,1,1)$.

Hence $A=\left(\begin{array}{c}211\\ 1\beta 3\\ 311\end{array}\right)$ for some unknown $\beta$. Expanding $\det (A)=1$ gives $2(\beta -3)-1(1-9)+1(1-3\beta )=1$, which simplifies to $-\beta +3=1$, so $\beta =2$.

Computing $A^2=\left(\begin{array}{c}856\\ 13810\\ 1067\end{array}\right)$, so $A^2+A=\left(\begin{array}{c}1067\\ 141013\\ 1378\end{array}\right)$ with $\det (A^2+A)=8$.

For a $3\times 3$ matrix, $\det (\mathrm{adj}M)=(\det M{)}^2$, so $\det (\mathrm{adj}(A^2+A))=8^2=64$.