Make the substitution [tex]t=\ln x[/tex], then compute the derivatives of [tex]y[/tex] with respect to [tex]t[/tex] via the chain rule.
[tex]\dfrac{\mathrm dy}{\mathrm dx}=\dfrac{\mathrm dy}{\mathrm dt}\dfrac{\mathrm dt}{\mathrm dx}[/tex]
[tex]\implies\dfrac{\mathrm dy}{\mathrm dx}=\dfrac1x\dfrac{\mathrm dy}{\mathrm dt}[/tex]
Let [tex]f(t)=\frac{\mathrm dy}{\mathrm dt}[/tex].
[tex]\dfrac{\mathrm d^2y}{\mathrm dx^2}=\dfrac{\mathrm d}{\mathrm dx}\left[\dfrac fx\right]=\dfrac{x\frac{\mathrm df}{\mathrm dx}-f}{x^2}[/tex]
[tex]\dfrac{\mathrm df}{\mathrm dx}=\dfrac{\mathrm df}{\mathrm dt}\dfrac{\mathrm dt}{\mathrm dx}=\dfrac1x\dfrac{\mathrm d^2y}{\mathrm dt^2}[/tex]
[tex]\implies\dfrac{\mathrm d^2y}{\mathrm dx^2}=\dfrac1{x^2}\left(\dfrac{\mathrm d^2y}{\mathrm dt^2}-\dfrac{\mathrm dy}{\mathrm dt}\right)[/tex]
Let [tex]g(t)=\frac{\mathrm df}{\mathrm dt}=\frac{\mathrm d^2y}{\mathrm dt^2}[/tex].
[tex]\dfrac{\mathrm d^3y}{\mathrm dx^3}=\dfrac{\mathrm d}{\mathrm dx}\left[\dfrac{g-f}{x^2}\right]=\dfrac{x^2\left(\frac{\mathrm dg}{\mathrm dx}-\frac{\mathrm df}{\mathrm dx}\right)-2x(g-f)}{x^4}[/tex]
[tex]\dfrac{\mathrm dg}{\mathrm dx}=\dfrac{\mathrm dg}{\mathrm dt}\dfrac{\mathrm dt}{\mathrm dx}=\dfrac1x\dfrac{\mathrm d^3y}{\mathrm dt^3}[/tex]
[tex]\implies\dfrac{\mathrm d^3y}{\mathrm dx^3}=\dfrac{x^2\left(\frac1x\frac{\mathrm dg}{\mathrm dt}-\frac1x\frac{\mathrm df}{\mathrm dt}\right)-2x(g-f)}{x^4}=\dfrac1{x^3}\left(\dfrac{\mathrm d^3y}{\mathrm dt^3}-3\dfrac{\mathrm d^2y}{\mathrm dt^2}+2\dfrac{\mathrm dy}{\mathrm dt}\right)[/tex]
Substituting [tex]y(t)[/tex] and its derivatives into the ODE gives a new one that is linear in [tex]t[/tex]:
[tex]\left(\dfrac{\mathrm d^3y}{\mathrm dt^3}-3\dfrac{\mathrm d^2y}{\mathrm dt^2}+2\dfrac{\mathrm dy}{\mathrm dt}\right)+5\left(\dfrac{\mathrm d^2y}{\mathrm dt^2}-\dfrac{\mathrm dy}{\mathrm dt}\right)+7\dfrac{\mathrm dy}{\mathrm dt}+8y=0[/tex]
[tex]\dfrac{\mathrm d^3y}{\mathrm dt^3}+2\dfrac{\mathrm d^2y}{\mathmr dt^2}+4\dfrac{\mathrm dy}{\mathrm dt}+8y=0[/tex]
[tex]y'''+2y''+4y'+8y=0[/tex]
which has characteristic equation
[tex]r^3+2r^2+4r+8=(r+2)(r^2+4)=0[/tex]
with roots [tex]r=-2[/tex] and [tex]r=\pm2i[/tex], so that the characteristic solution is
[tex]y_c(t)=C_1e^{-2t}+C_2\cos2t+C_3\sin2t[/tex]
Replace [tex]t=\ln x[/tex] to solve for [tex]y(x)[/tex]:
[tex]y_c(x)=C_1e^{-2\ln x}+C_2\cos(2\ln x)+C_3\sin(2\ln x)[/tex]
[tex]\boxed{y(x)=\dfrac{C_1}{x^2}+C_2\cos(2\ln x)+C_3\sin(2\ln x)}[/tex]
