I think you meant to write
[tex]2t^2y''+10ty'+8y=0[/tex]
which is an ODE of Cauchy-Euler type. Let [tex]y=t^m[/tex]. Then
[tex]y'=mt^{m-1}[/tex]
[tex]y''=m(m-1)t^{m-2}[/tex]
Substituting [tex]y[/tex] and its derivatives into the ODE gives
[tex]2m(m-1)t^m+10mt^m+8t^m=0[/tex]
Divide through by [tex]t^m[/tex], which we can do because [tex]t\neq0[/tex]:
[tex]2m(m-1)+10m+8=2m^2+8m+8=2(m+2)^2=0\implies m=-2[/tex]
Since this root has multiplicity 2, we get the characteristic solution
[tex]y_c=C_1t^{-2}+C_2t^{-2}\ln t[/tex]
If you're not sure where the logarithm comes from, scroll to the bottom for a bit more in-depth explanation.
With the given initial values, we find
[tex]y(1)=1\implies1=C_1[/tex]
[tex]y'(1)=0\implies0=-2C_1+C_2\implies C_2=2[/tex]
so that the particular solution is
[tex]\boxed{y(t)=t^{-2}+2t^{-2}\ln t}[/tex]
# # #
Under the hood, we're actually substituting [tex]t=e^u[/tex], so that [tex]u=\ln t[/tex]. When we do this, we need to account for the derivative of [tex]y[/tex] wrt the new variable [tex]u[/tex]. By the chain rule,
[tex]\dfrac{\mathrm dy}{\mathrm dt}=\dfrac{\mathrm dy}{\mathrm du}\dfrac{\mathrm du}{\mathrm dt}=\dfrac1t\dfrac{\mathrm dy}{\mathrm du}[/tex]
Since [tex]\frac{\mathrm dy}{\mathrm dt}[/tex] is a function of [tex]t[/tex], we can treat [tex]\frac{\mathrm dy}{\mathrm du}[/tex] in the same way, so denote this by [tex]f(t)[/tex]. By the quotient rule,
[tex]\dfrac{\mathrm d^2y}{\mathrm dt^2}=\dfrac{\mathrm d}{\mathrm dt}\left[\dfrac ft\right]=\dfrac{t\frac{\mathrm df}{\mathrm dt}-f}{t^2}[/tex]
and by the chain rule,
[tex]\dfrac{\mathrm df}{\mathrm dt}=\dfrac{\mathrm df}{\mathrm du}\dfrac{\mathrm du}{\mathrm dt}=\dfrac1t\dfrac{\mathrm df}{\mathrm du}[/tex]
where
[tex]\dfrac{\mathrm df}{\mathrm du}=\dfrac{\mathrm d}{\mathrm du}\left[\dfrac{\mathrm dy}{\mathrm du}\right]=\dfrac{\mathrm d^2y}{\mathrm du^2}[/tex]
so that
[tex]\dfrac{\mathrm d^2y}{\mathrm dt^2}=\dfrac{\frac{\mathrm d^2y}{\mathrm du^2}-\frac{\mathrm dy}{\mathrm du}}{t^2}=\dfrac1{t^2}\left(\dfrac{\mathrm d^2y}{\mathrm du^2}-\dfrac{\mathrm dy}{\mathrm du}\right)[/tex]
Plug all this into the original ODE to get a new one that is linear in [tex]u[/tex] with constant coefficients:
[tex]2t^2\left(\dfrac{\frac{\mathrm d^2y}{\mathrm du^2}-\frac{\mathrm d y}{\mathrm du}}{t^2}\right)+10t\left(\dfrac{\frac{\mathrm dy}{\mathrm du}}t\right)+8y=0[/tex]
[tex]2y''+8y'+8y=0[/tex]
which has characteristic equation
[tex]2r^2+8r+8=2(r+2)^2=0[/tex]
and admits the characteristic solution
[tex]y_c(u)=C_1e^{-2u}+C_2ue^{-2u}[/tex]
Finally replace [tex]u=\ln t[/tex] to get the solution we found earlier,
[tex]y_c(t)=C_1t^{-2}+C_2t^{-2}\ln t[/tex]
