When routing velocity to Env Amt, think in terms of both the minimum amount and maximum amount of envelope you want at minimum and maximum velocity, respectively.
Use the Amount knob in the envelope section to set the minimum/starting envelope amount, that plays at the lowest velocity. Set the mod depth of Velocity->Env Amt in the matrix to get the max amount or the desired amount you want at a given velocity. It’ll take some fine tuning of both of these to get it the way you want, but it’s better than the “all or nothing” the velocity button gives you.