We present results of nonlinear compensation achieved by digital backward-propagation (DBP) for next-generation optical coherent transmission systems. In our numerical investigations, phase-encoded signals (DQPSK) as well as signals of higher-order modulation formats (16-QAM) have been analyzed with respect to degradation by fiber nonlinearity such as self-phase modulation (SPM). The influence of inline compensation schemes used is studied and we also showed that DBP performance depends strongly on its parameters. A careful adjustment of e.g. the nonlinear calculating position improves the eye opening up to 3.5dB for DQPSK signals. Our work also includes investigations on optimized step-size selection. The implementation of a logarithmic step-size distribution reduces the numerical effort by about 50% with respect to using constant step-size distributions in 112Gbps 16-QAM transmission. Furthermore, by using an additional low-pass-filter (LPF) in each DBP stage, the required number of DBP stages is significantly reduced, where the computational efficiency can be enhanced without significant loss of performance, which will be helpful in future deployment of DBP in real-time signal processing modules for non-linear compensation. In order to explore the capability of this filtered DBP in terms of complexity, the results of diverse baud rates for multiple channel DP-QPSK systems with a total transmission capacity of 1.12 Tbit/s are compared.