Reverberation has a severe impact on the intelligibility of a speech signal and, despite all recent advances in acoustic modeling, still deteriorates the performance of automatic speech recognition (ASR) systems significantly, even when trained on large scale data [1], [2]. In these challenging far-field scenarios, signal processing to dereverberate and thus enhance the signal can help to mitigate the performance losses. Many techniques have been proposed for signal dereverberation, which can be broadly categorized in linear filter ing approaches and spectral subtraction like approaches for magnitude or power spectrum manipulation [3].

Weighted prediction error (WPE) dereverberates the signal by estimating an inverse filter which is used to subtract the reverberation tail from the observation and thus falls into the first category. It can operate either on a single channel or in a multiple-input multiple-output fashion on multi-channel data. The quality of the estimated filter coefficients mainly depends on the estimation of the PSD of the "anechoic speech", i.e., the direct speech signal and its early reflections. Since this signal is unknown, the vanilla WPE works iteratively by alternating between two steps: (Step 1) Dereverberating the signal using the current estimate of the anechoic speech PSD, and, (Step 2) estimating the anechoic speech PSD using the current estimate of the dereverberated signal. Alternating these two steps gradually improves the estimate of both, the (dereverberated) target signal and the anechoic speech PSD. This, however, inherently makes the vanilla WPE an offline method and computationally expensive.

To overcome this dependency issue – and enable an on-line usage of WPE – we recently proposed to utilize a neural network to directly estimate the PSD from the observation [4] [5]. We could show that this leads to improved performance for low-latency solutions compared to a more simple PSD estimation [6]. However, we now need parallel reverberated and non-reverberated data in order to train the PSD estimation network, limiting the applicability of the approach.

In this work, we lift this restriction by combining the WPE front-end with the acoustic model already during training. This allows us to train the estimator directly with a suitable ASR loss. Apart from an expected performance improvement like we saw with e.g. beamforming [7], the motivation for this is threefold:

1. Calculating a training target for the PSD estimator requires a corpus of parallel data. Building such a corpus is almost only possible by simulating the observed data, inevitably leading to a mismatch between training and test data.

2. There is no clear notion which part of the signal should be considered as anechoic or the target signal respectively. Thus, the training target for the PSD estimator is not well-defined. Where as the network can adjust to the acoustic model needs when trained jointly.

3. If other directed noise sources are present, their contribution to the covariance statistics could be optimized by an appropriate weighting factor for the tf-bins in question.

We investigate the performance of such a joint system and compare it with one separately trained on oracle PSD information.

Using D microphones, we observe a signal which is represented as the D-dimensional vector y_t,f at time frame index t and frequency bin index f in the short time Fourier transformation (STFT) domain. In a far-field scenario, this signal is impaired by (convolutive) reverberation. We assume, that for ASR the early part of the room impulse response (RIR) is beneficial whereas the reverberation tail deteriorates the recognition and should therefore be suppressed. Specifically, we consider the first 50 ms after the main peak of the RIR (h^(early)) to contribute to the anechoic signal whereas the remaining part (h^(tail)) is assumed to cause the distortions. In the STFT domain we denote this model as follows:

View Source\begin{equation*}{{\mathbf{y}}_{t,f}} = {\mathbf{x}}_{t,f}^{\left( {{\text{early}}} \right)} + {\mathbf{x}}_{t,f}^{\left( {{\text{tail}}} \right)},\tag{1}\end{equation*}

WPE estimates the reverberation tail of the signal from previous samples and subtracts it from the observation to obtain an optimal estimate of the anechoic speech in a maximum likelihood sense:

View Source\begin{equation*}\hat x_{t,f,d}^{\left( {{\text{early}}} \right)} = {y_t},f,d - {\mathbf{g}}_{f,d}^{\text{H}}{{\mathbf{\tilde y}}_{t - \Delta ,f}},\tag{2}\end{equation*}

Using ∆ ≥ 1 avoids whitening of the speech source. WPE maximizes the likelihood of the model under the assumption that the anechoic signal is a realization of a zero-mean circularly-symmetric complex Gaussian with an unknown time-varying variance λ_t,f.

3.1. Iterative WPE

There is no closed form solution for the likelihood optimization, but an iterative procedure which alternates between estimating the filter coefficients g_fd and the time-varying variance λ_tf exists:

Step 1)

$$\begin{align*} & {{\mathbf{R}}_f} = \sum\limits_t {\frac{{{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}}}{{{\lambda _{t,f}}}}} ,\tag{3} \\ & {{\mathbf{p}}_{f,d}} = \sum\limits_t {\frac{{{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}y_{t,f,d}^{\text{*}}}}{{{\lambda _{t,f}}}}} ,\tag{4} \\ & {{\mathbf{g}}_{f,d}} = {\mathbf{R}}_f^{ - 1}{{\mathbf{p}}_{f,d}}\tag{5}\end{align*}$$ View Source\begin{align*} & {{\mathbf{R}}_f} = \sum\limits_t {\frac{{{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}}}{{{\lambda _{t,f}}}}} ,\tag{3} \\ & {{\mathbf{p}}_{f,d}} = \sum\limits_t {\frac{{{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}y_{t,f,d}^{\text{*}}}}{{{\lambda _{t,f}}}}} ,\tag{4} \\ & {{\mathbf{g}}_{f,d}} = {\mathbf{R}}_f^{ - 1}{{\mathbf{p}}_{f,d}}\tag{5}\end{align*}

Step 2)

$$\begin{equation*}{\lambda _{t,f}} = \frac{1}{{\left( {\delta + 1 + \delta } \right)D}}\sum\limits_{\tau = t - \delta }^{t + \delta } {\sum\limits_d {{{\left| {\hat x_{\tau ,f,d}^{\left( {{\text{early}}} \right)}} \right|}^2}} .} \tag{6}\end{equation*}$$ View Source\begin{equation*}{\lambda _{t,f}} = \frac{1}{{\left( {\delta + 1 + \delta } \right)D}}\sum\limits_{\tau = t - \delta }^{t + \delta } {\sum\limits_d {{{\left| {\hat x_{\tau ,f,d}^{\left( {{\text{early}}} \right)}} \right|}^2}} .} \tag{6}\end{equation*}

The heuristic context of (δ + 1 + δ) frames helps to improve the variance estimate in this iterative scheme [8].

3.2. Recursive WPE

To derive a recursive formulation, the correlation matrix is estimated with a decaying window:

$$\begin{equation*}{{\mathbf{R}}_{t,f}} = \sum\limits_{\tau = 0}^t {{\alpha ^{t - \tau }}} \frac{{{{{\mathbf{\tilde y}}}_{\tau - \Delta ,f}}{\mathbf{\tilde y}}_{\tau - \Delta ,f}^{\text{H}}}}{{{\lambda _{\tau ,f}}}}.\tag{7}\end{equation*}$$ View Source\begin{equation*}{{\mathbf{R}}_{t,f}} = \sum\limits_{\tau = 0}^t {{\alpha ^{t - \tau }}} \frac{{{{{\mathbf{\tilde y}}}_{\tau - \Delta ,f}}{\mathbf{\tilde y}}_{\tau - \Delta ,f}^{\text{H}}}}{{{\lambda _{\tau ,f}}}}.\tag{7}\end{equation*}

This leads to a recursive solution with the following rank-one updates [9]:

$$\begin{align*} & {{\mathbf{K}}_{t,f}} = \frac{{{\mathbf{R}}_{t - 1,f}^{ - 1}{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}}}{{\alpha {\lambda _{t,f}} + {\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}{\mathbf{R}}_{t - 1,f}^{ - 1}{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}}}\tag{8} \\ & {\mathbf{R}}_{t,f}^{ - 1} = \frac{1}{\alpha }\left( {{\mathbf{R}}_{t - 1,f}^{ - 1} - {{\mathbf{K}}_{t,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}{\mathbf{R}}_{t - 1,f}^{ - 1}} \right)\tag{9} \\ & {{\mathbf{G}}_{t,f}} = {{\mathbf{G}}_{t - 1,f}} + {{\mathbf{K}}_{t,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}.\tag{10}\end{align*}$$ View Source\begin{align*} & {{\mathbf{K}}_{t,f}} = \frac{{{\mathbf{R}}_{t - 1,f}^{ - 1}{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}}}{{\alpha {\lambda _{t,f}} + {\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}{\mathbf{R}}_{t - 1,f}^{ - 1}{{{\mathbf{\tilde y}}}_{t - \Delta ,f}}}}\tag{8} \\ & {\mathbf{R}}_{t,f}^{ - 1} = \frac{1}{\alpha }\left( {{\mathbf{R}}_{t - 1,f}^{ - 1} - {{\mathbf{K}}_{t,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}{\mathbf{R}}_{t - 1,f}^{ - 1}} \right)\tag{9} \\ & {{\mathbf{G}}_{t,f}} = {{\mathbf{G}}_{t - 1,f}} + {{\mathbf{K}}_{t,f}}{\mathbf{\tilde y}}_{t - \Delta ,f}^{\text{H}}.\tag{10}\end{align*}

Here, G_t,f are the stacked filter taps g_f,d for each microphone. Note that these are now time variant. This is in essence a Recursive Least Squares (RLS) adaptive filter for the reverberation estimation. The authors of [6] approximate the PSD of the target signal using a smoothed PSD of the observation averaged over the microphones using a left and right context δ_L and δ_R:

$$\begin{equation*}{\lambda _{t,f}} = \frac{1}{D} \cdot \frac{1}{{{\delta _{\text{L}}} + 1 + {\delta _{\text{R}}}}}\sum\limits_{\tau = t - {\delta _{\text{L}}}}^{t + {\delta _{\text{R}}}} {\sum\limits_d {{{\left| {{y_{\tau .f,d}}} \right|}^2}.} } \tag{11}\end{equation*}$$ View Source\begin{equation*}{\lambda _{t,f}} = \frac{1}{D} \cdot \frac{1}{{{\delta _{\text{L}}} + 1 + {\delta _{\text{R}}}}}\sum\limits_{\tau = t - {\delta _{\text{L}}}}^{t + {\delta _{\text{R}}}} {\sum\limits_d {{{\left| {{y_{\tau .f,d}}} \right|}^2}.} } \tag{11}\end{equation*}

To demonstrate the versatility of the described approach, we evaluate the proposed systems in terms of WERs on the data of the REVERB challenge as well as on WSJ+VoiceHome data.

The REVERB challenge dataset [12] contains simulated and real utterances. For simulated data WSJCAM0 utterances [13] are convolved with measured RIRs. Noise is added with ∼20 dB signal to noise ratio (SNR). Reverberation times (T60) are in the range of 0.25 – 0.7 s. The real data consists of utterances from the MC-WSJ-AV corpus [14] which are recorded in a noisy reverberant room with a reverberation time of ∼0.7 s. The corpus is known for its mismatch between the simulated data used during training and the real recordings for evaluation. To reduce this discrepancy, we randomly sample the SNR to be in the range of 5 dB – 30 dB and scale the signal with 0.2 for the training of the PSD estimators and all finetuning experiments [4]. The initial acoustic model is trained on unscaled data without SNR perturbation.

For WSJ+VoiceHome we convolve WSJ utterances (5 k vocabulary) with VoiceHome RIRs and VoiceHome background noise [15] with reverberation times (T60) in the range of 395 – 585 ms. This is similar to the simulation setup proposed by Bertin et al. [16]. The VoiceHome background noise is very dynamic and typically found in households e.g. vacuum cleaner, dish washing or interviews on television and the SNR ranges from 0 dB – 10 dB.

We evaluate the performance for two and eight microphone channels. Since WPE preserves the number of channels, we always take the first one and use it for decoding with the acoustic model. For decoding, we use the 3-gram language model from the WSJ0 corpus without any rescoring afterwards.

Table 1. WERs/% for all systems evaluated on REVERB (real data eval set, averaged over near and far) and WSJ+VoiceHome. For the MSE loss parallel data is needed to calculate the anechoic PSD target, while the CE loss uses senone targets.

For WPE we use a DFT window size of 512 (32 ms) and a shift of 128 (8 ms). For the recursive WPE variant, we set α = 0.9999 and for vanilla WPE the number of iteration to 3. For all variants, we vary the delay parameter in a range between 1 and 4 and the number of filter taps is set to either 5 or 10. These values are determined on the development set and can be different for each configuration.

Our baselines for evaluation are Unprocessed, Iteration and Smooth. These use the first channel without any enhancement, vanilla WPE iterations and the smoothing PSD estimator (see. Eq. 11) respectively. These are compared with the neural network variants trained as described in Subsec. 4.3.

These systems are evaluated for two different latency constraints (where applicable): offline and online. Offline means, that the whole utterance is available for processing and this is our baseline scenario. For the online setting, which is our target scenario, we use the recursive formulation of WPE (Eq. 8–Eq. 10) and the system operates on a frame-by-frame basis.

All results are shown in Tbl. 1. First note, that the unprocessed baseline itself already achieves very good performance. For comparison, the recently updated Kaldi system achieves a WER of around 19.7 %² on the Reverb dataset. All fine-tuned systems improve upon the unprocessed baseline irrespective of the PSD estimator or the latency constraint showing the effectiveness of a WPE front-end. As another general tendency we can see that the DNN supported systems outperform the baselines, especially in the online case we were focusing on. For the offline use-case, using the vanilla WPE formulation with iterations seems to be most suitable though, especially when considering the system complexity.

For the online use-case however, using a DNN-based PSD estimator and training it with the ASR criterion is more effective. One reason for this might be that the PSD can be implicitly tuned for faster convergence of the filter estimation in this case. The best results are achieved if the PSD estimator is pre-trained using parallel data and then finetuned jointly with the acoustic model. If no parallel data is available, the PSD estimator can also be trained from scratch with little to no loss in performance. All this applies to both tested corpora and therefore for highly reverberant as well as reverberant and noisy household-like scenarios.

In this paper we demonstrate that jointly optimizing a DNN based PSD estimator and the acoustic model improves the performance for online dereverberation with WPE by 8 % – 18 % in highly reverberant as well as noisy reverberant scenarios compared to a baseline smoothing PSD estimator. Being able to backpropagate through the filter estimation further lifts the requirement for parallel training data and further allows to potentially train the PSD estimator on real data to reduce the mismatch between training and inference with minimal impact on the overall performance.