support ivector training in pytorch model#3969
Merged
danpovey merged 4 commits intokaldi-asr:pybind11from Mar 3, 2020
Merged
Conversation
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
Contributor
|
Thanks a lot for reviewing!
…On Tue, Mar 3, 2020 at 9:47 AM Fangjun Kuang ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In egs/aishell/s10/local/run_ivector_common.sh
<#3969 (comment)>:
> + ${temp_data_root}/${train_set}_sp_hires_max2 \
+ exp/nnet3${nnet3_affix}/extractor $ivectordir
+
+fi
+
+if [[ $stage -le 8 ]]; then
+ # Also extract iVectors for the test data, but in this case we don't need the speed
+ # perturbation (sp) or small-segment concatenation (comb).
+ for data in dev test; do
+ steps/online/nnet2/extract_ivectors_online.sh --cmd "$train_cmd" --nj 10 \
+ data/${data}_hires exp/nnet3${nnet3_affix}/extractor \
+ exp/nnet3${nnet3_affix}/ivectors_${data}_hires
+ done
+fi
+
+exit 0;
this file should end with a newline.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#3969?email_source=notifications&email_token=AAZFLO2DVOTQ4UXOGPS7GC3RFROS5A5CNFSM4K7XDWTKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCXU67ZI#pullrequestreview-367652837>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLO46SQT5HSXGZ6DFFHTRFROS5ANCNFSM4K7XDWTA>
.
|
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
csukuangfj
reviewed
Mar 3, 2020
Contributor
|
Guys, I just want to mention something...
I think it would be better if we shifted (not necessarily right now..) to,
instead of exposing the Kaldi egs as a Dataset,
exposing them as a DataLoader. That way we could use the existing
command-line tools for things like shuffling and
time-shifting, and it will be much more efficient for I/O.
The idea is that the dataloader would, on every epoch, create a suitable
command line and read from it as a pipe.
If it was a distributed data-loader, probably the easiest way to do it
would be to make sure there is an appropriately
split scp file and give it the appropriate one. We could the scripts here
#3765
to generate the scp files. I want to merge this soon; one option is to
merge into pybind11 first to test it.
…On Tue, Mar 3, 2020 at 10:11 AM fanlu ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In egs/aishell/s10/chain/feat_dataset.py
<#3969 (comment)>:
>
with open(feats_scp, 'r') as f:
for line in f:
split = line.split()
assert len(split) == 2
- items.append(split)
-
- self.items = items
+ uttid, rxfilename =split
OK
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#3969?email_source=notifications&email_token=AAZFLO6XPUG7BA6VEDOUVETRFRRNRA5CNFSM4K7XDWTKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCXVAYYY#discussion_r386762254>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLO2XIWLBFIYM4T3DZQ3RFRRNRANCNFSM4K7XDWTA>
.
|
Author
|
@csukuangfj I have fixed the code with your suggestion. please have a look. |
Contributor
|
OK, I'll run it after it's merged. |
Contributor
|
I'll take a look at that pr and start to do this.
|
Contributor
|
OK, merging. |
Contributor
|
Great, thanks! Firstly, just doing the merge and figuring out how to use
those newer scripts to prepare the egs would be a great start.
…On Tue, Mar 3, 2020 at 11:19 AM Haowen Qiu ***@***.***> wrote:
I'll take a look at that pr and start to do this.
Guys, I just want to mention something... I think it would be better if we
shifted (not necessarily right now..) to, instead of exposing the Kaldi egs
as a Dataset, exposing them as a DataLoader. That way we could use the
existing command-line tools for things like shuffling and time-shifting,
and it will be much more efficient for I/O. The idea is that the dataloader
would, on every epoch, create a suitable command line and read from it as a
pipe. If it was a distributed data-loader, probably the easiest way to do
it would be to make sure there is an appropriately split scp file and give
it the appropriate one. We could the scripts here #3765
<#3765> to generate the scp files.
I want to merge this soon; one option is to merge into pybind11 first to
test it.
… <#m_3185835891003743640_>
On Tue, Mar 3, 2020 at 10:11 AM fanlu *@*.*> wrote: @.** commented on
this pull request. ------------------------------ In
egs/aishell/s10/chain/feat_dataset.py <#3969 (comment)
<#3969 (comment)>>: >
with open(feats_scp, 'r') as f: for line in f: split = line.split() assert
len(split) == 2 - items.append(split) - - self.items = items + uttid,
rxfilename =split OK — You are receiving this because you commented. Reply
to this email directly, view it on GitHub <#3969
<#3969>?email_source=notifications&email_token=AAZFLO6XPUG7BA6VEDOUVETRFRRNRA5CNFSM4K7XDWTKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCXVAYYY#discussion_r386762254>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO2XIWLBFIYM4T3DZQ3RFRRNRANCNFSM4K7XDWTA
.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#3969?email_source=notifications&email_token=AAZFLO22JMLLIJ7M3UMK42LRFRZLFA5CNFSM4K7XDWTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENR5TZI#issuecomment-593746405>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLO63INWGRC6Y5NEM6SLRFRZLFANCNFSM4K7XDWTA>
.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
update latest result