Skip to content

modify clip grad#10441

Closed
hanwen-sun wants to merge 2 commits intomasterfrom
modify_clip_grad
Closed

modify clip grad#10441
hanwen-sun wants to merge 2 commits intomasterfrom
modify_clip_grad

Conversation

@hanwen-sun
Copy link

修改clip_grad中tensor parallel的通信策略:

  • L2 norm计算修改:

    • 每张卡先在本地计算梯度的squaresum
    • 调用all_reduce_sum, 每张卡得到累加后的squaresum
    • 开根号
  • TODO:

    • 更好的写法;

@hanwen-sun hanwen-sun requested a review from MARD1NO March 6, 2024 11:31
@github-actions
Copy link
Contributor

github-actions bot commented Mar 6, 2024

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

@levi131
Copy link
Contributor

levi131 commented Apr 16, 2024

问题在 #10443 中解决

@levi131 levi131 closed this Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants