atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[薛定谔的风口猪]]></title>
  <link href="https://Jaskey.github.io/atom.xml" rel="self"/>
  <link href="https://Jaskey.github.io/"/>
  <updated>2022-04-14T18:21:20+08:00</updated>
  <id>https://Jaskey.github.io/</id>
  <author>
    <name><![CDATA[Jaskey Lam]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[挑战大型系统的缓存设计——应对一致性问题]]></title>
    <link href="https://Jaskey.github.io/blog/2022/04/14/cache-consistency/"/>
    <updated>2022-04-14T17:03:26+08:00</updated>
    <id>https://Jaskey.github.io/blog/2022/04/14/cache-consistency</id>
    <content type="html"><![CDATA[<p>在真实的业务场景中，我们业务的数据——例如订单、会员、支付等——都是持久化到数据库中的，因为数据库能有很好的事务保证、持久化保证。但是，正因为数据库要能够满足这么多优秀的功能特性，使得数据库在设计上通常难以兼顾到性能，因此往往不能满足大型流量下的性能要求，像是 MySQL 数据库只能承担“千”这个级别的 QPS，否则很可能会不稳定，进而导致整个系统的故障。</p>

<p>但是客观上，我们的业务规模很可能要求着更高的 QPS，有些业务的规模本身就非常大，也有些业务会遇到一些流量高峰，比如电商会遇到大促的情况。</p>

<p>而这时候大部分的流量实际上都是<strong>读请求</strong>，而且大部分数据也是没有那么多变化的，如热门商品信息、微博的内容等常见数据就是如此。此时，<strong>缓存就是我们应对此类场景的利器</strong>。</p>

<h2>缓存的意义</h2>

<p>所谓缓存，实际上就是用空间换时间，准确地说是用<strong>更高速的空间来换时间</strong>，从而<strong>整体</strong><strong>上</strong><strong>提升读的性能</strong>。</p>

<p>何为更高速的空间呢？</p>

<ol>
<li>更快的存储介质。通常情况下，如果说数据库的速度慢，就得用更快的存储介质去替代它，目前最常见的就是Redis。Redis 单实例的读 QPS 可以高达 10w/s，90% 的场景下只需要正确使用 Redis 就能应对。</li>
<li>就近使用本地内存。就像 CPU 也有高速缓存一样，缓存也可以分为一级缓存、二级缓存。即便 Redis 本身性能已经足够高了，但访问一次 Redis 毕竟也需要一次网络 IO，而使用本地内存无疑有更快的速度。不过单机的内存是十分有限的，所以这种一级缓存只能存储非常少量的数据，通常是最热点的那些 key 对应的数据。这就相当于额外消耗宝贵的服务内存去换取高速的读取性能。</li>
</ol>


<h2>引入缓存后的一致性挑战</h2>

<p>用空间换时间，意味着数据同时存在于多个空间。最常见的场景就是数据同时存在于 Redis 与 MySQL 上（为了问题的普适性，后面举例中若没有特别说明，缓存均指 Redis 缓存）。</p>

<p>实际上，最权威最全的数据还是在 MySQL 里的，只要 Redis 数据没有得到及时的更新而导致最新数据没有同步到 Redis 中，就出现了数据不一致。</p>

<p>大部分情况下，只要使用了缓存，就必然会有不一致的情况出现，只是说这个不一致的时间窗口是否能做到足够的小。有些不合理的设计可能会导致数据持续不一致，这是我们需要改善设计去避免的。</p>

<h2>缓存不一致性无法客观地完全消灭</h2>

<p>为什么我们几乎没办法做到缓存和数据库之间的强一致呢？</p>

<p>正常情况下，我们需要在数据库更新完后，把对应的最新数据同步到缓存中，以便在读请求的时候，能读到新的数据而不是旧的数据（脏数据）。但是很可惜，由于数据库和 Redis 之间是没有事务保证的，所以我们无法确保写入数据库成功后，写入 Redis 也是一定成功的；即便 Redis 写入能成功，在数据库写入成功后到 Redis 写入成功前的这段时间里，Redis 数据也肯定是和 MySQL 不一致的。如下图：</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-1.png" title="image-1" alt="图片" /></p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-2.png" title="image-2" alt="图片" />
所以说这个时间窗口是没办法完全消灭的，除非我们付出极大的代价，使用分布式事务等各种手段去维持强一致，但是这样会使得系统的整体性能大幅度下降，甚至比不用缓存还慢，这样不就与我们使用缓存的目标背道而驰了吗？</p>

<p>不过虽然无法做到强一致，但是我们能做到的是缓存与数据库达到最终一致，而且不一致的时间窗口我们能做到尽可能短，按照经验来说，如果能将时间优化到 1ms 之内，这个一致性问题带来的影响我们就可以忽略不计。</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-3.png" title="image-3" alt="图片" /></p>

<h2>更新缓存的手段</h2>

<p>通常情况下，我们在处理查询请求的时候，使用缓存的逻辑如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>data = queryDataRedis(key);
</span><span class='line'>if (data ==null) {
</span><span class='line'>     data = queryDataMySQL(key); //缓存查询不到，从MySQL做查询
</span><span class='line'>     if (data!=null) {
</span><span class='line'>         updateRedis(key, data);//查询完数据后更新到MySQL
</span><span class='line'>     }
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>也就是说<strong>优先查询缓存，查询不到才查询数据库</strong>。如果这时候数据库查到数据了，就将缓存的数据进行更新。
这样的逻辑是正确的，而一致性的问题一般不来源于此，而是出现在处理<strong>写请求</strong>的时候。所以我们简化成最简单的写请求的逻辑，此时你可能会面临多个选择，究竟是直接更新缓存，还是失效缓存？而无论是更新缓存还是失效缓存，都可以选择在更新数据库之前，还是之后操作。</p>

<p>这样就演变出 4 个策略：<strong>更新数据库后更新缓存、更新数据库前更新缓存、更新数据库后删除缓存、更新数据库前删除缓存</strong><strong>。</strong>下面我们来分别讲述。</p>

<h3>更新数据库后更新缓存的不一致问题</h3>

<p>一种常见的操作是，设置一个过期时间，让写请求以数据库为准，过期后，读请求同步数据库中的最新数据给缓存。那么在加入了过期时间后，是否就不会有问题了呢？并不是这样。</p>

<p>大家设想一下这样的场景。</p>

<p>假如这里有一个计数器，把数据库自减 1，原始数据库数据是 100，同时有两个写请求申请计数减一，假设线程 A 先减数据库成功，线程 B 后减数据库成功。那么这时候数据库的值是 98，缓存里正确的值应该也要是 98。</p>

<p>但是特殊场景下，你可能会遇到这样的情况：</p>

<ol>
<li>线程 A 和线程 B 同时更新这个数据</li>
<li>更新数据库的顺序是先 A 后 B</li>
<li>更新缓存时顺序是先 B 后 A
如果我们的代码逻辑还是更新数据库后立刻更新缓存的数据，那么——</li>
</ol>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>updateMySQL();
</span><span class='line'>updateRedis(key, data);</span></code></pre></td></tr></table></div></figure>


<p>就可能出现：数据库的值是 100->99->98，但是缓存的数据却是 100->98->99，也就是数据库与缓存的不一致。而且这个不一致只能等到下一次数据库更新或者缓存失效才可能修复。</p>

<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong> </th>
<th style="text-align:left;"> <strong>线程B（写请求）</strong> </th>
<th style="text-align:left;"> <strong>问题</strong>                                                     </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T1       </td>
<td style="text-align:left;"> 更新数据库为99      </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T2       </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 更新数据库为98      </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T3       </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 更新缓存数据为98    </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T4       </td>
<td style="text-align:left;"> 更新缓存数据为99    </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 此时缓存的值被显式更新为99，但是实际上数据库的值已经是98，数据不一致 </td>
</tr>
</tbody>
</table>


<h3>更新数据库前更新缓存的不一致问题</h3>

<p>那你可能会想，这是否表示，我应该先让缓存更新，之后再去更新数据库呢？类似这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>updateRedis(key, data);//先更新缓存
</span><span class='line'>updateMySQL();//再更新数据库</span></code></pre></td></tr></table></div></figure>


<p>这样操作产生的问题更是显而易见的，因为我们无法保证数据库的更新成功，万一数据库更新失败了，你缓存的数据就不只是脏数据，而是错误数据了。
你可能会想，是否我在更新数据库失败的时候做 Redis 回滚的操作能够解决呢？这其实也是不靠谱的，因为我们也不能保证这个回滚的操作 100% 被成功执行。</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-4.png" title="image-4" alt="图片" /></p>

<p>同时，在写写并发的场景下，同样有类似的一致性问题，请看以下情况：</p>

<ol>
<li>线程 A 和线程 B 同时更新同这个数据</li>
<li>更新缓存的顺序是先 A 后 B</li>
<li>更新数据库的顺序是先 B 后 A
举个例子。线程 A 希望把计数器置为 0，线程 B 希望置为 1。而按照以上场景，缓存确实被设置为 1，但数据库却被设置为 0。</li>
</ol>


<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong> </th>
<th style="text-align:left;"> <strong>线程B（写请求）</strong> </th>
<th style="text-align:left;"> <strong>问题</strong>                                                     </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T1       </td>
<td style="text-align:left;"> 更新缓存为0         </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T2       </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 更新缓存为1         </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T3       </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 更新数据库为1       </td>
<td style="text-align:left;">                                                              </td>
</tr>
<tr>
<td style="text-align:left;"> T4       </td>
<td style="text-align:left;"> 更新数据库数据为0   </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;"> 此时缓存的值被显式更新为1，但是实际上数据库的值是0，数据不一致 </td>
</tr>
</tbody>
</table>


<p>所以<strong>通常情况下，更新缓存再更新数据库是我们</strong><strong>应该避免使用</strong><strong>的</strong><strong>一种</strong><strong>手段</strong>。</p>

<h3>更新数据库前删除缓存的问题</h3>

<p>那如果采取删除缓存的策略呢？也就是说我们在更新数据库的时候失效对应的缓存，让缓存在下次触发读请求时进行更新，是否会更好呢？同样地，针对在更新数据库前和数据库后这两个删除时机，我们来比较下其差异。</p>

<p>最直观的做法，我们可能会先让缓存失效，然后去更新数据库，代码逻辑如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>deleteRedis(key);//先删除缓存让缓存失效
</span><span class='line'>updateMySQL();//再更新数据库</span></code></pre></td></tr></table></div></figure>


<p>这样的逻辑看似没有问题，毕竟删除缓存后即便数据库更新失败了，也只是缓存上没有数据而已。然后并发两个写请求过来，无论怎么样的执行顺序，缓存最后的值也都是会被删除的，也就是说在并发写写的请求下这样的处理是没问题的。
然而，这种处理在读写并发的场景下却存在着隐患。</p>

<p>还是刚刚更新计数的例子。例如现在缓存的数据是 100，数据库也是 100，这时候需要对此计数减 1，减成功后，数据库应该是 99。如果这之后触发读请求，缓存如果有效的话，里面应该也要被更新为 99 才是正确的。</p>

<p>那么思考下这样的请求情况：</p>

<ol>
<li>线程 A 更新这个数据的同时，线程 B 读取这个数据</li>
<li>线程 A 成功删除了缓存里的老数据，这时候线程 B 查询数据发现缓存失效</li>
<li>线程 A 更新数据库成功</li>
</ol>


<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong>         </th>
<th style="text-align:left;"> <strong>线程B（读请求）</strong>                           </th>
<th style="text-align:left;"> <strong>问题</strong>                                                    </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T1       </td>
<td style="text-align:left;"> 删除缓存值                  </td>
<td style="text-align:left;">                                               </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T2       </td>
<td style="text-align:left;">                             </td>
<td style="text-align:left;"> 1.读取缓存数据，缓存缺失，从数据库读取数据100 </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T3       </td>
<td style="text-align:left;"> 更新数据库中的数据X的值为99 </td>
<td style="text-align:left;">                                               </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T4       </td>
<td style="text-align:left;">                             </td>
<td style="text-align:left;"> 将数据100的值写入缓存                         </td>
<td style="text-align:left;"> 此时缓存的值被显式更新为100，但是实际上数据库的值已经是99了 </td>
</tr>
</tbody>
</table>


<p>可以看到，在读写并发的场景下，一样会有不一致的问题。</p>

<p>针对这种场景，有个做法是所谓的“<strong>延迟双删策略</strong>”，就是说，既然可能因为读请求把一个旧的值又写回去，那么我在写请求处理完之后，等到差不多的时间延迟再重新删除这个缓存值。</p>

<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong> </th>
<th style="text-align:left;"> <strong>线程C（新的读请求）</strong>     </th>
<th style="text-align:left;"> <strong>线程D（新的读请求）</strong>                  </th>
<th style="text-align:left;"> <strong>问题</strong>                           </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T5       </td>
<td style="text-align:left;"> sleep(N)            </td>
<td style="text-align:left;"> 缓存存在，读取到缓存旧值100 </td>
<td style="text-align:left;">                                          </td>
<td style="text-align:left;"> 其他线程可能在双删成功前读到脏数据 </td>
</tr>
<tr>
<td style="text-align:left;"> T6       </td>
<td style="text-align:left;"> 删除缓存值          </td>
<td style="text-align:left;">                             </td>
<td style="text-align:left;">                                          </td>
<td style="text-align:left;">                                    </td>
</tr>
<tr>
<td style="text-align:left;"> T7       </td>
<td style="text-align:left;">                     </td>
<td style="text-align:left;">                             </td>
<td style="text-align:left;"> 缓存缺失，从数据库读取数据的最新值（99） </td>
<td style="text-align:left;">                                    </td>
</tr>
</tbody>
</table>


<p>这种解决思路的关键在于对 N 的时间的判断，如果 N 时间太短，线程 A 第二次删除缓存的时间依旧早于线程 B 把脏数据写回缓存的时间，那么相当于做了无用功。而 N 如果设置得太长，那么在触发双删之前，新请求看到的都是脏数据。</p>

<h3>更新数据库后删除缓存</h3>

<p>那如果我们把更新数据库放在删除缓存之前呢，问题是否解决？我们继续从读写并发的场景看下去，有没有类似的问题。</p>

<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong>             </th>
<th style="text-align:left;"> <strong>线程B（读请求）</strong>                        </th>
<th style="text-align:left;"> <strong>线程C（读请求）</strong>                   </th>
<th style="text-align:left;"> <strong>潜在问题</strong>                            </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T1       </td>
<td style="text-align:left;"> 更新主库 X = 99（原值 X = 100） </td>
<td style="text-align:left;">                                            </td>
<td style="text-align:left;">                                       </td>
<td style="text-align:left;">                                         </td>
</tr>
<tr>
<td style="text-align:left;"> T2       </td>
<td style="text-align:left;">                                 </td>
<td style="text-align:left;">                                            </td>
<td style="text-align:left;"> 读取数据，查询到缓存还有数据，返回100 </td>
<td style="text-align:left;"> 线程C实际上读取到了和数据库不一致的数据 </td>
</tr>
<tr>
<td style="text-align:left;"> T3       </td>
<td style="text-align:left;"> 删除缓存                        </td>
<td style="text-align:left;">                                            </td>
<td style="text-align:left;">                                       </td>
<td style="text-align:left;">                                         </td>
</tr>
<tr>
<td style="text-align:left;"> T4       </td>
<td style="text-align:left;">                                 </td>
<td style="text-align:left;"> 查询缓存，缓存缺失，查询数据库得到当前值99 </td>
<td style="text-align:left;">                                       </td>
<td style="text-align:left;">                                         </td>
</tr>
<tr>
<td style="text-align:left;"> T5       </td>
<td style="text-align:left;">                                 </td>
<td style="text-align:left;"> 将99写入缓存                               </td>
<td style="text-align:left;">                                       </td>
<td style="text-align:left;">                                         </td>
</tr>
</tbody>
</table>


<p>可以看到，大体上，采取先更新数据库再删除缓存的策略是没有问题的，仅在更新数据库成功到缓存删除之间的时间差内，可能会被别的线程读取到老值。</p>

<p>而在开篇的时候我们说过，缓存不一致性的问题无法在客观上完全消灭，因为我们无法保证数据库和缓存的操作是一个事务里的，而我们能做到的只是尽量缩短不一致的时间窗口。</p>

<p>在更新数据库后删除缓存这个场景下，不一致窗口仅仅是 T2 到 T3 的时间，大概是 1ms 左右，在大部分业务场景下我们都可以忽略不计。</p>

<p>但是真实场景下，还是会有一个情况存在不一致的可能性，这个场景是读线程发现缓存不存在，于是读写并发时，读线程回写进去老值。并发情况如下：</p>

<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>时间</strong> </th>
<th style="text-align:left;"> <strong>线程A（写请求）</strong>             </th>
<th style="text-align:left;"> <strong>线程B（读请求&ndash;缓存不存在场景）</strong>         </th>
<th style="text-align:left;"> <strong>潜在问题</strong>                                                </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> T1       </td>
<td style="text-align:left;">                                 </td>
<td style="text-align:left;"> 查询缓存，缓存缺失，查询数据库得到当前值100 </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T2       </td>
<td style="text-align:left;"> 更新主库 X = 99（原值 X = 100） </td>
<td style="text-align:left;">                                             </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T3       </td>
<td style="text-align:left;"> 删除缓存                        </td>
<td style="text-align:left;">                                             </td>
<td style="text-align:left;">                                                             </td>
</tr>
<tr>
<td style="text-align:left;"> T4       </td>
<td style="text-align:left;">                                 </td>
<td style="text-align:left;"> 将100写入缓存                               </td>
<td style="text-align:left;"> 此时缓存的值被显式更新为100，但是实际上数据库的值已经是99了 </td>
</tr>
</tbody>
</table>


<p>总的来说，这个不一致场景出现条件非常严格，因为并发量很大时，缓存不太可能不存在；如果并发很大，而缓存真的不存在，那么很可能是这时的写场景很多，因为写场景会删除缓存。所以待会我们会提到，写场景很多时候实际上并不适合采取删除策略。</p>

<h3>总结四种更新策略</h3>

<p>终上所述，我们对比了四个更新缓存的手段，做一个总结对比，如下图：</p>

<table>
<thead>
<tr>
<th style="text-align:left;"> <strong>策略</strong>              </th>
<th style="text-align:left;"> <strong>并发场景</strong>        </th>
<th style="text-align:left;"> <strong>潜在问题</strong>                                                 </th>
<th style="text-align:left;"> <strong>应对方案</strong>                                                 </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;"> 更新数据库+更新缓存   </td>
<td style="text-align:left;"> 写+读               </td>
<td style="text-align:left;"> 线程A未更新完缓存之前，线程B的读请求会短暂读到旧值           </td>
<td style="text-align:left;"> 可以忽略                                                     </td>
</tr>
<tr>
<td style="text-align:left;">                       </td>
<td style="text-align:left;"> 写+写               </td>
<td style="text-align:left;"> 更新数据库的顺序是先A后B，但更新缓存时顺序是先B后A，数据库和缓存数据不一致 </td>
<td style="text-align:left;"> 分布式锁（操作重）                                           </td>
</tr>
<tr>
<td style="text-align:left;"> 更新缓存+更新数据库   </td>
<td style="text-align:left;"> 无并发              </td>
<td style="text-align:left;"> 线程A还未更新完缓存但是更新数据库可能失败                    </td>
<td style="text-align:left;"> 利用MQ确认数据库更新成功（较复杂）                           </td>
</tr>
<tr>
<td style="text-align:left;">                       </td>
<td style="text-align:left;"> 写+写               </td>
<td style="text-align:left;"> 更新缓存的顺序是先A后B，但更新数据库时顺序是先B后A           </td>
<td style="text-align:left;"> 分布式锁（操作很重）                                         </td>
</tr>
<tr>
<td style="text-align:left;"> 删除缓存值+更新数据库 </td>
<td style="text-align:left;"> 写+读               </td>
<td style="text-align:left;"> 写请求的线程A删除了缓存在更新数据库之前，这时候读请求线程B到来，因为缓存缺失，则把当前数据读取出来放到缓存，而后线程A更新成功了数据库 </td>
<td style="text-align:left;"> 延迟双删（但是延迟的时间不好估计，且延迟的过程中依旧有不一致的时间窗口） </td>
</tr>
<tr>
<td style="text-align:left;"> 更新数据库+删除缓存值 </td>
<td style="text-align:left;"> 写+读（缓存命中）   </td>
<td style="text-align:left;"> 线程A完成数据库更新成功后，尚未删除缓存，线程B有并发读请求会读到旧的脏数据<br> </td>
<td style="text-align:left;"> 可以忽略                                                     </td>
</tr>
<tr>
<td style="text-align:left;">                       </td>
<td style="text-align:left;"> 写+读（缓存不命中） </td>
<td style="text-align:left;"> 读请求不命中缓存，写请求处理完之后读请求才回写缓存，此时缓存不一致 </td>
<td style="text-align:left;"> 分布式锁（操作重）                                           </td>
</tr>
</tbody>
</table>


<p><strong>从一致性的角度来看，采取更新数据库后删除缓存值，是更为适合的策略</strong><strong>。</strong>因为出现不一致的场景的条件更为苛刻，概率相比其他方案更低。</p>

<p>那么是否更新缓存这个策略就一无是处呢？不是的！</p>

<p>删除缓存值意味着对应的 key 会失效，那么这时候读请求都会打到数据库。如果这个数据的写操作非常频繁，就会导致缓存的作用变得非常小。而如果这时候某些 Key 还是非常大的热 key，就可能因为扛不住数据量而导致系统不可用。</p>

<p>如下图所示：</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-5.png" title="删除策略频繁的缓存失效导致读请求无法利用缓存" alt="图片" /></p>

<p>所以做个简单总结，足以适应绝大部分的互联网开发场景的决策：</p>

<ul>
<li><p><strong>针对大部分读多写少场景，建议选择更新数据库后删除缓存的策略。</strong></p></li>
<li><p><strong>针对读写相当或者写多读少的场景，建议选择更新数据库后更新缓存的策略。</strong></p></li>
</ul>


<h2>最终一致性如何保证？</h2>

<h3>缓存设置过期时间</h3>

<p>第一个方法便是我们上面提到的，当我们无法确定 MySQL 更新完成后，缓存的更新/删除一定能成功，例如 Redis 挂了导致写入失败了，或者当时网络出现故障，更常见的是服务当时刚好发生重启了，没有执行这一步的代码。</p>

<p>这些时候 MySQL 的数据就无法刷到 Redis 了。为了避免这种不一致性永久存在，使用缓存的时候，我们必须要给缓存设置一个过期时间，例如 1 分钟，这样即使出现了更新 Redis 失败的极端场景，不一致的时间窗口最多也只是 1 分钟。</p>

<p>这是我们最终一致性的兜底方案，万一出现任何情况的不一致问题，最后都能通过缓存失效后重新查询数据库，然后回写到缓存，来做到缓存与数据库的最终一致。</p>

<h3>如何减少缓存删除/更新的失败？</h3>

<p>万一删除缓存这一步因为服务重启没有执行，或者 Redis 临时不可用导致删除缓存失败了，就会有一个较长的时间（缓存的剩余过期时间）是数据不一致的。</p>

<p>那我们有没有什么手段来减少这种不一致的情况出现呢？这时候借助一个可靠的消息中间件就是一个不错的选择。</p>

<p>因为消息中间件有 ATLEAST-ONCE 的机制，如下图所示。</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-6.png" title="image-6" alt="图片" /></p>

<p>我们把删除 Redis 的请求以消费 MQ 消息的手段去失效对应的 Key 值，如果 Redis 真的存在异常导致无法删除成功，我们依旧可以依靠 MQ 的重试机制来让最终 Redis 对应的 Key 失效。</p>

<p>而你们或许会问，极端场景下，是否存在更新数据库后 MQ 消息没发送成功，或者没机会发送出去机器就重启的情况？</p>

<p>这个场景的确比较麻烦，如果 MQ 使用的是 RocketMQ，我们可以借助 RocketMQ 的事务消息，来让删除缓存的消息最终一定发送出去。而如果你没有使用 RocketMQ，或者你使用的消息中间件并没有事务消息的特性，则可以采取消息表的方式让更新数据库和发送消息一起成功。事实上这个话题比较大了，我们不在这里展开。</p>

<h3>如何处理复杂的多缓存场景？</h3>

<p>有些时候，真实的缓存场景并不是数据库中的一个记录对应一个 Key 这么简单，有可能一个数据库记录的更新会牵扯到多个 Key 的更新。还有另外一个场景是，更新不同的数据库的记录时可能需要更新同一个 Key 值，这常见于一些 App 首页数据的缓存。</p>

<p>我们以一个数据库记录对应多个 Key 的场景来举例。</p>

<p>假如系统设计上我们缓存了一个粉丝的主页信息、主播打赏榜 TOP10 的粉丝、单日 TOP 100 的粉丝等多个信息。如果这个粉丝注销了，或者这个粉丝触发了打赏的行为，上面多个 Key 可能都需要更新。只是一个打赏的记录，你可能就要做：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>updateMySQL();//更新数据库一条记录
</span><span class='line'>deleteRedisKey1();//失效主页信息的缓存
</span><span class='line'>updateRedisKey2();//更新打赏榜TOP10
</span><span class='line'>deleteRedisKey3();//更新单日打赏榜TOP100</span></code></pre></td></tr></table></div></figure>


<p>这就涉及多个 Redis 的操作，每一步都可能失败，影响到后面的更新。甚至从系统设计上，更新数据库可能是单独的一个服务，而这几个不同的 Key 的缓存维护却在不同的 3 个微服务中，这就大大增加了系统的复杂度和提高了缓存操作失败的可能性。最可怕的是，操作更新记录的地方很大概率不只在一个业务逻辑中，而是散发在系统各个零散的位置。
针对这个场景，解决方案和上文提到的保证最终一致性的操作一样，就是把更新缓存的操作以 MQ 消息的方式发送出去，由不同的系统或者专门的一个系统进行订阅，而做聚合的操作。如下图：</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-7.png" title="不同业务系统订阅MQ消息单独维护各自的缓存Key" alt="图片" /></p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-8.png" title="专门更新缓存的服务订阅MQ消息维护所有相关Key的缓存操作" alt="图片" /></p>

<h3>通过订阅 MySQL binlog 的方式处理缓存</h3>

<p>上面讲到的 MQ 处理方式需要业务代码里面显式地发送 MQ 消息。还有一种优雅的方式便是订阅 MySQL 的 binlog，监听数据的真实变化情况以处理相关的缓存。</p>

<p>例如刚刚提到的例子中，如果粉丝又触发打赏了，这时候我们利用 binlog 表监听是能及时发现的，发现后就能集中处理了，而且无论是在什么系统什么位置去更新数据，都能做到集中处理。</p>

<p>目前业界类似的产品有 Canal，具体的操作图如下：</p>

<p><img src="http://jaskey.github.io/images/cache-consistency/image-9.png" title="利用Canel订阅数据库binlog变更从而发出MQ消息，让一个专门消费者服务维护所有相关Key的缓存操作" alt="图片" /></p>

<p>到这里，针对大型系统缓存设计如何保证最终一致性，我们已经从策略、场景、操作方案等角度进行了细致的讲述，这些是我根据多年开发经验进行总结的，希望能对你起到帮助。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[为什么在一段时间内RocketMQ的队列同时分配给了两个消费者？详细剖析消费者负载均衡中的坑（上）]]></title>
    <link href="https://Jaskey.github.io/blog/2020/11/26/rocketmq-consumer-allocate/"/>
    <updated>2020-11-26T15:37:53+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/11/26/rocketmq-consumer-allocate</id>
    <content type="html"><![CDATA[<p>之前的文章有提到过，消费者大概是怎么做负载均衡的（集群模式），如下图所示：</p>

<p><img src="https://jaskey.github.io/images/rocketmq/consumer-loadbalance1.png" alt="消费者负载均衡" /></p>

<p>集群模式下，每个消费者实例会被分配到若干条队列。正因为消费者拿到了明确的队列，所以它们才能针对对应的队列做循环拉取消息的处理，以下是消费者客户端和broker通信的部分代码，可以看到通信的参数里有一个重要的参数，就是queueId</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'>        <span class="n">PullMessageRequestHeader</span> <span class="n">requestHeader</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PullMessageRequestHeader</span><span class="o">();</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setConsumerGroup</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">consumerGroup</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setTopic</span><span class="o">(</span><span class="n">mq</span><span class="o">.</span><span class="na">getTopic</span><span class="o">());</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setQueueId</span><span class="o">(</span><span class="n">mq</span><span class="o">.</span><span class="na">getQueueId</span><span class="o">());</span><span class="c1">//消息拉取必须显示的告诉broker拉取哪个queue的消息</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setQueueOffset</span><span class="o">(</span><span class="n">offset</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setMaxMsgNums</span><span class="o">(</span><span class="n">maxNums</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setSysFlag</span><span class="o">(</span><span class="n">sysFlagInner</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setCommitOffset</span><span class="o">(</span><span class="n">commitOffset</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setSuspendTimeoutMillis</span><span class="o">(</span><span class="n">brokerSuspendMaxTimeMillis</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setSubscription</span><span class="o">(</span><span class="n">subExpression</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setSubVersion</span><span class="o">(</span><span class="n">subVersion</span><span class="o">);</span>
</span><span class='line'>        <span class="n">requestHeader</span><span class="o">.</span><span class="na">setExpressionType</span><span class="o">(</span><span class="n">expressionType</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">String</span> <span class="n">brokerAddr</span> <span class="o">=</span> <span class="n">findBrokerResult</span><span class="o">.</span><span class="na">getBrokerAddr</span><span class="o">();</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="n">PullSysFlag</span><span class="o">.</span><span class="na">hasClassFilterFlag</span><span class="o">(</span><span class="n">sysFlagInner</span><span class="o">))</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">brokerAddr</span> <span class="o">=</span> <span class="n">computPullFromWhichFilterServer</span><span class="o">(</span><span class="n">mq</span><span class="o">.</span><span class="na">getTopic</span><span class="o">(),</span> <span class="n">brokerAddr</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">PullResult</span> <span class="n">pullResult</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">mQClientFactory</span><span class="o">.</span><span class="na">getMQClientAPIImpl</span><span class="o">().</span><span class="na">pullMessage</span><span class="o">(</span>
</span><span class='line'>            <span class="n">brokerAddr</span><span class="o">,</span>
</span><span class='line'>            <span class="n">requestHeader</span><span class="o">,</span>
</span><span class='line'>            <span class="n">timeoutMillis</span><span class="o">,</span>
</span><span class='line'>            <span class="n">communicationMode</span><span class="o">,</span>
</span><span class='line'>            <span class="n">pullCallback</span><span class="o">);</span>
</span></code></pre></td></tr></table></div></figure>


<p>这侧面也再次印证，RocketMQ的消费模型是Pull模式。</p>

<p>同时，对于每个消费者实例来说，在每个消息拉取之前，实际上都是确定了队列的（不会轻易发生改变），如下图控制台所示：</p>

<p><img src="https://jaskey.github.io/images/rocketmq/rocketmq-queue-allocation.png" alt="消费者负载均衡控制台示例" /></p>

<p>本文尝试对RocketMQ负载均衡（哪个消费者消费哪些队列）的原理进行解析，希望能让大家对其中的基本原理进行了解，并对部分问题能作出合理解析和正确规避。</p>

<h2>所谓Rebalance到底在解决什么问题</h2>

<p>RocketMQ每次分配队列的过程，代码里叫Relalance，本文在某些场景下也称为重排，实际上是一个负载均衡的过程。之所以说分配队列的过程就是负载均衡的过程的原因是，RocketMQ是负载均衡分配的就是队列，而不是消息。如果这个过程RocketMQ给了较高负载高，其实并不肯定意味着你能接受更多的消息（虽然绝大部分场景你可以这样理解），而只是说我给你分配了更多的队列。为什么说有更多的队列可能并不代表你有更多消息消费呢？</p>

<p>例如我们举一个例子，两个消费者一个消费者实例A获得了1个队列q0，一个消费者实例B获得了两个队列，这个负载均衡的过程分配了给B更多的&#8221;负载&#8221;（队列），但是假设消费者B获得的两个队列q1 q2中的q2本身是不可写的（topic可以配置读队列数量，写队列数量，所以是可能存在一些队列可读，但是不可写的情况），又或者生产者手动的选择了发送topic的queue目标（利用selector），这个过程从来都不选择q2，只有q0,和q1在做发送，甚至大部分情况下都往q0发，这时候消费者B实例其实都没有真正意义上的更高负载。</p>

<p>总结一下：就是所谓的消费者Rebalance，其实是分配队列的过程，它本质上希望解决的是一个消费者的负载问题，但是实际的工作其并不直接改变一个消费者实例的真实负载（消息），而是间接的决定的——通过管理分配队列的数量。而平时我们绝大部分可以认为队列的负载就是真实的消息负载的原因是基于这样一个前提：消息的分布基本是均匀分配在不同的队列上的，所以在这个前提下，获得了更多的队列实际上就是获得了更多的消息负载。</p>

<h2>Relance具体是如何决定分配的数量的</h2>

<p>RocketMQ的Rebalance实际上是<strong>无中心</strong>的，这和Kafka有本质区别，Kafka虽然也是客户端做的负载均衡，但是Kafka在做负载均衡之前会选定一个Leader，由这个Leader全局把控分配的过程，而后再把每个消费者对partion的分配结果广播给各个消费者。</p>

<p>而RocketMQ实际上没有人做这个统一分配的，而是每个消费者自己&#8221;有秩序地&#8221;计算出自己应该获取哪些队列，你可能会觉得很神奇，到底为啥大家能如此有秩序而不打架呢？我们下面来看看。</p>

<p>你可能知道RocketMQ是支持很多负载均衡的算法的，甚至还支持用户自己实现一个负载均衡算法。具体的这个分配算法需要实现以下接口：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="cm">/** * Strategy Algorithm for message allocating between consumers */</span><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">AllocateMessageQueueStrategy</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>    <span class="cm">/**     </span>
</span><span class='line'><span class="cm">    * Allocating by consumer id     </span>
</span><span class='line'><span class="cm">    *     </span>
</span><span class='line'><span class="cm">    * @param consumerGroup current consumer group     </span>
</span><span class='line'><span class="cm">    * @param currentCID current consumer id     </span>
</span><span class='line'><span class="cm">    * @param mqAll message queue set in current topic     </span>
</span><span class='line'><span class="cm">    * @param cidAll consumer set in current consumer group     </span>
</span><span class='line'><span class="cm">    * @return The allocate result of given strategy     */</span>
</span><span class='line'>    <span class="n">List</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;</span> <span class="nf">allocate</span><span class="o">(</span><span class="kd">final</span> <span class="n">String</span> <span class="n">consumerGroup</span><span class="o">,</span><span class="kd">final</span> <span class="n">String</span> <span class="n">currentCID</span><span class="o">,</span>        <span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;</span> <span class="n">mqAll</span><span class="o">,</span> <span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">cidAll</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>    <span class="cm">/** * Algorithm name    </span>
</span><span class='line'><span class="cm">    *     * @return The strategy name     </span>
</span><span class='line'><span class="cm">    */</span>
</span><span class='line'>    <span class="n">String</span> <span class="nf">getName</span><span class="o">();}</span>
</span></code></pre></td></tr></table></div></figure>


<p>这个接口的getName()只是一个唯一标识，用以标识该消费者实例是用什么负载均衡算法去分配队列。</p>

<p>关键在于<code>allocate</code>这个方法，这个方法的出参就是这次Rebalace的结果——本消费者实例应该去获取的队列列表。</p>

<p>其余四个入参分别是：</p>

<p>1.消费者组名</p>

<p>2.当前的消费者实例的唯一ID，实际上就是client 的ip@instanceName。</p>

<p>3.全局这个消费者组可以分配的队列集合</p>

<p>4.当前这个消费者组消费者集合（值是消费者实例的唯一id）</p>

<p>试想下，假设要你去做一个分配队列的算法，实际上最关键的就是两个视图：1.这个topic下全局当前在线的消费者列表，2.topic在全局下有哪些队列。</p>

<p>例如，你知道当前有4个消费者 c1 c2 c3 c4在线，也知道topic 下有 8个队列 q0,q1,q2,q3,q4,&hellip;q6，那么8/4=2，你就能知道每个消费者应该获取两个队列。例如： c1&ndash;>q0,q1, c2&ndash;>q2,q3, c3&ndash;>q4,q5, c4&ndash;>q5,q6。</p>

<p>实际上，这就是rocketmq默认的分配方案。</p>

<p>但现在唯一的问题在于，我们刚刚说的，我们没有一个中心节点统一地做分配，所以RocketMQ需要做一定的修改。如对于C1：</p>

<p>“我是C1，我知道当前有4个消费者 c1 c2 c3 c4在线，也知道topic 下有 8个队列 q0,q1,q2,q3,q4,&hellip;q6，那么8/4=2，我就能知道每个消费者应该获取两个队列，而我算出来我要的队列是c1&ndash;>q0,q1&#8221;。</p>

<p>同理对于C2：</p>

<p>“我是C2，我知道当前有4个消费者 c1 c2 c3 c4在线，也知道topic 下有 8个队列 q0,q1,q2,q3,q4,&hellip;q6，那么8/4=2，我就能知道每个消费者应该获取两个队列，而我算出来我要的队列是c2&ndash;>q2,q3。</p>

<p>要做到无中心的完成这个目标，唯一需要增加的输入项就是“我是C1”，&#8221;我是C2&#8221;这样的入参，所以上文提到的<code>allocate</code>方法下面<strong>当前的消费者实例</strong>的唯一ID就是干这个事用的。以下是一个默认的策略，本人添加了中文注释，以达到的就是上文例子中的分配结果：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="nd">@Override</span>
</span><span class='line'><span class="kd">public</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;</span> <span class="nf">allocate</span><span class="o">(</span><span class="n">String</span> <span class="n">consumerGroup</span><span class="o">,</span> <span class="n">String</span> <span class="n">currentCID</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;</span> <span class="n">mqAll</span><span class="o">,</span><span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">cidAll</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//START: 一些前置的判断</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">currentCID</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">currentCID</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">&lt;</span> <span class="mi">1</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalArgumentException</span><span class="o">(</span><span class="s">&quot;currentCID is empty&quot;</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">mqAll</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>        <span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalArgumentException</span><span class="o">(</span><span class="s">&quot;mqAll is null or mqAll empty&quot;</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">cidAll</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>        <span class="k">throw</span> <span class="k">new</span> <span class="nf">IllegalArgumentException</span><span class="o">(</span><span class="s">&quot;cidAll is null or cidAll empty&quot;</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">List</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;</span> <span class="n">result</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;</span><span class="n">MessageQueue</span><span class="o">&gt;();</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(!</span><span class="n">cidAll</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="n">currentCID</span><span class="o">))</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[BUG] ConsumerGroup: {} The consumerId: {} not in cidAll: {}&quot;</span><span class="o">,</span>
</span><span class='line'>            <span class="n">consumerGroup</span><span class="o">,</span>
</span><span class='line'>            <span class="n">currentCID</span><span class="o">,</span>
</span><span class='line'>            <span class="n">cidAll</span><span class="o">);</span>
</span><span class='line'>        <span class="k">return</span> <span class="n">result</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>    <span class="c1">//END: 一些前置的判断</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//核心分配逻辑开始</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">index</span> <span class="o">=</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">indexOf</span><span class="o">(</span><span class="n">currentCID</span><span class="o">);</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">mod</span> <span class="o">=</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">%</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">size</span><span class="o">();</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">averageSize</span> <span class="o">=</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">&lt;=</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">?</span> <span class="mi">1</span> <span class="o">:</span> <span class="o">(</span><span class="n">mod</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="n">index</span> <span class="o">&lt;</span> <span class="n">mod</span> <span class="o">?</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">/</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">:</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">/</span> <span class="n">cidAll</span><span class="o">.</span><span class="na">size</span><span class="o">());</span><span class="c1">//平均分配，每个cid分配多少队列</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">startIndex</span> <span class="o">=</span> <span class="o">(</span><span class="n">mod</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="n">index</span> <span class="o">&lt;</span> <span class="n">mod</span><span class="o">)</span> <span class="o">?</span> <span class="n">index</span> <span class="o">*</span> <span class="n">averageSize</span> <span class="o">:</span> <span class="n">index</span> <span class="o">*</span> <span class="n">averageSize</span> <span class="o">+</span> <span class="n">mod</span><span class="o">;</span> <span class="c1">//从哪里开始分配，分配的位点index是什么。</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">range</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">min</span><span class="o">(</span><span class="n">averageSize</span><span class="o">,</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">-</span> <span class="n">startIndex</span><span class="o">);</span><span class="c1">//真正分配的数量，避免除不尽的情况（实际上，有除不尽的情况）</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//开始分配本cid应该拿的队列列表</span>
</span><span class='line'>    <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">range</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">result</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">mqAll</span><span class="o">.</span><span class="na">get</span><span class="o">((</span><span class="n">startIndex</span> <span class="o">+</span> <span class="n">i</span><span class="o">)</span> <span class="o">%</span> <span class="n">mqAll</span><span class="o">.</span><span class="na">size</span><span class="o">()));</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">result</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h2>Rebalance是怎么对多Topic做分配</h2>

<p>细心地你可能会提一个问题，上面的提到的策略分配接口里，没有Topic的订阅关系的信息，那么如果一个消费者组订阅了topic1也订阅了topic2，topic下的队列数量可能是不一样的，那么最后分配的结果肯定也是不同的，那么怎么分配的呢？</p>

<p>答案是：一次topic的分配就单独调用一次分配接口，每次rebalance，实际上都会被RebalanceImpl里的rebalanceByTopic调用，而每订阅一个topic就会调用rebalanceByTopic，从而触发一次上文讲到的分配策略</p>

<h2>Rebalance什么时候触发</h2>

<p>其实看完上文，我们已经知道RocketMQ客户端是怎么无中心地做队列分配的了。现在还有一个问题，就是这个触发时机是什么时候？</p>

<p>为什么触发时机很重要呢？试想一下，突然间假设有一个消费者实例扩容了，从4个变成5个。如果有一个实例以5个去做负载均衡，其他四个老消费者以为在线的消费者还是只有四个，最后分配的结果肯定是会有重复的（某些情况甚至会漏分配），所以这个“节奏”很重要。</p>

<p>简单地来说，RocketMQ有三个时机会触发负载均衡：</p>

<ol>
<li><p>启动的时候，会立即触发</p></li>
<li><p>有消费实例数量的变更的时候。broker在接受到消费者的心跳包的时候如果发现这个实例是新的实例的时候，会广播一个消费者数量变更的事件给所有消费者实例；同理，当发现一个消费者实例的连接断了，也会广播这样的一个事件</p></li>
<li>定期触发（默认20秒）。</li>
</ol>


<p>第一个时机很好理解。启动的时候，消费者需要需要知道自己要分配什么队列，所以要触发Rebalance。</p>

<p>第二个时机实际也很好理解。因为有实例的数量变更，所以分配的结果肯定也需要调整的，这时候就要广播给各消费者。</p>

<p>第三点定期触发的原因实际上是一个补偿机制，为了避免第二点广播的时候因为网络异常等原因丢失了重分配的信号，或者还有别的场景实际上也需要重新计算分配结果（例如队列的数量变化、权限变化），所以需要一个定时任务做补偿。</p>

<p>从以上的触发时机可以看出，大部分情况下，消费者实例应该都是“节奏一致的”，如果出现异常场景或某些特殊场景，也会因为定时任务的补偿而达到最终一致的状态。所以如果你发现消费者分配有重复/漏分，很有可能这个消费者有短暂异常，没有及时地触发Rebalance，这个也可以从客户端日志中看出问题以便具体排查：如果一个消费者负载均衡后发现自己的分配的队列发生了变化：会有类似的日志（每一个Topic都会单独打印）：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">rebalanced</span> <span class="n">result</span> <span class="n">changed</span><span class="o">.</span> <span class="n">allocateMessageQueueStrategyName</span><span class="o">=</span><span class="n">AVG</span><span class="o">,</span> <span class="n">group</span><span class="o">=</span><span class="n">my</span><span class="o">-</span><span class="n">consumer</span><span class="o">,</span> <span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">clientId</span><span class="o">=</span><span class="mf">10.22</span><span class="o">.</span><span class="mf">224.39</span><span class="err">@</span><span class="mi">114452</span><span class="o">,</span> <span class="n">mqAllSize</span><span class="o">=</span><span class="mi">9</span><span class="o">,</span> <span class="n">cidAllSize</span><span class="o">=</span><span class="mi">1</span><span class="o">,</span> <span class="n">rebalanceResultSize</span><span class="o">=</span><span class="mi">9</span><span class="o">,</span> <span class="n">rebalanceResultSet</span><span class="o">=[</span><span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">2</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">1</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">2</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">3</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">0</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">0</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">1</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">3</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">2</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">0</span><span class="o">],</span> <span class="n">MessageQueue</span> <span class="o">[</span><span class="n">topic</span><span class="o">=</span><span class="n">topic_event_repay</span><span class="o">,</span> <span class="n">brokerName</span><span class="o">=</span><span class="n">broker</span><span class="o">-</span><span class="mi">3</span><span class="o">,</span> <span class="n">queueId</span><span class="o">=</span><span class="mi">1</span><span class="o">]]</span>
</span></code></pre></td></tr></table></div></figure>


<p>从而判断是否及时地触发了负载均衡。</p>

<p>注：虽然每次Rebalance都会触发，但是如果重新分配后发现和原来已分配的队列是一致的，并不会有实际的重排动作。如：上次分配的是q0,q1，这次分配的也是q0,q1意味着整体的外部状态并没有修改，是不会有真正的重排动作的，这时候在日志上并不会有所表现。</p>

<h2>Rebalance可能会到来消息的重复</h2>

<p>实际上，Rebalance如果真的发现前后有变化（重排），这是一个很重的操作。因为它需要drop掉当前分配的队列以及其中的任务，还需要同步消费进度等。<strong>而由于这个过程比较长，且很可能每个消费者实际drop队列和分配队列是不一致的，所以通常情况下，重排都意味着有消息的重复投递。</strong>所以消费者端必须要做好消费的幂等。</p>

<p>我们不妨假设这样一个分配过程：A1本来拥有q0，这次重排需要拿q1，A2本来拥有q1，这次重排不需要q1了。那么对于A2来说，他首先要做的是：把q1的任务中断（drop队列），然后在合适的时机把q1的消费进度同步一下，再重新分配（这个例子这里不太重要），同样的A1也是要经历一样的过程：把q0的任务中断（drop队列），然后在合适的时机把q0的消费进度同步一下，然后重新分配——拿到q1。</p>

<p>我们假设A1的过程比A2要快，这里有两个可能：</p>

<p>1.一种情况是A1在A2把q1队列drop掉之前，A1就又拿到了q1，所以在这个时间窗口上观察，你会发现q1短暂地同时分配给了A1和A2。而由于RocketMQ的消费模型是Pull模式，所以A1、A2会同时拉取消息，消息就重复了。</p>

<p>2.另一种情况可能性更大，A2的确drop掉了队列不拉取了，但是消费进度（假设为OF1）还没及时同步到broker。那么A1拿到了q1之后，他需要第一时间知道自己从哪里（位点）拉取消息，所以他会询问一次broker，而broker这时候他的信息也是落后的，就会返回一个较老的消息位点OF2，那么[OF2,OF1]之间的消息就会重复。</p>

<p>可以看到，光负载均衡的这个实现原理，就会导致RocketMQ消息重复比一般的消息中间件概率要大，而且严重不少（消息是批量重复的）。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[消息幂等（去重）通用解决方案，RocketMQ]]></title>
    <link href="https://Jaskey.github.io/blog/2020/06/08/rocketmq-message-dedup/"/>
    <updated>2020-06-08T15:37:53+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/06/08/rocketmq-message-dedup</id>
    <content type="html"><![CDATA[<p>消息中间件是分布式系统常用的组件，无论是异步化、解耦、削峰等都有广泛的应用价值。我们通常会认为，消息中间件是一个可靠的组件——这里所谓的可靠是指，只要我把消息成功投递到了消息中间件，消息就不会丢失，即消息肯定会至少保证消息能被消费者成功消费一次，这是消息中间件最基本的特性之一，也就是我们常说的“AT LEAST ONCE”，即消息至少会被“成功消费一遍”。</p>

<p>举个例子，一个消息M发送到了消息中间件，消息投递到了消费程序A，A接受到了消息，然后进行消费，但在消费到一半的时候程序重启了，这时候这个消息并没有标记为消费成功，这个消息还会继续投递给这个消费者，直到其消费成功了，消息中间件才会停止投递。</p>

<p>然而这种可靠的特性导致，消息可能被多次地投递。举个例子，还是刚刚这个例子，程序A接受到这个消息M并完成消费逻辑之后，正想通知消息中间件“我已经消费成功了”的时候，程序就重启了，那么对于消息中间件来说，这个消息并没有成功消费过，所以他还会继续投递。这时候对于应用程序A来说，看起来就是这个消息明明消费成功了，但是消息中间件还在重复投递。</p>

<p>这在RockectMQ的场景来看，就是同一个messageId的消息重复投递下来了。</p>

<p>基于消息的投递可靠（消息不丢）是优先级更高的，所以消息不重的任务就会转移到应用程序自我实现，这也是为什么RocketMQ的文档里强调的，消费逻辑需要自我实现幂等。背后的逻辑其实就是：不丢和不重是矛盾的（在分布式场景下），但消息重复是有解决方案的，而消息丢失是很麻烦的。</p>

<h2>简单的消息去重解决方案</h2>

<p>例如：假设我们业务的消息消费逻辑是：插入某张订单表的数据，然后更新库存：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>insert into t_order values .....
</span><span class='line'>update t_inv set count = count-1 where good_id = 'good123';</span></code></pre></td></tr></table></div></figure>


<p>要实现消息的幂等，我们可能会采取这样的方案：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>select * from t_order where order_no = 'order123'
</span><span class='line'>
</span><span class='line'>if(order  != null) {
</span><span class='line'>
</span><span class='line'>    return ;//消息重复，直接返回
</span><span class='line'>
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>这对于很多情况下，的确能起到不错的效果，但是在并发场景下，还是会有问题。</p>

<h2>并发重复消息</h2>

<p>假设这个消费的所有代码加起来需要1秒，有重复的消息在这1秒内（假设100毫秒）内到达（例如生产者快速重发，Broker重启等），那么很可能，上面去重代码里面会发现，数据依然是空的（因为上一条消息还没消费完，还没成功更新订单状态），</p>

<p>那么就会穿透掉检查的挡板，最后导致重复的消息消费逻辑进入到非幂等安全的业务代码中，从而引发重复消费的问题（如主键冲突抛出异常、库存被重复扣减而没释放等）</p>

<h3>并发去重的解决方案之一</h3>

<p>要解决上面并发场景下的消息幂等问题，一个可取的方案是开启事务把select 改成 select for update语句，把记录进行锁定。</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>select * from t_order where order_no = 'THIS_ORDER_NO' for update  //开启事务
</span><span class='line'>if(order.status != null) {
</span><span class='line'>    return ;//消息重复，直接返回
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>但这样消费的逻辑会因为引入了事务包裹而导致整个消息消费可能变长，并发度下降。</p>

<p>当然还有其他更高级的解决方案，例如更新订单状态采取乐观锁，更新失败则消息重新消费之类的。但这需要针对具体业务场景做更复杂和细致的代码开发、库表设计，不在本文讨论的范围。</p>

<p>但无论是select for update， 还是乐观锁这种解决方案，实际上都是基于业务表本身做去重，这无疑增加了业务开发的复杂度，  一个业务系统里面很大部分的请求处理都是依赖MQ的，如果每个消费逻辑本身都需要基于业务本身而做去重/幂等的开发的话，这是繁琐的工作量。本文希望探索出一个通用的消息幂等处理的方法，从而抽象出一定的工具类用以适用各个业务场景。</p>

<h1>Exactly Once</h1>

<p>在消息中间件里，有一个投递语义的概念，而这个语义里有一个叫&#8221;Exactly Once&#8221;，即消息肯定会被成功消费，并且只会被消费一次。以下是阿里云里对Exactly Once的解释：</p>

<blockquote><p>Exactly-Once 是指发送到消息系统的消息只能被消费端处理且仅处理一次，即使生产端重试消息发送导致某消息重复投递，该消息在消费端也只被消费一次。</p></blockquote>

<p>在我们业务消息幂等处理的领域内，可以认为业务消息的代码肯定会被执行，并且只被执行一次，那么我们可以认为是Exactly Once。</p>

<p>但这在分布式的场景下想找一个通用的方案几乎是不可能的。不过如果是针对基于数据库事务的消费逻辑，实际上是可行的。</p>

<h2>基于关系数据库事务插入消息表</h2>

<p>假设我们业务的消息消费逻辑是：更新MySQL数据库的某张订单表的状态：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>update t_order set status = 'SUCCESS' where order_no= 'order123';</span></code></pre></td></tr></table></div></figure>


<p>要实现Exaclty Once即这个消息只被消费一次（并且肯定要保证能消费一次），我们可以这样做：在这个数据库中增加一个消息消费记录表，把消息插入到这个表，并且把原来的订单更新和这个插入的动作放到同一个事务中一起提交，就能保证消息只会被消费一遍了。</p>

<ol>
<li>开启事务</li>
<li>插入消息表（处理好主键冲突的问题）</li>
<li>更新订单表（原消费逻辑）</li>
<li>提交事务</li>
</ol>


<p>说明：</p>

<ol>
<li><p>这时候如果消息消费成功并且事务提交了，那么消息表就插入成功了，这时候就算RocketMQ还没有收到消费位点的更新再次投递，也会插入消息失败而视为已经消费过，后续就直接更新消费位点了。这保证我们消费代码只会执行一次。</p></li>
<li><p>如果事务提交之前服务挂了（例如重启），对于本地事务并没有执行所以订单没有更新，消息表也没插入成功；而对于RocketMQ服务端来说，消费位点也没更新，所以消息还会继续投递下来，投递下来发现这个消息插入消息表也是成功的，所以可以继续消费。这保证了消息不丢失。</p></li>
</ol>


<p>事实上，阿里云ONS的EXACTLY-ONCE语义的实现上，就是类似这个方案基于数据库的事务特性实现的。更多详情可参考：<a href="https://help.aliyun.com/document_detail/102777.html">https://help.aliyun.com/document_detail/102777.html</a></p>

<p>基于这种方式，的确这是有能力拓展到不同的应用场景，因为他的实现方案与具体业务本身无关——而是依赖一个消息表。</p>

<p>但是这里有它的局限性</p>

<ol>
<li>消息的消费逻辑必须是依赖于关系型数据库事务。如果消费的消费过程中还涉及其他数据的修改，例如Redis这种不支持事务特性的数据源，则这些数据是不可回滚的。</li>
<li>数据库的数据必须是在一个库，跨库无法解决</li>
</ol>


<p>注：业务上，消息表的设计不应该以消息ID作为标识，而应该以业务的业务主键作为标识更为合理，以应对生产者的重发。阿里云上的消息去重只是RocketMQ的messageId，在生产者因为某些原因手动重发（例如上游针对一个交易重复请求了）的场景下起不到去重/幂等的效果（因消息id不同）。</p>

<h2>更复杂的业务场景</h2>

<p>如上所述，这种方式Exactly Once语义的实现，实际上有很多局限性，这种局限性使得这个方案基本不具备广泛应用的价值。并且由于基于事务，可能导致锁表时间过长等性能问题。</p>

<p>例如我们以一个比较常见的一个订单申请的消息来举例，可能有以下几步（以下统称为步骤X）：</p>

<ol>
<li><p>检查库存（RPC）</p></li>
<li><p>锁库存（RPC）</p></li>
<li><p>开启事务，插入订单表（MySQL）</p></li>
<li><p>调用某些其他下游服务（RPC）</p></li>
<li><p>更新订单状态</p></li>
<li><p>commit 事务（MySQL）</p></li>
</ol>


<p>这种情况下，我们如果采取消息表+本地事务的实现方式，消息消费过程中很多子过程是不支持回滚的，也就是说就算我们加了事务，实际上这背后的操作并不是原子性的。怎么说呢，就是说有可能第一条小在经历了第二步锁库存的时候，服务重启了，这时候实际上库存是已经在另外的服务里被锁定了，这并不能被回滚。当然消息还会再次投递下来，要保证消息能至少消费一遍，换句话说，锁库存的这个RPC接口本身依旧要支持“幂等”。</p>

<p>再者，如果在这个比较耗时的长链条场景下加入事务的包裹，将大大的降低系统的并发。所以通常情况下，我们处理这种场景的消息去重的方法还是会使用一开始说的业务自己实现去重逻辑的方式，如前面加select for update，或者使用乐观锁。</p>

<p>那我们有没有方法抽取出一个公共的解决方案，能兼顾去重、通用、高性能呢？</p>

<h2>拆解消息执行过程</h2>

<p>其中一个思路是把上面的几步，拆解成几个不同的子消息，例如：</p>

<ol>
<li><p>库存系统消费A：检查库存并做锁库存，发送消息B给订单服务</p></li>
<li><p>订单系统消费消息B：插入订单表（MySQL），发送消息C给自己（下游系统）消费</p></li>
<li><p>下游系统消费消息C：处理部分逻辑，发送消息D给订单系统</p></li>
<li><p>订单系统消费消息D：更新订单状态</p></li>
</ol>


<p>注：上述步骤需要保证本地事务和消息是一个事务的（至少是最终一致性的），这其中涉及到分布式事务消息相关的话题，不在本文论述。</p>

<p>可以看到这样的处理方法会使得每一步的操作都比较原子，而原子则意味着是小事务，小事务则意味着使用消息表+事务的方案显得可行。</p>

<p>然而，这太复杂了！这把一个本来连续的代码逻辑割裂成多个系统多次消息交互！那还不如业务代码层面上加锁实现呢。</p>

<h2>更通用的解决方案</h2>

<p>上面消息表+本地事务的方案之所以有其局限性和并发的短板，究其根本是因为它<strong>依赖于关系型数据库的事务</strong>，且必须要把事务包裹于整个消息消费的环节。</p>

<p>如果我们能不依赖事务而实现消息的去重，那么方案就能推广到更复杂的场景例如：RPC、跨库等。</p>

<p>例如，我们依旧使用消息表，但是不依赖事务，而是针对消息表增加消费状态，是否可以解决问题呢？</p>

<h3>基于消息幂等表的非事务方案</h3>

<p><img src="http://jaskey.github.io/images/message-dedup/dedup-solution-01.png" title="dedup-solution-01" alt="dedup-solution-01" /></p>

<p>以上是去事务化后的消息幂等方案的流程，可以看到，此方案是无事务的，而是针对消息表本身做了状态的区分：消费中、消费完成。<strong>只有消费完成的消息才会被幂等处理掉</strong>。而对于已有消费中的消息，后面重复的消息会触发延迟消费（在RocketMQ的场景下即发送到RETRY TOPIC），之所以触发延迟消费是为了控制并发场景下，第二条消息在第一条消息没完成的过程中，去控制消息不丢（如果直接幂等，那么会丢失消息（同一个消息id的话），因为上一条消息如果没有消费完成的时候，第二条消息你已经告诉broker成功了，那么第一条消息这时候失败broker也不会重新投递了）</p>

<p>上面的流程不再细说，后文有github源码的地址，读者可以参考源码的实现，这里我们回头看看我们一开始想解决的问题是否解决了：</p>

<ol>
<li>消息已经消费成功了，第二条消息将被直接幂等处理掉（消费成功）。</li>
<li>并发场景下的消息，依旧能满足不会出现消息重复，即穿透幂等挡板的问题。</li>
<li>支持上游业务生产者重发的业务重复的消息幂等问题。</li>
</ol>


<p>关于第一个问题已经很明显已经解决了，在此就不讨论了。</p>

<p>关于第二个问题是如何解决的？主要是依靠插入消息表的这个动作做控制的，假设我们用MySQL作为消息表的存储媒介（设置消息的唯一ID为主键），那么插入的动作只有一条消息会成功，后面的消息插入会由于主键冲突而失败，走向延迟消费的分支，然后后面延迟消费的时候就会变成上面第一个场景的问题。</p>

<p>关于第三个问题，只要我们设计去重的消息键让其支持业务的主键（例如订单号、请求流水号等），而不仅仅是messageId即可。所以也不是问题。</p>

<h3>此方案是否有消息丢失的风险？</h3>

<p>如果细心的读者可能会发现这里实际上是有逻辑漏洞的，问题出在上面聊到的个三问题中的第2个问题（并发场景），在并发场景下我们依赖于消息状态是做并发控制使得第2条消息重复的消息会不断延迟消费（重试）。但如果这时候第1条消息也由于一些异常原因（例如机器重启了、外部异常导致消费失败）没有成功消费成功呢？也就是说这时候延迟消费实际上每次下来看到的都是<em>消费中</em>的状态，最后消费就会被视为消费失败而被投递到死信Topic中（RocketMQ默认可以重复消费16次）。</p>

<p>有这种顾虑是正确的！对于此，我们解决的方法是，插入的消息表必须要带一个最长消费过期时间，例如10分钟，意思是如果一个消息处于<em>消费中</em>超过10分钟，就需要从消息表中删除（需要程序自行实现）。所以最后这个消息的流程会是这样的：</p>

<p><img src="http://jaskey.github.io/images/message-dedup/dedup-solution-02.png" title="dedup-solution-02" alt="dedup-solution-01" /></p>

<h2>更灵活的消息表存储媒介</h2>

<p>我们这个方案实际上没有事务的，只需要一个存储的中心媒介，那么自然我们可以选择更灵活的存储媒介，例如Redis。使用Redis有两个好处：</p>

<ol>
<li>性能上损耗更低</li>
<li>上面我们讲到的超时时间可以直接利用Redis本身的ttl实现</li>
</ol>


<p>当然Redis存储的数据可靠性、一致性等方面是不如MySQL的，需要用户自己取舍。</p>

<h1>源码：RocketMQDedupListener</h1>

<p>以上方案针对RocketMQ的Java实现已经开源放到Github中，具体的使用文档可以参考<a href="https://github.com/Jaskey/RocketMQDedupListener">https://github.com/Jaskey/RocketMQDedupListener</a> ，</p>

<p>以下仅贴一个Readme中利用Redis去重的使用样例，用以意业务中如果使用此工具加入消息去重幂等的是多么简单：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'>        <span class="c1">//利用Redis做幂等表</span>
</span><span class='line'>        <span class="n">DefaultMQPushConsumer</span> <span class="n">consumer</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">DefaultMQPushConsumer</span><span class="o">(</span><span class="s">&quot;TEST-APP1&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="n">consumer</span><span class="o">.</span><span class="na">subscribe</span><span class="o">(</span><span class="s">&quot;TEST-TOPIC&quot;</span><span class="o">,</span> <span class="s">&quot;*&quot;</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">String</span> <span class="n">appName</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">getConsumerGroup</span><span class="o">();</span><span class="c1">// 大部分情况下可直接使用consumer group名</span>
</span><span class='line'>        <span class="n">StringRedisTemplate</span> <span class="n">stringRedisTemplate</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span><span class="c1">// 这里省略获取StringRedisTemplate的过程</span>
</span><span class='line'>        <span class="n">DedupConfig</span> <span class="n">dedupConfig</span> <span class="o">=</span> <span class="n">DedupConfig</span><span class="o">.</span><span class="na">enableDedupConsumeConfig</span><span class="o">(</span><span class="n">appName</span><span class="o">,</span> <span class="n">stringRedisTemplate</span><span class="o">);</span>
</span><span class='line'>        <span class="n">DedupConcurrentListener</span> <span class="n">messageListener</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">SampleListener</span><span class="o">(</span><span class="n">dedupConfig</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">consumer</span><span class="o">.</span><span class="na">registerMessageListener</span><span class="o">(</span><span class="n">messageListener</span><span class="o">);</span>
</span><span class='line'>        <span class="n">consumer</span><span class="o">.</span><span class="na">start</span><span class="o">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>以上代码大部分是原始RocketMQ的必须代码，唯一需要修改的仅仅是创建一个<code>DedupConcurrentListener</code>示例，在这个示例中指明你的消费逻辑和去重的业务键（默认是messageId）。</p>

<p>更多使用详情请参考Github上的说明。</p>

<h1>这种实现是否一劳永逸？</h1>

<p>实现到这里，似乎方案挺完美的，所有的消息都能快速的接入去重，且与具体业务实现也完全解耦。那么这样是否就完美的完成去重的所有任务呢？</p>

<p>很可惜，其实不是的。原因很简单：因为要保证消息至少被成功消费一遍，那么消息就有机会消费到一半的时候失败触发消息重试的可能。还是以上面的订单流程X：</p>

<blockquote><ol>
<li><p>检查库存（RPC）</p></li>
<li><p>锁库存（RPC）</p></li>
<li><p>开启事务，插入订单表（MySQL）</p></li>
<li><p>调用某些其他下游服务（RPC）</p></li>
<li><p>更新订单状态</p></li>
<li><p>commit 事务（MySQL）</p></li>
</ol>
</blockquote>

<p>当消息消费到步骤3的时候，我们假设MySQL异常导致失败了，触发消息重试。因为在重试前我们会删除幂等表的记录，所以消息重试的时候就会重新进入消费代码，那么步骤1和步骤2就会重新再执行一遍。如果步骤2本身不是幂等的，那么这个业务消息消费依旧没有做好完整的幂等处理。</p>

<h1>本实现方式的价值？</h1>

<p>那么既然这个并不能完整的完成消息幂等，还有什么价值呢？价值可就大了！虽然这不是解决消息幂等的银弹（事实上，软件工程领域里基本没有银弹），但是他能以便捷的手段解决：</p>

<p>1.各种由于Broker、负载均衡等原因导致的消息重投递的重复问题</p>

<p>2.各种上游生产者导致的业务级别消息重复问题</p>

<p>3.重复消息并发消费的控制窗口问题，就算重复，重复也不可能同一时间进入消费逻辑</p>

<h1>一些其他的消息去重的建议</h1>

<p>也就是说，使用这个方法能保证正常的消费逻辑场景下（无异常，无异常退出），消息的幂等工作全部都能解决，无论是业务重复，还是rocketmq特性带来的重复。</p>

<p>事实上，这已经能解决99%的消息重复问题了，毕竟异常的场景肯定是少数的。那么如果希望异常场景下也能处理好幂等的问题，可以做以下工作降低问题率：</p>

<ol>
<li>消息消费失败做好回滚处理。如果消息消费失败本身是带回滚机制的，那么消息重试自然就没有副作用了。</li>
<li>消费者做好优雅退出处理。这是为了尽可能避免消息消费到一半程序退出导致的消息重试。</li>
<li>一些无法做到幂等的操作，至少要做到终止消费并告警。例如锁库存的操作，如果统一的业务流水锁成功了一次库存，再触发锁库存，如果做不到幂等的处理，至少要做到消息消费触发异常（例如主键冲突导致消费异常等）</li>
<li>在#3做好的前提下，做好消息的消费监控，发现消息重试不断失败的时候，手动做好#1的回滚，使得下次重试消费成功。</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[记一次因索引合并导致的MySQL死锁分析过程]]></title>
    <link href="https://Jaskey.github.io/blog/2020/06/01/mysql-deadlock-index-merge/"/>
    <updated>2020-06-01T19:49:26+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/06/01/mysql-deadlock-index-merge</id>
    <content type="html"><![CDATA[<p>生产上偶现这段代码会出现死锁，死锁日志如下。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
<span class='line-number'>84</span>
<span class='line-number'>85</span>
<span class='line-number'>86</span>
<span class='line-number'>87</span>
<span class='line-number'>88</span>
<span class='line-number'>89</span>
<span class='line-number'>90</span>
<span class='line-number'>91</span>
<span class='line-number'>92</span>
<span class='line-number'>93</span>
<span class='line-number'>94</span>
<span class='line-number'>95</span>
<span class='line-number'>96</span>
<span class='line-number'>97</span>
<span class='line-number'>98</span>
<span class='line-number'>99</span>
<span class='line-number'>100</span>
<span class='line-number'>101</span>
<span class='line-number'>102</span>
<span class='line-number'>103</span>
<span class='line-number'>104</span>
<span class='line-number'>105</span>
<span class='line-number'>106</span>
<span class='line-number'>107</span>
<span class='line-number'>108</span>
<span class='line-number'>109</span>
<span class='line-number'>110</span>
<span class='line-number'>111</span>
<span class='line-number'>112</span>
<span class='line-number'>113</span>
<span class='line-number'>114</span>
<span class='line-number'>115</span>
<span class='line-number'>116</span>
<span class='line-number'>117</span>
<span class='line-number'>118</span>
<span class='line-number'>119</span>
<span class='line-number'>120</span>
<span class='line-number'>121</span>
<span class='line-number'>122</span>
<span class='line-number'>123</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">TRANSACTION</span><span class="p">:</span>
</span><span class='line'><span class="n">TRANSACTION</span> <span class="mi">424487272</span><span class="p">,</span> <span class="n">ACTIVE</span> <span class="mi">0</span> <span class="n">sec</span> <span class="n">fetching</span> <span class="n">rows</span>
</span><span class='line'><span class="n">mysql</span> <span class="kp">tables</span> <span class="k">in</span> <span class="k">use</span> <span class="mi">3</span><span class="p">,</span> <span class="n">locked</span> <span class="mi">3</span>
</span><span class='line'><span class="k">LOCK</span> <span class="n">WAIT</span> <span class="mi">6</span> <span class="k">lock</span> <span class="nf">struct</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">heap</span> <span class="n">size</span> <span class="mi">1184</span><span class="p">,</span> <span class="mi">4</span> <span class="n">row</span> <span class="k">lock</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
</span><span class='line'><span class="n">MySQL</span> <span class="n">thread</span> <span class="n">id</span> <span class="mi">3205005</span><span class="p">,</span> <span class="n">OS</span> <span class="n">thread</span> <span class="n">handle</span> <span class="mi">0</span><span class="n">x7f39c21c8700</span><span class="p">,</span> <span class="n">query</span> <span class="n">id</span> <span class="mi">567774892</span> <span class="mi">10</span><span class="p">.</span><span class="mi">14</span><span class="p">.</span><span class="mi">34</span><span class="p">.</span><span class="mi">30</span> <span class="n">finance</span> <span class="n">Searching</span> <span class="n">rows</span> <span class="k">for</span> <span class="k">update</span>
</span><span class='line'><span class="k">update</span> <span class="n">repay_plan_info_1</span>
</span><span class='line'>     <span class="kt">SET</span> <span class="n">actual_pay_period_amount</span> <span class="o">=</span> <span class="mi">38027</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_principal_amount</span> <span class="o">=</span> <span class="mi">36015</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_interest_amount</span> <span class="o">=</span> <span class="mi">1980</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_fee</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_fine</span> <span class="o">=</span> <span class="mi">32</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_discount_amount</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
</span><span class='line'>        <span class="n">repay_status</span> <span class="o">=</span> <span class="s1">&#39;PAYOFF&#39;</span><span class="p">,</span>
</span><span class='line'>        <span class="n">repay_type</span> <span class="o">=</span> <span class="s1">&#39;OVERDUE&#39;</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_repay_time</span> <span class="o">=</span> <span class="s1">&#39;2019-08-12 15:48:15.025&#39;</span>
</span><span class='line'>
</span><span class='line'>     <span class="k">WHERE</span> <span class="p">(</span>  <span class="n">user_id</span> <span class="o">=</span> <span class="s1">&#39;938467411690006528&#39;</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">loan_order_no</span> <span class="o">=</span> <span class="s1">&#39;LN201907120655461690006528458116&#39;</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">seq_no</span> <span class="o">=</span> <span class="mi">1</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">repay_status</span> <span class="o">&lt;&gt;</span> <span class="s1">&#39;PAYOFF&#39;</span> <span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">WAITING</span> <span class="k">FOR</span> <span class="n">THIS</span> <span class="k">LOCK</span> <span class="k">TO</span> <span class="n">BE</span> <span class="n">GRANTED</span><span class="p">:</span>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">30</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">136</span> <span class="k">index</span> <span class="ss">`PRIMARY`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487272</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span> <span class="n">waiting</span>
</span><span class='line'><span class="n">Record</span> <span class="k">lock</span><span class="p">,</span> <span class="n">heap</span> <span class="n">no</span> <span class="mi">64</span> <span class="n">PHYSICAL</span> <span class="n">RECORD</span><span class="p">:</span> <span class="n">n_fields</span> <span class="mi">33</span><span class="p">;</span> <span class="n">compact</span> <span class="n">format</span><span class="p">;</span> <span class="n">info</span> <span class="n">bits</span> <span class="mi">0</span>
</span><span class='line'> <span class="mi">0</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">800000000000051</span><span class="n">e</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">1</span><span class="p">:</span> <span class="n">len</span> <span class="mi">6</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">0000193</span><span class="n">d35df</span><span class="p">;</span> <span class="k">asc</span>    <span class="o">=</span><span class="mi">5</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">2</span><span class="p">:</span> <span class="n">len</span> <span class="mi">7</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">06000001</span><span class="n">d402e7</span><span class="p">;</span> <span class="k">asc</span>        <span class="p">;;</span>
</span><span class='line'> <span class="mi">3</span><span class="p">:</span> <span class="n">len</span> <span class="mi">30</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">323031393036313332303532303634323936303534323130353730303030</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">201906132052064296054210570000</span><span class="p">;</span> <span class="p">(</span><span class="n">total</span> <span class="mi">32</span> <span class="n">bytes</span><span class="p">);</span>
</span><span class='line'> <span class="mi">4</span><span class="p">:</span> <span class="n">len</span> <span class="mi">30</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">4</span><span class="n">c4e32303139303631333031323934303136393030303635323831373534</span><span class="p">;</span> <span class="k">asc</span> <span class="n">LN2019061301294016900065281754</span><span class="p">;</span> <span class="p">(</span><span class="n">total</span> <span class="mi">32</span> <span class="n">bytes</span><span class="p">);</span>
</span><span class='line'> <span class="mi">5</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000002</span><span class="p">;</span> <span class="k">asc</span>     <span class="p">;;</span>
</span><span class='line'> <span class="mi">6</span><span class="p">:</span> <span class="n">len</span> <span class="mi">18</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">393338343637343131363930303036353238</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">938467411690006528</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">7</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000003</span><span class="p">;</span> <span class="k">asc</span>     <span class="p">;;</span>
</span><span class='line'> <span class="mi">8</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000258</span><span class="p">;</span> <span class="k">asc</span>    <span class="n">X</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">9</span><span class="p">:</span> <span class="n">len</span> <span class="mi">3</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">646179</span><span class="p">;</span> <span class="k">asc</span> <span class="n">day</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">10</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">11</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">12</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000005106</span><span class="p">;</span> <span class="k">asc</span>       <span class="n">Q</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">13</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">14</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000004</span><span class="n">e1e</span><span class="p">;</span> <span class="k">asc</span>       <span class="n">N</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">15</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">16</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000000000002</span><span class="n">d6</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">17</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">18</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">19</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">20</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000012</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">21</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">22</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">23</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">24</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">3230313930383131</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">20190811</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">25</span><span class="p">:</span> <span class="n">len</span> <span class="mi">7</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">4</span><span class="n">f564552445545</span><span class="p">;</span> <span class="k">asc</span> <span class="n">OVERDUE</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">26</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">27</span><span class="p">:</span> <span class="n">len</span> <span class="mi">1</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">59</span><span class="p">;</span> <span class="k">asc</span> <span class="n">Y</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">28</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">29</span><span class="p">:</span> <span class="n">len</span> <span class="mi">5</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">99</span><span class="n">a35a1768</span><span class="p">;</span> <span class="k">asc</span>   <span class="n">Z</span> <span class="n">h</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">30</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">5</span><span class="n">d503dd8</span><span class="p">;</span> <span class="k">asc</span> <span class="p">]</span><span class="n">P</span><span class="o">=</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">31</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">32</span><span class="p">:</span> <span class="n">len</span> <span class="mi">5</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">99</span><span class="n">a3d80281</span><span class="p">;</span> <span class="k">asc</span>      <span class="p">;;</span>
</span><span class='line'>
</span><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="n">TRANSACTION</span><span class="p">:</span>
</span><span class='line'><span class="n">TRANSACTION</span> <span class="mi">424487271</span><span class="p">,</span> <span class="n">ACTIVE</span> <span class="mi">0</span> <span class="n">sec</span> <span class="n">fetching</span> <span class="n">rows</span>
</span><span class='line'><span class="n">mysql</span> <span class="kp">tables</span> <span class="k">in</span> <span class="k">use</span> <span class="mi">3</span><span class="p">,</span> <span class="n">locked</span> <span class="mi">3</span>
</span><span class='line'><span class="mi">5</span> <span class="k">lock</span> <span class="nf">struct</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">heap</span> <span class="n">size</span> <span class="mi">1184</span><span class="p">,</span> <span class="mi">3</span> <span class="n">row</span> <span class="k">lock</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
</span><span class='line'><span class="n">MySQL</span> <span class="n">thread</span> <span class="n">id</span> <span class="mi">3204980</span><span class="p">,</span> <span class="n">OS</span> <span class="n">thread</span> <span class="n">handle</span> <span class="mi">0</span><span class="n">x7f3db0cf6700</span><span class="p">,</span> <span class="n">query</span> <span class="n">id</span> <span class="mi">567774893</span> <span class="mi">10</span><span class="p">.</span><span class="mi">14</span><span class="p">.</span><span class="mi">34</span><span class="p">.</span><span class="mi">30</span> <span class="n">finance</span> <span class="n">Searching</span> <span class="n">rows</span> <span class="k">for</span> <span class="k">update</span>
</span><span class='line'><span class="k">update</span> <span class="n">repay_plan_info_1</span>
</span><span class='line'>     <span class="kt">SET</span> <span class="n">actual_pay_period_amount</span> <span class="o">=</span> <span class="mi">20742</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_principal_amount</span> <span class="o">=</span> <span class="mi">19998</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_interest_amount</span> <span class="o">=</span> <span class="mi">726</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_fee</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_pay_fine</span> <span class="o">=</span> <span class="mi">18</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_discount_amount</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
</span><span class='line'>        <span class="n">repay_status</span> <span class="o">=</span> <span class="s1">&#39;PAYOFF&#39;</span><span class="p">,</span>
</span><span class='line'>        <span class="n">repay_type</span> <span class="o">=</span> <span class="s1">&#39;OVERDUE&#39;</span><span class="p">,</span>
</span><span class='line'>        <span class="n">actual_repay_time</span> <span class="o">=</span> <span class="s1">&#39;2019-08-12 15:48:15.025&#39;</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>     <span class="k">WHERE</span> <span class="p">(</span>  <span class="n">user_id</span> <span class="o">=</span> <span class="s1">&#39;938467411690006528&#39;</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">loan_order_no</span> <span class="o">=</span> <span class="s1">&#39;LN201906130129401690006528175485&#39;</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">seq_no</span> <span class="o">=</span> <span class="mi">2</span>
</span><span class='line'>                  <span class="k">and</span> <span class="n">repay_status</span> <span class="o">&lt;&gt;</span> <span class="s1">&#39;PAYOFF&#39;</span> <span class="p">)</span>
</span><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="n">HOLDS</span> <span class="n">THE</span> <span class="k">LOCK</span><span class="p">(</span><span class="n">S</span><span class="p">):</span>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">30</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">136</span> <span class="k">index</span> <span class="ss">`PRIMARY`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487271</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span>
</span><span class='line'><span class="n">Record</span> <span class="k">lock</span><span class="p">,</span> <span class="n">heap</span> <span class="n">no</span> <span class="mi">64</span> <span class="n">PHYSICAL</span> <span class="n">RECORD</span><span class="p">:</span> <span class="n">n_fields</span> <span class="mi">33</span><span class="p">;</span> <span class="n">compact</span> <span class="n">format</span><span class="p">;</span> <span class="n">info</span> <span class="n">bits</span> <span class="mi">0</span>
</span><span class='line'> <span class="mi">0</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">800000000000051</span><span class="n">e</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">1</span><span class="p">:</span> <span class="n">len</span> <span class="mi">6</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">0000193</span><span class="n">d35df</span><span class="p">;</span> <span class="k">asc</span>    <span class="o">=</span><span class="mi">5</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">2</span><span class="p">:</span> <span class="n">len</span> <span class="mi">7</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">06000001</span><span class="n">d402e7</span><span class="p">;</span> <span class="k">asc</span>        <span class="p">;;</span>
</span><span class='line'> <span class="mi">3</span><span class="p">:</span> <span class="n">len</span> <span class="mi">30</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">323031393036313332303532303634323936303534323130353730303030</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">201906132052064296054210570000</span><span class="p">;</span> <span class="p">(</span><span class="n">total</span> <span class="mi">32</span> <span class="n">bytes</span><span class="p">);</span>
</span><span class='line'> <span class="mi">4</span><span class="p">:</span> <span class="n">len</span> <span class="mi">30</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">4</span><span class="n">c4e32303139303631333031323934303136393030303635323831373534</span><span class="p">;</span> <span class="k">asc</span> <span class="n">LN2019061301294016900065281754</span><span class="p">;</span> <span class="p">(</span><span class="n">total</span> <span class="mi">32</span> <span class="n">bytes</span><span class="p">);</span>
</span><span class='line'> <span class="mi">5</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000002</span><span class="p">;</span> <span class="k">asc</span>     <span class="p">;;</span>
</span><span class='line'> <span class="mi">6</span><span class="p">:</span> <span class="n">len</span> <span class="mi">18</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">393338343637343131363930303036353238</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">938467411690006528</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">7</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000003</span><span class="p">;</span> <span class="k">asc</span>     <span class="p">;;</span>
</span><span class='line'> <span class="mi">8</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000258</span><span class="p">;</span> <span class="k">asc</span>    <span class="n">X</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">9</span><span class="p">:</span> <span class="n">len</span> <span class="mi">3</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">646179</span><span class="p">;</span> <span class="k">asc</span> <span class="n">day</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">10</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">11</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">12</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000005106</span><span class="p">;</span> <span class="k">asc</span>       <span class="n">Q</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">13</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">14</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000004</span><span class="n">e1e</span><span class="p">;</span> <span class="k">asc</span>       <span class="n">N</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">15</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">16</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">80000000000002</span><span class="n">d6</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">17</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">18</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">19</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">20</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000012</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">21</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">22</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">23</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">8000000000000000</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'> <span class="mi">24</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">3230313930383131</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">20190811</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">25</span><span class="p">:</span> <span class="n">len</span> <span class="mi">7</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">4</span><span class="n">f564552445545</span><span class="p">;</span> <span class="k">asc</span> <span class="n">OVERDUE</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">26</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">27</span><span class="p">:</span> <span class="n">len</span> <span class="mi">1</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">59</span><span class="p">;</span> <span class="k">asc</span> <span class="n">Y</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">28</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">29</span><span class="p">:</span> <span class="n">len</span> <span class="mi">5</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">99</span><span class="n">a35a1768</span><span class="p">;</span> <span class="k">asc</span>   <span class="n">Z</span> <span class="n">h</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">30</span><span class="p">:</span> <span class="n">len</span> <span class="mi">4</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">5</span><span class="n">d503dd8</span><span class="p">;</span> <span class="k">asc</span> <span class="p">]</span><span class="n">P</span><span class="o">=</span> <span class="p">;;</span>
</span><span class='line'> <span class="mi">31</span><span class="p">:</span> <span class="k">SQL</span> <span class="no">NULL</span><span class="p">;</span>
</span><span class='line'> <span class="mi">32</span><span class="p">:</span> <span class="n">len</span> <span class="mi">5</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">99</span><span class="n">a3d80281</span><span class="p">;</span> <span class="k">asc</span>      <span class="p">;;</span>
</span><span class='line'>
</span><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="n">WAITING</span> <span class="k">FOR</span> <span class="n">THIS</span> <span class="k">LOCK</span> <span class="k">TO</span> <span class="n">BE</span> <span class="n">GRANTED</span><span class="p">:</span>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">137</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">464</span> <span class="k">index</span> <span class="ss">`idx_user_id`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487271</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span> <span class="n">waiting</span>
</span><span class='line'><span class="n">Record</span> <span class="k">lock</span><span class="p">,</span> <span class="n">heap</span> <span class="n">no</span> <span class="mi">161</span> <span class="n">PHYSICAL</span> <span class="n">RECORD</span><span class="p">:</span> <span class="n">n_fields</span> <span class="mi">2</span><span class="p">;</span> <span class="n">compact</span> <span class="n">format</span><span class="p">;</span> <span class="n">info</span> <span class="n">bits</span> <span class="mi">0</span>
</span><span class='line'> <span class="mi">0</span><span class="p">:</span> <span class="n">len</span> <span class="mi">18</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">393338343637343131363930303036353238</span><span class="p">;</span> <span class="k">asc</span> <span class="mi">938467411690006528</span><span class="p">;;</span>
</span><span class='line'> <span class="mi">1</span><span class="p">:</span> <span class="n">len</span> <span class="mi">8</span><span class="p">;</span> <span class="n">hex</span> <span class="mi">800000000000051</span><span class="n">e</span><span class="p">;</span> <span class="k">asc</span>         <span class="p">;;</span>
</span><span class='line'>
</span><span class='line'><span class="o">***</span> <span class="n">WE</span> <span class="n">ROLL</span> <span class="n">BACK</span> <span class="nf">TRANSACTION</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h1>代码定位</h1>

<p>按照死锁的update sql语句，我们先定位这个死锁SQL中代码是哪个代码片段导致的。后面我们定位到，是如下代码片段导致的：</p>

<p><img src="http://jaskey.github.io/images/mysql-deadlock/deadlock-code.png" alt="deadlock-codedeadlock-code" /></p>

<p>实际上一眼看上去，这段代码有一个很典型的业务开发场景问题：开启事务在for循环写SQL。</p>

<p>注：这在实际的问题定位过程中并不容易，因为死锁日志并不能反向直接定位到方法的对账、线程名等，如果一个库被多个服务同时连接，甚至定位是哪个服务都不容易。</p>

<h1>死锁分析（1）——猜测可能消息重发</h1>

<p>按照死锁的必要条件：<strong>循环等待条件</strong>。即 T1事务应该持有了某把锁L1，然后去申请锁L2，而这时候发现T2事务已经持有了L2，而T2事务又去申请L1，这时候就发生循环等待而死锁。</p>

<p>一开始会猜测，是否我们更新表的顺序在两个事务里面是反方向的，即T1事务更新ta、tb表，锁ta表的记录，准备去拿tb表记录的锁；T2事务更新tb、ta表，锁了tb记录准备去拿ta的锁，这是比较常见的死锁情况。但是从SQL看，我们死锁的SQL是同一张表的，即同一张表不同的记录。</p>

<p>而且从死锁日志中可以发现，两个死锁的SQL居然是“一样”的，也就是说是“同一条”SQL/同一段代码（不同的where条件参数）导致的，。即上图代码中的这段for循环更新还款计划的代码。</p>

<p>但是光这段For循环来看，如果要发生死锁，有可能同一批请求，更新记录的顺序是反过来的，然后又并发执行的时候，可能出现。</p>

<p>一开始会猜测上游触发了两条一样的请求（我们这个场景是MQ重发），出现了并发，两条消息分在两个事务中并发执行。但是如果是MQ导致的原因，FOR循环更新的记录顺序是一样的，一样的顺序意味着一样的一样的加锁顺序，一样的加锁顺序意味着最多出现获取锁超时，不会满足【循环等待】的条件，不可能死锁。所以排除MQ重发的可能。</p>

<h1>死锁分析（2）</h1>

<p>仔细阅读出现问题的两条SQL，可以发现一个规律，这里面都带一个相同的where条件：<code>userId= 938467411690006528</code>，意味着这两个事务的请求都来自一个用户发起的，然后从<code>actual_repay_time = '2019-08-12 15:48:15.025'</code> 来看，的确是瞬间一起执行的两个事务，但是却是不一样的两个借据。对应到真实的用户的操作上，用户的确有可能发起两个借据的同时还款，例如同时结清多笔借据。</p>

<p>通过出现了几次的死锁，总结出了其相同的规律：每次的死锁SQL条件都有一样的特征——<strong>相同的userId+不同的借据+并发</strong>。基本可以断定，相同的用户在同时还款多笔的时候，可能会发现死锁，但很可惜，测试环境、生产环境我们模拟这个场景都无法复现死锁的情况。</p>

<p>只能靠技术手段分析原因了。</p>

<p>思路：这是了两个完全不同的借据环境计划，操作完全不一样的数据记录，为什么会发生死锁呢？是不是锁的不是行而是锁了表？</p>

<h1>死锁日志分析</h1>

<p>从事务1中的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">WAITING</span> <span class="k">FOR</span> <span class="n">THIS</span> <span class="k">LOCK</span> <span class="k">TO</span> <span class="n">BE</span> <span class="n">GRANTED</span><span class="p">:</span>
</span><span class='line'>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">30</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">136</span> <span class="k">index</span> <span class="ss">`PRIMARY`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487272</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span> <span class="n">waiting</span>
</span></code></pre></td></tr></table></div></figure>


<p>事务2中的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="n">HOLDS</span> <span class="n">THE</span> <span class="k">LOCK</span><span class="p">(</span><span class="n">S</span><span class="p">):</span>
</span><span class='line'>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">30</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">136</span> <span class="k">index</span> <span class="ss">`PRIMARY`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487271</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span>
</span></code></pre></td></tr></table></div></figure>


<p>从RECORD LOCKS的标示可知，的确<strong>锁的是行锁</strong>不是表锁。且从&#8221;but not gap&#8221;的信息来看，也不存在间隙锁（注：我们线上隔离级别是read committed,本来就不存在间隙锁问题）。所以锁的位置应该的确是我们操作的行记录才对。但是非常奇怪的是，实际业务上操作的记录的确是完全隔离的（因为是不同的借据，记录没有交集），为什么会冲突呢？</p>

<p>  再细节阅读死锁日志从事务2中获取到了一点线索：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="o">***</span> <span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="n">WAITING</span> <span class="k">FOR</span> <span class="n">THIS</span> <span class="k">LOCK</span> <span class="k">TO</span> <span class="n">BE</span> <span class="n">GRANTED</span><span class="p">:</span>
</span><span class='line'>
</span><span class='line'><span class="n">RECORD</span> <span class="n">LOCKS</span> <span class="n">space</span> <span class="n">id</span> <span class="mi">3680</span> <span class="n">page</span> <span class="n">no</span> <span class="mi">137</span> <span class="n">n</span> <span class="n">bits</span> <span class="mi">464</span> <span class="k">index</span> <span class="ss">`idx_user_id`</span> <span class="n">of</span> <span class="k">table</span> <span class="ss">`db_loan_core_2`</span><span class="p">.</span><span class="ss">`repay_plan_info_1`</span> <span class="n">trx</span> <span class="n">id</span> <span class="mi">424487271</span> <span class="n">lock_mode</span> <span class="n">X</span> <span class="n">locks</span> <span class="n">rec</span> <span class="n">but</span> <span class="k">not</span> <span class="n">gap</span> <span class="n">waiting</span> <span class="n">Record</span> <span class="k">lock</span><span class="p">,</span> <span class="n">heap</span> <span class="n">no</span> <span class="mi">161</span> <span class="n">PHYSICAL</span> <span class="n">RECORD</span><span class="p">:</span> <span class="n">n_fields</span> <span class="mi">2</span><span class="p">;</span> <span class="n">compact</span> <span class="n">format</span><span class="p">;</span> <span class="n">info</span> <span class="n">bits</span> <span class="mi">0</span>
</span></code></pre></td></tr></table></div></figure>


<p>这个索引很奇怪，是userid的索引？</p>

<p>分析之前，我们先看先看锁持有情况：</p>

<p>T1等待锁space id 3680 page no 30</p>

<p>T2持有锁space id 3680 page no 30</p>

<p>T2等待锁space id 3680 page no 137</p>

<p>最后回滚了T2</p>

<p>可以<strong>推断space id 3680 page no 137应该被T1持有了</strong>，但是日志中没有显示出来。</p>

<ul>
<li><p>3680 page no 30这个锁是一个主键索引PRIMARY导致的，实际上我们没有用到我们的自增主键，是非聚集索引，所以这是先锁的非主键索引最后找到的主键去加锁。</p></li>
<li><p>3680 page no 137这个锁就比较奇怪了，他锁在了idx_user_id这个索引，这个索引是加在userId上的，也就是T2<strong>他正在尝试锁所有这个用户的还款计划的记录！</strong></p></li>
</ul>


<p>如果是这样，问题就解释通了：</p>

<p>T1： 锁了某行记录X（具体怎么锁的，从死锁日志中未能获取），然后准备去获取LN201907120655461690006528458116，SEQ=1的记录的锁。</p>

<p>T2： 锁了LN201907120655461690006528458116，SEQ=1的锁，而他想去锁所有userId=938467411690006528的记录，这里面肯定包含了记录X，所以他无法获得X的锁。</p>

<p>这样就造成死锁了，因为X已经被T1持有了，而T1又在等T2释放LN201907120655461690006528458116，SEQ=1这个锁。</p>

<p>至于为什么T2明明准备操作LN201906130129401690006528175485，SEQ=2的记录，却之前持有了LN201907120655461690006528458116，SEQ=1的锁，大概率不是因为之前的SQL真的操作LN201907120655461690006528458116，SEQ=1的记录，也是因为他之前本想持有别的记录（从锁的详细信息上猜，可能是LN2019061301294016900065281754的相关记录），但是因为这个idx_user_id的索引问题，顺带锁着了LN201907120655461690006528458116，SEQ=1，因为都属于一个userId。</p>

<p>所以从时间线上分析，顺序应该是：</p>

<ol>
<li><p>T1锁了某记录X</p></li>
<li><p>T2锁了某记录Y（从hold this lock的日志细节中推断，是LN2019061301294016900065281754），然后准备锁LN201906130129401690006528175485，SEQ=2，这时候的这条SQL触发了idx_user_id，连带一起锁锁住了LN201907120655461690006528458116，SEQ=1并准备锁其它同用户记录</p></li>
<li><p>T1 执行下一条sql，准备获取LN201907120655461690006528458116，SEQ=1的锁，发现被T2获取了，等待。</p></li>
<li><p>T2在锁其它记录的过程中发现了X，但是锁不住，发现X被T1持有。而自己又持有了LN201907120655461690006528458116，SEQ=1这行记录的锁。</p></li>
</ol>


<p>这时候循环等待，死锁！</p>

<p>所以根源是为什么SQL会使用idx_user_id这个索引呢？</p>

<h2>索引信息</h2>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='mysql'><span class='line'><span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="ss">`id`</span><span class="p">),</span>
</span><span class='line'><span class="k">UNIQUE</span> <span class="k">KEY</span> <span class="ss">`uk_repay_order`</span> <span class="p">(</span><span class="ss">`loan_order_no`</span><span class="p">,</span><span class="ss">`seq_no`</span><span class="p">),</span>
</span><span class='line'><span class="k">UNIQUE</span> <span class="k">KEY</span> <span class="ss">`uk_repay_plan_no`</span> <span class="p">(</span><span class="ss">`repay_plan_no`</span><span class="p">),</span>
</span><span class='line'><span class="k">KEY</span> <span class="ss">`idx_user_id`</span> <span class="p">(</span><span class="ss">`user_id`</span><span class="p">),</span>
</span><span class='line'><span class="k">KEY</span> <span class="ss">`idx_create_time`</span> <span class="p">(</span><span class="ss">`create_time`</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>从唯一主键是UNIQUE KEY <code>uk_repay_order</code> (<code>loan_order_no</code>,<code>seq_no</code>)</p>

<p>从我们SQL上看<code>loan_order_no</code>+<code>seq_no</code>是唯一主键，应该肯定能唯一定位一行记录，索引应该使用这个是最优的才对。</p>

<p>这时候我们去看T2的那条SQL的执行计划得知：其没有使用索引uk_repay_order，而使用了一个type: index_merge，走的索引是uk_repay_order_,idx_user_id，也就是他居然两个索引同时生效了。</p>

<p><img src="http://jaskey.github.io/images/mysql-deadlock/deadlock-explain.png" alt="deadlock-codedeadlock-code" /></p>

<h1>解决方案</h1>

<p>实际上，由于index merge，客观上就会增加update语句的死锁可能性，相关bug连接如下：<a href="https://bugs.mysql.com/bug.php?id=77209">https://bugs.mysql.com/bug.php?id=77209</a></p>

<p>而其实如果出现了index merge，在很多情况下意味着我们索引的建立可能并不合理。</p>

<p>解决方案有两个：</p>

<ol>
<li>建立联合索引，以避免index merge，让联合索引生效则不会因此锁住所有该userId的记录</li>
<li>取消index merge的优化</li>
</ol>


<h1>遗留问题</h1>

<p>什么时候才会触发index merge，这个在文档中似乎并没有很明确的触发实际，从这些死锁的SQL来看，某些SQL在事后explain的时候，并没有走index merge，而有些却走了。从本案例来看，事务1的SQL并没有走index merge，但是事务2这样类似的SQL却走了。</p>

<p>只查到一个必要条件是：</p>

<blockquote><p>Intersect和Union都需要使用的索引是ROR的，也就时ROWID ORDERED，即针对不同的索引扫描出来的数据必须是同时按照ROWID排序的，这里的 ROWID其实也就是InnoDB的主键(如果不定义主键，InnoDB会隐式添加ROWID列作为主键)。只有每个索引是ROR的，才能进行归并排序，你懂的。 当然你可能会有疑惑，查不记录后内部进行一次sort不一样么，何必必须要ROR呢，不错，所以有了SORT-UNION。SORT-UNION就是每个非ROR的索引 排序后再进行Merge
– 来自 <a href="http://www.cnblogs.com/nocode/archive/2013/01/28/2880654.html">http://www.cnblogs.com/nocode/archive/2013/01/28/2880654.html</a></p></blockquote>

<p>为此我在stackoverflow 提了一个问题看后续有结论再更新：<a href="https://stackoverflow.com/questions/57987713/why-mysql-decide-to-use-index-merge-though-i-have-already-use-a-unique-key-index">https://stackoverflow.com/questions/57987713/why-mysql-decide-to-use-index-merge-though-i-have-already-use-a-unique-key-index</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Elastic Job从单点到高可用、同城主备、同城双活]]></title>
    <link href="https://Jaskey.github.io/blog/2020/05/25/elastic-job-timmer-active-standby/"/>
    <updated>2020-05-25T20:49:13+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/05/25/elastic-job-timmer-active-standby</id>
    <content type="html"><![CDATA[<p>在使用Elastic  Job Lite做定时任务的时候，我发现很多开发的团队都是直接部署单点，这对于一些离线的非核心业务（如对账、监控等）或许无关紧要，但对于一些高可用补偿、核心数据定时修改（如金融场景的利息更新等），单点部署则“非常危险”。实际上，Elastic  Job Lite是支持高可用的。网上关于Elastic Job的较高级的博文甚少，本文试图结合自身实践的一些经验，大致讲解其方案原理，并延伸至同城双机房的架构实践。</p>

<p>注：本文所有讨论均基于开源版本的Elastic Job Lite， 不涉及Elastic Job Cloud部分。</p>

<h2>单点部署到高可用</h2>

<p>如本文开头所说，很多系统的部署是采取以下部署架构：</p>

<p><img src="http://jaskey.github.io/images/esjob/esjob-single.png" alt="esjob-single" /></p>

<p>原因是开发者<strong>担心定时任务在同一时刻被触发多次</strong>，导致业务有问题。实际上这是对于框架最基本的原理不了解。在官方文档的功能列表里<a href="http://elasticjob.io/docs/elastic-job-lite/00-overview/">http://elasticjob.io/docs/elastic-job-lite/00-overview/</a>  就已说明其最基本的功能之一就是：</p>

<blockquote><p>作业分片一致性，保证同一分片在分布式环境中仅一个执行实例</p></blockquote>

<p>Elastic Job会依赖zookeeper选举出对应的实例做sharding，从而保证只有一个实例在执行同一个分片（如果任务没有采取分片（即分片数是0），就意味着这个任务只有一个实例在执行）</p>

<p><img src="https://camo.githubusercontent.com/f4d957e95b07c98cc1fe899b68915ad8e44c8f81/687474703a2f2f656c61737469636a6f622e696f2f646f63732f656c61737469632d6a6f622d6c6974652f696d672f6172636869746563747572652f656c61737469635f6a6f625f6c6974652e706e67" alt="elastic-job-架构" /></p>

<p>所以如下图所示的部署架构是完全没问题的——一来，服务只会被一个实例调用，二来，如果某个服务挂了，其他实例也能接管继续提供服务从而实现高可用。</p>

<p><img src="http://jaskey.github.io/images/esjob/esjob-cluster.png" alt="esjob-single" /></p>

<h1>双机房高可用</h1>

<p>随着互联网业务的发展，慢慢地，对架构的高可用会有更高的要求。下一步可能就是需要同城两机房部署，那这时候为了保证定时服务在两个机房的高可用，我们架构上可能会变成这样的：</p>

<p><img src="http://jaskey.github.io/images/esjob/esjob-cluster-2idc.png" alt="esjob-single" /></p>

<p>这样如果A机房的定时任务全部不可用了，B机房的确也能接手提供服务。而且由于集群是一个，Elastic Job能保证同一个分片在两个机房也只有一个实例运行。看似挺完美的。</p>

<p>注：本文不讨论zookeeper如何实现双机房的高可用，实际上从zookeeper的原理来看，仅仅两个机房组成一个大集群并不可以实现双机房高可用。</p>

<h1>优先级调度？</h1>

<p>以上的架构解决了定时任务在两个机房都可用的问题，但是实际的生产中，定时任务很可能是依赖存储的数据源的。而这个数据源，通常是有主备之分（这里不考虑单元化的架构的情况）：例如主在A机房，备在B机房做实时同步。</p>

<p>如果这个定时任务只有读操作，可能没问题，因为只要配置数据源连接同机房的数据源即可。但是如果是要写入的，就有一个问题——如果所有任务都在B机房被调度了，那么这些数据的写入都会跨机房地往A机房写入，这样延迟就大大提升了，如下图所示。</p>

<p><img src="http://jaskey.github.io/images/esjob/esjob-cluster-2idc-problem.png" alt="esjob-single" /></p>

<p>如图所示，如果Elastic Job把任务都调度到了B机房，那么流量就一直跨机房写了，这样对于性能来说是不好的事情。</p>

<p>那么有没有办法达到如下效果了：</p>

<ol>
<li>保证两个机房都随时可用，也就是一个机房的服务如果全部不可用了，另外一个机房能提供对等的服务</li>
<li>但一个任务可以优先指定A机房执行</li>
</ol>


<h2>Elastic Job分片策略</h2>

<p>在回答这个问题之前，我们需要了解下Elastic Job的分片策略，根据官网的说明（<a href="http://elasticjob.io/docs/elastic-job-lite/02-guide/job-sharding-strategy/">http://elasticjob.io/docs/elastic-job-lite/02-guide/job-sharding-strategy/</a>  ） ，Elastic Job是内置了一些分片策略可选的，其中有平均分配算法，作业名的哈希值奇偶数决定IP升降序算法和作业名的哈希值对服务器列表进行轮转；同时也是支持自定义的策略，实现实现<code>JobShardingStrategy</code>接口并实现<code>sharding</code>方法即可。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">sharding</span><span class="o">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">&gt;</span> <span class="n">jobInstances</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">shardingTotalCount</span><span class="o">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>假设我们可以实现这一的自定义策略：让做分片的时候知道哪些实例是A机房的，哪些是B机房的，然后我们知道A机房是优先的，在做分片策略的时候先把B机房的实例踢走，再复用原来的策略做分配。这不就解决我们的就近接入问题（接近数据源）了吗？</p>

<p>以下是利用装饰器模式自定义的一个装饰器类（抽象类，由子类判断哪些实例属于standby的实例），读者可以结合自身业务场景配合使用。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">JobShardingStrategyActiveStandbyDecorator</span> <span class="kd">implements</span> <span class="n">JobShardingStrategy</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//内置的分配策略采用原来的默认策略：平均</span>
</span><span class='line'>    <span class="kd">private</span> <span class="n">JobShardingStrategy</span> <span class="n">inner</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">AverageAllocationJobShardingStrategy</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>    <span class="cm">/**</span>
</span><span class='line'><span class="cm">     * 判断一个实例是否是备用的实例，在每次触发sharding方法之前会遍历所有实例调用此方法。</span>
</span><span class='line'><span class="cm">     * 如果主备实例同时存在于列表中，那么备实例将会被剔除后才进行sharding</span>
</span><span class='line'><span class="cm">     * @param jobInstance</span>
</span><span class='line'><span class="cm">     * @return</span>
</span><span class='line'><span class="cm">     */</span>
</span><span class='line'>    <span class="kd">protected</span> <span class="kd">abstract</span> <span class="kt">boolean</span> <span class="nf">isStandby</span><span class="o">(</span><span class="n">JobInstance</span> <span class="n">jobInstance</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">public</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">sharding</span><span class="o">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">&gt;</span> <span class="n">jobInstances</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">shardingTotalCount</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">List</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">&gt;</span> <span class="n">jobInstancesCandidates</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;(</span><span class="n">jobInstances</span><span class="o">);</span>
</span><span class='line'>        <span class="n">List</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">&gt;</span> <span class="n">removeInstance</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;();</span>
</span><span class='line'>
</span><span class='line'>        <span class="kt">boolean</span> <span class="n">removeSelf</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
</span><span class='line'>        <span class="k">for</span> <span class="o">(</span><span class="n">JobInstance</span> <span class="n">jobInstance</span> <span class="o">:</span> <span class="n">jobInstances</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="kt">boolean</span> <span class="n">isStandbyInstance</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
</span><span class='line'>            <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>                <span class="n">isStandbyInstance</span> <span class="o">=</span> <span class="n">isStandby</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">,</span> <span class="n">jobName</span><span class="o">);</span>
</span><span class='line'>            <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>                <span class="n">log</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">&quot;isStandBy throws error, consider as not standby&quot;</span><span class="o">,</span><span class="n">e</span><span class="o">);</span>
</span><span class='line'>            <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="k">if</span> <span class="o">(</span><span class="n">isStandbyInstance</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>                <span class="k">if</span> <span class="o">(</span><span class="n">IpUtils</span><span class="o">.</span><span class="na">getIp</span><span class="o">().</span><span class="na">equals</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">.</span><span class="na">getIp</span><span class="o">()))</span> <span class="o">{</span>
</span><span class='line'>                    <span class="n">removeSelf</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
</span><span class='line'>                <span class="o">}</span>
</span><span class='line'>                <span class="n">jobInstancesCandidates</span><span class="o">.</span><span class="na">remove</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">);</span>
</span><span class='line'>                <span class="n">removeInstance</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">);</span>
</span><span class='line'>            <span class="o">}</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="n">jobInstancesCandidates</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span><span class="c1">//移除后发现没有实例了，就不移除了，用原来的列表（后备）的顶上</span>
</span><span class='line'>            <span class="n">jobInstancesCandidates</span> <span class="o">=</span> <span class="n">jobInstances</span><span class="o">;</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[{}] ATTENTION!! Only backup job instances exist, but do sharding with them anyway {}&quot;</span><span class="o">,</span> <span class="n">jobName</span><span class="o">,</span> <span class="n">JSON</span><span class="o">.</span><span class="na">toJSONString</span><span class="o">(</span><span class="n">jobInstancesCandidates</span><span class="o">));</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="o">(!</span><span class="n">jobInstancesCandidates</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">jobInstances</span><span class="o">))</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[{}] remove backup before really do sharding, removeSelf :{} , remove instances: {}&quot;</span><span class="o">,</span> <span class="n">jobName</span><span class="o">,</span> <span class="n">removeSelf</span><span class="o">,</span> <span class="n">JSON</span><span class="o">.</span><span class="na">toJSONString</span><span class="o">(</span><span class="n">removeInstance</span><span class="o">));</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[{}] after remove backups :{}&quot;</span><span class="o">,</span> <span class="n">jobName</span><span class="o">,</span> <span class="n">JSON</span><span class="o">.</span><span class="na">toJSONString</span><span class="o">(</span><span class="n">jobInstancesCandidates</span><span class="o">));</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span><span class="c1">//全部都是master或者全部都是slave</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[{}] job instances just remain the same {}&quot;</span><span class="o">,</span> <span class="n">jobName</span><span class="o">,</span> <span class="n">JSON</span><span class="o">.</span><span class="na">toJSONString</span><span class="o">(</span><span class="n">jobInstancesCandidates</span><span class="o">));</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">//保险一点，排序一下，保证每个实例拿到的列表肯定是一样的</span>
</span><span class='line'>        <span class="n">jobInstancesCandidates</span><span class="o">.</span><span class="na">sort</span><span class="o">((</span><span class="n">o1</span><span class="o">,</span> <span class="n">o2</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">o1</span><span class="o">.</span><span class="na">getJobInstanceId</span><span class="o">().</span><span class="na">compareTo</span><span class="o">(</span><span class="n">o2</span><span class="o">.</span><span class="na">getJobInstanceId</span><span class="o">()));</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">return</span> <span class="n">inner</span><span class="o">.</span><span class="na">sharding</span><span class="o">(</span><span class="n">jobInstancesCandidates</span><span class="o">,</span> <span class="n">jobName</span><span class="o">,</span> <span class="n">shardingTotalCount</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h2>利用自定义策略实现同城双机房下的优先级调度</h2>

<p>以下是一个很简单的就近接入的例子： 指定在ip白名单的，就是优先执行的，不在的都认为是备用的。我们看如何实现。</p>

<h3>一、继承此装饰器策略，指定哪些实例是standby实例</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ActiveStandbyESJobStrategy</span> <span class="kd">extends</span> <span class="n">JobShardingStrategyActiveStandbyDecorator</span><span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">protected</span> <span class="kt">boolean</span> <span class="nf">isStandby</span><span class="o">(</span><span class="n">JobInstance</span> <span class="n">jobInstance</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">activeIps</span> <span class="o">=</span> <span class="s">&quot;10.10.10.1,10.10.10.2&quot;</span><span class="o">;</span><span class="c1">//只有这两个ip的实例才是优先执行的，其他都是备用的</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">ss</span><span class="o">[]</span> <span class="o">=</span> <span class="n">activeIps</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">&quot;,&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="k">return</span> <span class="o">!</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">ss</span><span class="o">).</span><span class="na">contains</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">.</span><span class="na">getIp</span><span class="o">());</span><span class="c1">//不在active名单的就是后备</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>很简单吧！这样实现之后，就能达到以下类似的效果</p>

<p><img src="http://jaskey.github.io/images/esjob/esjob-cluster-2idc-active-standby.png" alt="esjob-single" /></p>

<h3>二、 在任务启动前，指定使用这个策略</h3>

<p>以下以Java的方式示意，</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">JobCoreConfiguration</span> <span class="n">simpleCoreConfig</span> <span class="o">=</span> <span class="n">JobCoreConfiguration</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">jobClass</span><span class="o">.</span><span class="na">getName</span><span class="o">(),</span> <span class="n">cron</span><span class="o">,</span> <span class="n">shardingTotalCount</span><span class="o">).</span><span class="na">shardingItemParameters</span><span class="o">(</span><span class="n">shardingItemParameters</span><span class="o">).</span><span class="na">build</span><span class="o">();</span>
</span><span class='line'><span class="n">SimpleJobConfiguration</span> <span class="n">simpleJobConfiguration</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">SimpleJobConfiguration</span><span class="o">(</span><span class="n">simpleCoreConfig</span><span class="o">,</span> <span class="n">jobClass</span><span class="o">.</span><span class="na">getCanonicalName</span><span class="o">());</span>
</span><span class='line'><span class="k">return</span> <span class="n">LiteJobConfiguration</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">simpleJobConfiguration</span><span class="o">)</span>
</span><span class='line'>        <span class="o">.</span><span class="na">jobShardingStrategyClass</span><span class="o">(</span><span class="s">&quot;com.xxx.yyy.job.ActiveStandbyESJobStrategy&quot;</span><span class="o">)</span><span class="c1">//使用主备的分配策略，分主备实例（输入你的实现类类名）</span>
</span><span class='line'>        <span class="o">.</span><span class="na">build</span><span class="o">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样就大功告成了。</p>

<h1>同城双活模式</h1>

<p>以上这样改造后，针对定时任务就已经解决了两个问题：</p>

<p>1、定时任务能实现在两个机房下的高可用</p>

<p>2、任务能优先调度到指定机房</p>

<p>这种模式下，对于定时任务来说，B机房其实只是个备机房——因为A机房永远都是优先调度的。</p>

<p>对于B机房是否有一些实际问题其实我们可能是不知道的（常见的例如数据库权限没申请），由于没有流量的验证，这时候真的出现容灾问题，B机房是否能安全接受其实并不是100%稳妥的。</p>

<p>我们能否再进一步做到同城双活呢？也就是，B机房也会承担一部分的流量？例如10%？</p>

<p>回到自定义策略的sharding接口：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'> <span class="kd">public</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">sharding</span><span class="o">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">JobInstance</span><span class="o">&gt;</span> <span class="n">jobInstances</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">shardingTotalCount</span><span class="o">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>在做分配的时候，是能拿到一个任务实例的全景图（所有实例列表），当前的任务名，和分片数。</p>

<p>基于此其实是可以做一些事情把流量引流到B机房实例的，例如：</p>

<ol>
<li>指定任务的主机房让其是B机房优先调度（例如挑选部分只读任务，占10%的任务数）</li>
<li>对于分片的分配，把末尾（如1/10）的分片优先分配给B机房。</li>
</ol>


<p>以上两种方案都能实现让A、B两个机房都有流量（有任务在被调度），从而实现所谓的双活。</p>

<p>以下针对上面抛出来的方案一，给出一个双活的示意代码和架构。</p>

<p>假设我们定时任务有两个任务，TASK_A_FIRST，TASK_B_FIRST，其中TASK_B_FIRST是一个只读的任务，那么我们可以让他配置读B机房的备库让他优先运行在B机房，而TASK_A_FIRST是一个更为频繁的任务，而且带有写操作，我们则优先运行在A机房，从而实现双机房均有流量。</p>

<p>注：这里任意一个机房不可用了，任务均能在另外一个机房调度，这里增强的只是对于不同任务做针对性的优先调度实现双活</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ActiveStandbyESJobStrategy</span> <span class="kd">extends</span> <span class="n">JobShardingStrategyActiveStandbyDecorator</span><span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">protected</span> <span class="kt">boolean</span> <span class="nf">isStandby</span><span class="o">(</span><span class="n">JobInstance</span> <span class="n">jobInstance</span><span class="o">,</span> <span class="n">String</span> <span class="n">jobName</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>         <span class="n">String</span> <span class="n">activeIps</span> <span class="o">=</span> <span class="s">&quot;10.10.10.1,10.10.10.2&quot;</span><span class="o">;</span><span class="c1">//默认只有这两个ip的实例才是优先执行的，其他都是备用的</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="s">&quot;TASK_B_FIRST&quot;</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">jobName</span><span class="o">)){</span><span class="c1">//选择这个任务优先调度到B机房</span>
</span><span class='line'>           <span class="n">activeIps</span> <span class="o">=</span> <span class="s">&quot;10.11.10.1,10.11.10.2&quot;</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">String</span> <span class="n">ss</span><span class="o">[]</span> <span class="o">=</span> <span class="n">activeIps</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">&quot;,&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="k">return</span> <span class="o">!</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">ss</span><span class="o">).</span><span class="na">contains</span><span class="o">(</span><span class="n">jobInstance</span><span class="o">.</span><span class="na">getIp</span><span class="o">());</span><span class="c1">//不在active名单的就是后备</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p><img src="http://jaskey.github.io/images/esjob/esjob-cluster-2idc-2live.png" alt="esjob-single" /></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[[DUBBO] ReferenceConfig(null) Is Not DESTROYED When FINALIZE分析及解决]]></title>
    <link href="https://Jaskey.github.io/blog/2020/05/22/dubbo-refernececonfig-is-not-destroyed-when-finalize/"/>
    <updated>2020-05-22T18:14:58+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/05/22/dubbo-refernececonfig-is-not-destroyed-when-finalize</id>
    <content type="html"><![CDATA[<p>最近发现经常有类似告警：</p>

<p>​    [DUBBO] ReferenceConfig(null) is not DESTROYED when FINALIZE，dubbo version 2.6.2</p>

<p>在此记录一下分析的过程和解决方案。</p>

<h2>从日志定位源码位置</h2>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">&quot;unused&quot;</span><span class="o">)</span>
</span><span class='line'><span class="kd">private</span> <span class="kd">final</span> <span class="n">Object</span> <span class="n">finalizerGuardian</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Object</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">finalize</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Throwable</span> <span class="o">{</span>
</span><span class='line'>        <span class="kd">super</span><span class="o">.</span><span class="na">finalize</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="o">(!</span><span class="n">ReferenceConfig</span><span class="o">.</span><span class="na">this</span><span class="o">.</span><span class="na">destroyed</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">logger</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">&quot;ReferenceConfig(&quot;</span> <span class="o">+</span> <span class="n">url</span> <span class="o">+</span> <span class="s">&quot;) is not DESTROYED when FINALIZE&quot;</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>            <span class="cm">/* don&#39;t destroy for now</span>
</span><span class='line'><span class="cm">            try {</span>
</span><span class='line'><span class="cm">                ReferenceConfig.this.destroy();</span>
</span><span class='line'><span class="cm">            } catch (Throwable t) {</span>
</span><span class='line'><span class="cm">                    logger.warn(&quot;Unexpected err when destroy invoker of ReferenceConfig(&quot; + url + &quot;) in finalize method!&quot;, t);</span>
</span><span class='line'><span class="cm">            }</span>
</span><span class='line'><span class="cm">            */</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">};</span>
</span></code></pre></td></tr></table></div></figure>


<p>通过日志搜索源码，发现这个日志是<code>ReferenceConfig</code>类中一个<code>finalizerGuardian</code>的实例变量下复写了<code>finalize</code>而打印出来的。而从其中的源码来看，原本应该是希望做到：在发现原本的对象没有被释放资源的时候，手动回收资源。但是后面缺代码被注释了（猜测是有巨坑），就只保留了一个告警。所以实际上这个对象现在只起到了WARN日志提示的作用，并无实际作用。</p>

<p>注：复写<code>finalize</code>方法会导致一定的GC回收的性能问题，因为一个对象如果其中<code>finalize</code>被复写（哪怕只是写一个分号空实现），在垃圾回收的时候都会以单独的方法回收，简单说就是有一条独立的Finalizer线程（优先级很低）单独回收，如果对象分配频繁，会引起一定的性能问题。回到Dubbo的场景，假设在高并发的场景下不断创建ReferenceConfig对象，会影响这些对象的回收效率（并且这个过程中会产生一些<code>java.lang.ref.Finalizer</code>对象）甚至OOM，现在只是打印一个日志是一个不好的实践。对于finalize的原理和其对垃圾回收的影响可以参考<a href="https://blog.heaphero.io/2018/04/13/heaphero-user-manual-2/#ObjFin">https://blog.heaphero.io/2018/04/13/heaphero-user-manual-2/#ObjFin</a>  ，这里摘抄其中一段供参考：</p>

<blockquote><p>Objects that have finalize() method are treated differently during garbage collection process than the ones which don’t have. During garbage collection phase, objects with finalize() method aren’t immediately evicted from the memory. Instead, as the first step, those objects are added to an internal queue of java.lang.ref.Finalizer. For entire JVM, there is only one low priority JVM thread by name ‘Finalizer’ that executes finalize() method of each object in the queue. Only after the execution of finalize() method, object becomes eligible for Garbage Collection. Assume if your application is producing a lot of objects which has finalize() method and low priority “Finalizer” thread isn’t able to keep up with executing finalize() method, then significant amount unfinalized objects will start to build up in the internal queue of java.lang.ref.Finalize, which would result in significant amount of memory wastage.</p></blockquote>

<h2>问题分析</h2>

<p>从以上的源码上，这句日志打印的充分必要条件是：</p>

<p>1.ReferenceConfig被垃圾回收</p>

<p>2.垃圾回收的时候ReferenceConfig没有调用过destroy方法，即<code>!ReferenceConfig.this.destroyed</code></p>

<h2>代码始末</h2>

<p>基于以上的分析，需要知道哪里我们会创建<code>ReferenceConfig</code>对象。通常情况下，这个对象我们是不会代码显示创建的，因为正常都是Dubbo基于我们的配置（注解或者配置文件）去管理内部的对象，只有我们在泛化调用的时候，可能会手动创建。</p>

<p>Dubbo官方文档里<a href="http://dubbo.apache.org/zh-cn/docs/user/demos/generic-reference.html">http://dubbo.apache.org/zh-cn/docs/user/demos/generic-reference.html</a>  ，关于泛化调用有类似的代码，其中就会手动创建<code>ReferenceConfig</code>对象</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="c1">// 引用远程服务 </span>
</span><span class='line'><span class="c1">// 该实例很重量，里面封装了所有与注册中心及服务提供方连接，请缓存</span>
</span><span class='line'><span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;</span> <span class="n">reference</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;();</span>
</span><span class='line'><span class="c1">// 弱类型接口名</span>
</span><span class='line'><span class="n">reference</span><span class="o">.</span><span class="na">setInterface</span><span class="o">(</span><span class="s">&quot;com.xxx.XxxService&quot;</span><span class="o">);</span>
</span><span class='line'><span class="n">reference</span><span class="o">.</span><span class="na">setVersion</span><span class="o">(</span><span class="s">&quot;1.0.0&quot;</span><span class="o">);</span>
</span><span class='line'><span class="c1">// 声明为泛化接口 </span>
</span><span class='line'><span class="n">reference</span><span class="o">.</span><span class="na">setGeneric</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 用org.apache.dubbo.rpc.service.GenericService可以替代所有接口引用  </span>
</span><span class='line'><span class="n">GenericService</span> <span class="n">genericService</span> <span class="o">=</span> <span class="n">reference</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 基本类型以及Date,List,Map等不需要转换，直接调用 </span>
</span><span class='line'><span class="n">Object</span> <span class="n">result</span> <span class="o">=</span> <span class="n">genericService</span><span class="o">.</span><span class="na">$invoke</span><span class="o">(</span><span class="s">&quot;sayHello&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span><span class="s">&quot;java.lang.String&quot;</span><span class="o">},</span> <span class="k">new</span> <span class="n">Object</span><span class="o">[]</span> <span class="o">{</span><span class="s">&quot;world&quot;</span><span class="o">});</span>
</span><span class='line'>
</span></code></pre></td></tr></table></div></figure>


<p>由于以上代码会存在很容易导致连接等相关资源泄露等问题，详见：<a href="http://dubbo.apache.org/zh-cn/docs/user/demos/reference-config-cache.html">http://dubbo.apache.org/zh-cn/docs/user/demos/reference-config-cache.html</a> ，所以正常的泛化调用的使用方式则变成这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">XxxService</span><span class="o">&gt;</span> <span class="n">reference</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">XxxService</span><span class="o">&gt;();</span>
</span><span class='line'><span class="n">reference</span><span class="o">.</span><span class="na">setInterface</span><span class="o">(</span><span class="n">XxxService</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'><span class="n">reference</span><span class="o">.</span><span class="na">setVersion</span><span class="o">(</span><span class="s">&quot;1.0.0&quot;</span><span class="o">);</span>
</span><span class='line'><span class="o">......</span>
</span><span class='line'><span class="n">ReferenceConfigCache</span> <span class="n">cache</span> <span class="o">=</span> <span class="n">ReferenceConfigCache</span><span class="o">.</span><span class="na">getCache</span><span class="o">();</span>
</span><span class='line'><span class="c1">// cache.get方法中会缓存 Reference对象，并且调用ReferenceConfig.get方法启动ReferenceConfig</span>
</span><span class='line'><span class="n">XxxService</span> <span class="n">xxxService</span> <span class="o">=</span> <span class="n">cache</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">reference</span><span class="o">);</span>
</span><span class='line'><span class="c1">// 注意！ Cache会持有ReferenceConfig，不要在外部再调用ReferenceConfig的destroy方法，导致Cache内的ReferenceConfig失效！</span>
</span><span class='line'><span class="c1">// 使用xxxService对象</span>
</span><span class='line'><span class="n">xxxService</span><span class="o">.</span><span class="na">sayHello</span><span class="o">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>Dubbo官方对于相关建议的解释是：</p>

<blockquote><p><code>ReferenceConfig</code> 实例很重，封装了与注册中心的连接以及与提供者的连接，需要缓存。否则重复生成 <code>ReferenceConfig</code> 可能造成性能问题并且会有内存和连接泄漏。在 API 方式编程时，容易忽略此问题。</p></blockquote>

<p>但是，实际上这句话和其API的实际设计上存在一定的误解。Dubbo认为<code>ReferenceConfig</code> 实例很重，所以应该缓存这个对象，所以设计了一个<code>ReferenceConfigCache</code>类，这个类实际上可以认为就是一个Map，当第一次调用cache.get(reference)的时候，实际上会把这个<code>ReferenceConfig</code> 放到里面的Map中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">T</span> <span class="nf">get</span><span class="o">(</span><span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">referenceConfig</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">key</span> <span class="o">=</span> <span class="n">generator</span><span class="o">.</span><span class="na">generateKey</span><span class="o">(</span><span class="n">referenceConfig</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">ReferenceConfig</span><span class="o">&lt;?&gt;</span> <span class="n">config</span> <span class="o">=</span> <span class="n">cache</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">config</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="o">(</span><span class="n">T</span><span class="o">)</span> <span class="n">config</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">cache</span><span class="o">.</span><span class="na">putIfAbsent</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">referenceConfig</span><span class="o">);</span>
</span><span class='line'>    <span class="n">config</span> <span class="o">=</span> <span class="n">cache</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
</span><span class='line'>    <span class="k">return</span> <span class="o">(</span><span class="n">T</span><span class="o">)</span> <span class="n">config</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>而<code>config.get()</code>的时候，实际上会调用内部的各种初始化的代码。</p>

<p>但是，这里接口设计有一个自相矛盾的地方。怎么讲了，因为“<code>ReferenceConfig</code> 实例很重”，所以，<code>ReferenceConfigCache</code>帮我们做了缓存，但是使用的时候，接口的设计却只能接受一个<code>ReferenceConfig</code> 对象，那这个对象从何而来呢？也就是说，这个缓存其实只能给Dubbo内部使用——用户给一个<code>ReferenceConfig</code> 对象给Dubbo，Dubbo判断这个对象是不是和以前的对象等价，等价的话我就不用用户传递的，用以前创建好的（因为这个对象各种资源都创建好了，没必要重复创建）。</p>

<h2>原因呼之欲出</h2>

<p>分析到这里，其实这个问题的原因已经呼之欲出了：因为泛化调用而创建了<code>ReferenceConfig</code> 对象。实际上，要复现这个问题，只需要模拟不断创建临时<code>ReferenceConfig</code> 变量然后触发GC即可：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span><span class="o">&lt;</span><span class="mi">100000</span><span class="o">;</span><span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">new</span> <span class="n">ReferenceConfig</span><span class="o">&lt;&gt;();</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'><span class="n">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span><span class="c1">//手动触发一下GC，确保上面创建的ReferenceConfig能触发GC回收</span>
</span></code></pre></td></tr></table></div></figure>


<p>运行以上代码，你能发现和本文开头一模一样的告警日志。</p>

<h2>为什么是ReferenceConfig(null)，即URL为什么是null？</h2>

<p>你可能会问，上面的分析原因是清楚了，但是为什么ReferenceConfig打印的时候，url参数总是显示null? 其实上面的分析已经回答这个问题了。上面章节我们提到“而<code>config.get()</code>的时候，实际上会调用内部的各种初始化的代码。”而这个初始化的过程之一就是构建合适的url，所以当我们使用<code>`ReferenceConfigCache</code>做泛化调用的时候，除了第一次创建的ReferenceConfig被<code>ReferenceConfigCache</code>缓存起来并初始化了，其他的对象其实都没有初始化过，那自然URL就是空了，同时又因为没有被缓存（所以对象在方法运行结束后不可达了）必然后面会被触发GC，那日志看起来就是ReferenceConfig(null)。</p>

<p>实际上分析到这里，我们可以看出这里Dubbo有两个处理不好的地方</p>

<ol>
<li>API设计不合理——认为ReferenceConfig很重需要缓存，但是使用的时候必须要提供一个对象</li>
<li>ReferenceConfig回收的告警上，其实是存在优化空间的，像这种没有初始化过的对象，其实没必要打WARN日志，毕竟他没有初始化过就自然没有什么可destroy的</li>
</ol>


<h2>解决方案</h2>

<p>从这里分析可以看到这行告警其实是”误报“，只要日志里url显示是null，并没有什么特殊的实际影响（在不考虑上文讲的GC问题的前提下）</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="o">[</span><span class="n">DUBBO</span><span class="o">]</span> <span class="nf">ReferenceConfig</span><span class="o">(</span><span class="kc">null</span><span class="o">)</span> <span class="n">is</span> <span class="n">not</span> <span class="n">DESTROYED</span> <span class="n">when</span> <span class="n">FINALIZE</span><span class="err">，</span><span class="n">dubbo</span> <span class="n">version</span> <span class="mf">2.6</span><span class="o">.</span><span class="mi">2</span>
</span></code></pre></td></tr></table></div></figure>


<p>那如果要修复这个告警，则可考虑显示的缓存<code>ReferenceConfig</code>对象，不要每次泛化调用的时候都创建一个，可参考以下代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">private</span> <span class="kd">synchronized</span> <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;</span> <span class="nf">getOrNewReferenceConfig</span><span class="o">(</span><span class="n">String</span> <span class="n">interfaceClass</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">refConfigCacheKey</span> <span class="o">=</span> <span class="n">interfaceClass</span><span class="o">;</span>
</span><span class='line'>    <span class="n">WeakReference</span><span class="o">&lt;</span><span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;&gt;</span> <span class="n">referenceConfigWeakReference</span> <span class="o">=</span> <span class="n">refConfigCache</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">refConfigCacheKey</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">referenceConfigWeakReference</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span><span class="c1">//缓存有弱引用</span>
</span><span class='line'>        <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;</span> <span class="n">referenceConfigFromWR</span> <span class="o">=</span> <span class="n">referenceConfigWeakReference</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="n">referenceConfigFromWR</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span><span class="c1">//证明没人引用自己被GC了，需要重建</span>
</span><span class='line'>            <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;</span> <span class="n">referenceConfig</span> <span class="o">=</span> <span class="n">newRefConifg</span><span class="o">(</span><span class="n">interfaceClass</span><span class="o">);</span>
</span><span class='line'>            <span class="n">refConfigCache</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">refConfigCacheKey</span><span class="o">,</span> <span class="k">new</span> <span class="n">WeakReference</span><span class="o">&lt;&gt;(</span><span class="n">referenceConfig</span><span class="o">));</span><span class="c1">//放入缓存中，用弱应用hold住，不影响该有GC</span>
</span><span class='line'>            <span class="k">return</span> <span class="n">referenceConfig</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class='line'>            <span class="k">return</span> <span class="n">referenceConfigFromWR</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="o">}</span> <span class="k">else</span> <span class="o">{</span><span class="c1">//缓存没有，则创建</span>
</span><span class='line'>        <span class="n">ReferenceConfig</span><span class="o">&lt;</span><span class="n">GenericService</span><span class="o">&gt;</span> <span class="n">referenceConfig</span> <span class="o">=</span> <span class="n">newRefConifg</span><span class="o">(</span><span class="n">interfaceClass</span><span class="o">);</span>
</span><span class='line'>        <span class="n">refConfigCache</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">refConfigCacheKey</span><span class="o">,</span> <span class="k">new</span> <span class="n">WeakReference</span><span class="o">&lt;&gt;(</span><span class="n">referenceConfig</span><span class="o">));</span><span class="c1">//放入缓存中，用弱应用hold住，不影响该有GC</span>
</span><span class='line'>        <span class="k">return</span> <span class="n">referenceConfig</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>注：</p>

<ol>
<li>其中newRefConifg即为原先的创建<code>ReferenceConfig</code>的代码</li>
<li>之所以使用<code>WeakReference</code>是为了保证这个缓存的对象不会影响GC——即该回收的时候还是得回收</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[优雅地处理重复请求（并发请求）——附Java实现]]></title>
    <link href="https://Jaskey.github.io/blog/2020/05/19/handle-duplicate-request/"/>
    <updated>2020-05-19T19:52:01+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/05/19/handle-duplicate-request</id>
    <content type="html"><![CDATA[<p>对于一些用户请求，在某些情况下是可能重复发送的，如果是查询类操作并无大碍，但其中有些是涉及写入操作的，一旦重复了，可能会导致很严重的后果，例如交易的接口如果重复请求可能会重复下单。</p>

<p>重复的场景有可能是：</p>

<ol>
<li>黑客拦截了请求，重放</li>
<li>前端/客户端因为某些原因请求重复发送了，或者用户在很短的时间内重复点击了。</li>
<li>网关重发</li>
<li>&hellip;.</li>
</ol>


<p>本文讨论的是如果在服务端优雅地统一处理这种情况，如何禁止用户重复点击等客户端操作不在本文的讨论范畴。</p>

<h2>利用唯一请求编号去重</h2>

<p>你可能会想到的是，只要请求有唯一的请求编号，那么就能借用Redis做这个去重——只要这个唯一请求编号在redis存在，证明处理过，那么就认为是重复的</p>

<p>代码大概如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'>    <span class="n">String</span> <span class="n">KEY</span> <span class="o">=</span> <span class="s">&quot;REQ12343456788&quot;</span><span class="o">;</span><span class="c1">//请求唯一编号</span>
</span><span class='line'>    <span class="kt">long</span> <span class="n">expireTime</span> <span class="o">=</span>  <span class="mi">1000</span><span class="o">;</span><span class="c1">// 1000毫秒过期，1000ms内的重复请求会认为重复</span>
</span><span class='line'>    <span class="kt">long</span> <span class="n">expireAt</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">()</span> <span class="o">+</span> <span class="n">expireTime</span><span class="o">;</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">val</span> <span class="o">=</span> <span class="s">&quot;expireAt@&quot;</span> <span class="o">+</span> <span class="n">expireAt</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//redis key还存在的话要就认为请求是重复的</span>
</span><span class='line'>    <span class="n">Boolean</span> <span class="n">firstSet</span> <span class="o">=</span> <span class="n">stringRedisTemplate</span><span class="o">.</span><span class="na">execute</span><span class="o">((</span><span class="n">RedisCallback</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;)</span> <span class="n">connection</span> <span class="o">-&gt;</span> <span class="n">connection</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">KEY</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">val</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">Expiration</span><span class="o">.</span><span class="na">milliseconds</span><span class="o">(</span><span class="n">expireTime</span><span class="o">),</span> <span class="n">RedisStringCommands</span><span class="o">.</span><span class="na">SetOption</span><span class="o">.</span><span class="na">SET_IF_ABSENT</span><span class="o">));</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">final</span> <span class="kt">boolean</span> <span class="n">isConsiderDup</span><span class="o">;</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span><span class="n">firstSet</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="n">firstSet</span><span class="o">)</span> <span class="o">{</span><span class="c1">// 第一次访问</span>
</span><span class='line'>        <span class="n">isConsiderDup</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span> <span class="k">else</span> <span class="o">{</span><span class="c1">// redis值已存在，认为是重复了</span>
</span><span class='line'>        <span class="n">isConsiderDup</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h2>业务参数去重</h2>

<p>上面的方案能解决具备唯一请求编号的场景，例如每次写请求之前都是服务端返回一个唯一编号给客户端，客户端带着这个请求号做请求，服务端即可完成去重拦截。</p>

<p>但是，很多的场景下，请求并不会带这样的唯一编号！那么我们能否针对请求的参数作为一个请求的标识呢？</p>

<p>先考虑简单的场景，假设请求参数只有一个字段reqParam，我们可以利用以下标识去判断这个请求是否重复。 <strong>用户ID:接口名:请求参数</strong></p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">KEY</span> <span class="o">=</span> <span class="s">&quot;dedup:U=&quot;</span><span class="o">+</span><span class="n">userId</span> <span class="o">+</span> <span class="s">&quot;M=&quot;</span> <span class="o">+</span> <span class="n">method</span> <span class="o">+</span> <span class="s">&quot;P=&quot;</span> <span class="o">+</span> <span class="n">reqParam</span><span class="o">;</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么当同一个用户访问同一个接口，带着同样的reqParam过来，我们就能定位到他是重复的了。</p>

<p>但是问题是，我们的接口通常不是这么简单，以目前的主流，我们的参数通常是一个JSON。那么针对这种场景，我们怎么去重呢？</p>

<h3>计算请求参数的摘要作为参数标识</h3>

<p>假设我们把请求参数（JSON）按KEY做升序排序，排序后拼成一个字符串，作为KEY值呢？但这可能非常的长，所以我们可以考虑对这个字符串求一个MD5作为参数的摘要，以这个摘要去取代reqParam的位置。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">KEY</span> <span class="o">=</span> <span class="s">&quot;dedup:U=&quot;</span><span class="o">+</span><span class="n">userId</span> <span class="o">+</span> <span class="s">&quot;M=&quot;</span> <span class="o">+</span> <span class="n">method</span> <span class="o">+</span> <span class="s">&quot;P=&quot;</span> <span class="o">+</span> <span class="n">reqParamMD5</span><span class="o">;</span>
</span></code></pre></td></tr></table></div></figure>


<p>这样，请求的唯一标识就打上了！</p>

<p>注：MD5理论上可能会重复，但是去重通常是短时间窗口内的去重（例如一秒），一个短时间内同一个用户同样的接口能拼出不同的参数导致一样的MD5几乎是不可能的。</p>

<h3>继续优化，考虑剔除部分时间因子</h3>

<p>上面的问题其实已经是一个很不错的解决方案了，但是实际投入使用的时候可能发现有些问题：某些请求用户短时间内重复的点击了（例如1000毫秒发送了三次请求），但绕过了上面的去重判断（不同的KEY值）。</p>

<p>原因是这些请求参数的字段里面，<strong>是带时间字段的</strong>，这个字段标记用户请求的时间，服务端可以借此丢弃掉一些老的请求（例如5秒前）。如下面的例子，请求的其他参数是一样的，除了请求时间相差了一秒：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'>    <span class="c1">//两个请求一样，但是请求时间差一秒</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">req</span> <span class="o">=</span> <span class="s">&quot;{\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestTime\&quot; :\&quot;20190101120001\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestValue\&quot; :\&quot;1000\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestKey\&quot; :\&quot;key\&quot;\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;}&quot;</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">String</span> <span class="n">req2</span> <span class="o">=</span> <span class="s">&quot;{\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestTime\&quot; :\&quot;20190101120002\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestValue\&quot; :\&quot;1000\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestKey\&quot; :\&quot;key\&quot;\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;}&quot;</span><span class="o">;</span>
</span></code></pre></td></tr></table></div></figure>


<p>这种请求，我们也很可能需要挡住后面的重复请求。所以求业务参数摘要之前，需要剔除这类时间字段。还有类似的字段可能是GPS的经纬度字段（重复请求间可能有极小的差别）。</p>

<h2>请求去重工具类，Java实现</h2>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ReqDedupHelper</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="cm">/**</span>
</span><span class='line'><span class="cm">     *</span>
</span><span class='line'><span class="cm">     * @param reqJSON 请求的参数，这里通常是JSON</span>
</span><span class='line'><span class="cm">     * @param excludeKeys 请求参数里面要去除哪些字段再求摘要</span>
</span><span class='line'><span class="cm">     * @return 去除参数的MD5摘要</span>
</span><span class='line'><span class="cm">     */</span>
</span><span class='line'>    <span class="kd">public</span> <span class="n">String</span> <span class="nf">dedupParamMD5</span><span class="o">(</span><span class="kd">final</span> <span class="n">String</span> <span class="n">reqJSON</span><span class="o">,</span> <span class="n">String</span><span class="o">...</span> <span class="n">excludeKeys</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">decreptParam</span> <span class="o">=</span> <span class="n">reqJSON</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">TreeMap</span> <span class="n">paramTreeMap</span> <span class="o">=</span> <span class="n">JSON</span><span class="o">.</span><span class="na">parseObject</span><span class="o">(</span><span class="n">decreptParam</span><span class="o">,</span> <span class="n">TreeMap</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="n">excludeKeys</span><span class="o">!=</span><span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">dedupExcludeKeys</span> <span class="o">=</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">excludeKeys</span><span class="o">);</span>
</span><span class='line'>            <span class="k">if</span> <span class="o">(!</span><span class="n">dedupExcludeKeys</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>                <span class="k">for</span> <span class="o">(</span><span class="n">String</span> <span class="n">dedupExcludeKey</span> <span class="o">:</span> <span class="n">dedupExcludeKeys</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>                    <span class="n">paramTreeMap</span><span class="o">.</span><span class="na">remove</span><span class="o">(</span><span class="n">dedupExcludeKey</span><span class="o">);</span>
</span><span class='line'>                <span class="o">}</span>
</span><span class='line'>            <span class="o">}</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">String</span> <span class="n">paramTreeMapJSON</span> <span class="o">=</span> <span class="n">JSON</span><span class="o">.</span><span class="na">toJSONString</span><span class="o">(</span><span class="n">paramTreeMap</span><span class="o">);</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">md5deDupParam</span> <span class="o">=</span> <span class="n">jdkMD5</span><span class="o">(</span><span class="n">paramTreeMapJSON</span><span class="o">);</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">debug</span><span class="o">(</span><span class="s">&quot;md5deDupParam = {}, excludeKeys = {} {}&quot;</span><span class="o">,</span> <span class="n">md5deDupParam</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">deepToString</span><span class="o">(</span><span class="n">excludeKeys</span><span class="o">),</span> <span class="n">paramTreeMapJSON</span><span class="o">);</span>
</span><span class='line'>        <span class="k">return</span> <span class="n">md5deDupParam</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">private</span> <span class="kd">static</span> <span class="n">String</span> <span class="nf">jdkMD5</span><span class="o">(</span><span class="n">String</span> <span class="n">src</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">res</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
</span><span class='line'>        <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">MessageDigest</span> <span class="n">messageDigest</span> <span class="o">=</span> <span class="n">MessageDigest</span><span class="o">.</span><span class="na">getInstance</span><span class="o">(</span><span class="s">&quot;MD5&quot;</span><span class="o">);</span>
</span><span class='line'>            <span class="kt">byte</span><span class="o">[]</span> <span class="n">mdBytes</span> <span class="o">=</span> <span class="n">messageDigest</span><span class="o">.</span><span class="na">digest</span><span class="o">(</span><span class="n">src</span><span class="o">.</span><span class="na">getBytes</span><span class="o">());</span>
</span><span class='line'>            <span class="n">res</span> <span class="o">=</span> <span class="n">DatatypeConverter</span><span class="o">.</span><span class="na">printHexBinary</span><span class="o">(</span><span class="n">mdBytes</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">&quot;&quot;</span><span class="o">,</span><span class="n">e</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>        <span class="k">return</span> <span class="n">res</span><span class="o">;</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>下面是一些测试日志：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="c1">//两个请求一样，但是请求时间差一秒</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">req</span> <span class="o">=</span> <span class="s">&quot;{\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestTime\&quot; :\&quot;20190101120001\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestValue\&quot; :\&quot;1000\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestKey\&quot; :\&quot;key\&quot;\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;}&quot;</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">String</span> <span class="n">req2</span> <span class="o">=</span> <span class="s">&quot;{\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestTime\&quot; :\&quot;20190101120002\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestValue\&quot; :\&quot;1000\&quot;,\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;\&quot;requestKey\&quot; :\&quot;key\&quot;\n&quot;</span> <span class="o">+</span>
</span><span class='line'>            <span class="s">&quot;}&quot;</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//全参数比对，所以两个参数MD5不同</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">dedupMD5</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ReqDedupHelper</span><span class="o">().</span><span class="na">dedupParamMD5</span><span class="o">(</span><span class="n">req</span><span class="o">);</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">dedupMD52</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ReqDedupHelper</span><span class="o">().</span><span class="na">dedupParamMD5</span><span class="o">(</span><span class="n">req2</span><span class="o">);</span>
</span><span class='line'>    <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">&quot;req1MD5 = &quot;</span><span class="o">+</span> <span class="n">dedupMD5</span><span class="o">+</span><span class="s">&quot; , req2MD5=&quot;</span><span class="o">+</span><span class="n">dedupMD52</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//去除时间参数比对，MD5相同</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">dedupMD53</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ReqDedupHelper</span><span class="o">().</span><span class="na">dedupParamMD5</span><span class="o">(</span><span class="n">req</span><span class="o">,</span><span class="s">&quot;requestTime&quot;</span><span class="o">);</span>
</span><span class='line'>    <span class="n">String</span> <span class="n">dedupMD54</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ReqDedupHelper</span><span class="o">().</span><span class="na">dedupParamMD5</span><span class="o">(</span><span class="n">req2</span><span class="o">,</span><span class="s">&quot;requestTime&quot;</span><span class="o">);</span>
</span><span class='line'>    <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">&quot;req1MD5 = &quot;</span><span class="o">+</span> <span class="n">dedupMD53</span><span class="o">+</span><span class="s">&quot; , req2MD5=&quot;</span><span class="o">+</span><span class="n">dedupMD54</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>日志输出：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">req1MD5</span> <span class="o">=</span> <span class="mi">9</span><span class="n">E054D36439EBDD0604C5E65EB5C8267</span> <span class="o">,</span> <span class="n">req2MD5</span><span class="o">=</span><span class="n">A2D20BAC78551C4CA09BEF97FE468A3F</span>
</span><span class='line'><span class="n">req1MD5</span> <span class="o">=</span> <span class="n">C2A36FED15128E9E878583CAAAFEFDE9</span> <span class="o">,</span> <span class="n">req2MD5</span><span class="o">=</span><span class="n">C2A36FED15128E9E878583CAAAFEFDE9</span>
</span></code></pre></td></tr></table></div></figure>


<p>日志说明：</p>

<ul>
<li>一开始两个参数由于requestTime是不同的，所以求去重参数摘要的时候可以发现两个值是不一样的</li>
<li>第二次调用的时候，去除了requestTime再求摘要（第二个参数中传入了&#8221;requestTime&#8221;），则发现两个摘要是一样的，符合预期。</li>
</ul>


<h2>总结</h2>

<p>至此，我们可以得到完整的去重解决方案，如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">userId</span><span class="o">=</span> <span class="s">&quot;12345678&quot;</span><span class="o">;</span><span class="c1">//用户</span>
</span><span class='line'><span class="n">String</span> <span class="n">method</span> <span class="o">=</span> <span class="s">&quot;pay&quot;</span><span class="o">;</span><span class="c1">//接口名</span>
</span><span class='line'><span class="n">String</span> <span class="n">dedupMD5</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ReqDedupHelper</span><span class="o">().</span><span class="na">dedupParamMD5</span><span class="o">(</span><span class="n">req</span><span class="o">,</span><span class="s">&quot;requestTime&quot;</span><span class="o">);</span><span class="c1">//计算请求参数摘要，其中剔除里面请求时间的干扰</span>
</span><span class='line'><span class="n">String</span> <span class="n">KEY</span> <span class="o">=</span> <span class="s">&quot;dedup:U=&quot;</span> <span class="o">+</span> <span class="n">userId</span> <span class="o">+</span> <span class="s">&quot;M=&quot;</span> <span class="o">+</span> <span class="n">method</span> <span class="o">+</span> <span class="s">&quot;P=&quot;</span> <span class="o">+</span> <span class="n">dedupMD5</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="kt">long</span> <span class="n">expireTime</span> <span class="o">=</span>  <span class="mi">1000</span><span class="o">;</span><span class="c1">// 1000毫秒过期，1000ms内的重复请求会认为重复</span>
</span><span class='line'><span class="kt">long</span> <span class="n">expireAt</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">()</span> <span class="o">+</span> <span class="n">expireTime</span><span class="o">;</span>
</span><span class='line'><span class="n">String</span> <span class="n">val</span> <span class="o">=</span> <span class="s">&quot;expireAt@&quot;</span> <span class="o">+</span> <span class="n">expireAt</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// NOTE:直接SETNX不支持带过期时间，所以设置+过期不是原子操作，极端情况下可能设置了就不过期了，后面相同请求可能会误以为需要去重，所以这里使用底层API，保证SETNX+过期时间是原子操作</span>
</span><span class='line'><span class="n">Boolean</span> <span class="n">firstSet</span> <span class="o">=</span> <span class="n">stringRedisTemplate</span><span class="o">.</span><span class="na">execute</span><span class="o">((</span><span class="n">RedisCallback</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;)</span> <span class="n">connection</span> <span class="o">-&gt;</span> <span class="n">connection</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">KEY</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">val</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> <span class="n">Expiration</span><span class="o">.</span><span class="na">milliseconds</span><span class="o">(</span><span class="n">expireTime</span><span class="o">),</span>
</span><span class='line'>        <span class="n">RedisStringCommands</span><span class="o">.</span><span class="na">SetOption</span><span class="o">.</span><span class="na">SET_IF_ABSENT</span><span class="o">));</span>
</span><span class='line'>
</span><span class='line'><span class="kd">final</span> <span class="kt">boolean</span> <span class="n">isConsiderDup</span><span class="o">;</span>
</span><span class='line'><span class="k">if</span> <span class="o">(</span><span class="n">firstSet</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="n">firstSet</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="n">isConsiderDup</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class='line'>    <span class="n">isConsiderDup</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Dubbo Provider中获取调用者的应用名与IP]]></title>
    <link href="https://Jaskey.github.io/blog/2020/05/18/dubbo-filter-trace-consumer/"/>
    <updated>2020-05-18T19:25:21+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/05/18/dubbo-filter-trace-consumer</id>
    <content type="html"><![CDATA[<p>在Dubbo做微服务的架构后，对于应用请求的追踪是尤为重要的。试想一下你有一个服务在告警，但你却不知道你的请求是从哪个服务/ip上过来的，这对于问题的定位会造成极大的困难。这对于一个上游调用方多、实例多的系统来说，问题尤为明显。</p>

<p>本文仅讨论如何简单地用日志的形式做到追踪调用方的的应用名与IP，详细的调用链追踪是一个系统的话题，不在本文讨论。</p>

<p>要无缝的获取调用方的相关信息，我们可以借助Dubbo的Filter。通过在Provider端增加一个Filter做一个打印。但具体怎么获取呢？</p>

<h2>IP</h2>

<p>IP的获取比较简单，我们可以在Provier端直接使用如下代码获取：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">clientIp</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getRemoteHost</span><span class="o">();</span><span class="c1">//这次请求来自哪个ip</span>
</span></code></pre></td></tr></table></div></figure>


<h2>应用名</h2>

<p>应用名则没那么容易，或许你看到过url中是有一个application的参数的，那我们是否可以使用以下代码来获取呢？</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">applicationFromContextUrl</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getUrl</span><span class="o">().</span><span class="na">getParameter</span><span class="o">(</span><span class="s">&quot;application&quot;</span><span class="o">);</span><span class="c1">//得到的是本应用的名字</span>
</span><span class='line'><span class="n">String</span> <span class="n">applicationFromInvokerURL</span> <span class="o">=</span> <span class="n">invoker</span><span class="o">.</span><span class="na">getUrl</span><span class="o">().</span><span class="na">getParameter</span><span class="o">(</span><span class="n">Constants</span><span class="o">.</span><span class="na">APPLICATION_KEY</span><span class="o">);</span><span class="c1">//得到的也是本应用的名字</span>
</span><span class='line'><span class="n">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;applicationFromUrl = {}, applicationFromInvokerURL= {}&quot;</span><span class="o">,</span> <span class="n">applicationFromContextUrl</span><span class="o">,</span> <span class="n">applicationFromInvokerURL</span><span class="o">);</span>
</span></code></pre></td></tr></table></div></figure>


<p>答案是否定的，事实上，无论是Provider还是Consumer，当你使用这段代码获取的时候，拿到的都是本应用。</p>

<p>所以需要获取调用方的应用名，我们需要显示的设置进来，这里就需要增加一个Consumer的Filter，获取consumer的应用名放入attachment中带到Provider，Provider在filter中从attachment中获取即可。</p>

<p>Consumer Filter中传入应用名至attachment中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="c1">//手动设置consumer的应用名进attachment</span>
</span><span class='line'><span class="n">String</span> <span class="n">application</span> <span class="o">=</span> <span class="n">invoker</span><span class="o">.</span><span class="na">getUrl</span><span class="o">().</span><span class="na">getParameter</span><span class="o">(</span><span class="n">Constants</span><span class="o">.</span><span class="na">APPLICATION_KEY</span><span class="o">);</span>
</span><span class='line'><span class="k">if</span> <span class="o">(</span><span class="n">application</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>      <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">setAttachment</span><span class="o">(</span><span class="s">&quot;dubboApplication&quot;</span><span class="o">,</span> <span class="n">application</span><span class="o">);</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Provider Filter中从其中获取调用方的应用名：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">String</span> <span class="n">application</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getAttachment</span><span class="o">(</span><span class="s">&quot;dubboApplication&quot;</span><span class="o">);</span>
</span></code></pre></td></tr></table></div></figure>


<h2>一对Trace Filter示意</h2>

<p>以下是一对消费者和生产者的Filter示意，实现了以下功能：</p>

<ol>
<li><p>Provider端记录了打印了调用方的IP和应用名</p></li>
<li><p>Consumer端打印了服务提供方的IP即本次调用的耗时</p></li>
</ol>


<p>Consumer Filter：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="nd">@Activate</span><span class="o">(</span><span class="n">group</span> <span class="o">=</span> <span class="n">Constants</span><span class="o">.</span><span class="na">CONSUMER</span><span class="o">)</span>
</span><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">LogTraceConsumerFilter</span> <span class="kd">implements</span> <span class="n">Filter</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Logger</span> <span class="n">LOG</span> <span class="o">=</span> <span class="n">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="n">LogTraceConsumerFilter</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">public</span> <span class="n">Result</span> <span class="nf">invoke</span><span class="o">(</span><span class="n">Invoker</span><span class="o">&lt;?&gt;</span> <span class="n">invoker</span><span class="o">,</span> <span class="n">Invocation</span> <span class="n">invocation</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">RpcException</span> <span class="o">{</span>
</span><span class='line'>        <span class="c1">//手动设置consumer的应用名进attachment</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">application</span> <span class="o">=</span> <span class="n">invoker</span><span class="o">.</span><span class="na">getUrl</span><span class="o">().</span><span class="na">getParameter</span><span class="o">(</span><span class="n">Constants</span><span class="o">.</span><span class="na">APPLICATION_KEY</span><span class="o">);</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(</span><span class="n">application</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">setAttachment</span><span class="o">(</span><span class="s">&quot;dubboApplication&quot;</span><span class="o">,</span> <span class="n">application</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">Result</span> <span class="n">result</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">serverIp</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
</span><span class='line'>        <span class="kt">long</span> <span class="n">startTime</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">();</span>
</span><span class='line'>        <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">result</span> <span class="o">=</span> <span class="n">invoker</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="n">invocation</span><span class="o">);</span>
</span><span class='line'>            <span class="n">serverIp</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getRemoteHost</span><span class="o">();</span><span class="c1">//这次返回结果是哪个ip</span>
</span><span class='line'>            <span class="k">return</span> <span class="n">result</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">Throwable</span> <span class="n">throwable</span> <span class="o">=</span> <span class="o">(</span><span class="n">result</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">?</span> <span class="kc">null</span> <span class="o">:</span> <span class="n">result</span><span class="o">.</span><span class="na">getException</span><span class="o">();</span>
</span><span class='line'>            <span class="n">Object</span> <span class="n">resultObj</span> <span class="o">=</span> <span class="o">(</span><span class="n">result</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">?</span> <span class="kc">null</span> <span class="o">:</span> <span class="n">result</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span>
</span><span class='line'>            <span class="kt">long</span> <span class="n">costTime</span> <span class="o">=</span> <span class="n">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">()</span> <span class="o">-</span> <span class="n">startTime</span><span class="o">;</span>
</span><span class='line'>            <span class="n">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;[TRACE] Call {}, {}.{}() param:{}, return:{}, exception:{}, cost:{} ms!&quot;</span><span class="o">,</span> <span class="n">serverIp</span><span class="o">,</span> <span class="n">invoker</span><span class="o">.</span><span class="na">getInterface</span><span class="o">(),</span> <span class="n">invocation</span><span class="o">.</span><span class="na">getMethodName</span><span class="o">(),</span> <span class="n">invocation</span><span class="o">.</span><span class="na">getArguments</span><span class="o">(),</span> <span class="n">resultObj</span><span class="o">,</span> <span class="n">throwable</span><span class="o">,</span> <span class="n">costTime</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Provider Filter：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="nd">@Activate</span><span class="o">(</span><span class="n">group</span> <span class="o">=</span> <span class="n">Constants</span><span class="o">.</span><span class="na">PROVIDER</span><span class="o">)</span>
</span><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">LogTraceProviderFilter</span> <span class="kd">implements</span> <span class="n">Filter</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Logger</span> <span class="n">LOG</span> <span class="o">=</span> <span class="n">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="n">LogTraceProviderFilter</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="nd">@Override</span>
</span><span class='line'>    <span class="kd">public</span> <span class="n">Result</span> <span class="nf">invoke</span><span class="o">(</span><span class="n">Invoker</span><span class="o">&lt;?&gt;</span> <span class="n">invoker</span><span class="o">,</span> <span class="n">Invocation</span> <span class="n">invocation</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">RpcException</span> <span class="o">{</span>
</span><span class='line'>        <span class="c1">//上游如果手动设置了consumer的应用名进attachment，则取出来打印</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">clientIp</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getRemoteHost</span><span class="o">();</span><span class="c1">//这次请求来自哪个ip</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">application</span> <span class="o">=</span> <span class="n">RpcContext</span><span class="o">.</span><span class="na">getContext</span><span class="o">().</span><span class="na">getAttachment</span><span class="o">(</span><span class="s">&quot;dubboApplication&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="n">String</span> <span class="n">from</span> <span class="o">=</span> <span class="n">clientIp</span><span class="o">;</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(!</span><span class="n">StringUtils</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">(</span><span class="n">application</span><span class="o">))</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">from</span> <span class="o">=</span> <span class="n">application</span><span class="o">+</span><span class="s">&quot;(&quot;</span><span class="o">+</span><span class="n">clientIp</span><span class="o">+</span><span class="s">&quot;)&quot;</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">LOG</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">&quot;[Trace]From {}, {}.{}() param:{}&quot;</span><span class="o">,</span> <span class="n">from</span><span class="o">,</span> <span class="n">invoker</span><span class="o">.</span><span class="na">getInterface</span><span class="o">(),</span> <span class="n">invocation</span><span class="o">.</span><span class="na">getMethodName</span><span class="o">(),</span> <span class="n">invocation</span><span class="o">.</span><span class="na">getArguments</span><span class="o">());</span>
</span><span class='line'>        <span class="k">return</span> <span class="n">invoker</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="n">invocation</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h2>Filter 文件中配置启用（注：替换对应的包名）：</h2>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">logTraceProviderFilter</span><span class="o">=</span><span class="n">xxxx</span><span class="o">.</span><span class="na">LogTraceProviderFilter</span>
</span><span class='line'><span class="n">logTraceConsumerFilter</span><span class="o">=</span><span class="n">xxxx</span><span class="o">.</span><span class="na">LogTraceConsumerFilter</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[利用Java进程名进行jstat -gc]]></title>
    <link href="https://Jaskey.github.io/blog/2020/05/13/stat-gc-with-process-name/"/>
    <updated>2020-05-13T16:29:07+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/05/13/stat-gc-with-process-name</id>
    <content type="html"><![CDATA[<p>需要实时观看GC的情况，我们可以类似如下命令进行监控</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'> jstat -gc <span class="nv">$pid</span> <span class="m">100</span> <span class="m">10</span>
</span></code></pre></td></tr></table></div></figure>


<p>但是这里需要一个进程号，很麻烦，每个Java进程在不同机器或者启动不一样就会不一样，对于自动监控脚本或者是如果需要定位应用刚开始启动时候gc的问题时，当你手动敲完命令拿到pid的时候，可能都凉了。</p>

<p>对此写了一个简单的shell脚本可以传入进程名去执行jstat</p>

<p>gcstat.sh:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="c">#! /bin/bash</span>
</span><span class='line'>
</span><span class='line'><span class="nv">process</span><span class="o">=</span><span class="nv">$1</span>
</span><span class='line'><span class="nv">interval</span><span class="o">=</span><span class="nv">$2</span>
</span><span class='line'><span class="nv">count</span><span class="o">=</span><span class="nv">$3</span>
</span><span class='line'><span class="nv">pid</span><span class="o">=</span><span class="k">$(</span>ps -ef <span class="p">|</span> grep java <span class="p">|</span> grep <span class="nv">$process</span> <span class="p">|</span> grep -v grep <span class="p">|</span> awk <span class="s1">&#39;{print $2}&#39;</span><span class="k">)</span>
</span><span class='line'><span class="nb">echo</span> <span class="nv">$pid</span>
</span><span class='line'><span class="nb">echo</span> <span class="nv">$interval</span>
</span><span class='line'><span class="nb">echo</span> <span class="nv">$count</span>
</span></code></pre></td></tr></table></div></figure>


<p>使用：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>./gcstat.sh  processName <span class="m">1000</span> 5
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[自定义ShardingSphere的加解密器]]></title>
    <link href="https://Jaskey.github.io/blog/2020/04/29/user-defined-shardingsphere-encryptor/"/>
    <updated>2020-04-29T20:44:03+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/04/29/user-defined-shardingsphere-encryptor</id>
    <content type="html"><![CDATA[<p>默认的Sharding Sphere 支持AES和MD5两种加密器。有些时候可能需要自定义使用自己的加解密算法，如AES的具体实现不一样等。网上公开的并没有直接的指引，通过部分源码的阅读，找到了可行的方式。需要三步：</p>

<h2>1.实现自定义解密器 （实现ShardingEncryptor 接口）</h2>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TestShardingEncryptor</span> <span class="kd">implements</span> <span class="n">ShardingEncryptor</span> <span class="o">{</span>
</span><span class='line'>        <span class="kd">private</span> <span class="n">Properties</span> <span class="n">properties</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>         <span class="nd">@Override</span>
</span><span class='line'>         <span class="kd">public</span> <span class="n">String</span> <span class="nf">getType</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>                <span class="k">return</span> <span class="s">&quot;TEST&quot;</span><span class="o">;</span>
</span><span class='line'>          <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>          <span class="nd">@Override</span>
</span><span class='line'>         <span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>
</span><span class='line'>         <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>         <span class="nd">@Override</span>
</span><span class='line'>         <span class="kd">public</span> <span class="n">String</span> <span class="nf">encrypt</span><span class="o">(</span><span class="kd">final</span> <span class="n">Object</span> <span class="n">plaintext</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>             <span class="k">return</span> <span class="s">&quot;TEST-&quot;</span><span class="o">+</span><span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">plaintext</span><span class="o">);</span>
</span><span class='line'>         <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>         <span class="nd">@Override</span>
</span><span class='line'>        <span class="kd">public</span> <span class="n">Object</span> <span class="nf">decrypt</span><span class="o">(</span><span class="kd">final</span> <span class="n">String</span> <span class="n">ciphertext</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>             <span class="k">return</span> <span class="n">ciphertext</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s">&quot;TEST-&quot;</span><span class="o">,</span><span class="s">&quot;&quot;</span><span class="o">);</span>
</span><span class='line'>         <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>其中<code>getType</code>返回的字符串（本例为&#8221;TEST&#8221;）即为本加解密器的类型（后续使用的时候会使用）</p>

<h2>2.创建org.apache.shardingsphere.spi.encrypt.ShardingEncryptor 文件</h2>

<p>需要创建一个文件名为<code>org.apache.shardingsphere.spi.encrypt.ShardingEncryptor</code>放入resources路径下的<code>\META-INF\services</code></p>

<p><img src="http://jaskey.github.io/images/shardingsphere/sharding-encryptor-file-path.png" title="sharding-encryptor-file-path" alt="sharding-encryptor-file-path" /></p>

<p>文件的内容就是类名全称，如：</p>

<p>com.yourcompany.TestShardingEncryptor</p>

<h2>3.配置使用此自定义类</h2>

<h4>Java配置模式：</h4>

<p>如果未使用Spring Boot，需要显示用代码配置</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">EncryptorRuleConfiguration</span> <span class="n">encryptorConfig</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">EncryptorRuleConfiguration</span><span class="o">(</span><span class="s">&quot;TEST&quot;</span><span class="o">,</span> <span class="n">props</span><span class="o">);</span>
</span></code></pre></td></tr></table></div></figure>


<h4>Spring Boot配置模式：</h4>

<p>如果使用的是Spring Boot配置模式，则需要如下配置</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">spring</span><span class="o">.</span><span class="na">shardingsphere</span><span class="o">.</span><span class="na">encrypt</span><span class="o">.</span><span class="na">encryptors</span><span class="o">.</span><span class="na">my_encryptor</span><span class="o">.</span><span class="na">type</span><span class="o">=</span><span class="n">TEST</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Java GC垃圾收集器这点小事]]></title>
    <link href="https://Jaskey.github.io/blog/2020/04/27/gc-basics/"/>
    <updated>2020-04-27T15:08:05+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/04/27/gc-basics</id>
    <content type="html"><![CDATA[<p>​    对于大多数的应用来说，其实默认的JVM设置就够用了，但当你意识到有GC引起的性能问题、并且仅仅加大堆内存空间也解决不了的时候，那你就应该考虑下GC的调优了。但对于大多数程序员来说，这是很麻烦的，因为调优需要很多耐心，并且需要知道垃圾回收器的运作原理及背后对应用的影响。本文是high-level层面的关于Java垃圾回收器的总览，并以例子的形式去讲解性能定位的一些问题。</p>

<p>​    正文开始。</p>

<p>​    Java提供了很多种垃圾回收器，会在gc运行的线程中搭配着不同的算法。不同的回收器工作原理不一样，优缺点也不同。最重要的是无论哪一种回收器都会&#8221;stop the world&#8221;。就是说，在收集和回收的过程中，你的应用程序（或多或少）都会处于暂停状态，只不过不同算法的stop the world的表现有所不同。有些算法一直都会闲置不工作直到必须要垃圾收集才开始工作，然后就暂停应用程序很长的时间；而有一些则能和应用程序同步的进行所以在“stop the world”阶段就会有更少的暂停。选择最合适的算法要取决于你的目标：你是想优化整体的吞吐量即便不时地就会长时间暂停也是可以接受的，还是说你是想得到低延迟的优化通过分散各个时间以得到每时每刻都低延迟。</p>

<p>​    为了增强垃圾回收的过程，Java（准确的说是 HotSpot JVM）把堆分成了两个代，年轻代和年老代（还有一个叫永久代的区域不在我们本文讨论范围）</p>

<p><img src="http://jaskey.github.io/images/gc/hotspot-heap.png" title="hotspot-heap" alt="hotspot-heap" /></p>

<p>​    年轻代是一些“年轻”的对象存放的地方，而年轻代还会继续分为以下三个区域：</p>

<ol>
<li>伊甸区（Eden Space）</li>
<li>幸存区1（Survivor Space 1）</li>
<li>幸存区2（Survivor Space 2）</li>
</ol>


<p>​    默认情况下，伊甸区是大于两个幸存者区的总和的。例如在我的Mac OS X上的64位HotSpot JVM上，伊甸区占了大概年轻代76%的区域。所有的对象一开始都会在伊甸区创建，当伊甸区满了之后，就会触发一次次要的垃圾回收（minor gc），期间新对象会快速地被检查是否可以进行垃圾回收。一旦发现那些对象已经死亡（dead），也就是说没有再被其他对象引用了（这里先简单忽略掉引用的具体类型带来的一些差异，不在本文讨论），就会被标记为死亡然后被垃圾回收掉。而其中“幸存”的对象就会被移到其中的一个空的Survivor Space。你可能会问，具体移动到哪一个幸存区呢？要回答这个问题，首先我们先聊一下幸存区的设计。</p>

<p>​    之所以设计两个幸存区，是为了避免内存碎片。假设只有一个幸存区（然后我们把幸存区想象成一个内存中连续的数组），当年轻代的gc在这个数组上运行了一遍后，会标记一些死亡对象然后删除掉，这样的话势必会在内存中留下一些空洞的区域（原来的对象存活的位置），那么就有必要做压缩了。所以为了避免做压缩，HotSpot JVM就从一个幸存者区复制所有幸存的对象到另外一个（空的）幸存者区里，这样就没有空洞了。这里我们讲到了压缩，顺便提一下年老代的垃圾回收器（除了CMS）在进行年老代垃圾回收的时候都会进行压缩以避免内存碎片。</p>

<p>​    简单地说，次要的垃圾回收（当伊甸区满的时候）就会把存活的对象从伊甸区和其中一个幸存区（gc日志中以“from”呈现）左右捣腾地搬到另外一个幸存区（又叫“to”）。这样会一直的持续下去直到以下的条件发生：</p>

<ol>
<li>对象达到了最大的晋升时间阈值（<em>maximum tenuring threshold</em>），就是说在年轻代被左右捣腾得足够久了，媳妇熬成婆。</li>
<li>幸存区已经没有空间去接受新生的对象了（后面会讲到）</li>
</ol>


<p>​    以上条件发生后，对象就会被移动到年老代了。下面用一个具体的例子来理解下。假设我们有以下的应用程序，它会在初始化的时候创建一些长期存活的对象，也会在运行的过程中不断的创建很多存活时间很短的对象（例如我们的web服务器程序在处理请求的时候会不断分配存活时间很短的对象）</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">createFewLongLivedAndManyShortLivedObjects</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">HashSet</span><span class="o">&lt;</span><span class="n">Double</span><span class="o">&gt;</span> <span class="n">set</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashSet</span><span class="o">&lt;</span><span class="n">Double</span><span class="o">&gt;();</span>
</span><span class='line'>
</span><span class='line'>        <span class="kt">long</span> <span class="n">l</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
</span><span class='line'>        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">100</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">Double</span> <span class="n">longLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Double</span><span class="o">(</span><span class="n">l</span><span class="o">++);</span>
</span><span class='line'>            <span class="n">set</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">longLivedDouble</span><span class="o">);</span>  <span class="c1">// 加到集合里，让这些对象能持续的存活</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">while</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// 不断地创建一些存活时间短的对象（这里在实际代码中比较极端，仅为演示用）</span>
</span><span class='line'>            <span class="n">Double</span> <span class="n">shortLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Double</span><span class="o">(</span><span class="n">l</span><span class="o">++);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p> 在运行这个程序的过程中我们启用GC的部分日志参数：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="o">-</span><span class="n">Xmx100m</span>                     <span class="c1">// 分配100MV的堆内存</span>
</span><span class='line'><span class="o">-</span><span class="nl">XX:</span><span class="o">-</span><span class="n">PrintGC</span>                 <span class="c1">// 开启GC日志打印</span>
</span><span class='line'><span class="o">-</span><span class="nl">XX:</span><span class="o">+</span><span class="n">PrintHeapAtGC</span>           <span class="c1">// 开启GC日志打印堆信息</span>
</span><span class='line'><span class="o">-</span><span class="nl">XX:</span><span class="n">MaxTenuringThreshold</span><span class="o">=</span><span class="mi">15</span>  <span class="c1">// 为了让对象能在年轻代呆久一点</span>
</span><span class='line'><span class="o">-</span><span class="nl">XX:</span><span class="o">+</span><span class="n">UseConcMarkSweepGC</span>      <span class="c1">// 暂时先忽略这个配置，后面会讲到</span>
</span><span class='line'><span class="o">-</span><span class="nl">XX:</span><span class="o">+</span><span class="n">UseParNewGC</span>             <span class="c1">// 暂时先忽略这个配置，后面会讲到</span>
</span></code></pre></td></tr></table></div></figure>


<p>gc 日志会显示垃圾收集前后的情况如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'>Heap <span class="nt">&lt;b&gt;</span>before<span class="nt">&lt;/b&gt;</span> GC invocations=5 (full 0):
</span><span class='line'> par new (<span class="nt">&lt;u&gt;</span>young<span class="nt">&lt;/u&gt;</span>) generation total 30720K, used 28680K
</span><span class='line'>  eden space 27328K,   <span class="nt">&lt;b&gt;</span>100%<span class="nt">&lt;/b&gt;</span> used
</span><span class='line'>  from space 3392K,   <span class="nt">&lt;b&gt;</span>39%<span class="nt">&lt;/b&gt;</span> used
</span><span class='line'>  to   space 3392K,   0% used
</span><span class='line'> concurrent mark-sweep (<span class="nt">&lt;u&gt;</span>old<span class="nt">&lt;/u&gt;</span>) generation total 68288K, used <span class="nt">&lt;b&gt;</span>0K<span class="nt">&lt;/b&gt;</span> <span class="nt">&lt;br/&gt;</span>
</span><span class='line'>Heap <span class="nt">&lt;b&gt;</span>after<span class="nt">&lt;/b&gt;</span> GC invocations=6 (full 0):
</span><span class='line'> par new generation (<span class="nt">&lt;u&gt;</span>young<span class="nt">&lt;/u&gt;</span>) total 30720K, used 1751K
</span><span class='line'>  eden space 27328K,   <span class="nt">&lt;b&gt;</span>0%<span class="nt">&lt;/b&gt;</span> used
</span><span class='line'>  from space 3392K,   <span class="nt">&lt;b&gt;</span>51%<span class="nt">&lt;/b&gt;</span> used
</span><span class='line'>  to   space 3392K,   0% used
</span><span class='line'> concurrent mark-sweep (<span class="nt">&lt;u&gt;</span>old<span class="nt">&lt;/u&gt;</span>) generation total 68288K, used <span class="nt">&lt;b&gt;</span>0K<span class="nt">&lt;/b&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>​    从这个日志里我们能得到以下信息。第一，在这次gc之前，已经发生了5次的minor gc了（所以这次是第6次）。第二，伊甸区占用了100%所以触发了这次的gc。第三，其中一个幸存区域已经使用了39%的空间（还有不少可用空间）。而这次垃圾收集结束后，我们能看到伊甸区就被清空了（0%）然后幸存者区域上升到51%。这意味着伊甸区和其中一个幸存区里存活的对象已经被移动到另外一个幸存区了，然后死亡的对象已经被垃圾回收了。怎么推断的死亡对象被回收了呢？我们看到伊甸区原来是比幸存区要大的（27328K vs 3392K），而后面幸存区的空间大小仅仅是轻微的上升（伊甸区被清空了），所以大量的对象肯定是被垃圾回收了。而我们再看看年老代，年老代是一直都是空的，无论是这次垃圾回收前还是后（回想一下，我们设置了晋升阈值为15）。</p>

<p>​    下面我们再试另外一个实验。这次用多线程不断的创建存活时间很短的对象。直觉上判断，依旧应该没有对象会上升到年老代才对，因为minor gc就应该可以把这些对象清理干净。我们来看看实际情况如何</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">createManyShortLivedObjects</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>        <span class="kd">final</span> <span class="kt">int</span> <span class="n">NUMBER_OF_THREADS</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span>
</span><span class='line'>        <span class="kd">final</span> <span class="kt">int</span> <span class="n">NUMBER_OF_OBJECTS_EACH_TIME</span> <span class="o">=</span> <span class="mi">1000000</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span><span class="o">&lt;</span><span class="n">NUMBER_OF_THREADS</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
</span><span class='line'>            <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span>
</span><span class='line'>                    <span class="k">while</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>                        <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span><span class="o">&lt;</span><span class="n">NUMBER_OF_OBJECTS_EACH_TIME</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
</span><span class='line'>                            <span class="n">Double</span> <span class="n">shortLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Double</span><span class="o">(</span><span class="mf">1.0d</span><span class="o">);</span>
</span><span class='line'>                        <span class="o">}</span>
</span><span class='line'>                        <span class="n">sleepMillis</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
</span><span class='line'>                    <span class="o">}</span>
</span><span class='line'>                <span class="o">}</span>
</span><span class='line'>            <span class="o">}).</span><span class="na">start</span><span class="o">();</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>这次，我们只给10MB的内存，然后看看GC日志</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">Heap</span> <span class="o">&lt;</span><span class="n">b</span><span class="o">&gt;</span><span class="n">before</span><span class="o">&lt;/</span><span class="n">b</span><span class="o">&gt;</span> <span class="n">GC</span> <span class="n">invocations</span><span class="o">=</span><span class="mi">0</span> <span class="o">(</span><span class="n">full</span> <span class="mi">0</span><span class="o">):</span>
</span><span class='line'> <span class="n">par</span> <span class="nf">new</span> <span class="o">(&lt;</span><span class="n">u</span><span class="o">&gt;</span><span class="n">young</span><span class="o">&lt;/</span><span class="n">u</span><span class="o">&gt;)</span> <span class="n">generation</span> <span class="n">total</span> <span class="mi">3072</span><span class="n">K</span><span class="o">,</span> <span class="n">used</span> <span class="mi">2751</span><span class="n">K</span>
</span><span class='line'>  <span class="n">eden</span> <span class="n">space</span> <span class="mi">2752</span><span class="n">K</span><span class="o">,</span>  <span class="mi">99</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'>  <span class="n">from</span> <span class="n">space</span> <span class="mi">320</span><span class="n">K</span><span class="o">,</span>   <span class="mi">0</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'>  <span class="n">to</span>   <span class="n">space</span> <span class="mi">320</span><span class="n">K</span><span class="o">,</span>   <span class="mi">0</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'> <span class="n">concurrent</span> <span class="n">mark</span><span class="o">-</span><span class="n">sweep</span> <span class="o">(&lt;</span><span class="n">u</span><span class="o">&gt;</span><span class="n">old</span><span class="o">&lt;/</span><span class="n">u</span><span class="o">&gt;)</span> <span class="n">generation</span> <span class="n">total</span> <span class="mi">6848</span><span class="n">K</span><span class="o">,</span> <span class="n">used</span> <span class="o">&lt;</span><span class="n">b</span><span class="o">&gt;</span><span class="mi">0</span><span class="n">K</span><span class="o">&lt;/</span><span class="n">b</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">br</span><span class="o">/&gt;</span>
</span><span class='line'><span class="n">Heap</span> <span class="o">&lt;</span><span class="n">b</span><span class="o">&gt;</span><span class="n">after</span><span class="o">&lt;/</span><span class="n">b</span><span class="o">&gt;</span> <span class="n">GC</span> <span class="n">invocations</span><span class="o">=</span><span class="mi">1</span> <span class="o">(</span><span class="n">full</span> <span class="mi">0</span><span class="o">):</span>
</span><span class='line'> <span class="n">par</span> <span class="k">new</span> <span class="nf">generation</span>  <span class="o">(&lt;</span><span class="n">u</span><span class="o">&gt;</span><span class="n">young</span><span class="o">&lt;/</span><span class="n">u</span><span class="o">&gt;)</span>  <span class="n">total</span> <span class="mi">3072</span><span class="n">K</span><span class="o">,</span> <span class="n">used</span> <span class="mi">318</span><span class="n">K</span>
</span><span class='line'>  <span class="n">eden</span> <span class="n">space</span> <span class="mi">2752</span><span class="n">K</span><span class="o">,</span>   <span class="mi">0</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'>  <span class="n">from</span> <span class="n">space</span> <span class="mi">320</span><span class="n">K</span><span class="o">,</span>  <span class="mi">99</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'>  <span class="n">to</span>   <span class="n">space</span> <span class="mi">320</span><span class="n">K</span><span class="o">,</span>   <span class="mi">0</span><span class="o">%</span> <span class="n">used</span>
</span><span class='line'> <span class="n">concurrent</span> <span class="n">mark</span><span class="o">-</span><span class="n">sweep</span> <span class="o">(&lt;</span><span class="n">u</span><span class="o">&gt;</span><span class="n">old</span><span class="o">&lt;/</span><span class="n">u</span><span class="o">&gt;)</span> <span class="n">generation</span> <span class="n">total</span> <span class="mi">6848</span><span class="n">K</span><span class="o">,</span> <span class="n">used</span> <span class="o">&lt;</span><span class="n">b</span><span class="o">&gt;</span><span class="mi">76</span><span class="n">K</span><span class="o">&lt;/</span><span class="n">b</span><span class="o">&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>​    从日志上看，并不如我们一开始想的那样。这次，老年代在第一次minor gc之后，接受了一些对象。实际上这些对象都是存活时间很短的对象，并且我们设置了晋升阈值是15次，再而且日志里显示的gc只是第一次垃圾收集。这个现象背后实际上是这样的：应用程序创建了大量的对象在伊甸区，minor gc启动的时候尝试去回收，但是大多数的这些存活时间很短的对象实际上都是active的（被一个运行中的线程引用着）。那么年轻代的垃圾收集器就只好把这些对象移动到年老代了。这其实是一个不好的现象，因为这些被移到到年老代的对象其实是过早衰老了（prematurely aged），它们只有在老年代的major gc才能被回收，而major gc通常会耗时更长。对于某些垃圾算法例如CMS，major gc会在年老代70%内存占据后出发。这个值可以通过参数修改<code>-XX:CMSInitiatingOccupancyFraction=70</code></p>

<p>​    怎么样防止这些短暂存活的对象过早衰老呢？有几个方法，其中一个理论上可行的方法是估计这些活跃的短暂存活对象的数量，然后设置合理的年轻代大小。我们下面来试试：</p>

<ul>
<li>年轻代默认是整个堆大小的1/3，这次我们通过 <code>-XX:NewRatio=1</code> 来修改其大小让他内存更大些（大约3.4MB，原来是3MB）</li>
<li>同时调整幸存者区的大小：<code>-XX:SurvivorRatio=1</code> （大约1.6MB一个区，原来是0.3MB）</li>
</ul>


<p>问题就解决了。经过8次的minor gc，年老代依旧是空的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'>Heap <span class="nt">&lt;b&gt;</span>before<span class="nt">&lt;/b&gt;</span> GC invocations=7 (full 0):
</span><span class='line'> par new generation   total 3456K, used 2352K
</span><span class='line'>  eden space 1792K,  99% used
</span><span class='line'>  from space 1664K,  33% used
</span><span class='line'>  to   space 1664K,   0% used
</span><span class='line'> concurrent mark-sweep generation total 5120K, used <span class="nt">&lt;b&gt;</span>0K<span class="nt">&lt;/b&gt;</span> <span class="nt">&lt;br/&gt;</span>
</span><span class='line'>Heap <span class="nt">&lt;b&gt;</span>after<span class="nt">&lt;/b&gt;</span> GC invocations=8 (full 0):
</span><span class='line'> par new generation   total 3456K, used 560K
</span><span class='line'>  eden space 1792K,   0% used
</span><span class='line'>  from space 1664K,  33% used
</span><span class='line'>  to   space 1664K,   0% used [
</span><span class='line'> concurrent mark-sweep generation total 5120K, used <span class="nt">&lt;b&gt;</span>0K<span class="nt">&lt;/b&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>​    对于GC调优，没有银弹。这里只是简单地示意。对于实际的应用，需要不断的修改配置试错来找到最佳配置。例如，这次其实我们也可以将堆的总大小调大一倍来解决此问题。</p>

<h2>垃圾回收算法</h2>

<p>​    接下来我们来看看具体的垃圾回收算法。Hotspot JVM针对年轻代和年老代有多个不同的算法。从宏观层面上看，有三种类型的垃圾回收算法，每一类都有单独的<a href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/collectors.html">性能特性</a>:</p>

<p><strong>serial collector</strong> ：使用一条线程进行所有的垃圾回收工作，相对来说也是高效的因为没有线程之间的通信。适用于单处理器的机器。使用<code>-XX:+UseSerialGC.</code>启用</p>

<p><strong>parallel collector</strong> (同时也称作吞吐回收器) ：使用多线程进行垃圾回收，这样能显著的降低垃圾回收的负荷。设计来适用于这样的应用：拥有中等或大数据集的，运行在多核处理器或多线程的硬件</p>

<p><strong>concurrent collector</strong>： 大部分的垃圾回收工作会同步的进行（不阻塞应用的运行）以维持短暂的GC暂停时间。它是设计给中等或大数据集的、响应时间比整体的吞吐量要更重要的应用，因为用这种算法去降低GC的停顿会一定程度降低应用的性能。</p>

<p><img src="http://jaskey.github.io/images/gc/gc-compared.png" title="gc-compared" alt="gc-compared" /></p>

<p>​    HotSpot JVM可以让我们选择不同的GC算法去回收年轻代和年老代，但是某些算法是需要配套的使用才兼容的。例如，你不能选择<em>Parallel Scavenge</em>去回收年轻代的同时，使用CMS收集器去回收年老代因为这两个收集器是不兼容的。以下是兼容的收集器的示意图</p>

<p><img src="http://jaskey.github.io/images/gc/gc-collectors-pairing.jpg" title="gc-collectors-pairing" alt="gc-collectors-pairing" /></p>

<ol>
<li>“Serial”是一个stop-the-world，复制算法的垃圾收集器，使用一条GC线程。</li>
<li>“Parallel Scavenge”是一个stop-the-world、采用复制算法的垃圾收集器，但是使用多条GC线程。</li>
<li>ParNew是一个stop-the-world，复制算法的收集器，使用多条GC线程。它和Parallel Scavenge的区别是它做了一些增强以适应搭配CMS使用。例如ParNew会做必要的同步（synchronization ）以便它能在CMS的同步阶段运行。</li>
<li>Serial Old 是一个stop-the-world，采用标记-清除-压缩算法的回收器，使用一条GC线程</li>
<li>CMS（Concurrent Mark Sweep）是一个同步能力最强、低延迟的垃圾回收器</li>
<li>Parallel Old是一个压缩算法的垃圾回收器，使用多个GC线程。</li>
</ol>


<p>​    对于服务端的应用程序（需要处理客户端请求）来说，使用CMS+ParNew是不错的选择。</p>

<p>我在大概10GB堆内存的程序中使用过也能保持响应时间稳定和短暂的GC暂停时间。我认识的一些开发者使用Parallel collectors (<em>Parallel Scavenge</em> + <em>Parallel Old</em>) ，效果也不错。</p>

<p>​    其中一件需要注意的事是CMS已经宣布废弃了，会被Oralce推荐使用一个新的同步收集器取代， <a href="https://docs.oracle.com/javase/7/docs/technotes/guides/vm/G1.html">Garbage-First</a> 简称 <strong>G1</strong>, 一个最先由Java推出的垃圾收集器</p>

<p>​    G1是一个服务端类型（server-style）的垃圾回收器，针对多处理器、大内存的计算机使用。它能尽可能地满足一个GC延迟时间的目标，同时也有很高的吞吐量</p>

<p>​    <strong>G1</strong> 会同时在年轻代和年老代进行工作。它针对大堆有专门的优化（>10GB）。我没有亲身尝试过G1，我团队里的开发者仍然使用的CMS，所以我还不能对两者进行比较。但通过快速的搜索之后，我找到了一个性能对比说CMS会比G1更好（<a href="http://blog.novatec-gmbh.de/g1-action-better-cms/">CMS outperforming</a> <a href="https://dzone.com/articles/g1-vs-cms-vs-parallel-gc">G1</a>）。我倾向于谨慎，但G1应该是不错的。我们能靠以下参数启动</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'>-XX:+UseG1GC
</span></code></pre></td></tr></table></div></figure>


<p>注：以上由本人摘选翻译自<a href="https://codeahoy.com/2017/08/06/basics-of-java-garbage-collection/">https://codeahoy.com/2017/08/06/basics-of-java-garbage-collection/</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[基于Sharding Sphere实现数据“一键脱敏”]]></title>
    <link href="https://Jaskey.github.io/blog/2020/03/18/sharding-sphere-data-desensitization/"/>
    <updated>2020-03-18T20:53:00+08:00</updated>
    <id>https://Jaskey.github.io/blog/2020/03/18/sharding-sphere-data-desensitization</id>
    <content type="html"><![CDATA[<p>在真实业务场景中，数据库中经常需要存储某些客户的关键性敏感信息如：身份证号、银行卡号、姓名、手机号码等，此类信息按照合规要求，通常需要实现加密存储以满足合规要求。</p>

<h3>痛点一：</h3>

<p>通常的解决方案是我们书写SQL的时候，把对应的加密字段手动进行加密再进行插入，在查询的时候使用之前再手动进行解密。此方法固然可行，但是使用起来非常不便捷且繁琐，使得日常的业务开发与存储合规的细节紧耦合</p>

<h3>痛点二：</h3>

<p>对于一些为了快速上线而一开始没有实现合规脱敏的系统，如何比较快速的使得已有业务满足合规要求的同时，尽量减少对原系统的改造。（通常的这个过程至少包括：1.新增脱敏列的存储 2.同时做数据迁移 3.业务的代码做兼容逻辑等）。</p>

<p>Apache ShardingSphere下面存在一个数据脱敏模块，此模块集成的常用的数据脱敏的功能。其基本原理是对用户输入的SQL进行解析拦截，并依靠用户的脱敏配置进行SQL的改写，从而实现对原文字段的加密及加密字段的解密。最终实现对用户无感的加解密存储、查询。</p>

<h2>脱敏配置Quick Start——Spring 显示配置：</h2>

<p>以下介绍基于Spring如何快速让系统支持脱敏配置。</p>

<h3>1.引入依赖</h3>

<pre><code>&lt;!-- for spring namespace --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;org.apache.shardingsphere&lt;/groupId&gt;
    &lt;artifactId&gt;sharding-jdbc-spring-namespace&lt;/artifactId&gt;
    &lt;version&gt;${sharding-sphere.version}&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>

<h3>2.创建脱敏配置规则对象</h3>

<p>在创建数据源之前，需要准备一个EncryptRuleConfiguration进行脱敏的配置，以下是一个例子，对于同一个数据源里两张表card_info，pay_order的不同字段进行AES的加密</p>

<pre><code>private EncryptRuleConfiguration getEncryptRuleConfiguration() {
Properties props = new Properties();

//自带aes算法需要
props.setProperty("aes.key.value", aeskey);
EncryptorRuleConfiguration encryptorConfig = new EncryptorRuleConfiguration("AES", props);

//自定义算法
//props.setProperty("qb.finance.aes.key.value", aeskey);
//EncryptorRuleConfiguration encryptorConfig = new EncryptorRuleConfiguration("QB-FINANCE-AES", props);

EncryptRuleConfiguration encryptRuleConfig = new EncryptRuleConfiguration();
encryptRuleConfig.getEncryptors().put("aes", encryptorConfig);

//START: card_info 表的脱敏配置
{
    EncryptColumnRuleConfiguration columnConfig1 = new EncryptColumnRuleConfiguration("", "name", "", "aes");
    EncryptColumnRuleConfiguration columnConfig2 = new EncryptColumnRuleConfiguration("", "id_no", "", "aes");
    EncryptColumnRuleConfiguration columnConfig3 = new EncryptColumnRuleConfiguration("", "finshell_card_no", "", "aes");
    Map&lt;String, EncryptColumnRuleConfiguration&gt; columnConfigMaps = new HashMap&lt;&gt;();
    columnConfigMaps.put("name", columnConfig1);
    columnConfigMaps.put("id_no", columnConfig2);
    columnConfigMaps.put("finshell_card_no", columnConfig3);
    EncryptTableRuleConfiguration tableConfig = new EncryptTableRuleConfiguration(columnConfigMaps);
    encryptRuleConfig.getTables().put("card_info", tableConfig);
}
//END: card_info 表的脱敏配置

//START: pay_order 表的脱敏配置
{
    EncryptColumnRuleConfiguration columnConfig1 = new EncryptColumnRuleConfiguration("", "card_no", "", "aes");
    Map&lt;String, EncryptColumnRuleConfiguration&gt; columnConfigMaps = new HashMap&lt;&gt;();
    columnConfigMaps.put("card_no", columnConfig1);
    EncryptTableRuleConfiguration tableConfig = new EncryptTableRuleConfiguration(columnConfigMaps);
    encryptRuleConfig.getTables().put("pay_order", tableConfig);
}

log.info("脱敏配置构建完成:{} ", encryptRuleConfig);
return encryptRuleConfig;
</code></pre>

<p>}</p>

<p>说明：</p>

<ol>
<li>创建 EncryptColumnRuleConfiguration 的时候有四个参数，前两个参数分表叫plainColumn、cipherColumn，其意思是数据库存储里面真实的两个列（名文列、脱敏列），对于新的系统，只需要设置脱敏列即可，所以以上示例为plainColumn为&#8221;&ldquo;。</li>
<li>创建EncryptTableRuleConfiguration 的时候需要传入一个map，这个map存的value即#1中说明的EncryptColumnRuleConfiguration ，而其key则是一个逻辑列，对于新系统，此逻辑列即为真实的脱敏列。Sharding Shpere在拦截到SQL改写的时候，会按照用户的配置，把逻辑列映射为名文列或者脱敏列（默认）如下的示例</li>
</ol>


<p><img src="http://jaskey.github.io/images/shardingsphere/basic.png" title="shardings sphere basic" alt="shardings sphere basic" /></p>

<h3>3.使用Sharding Sphere的数据源进行管理</h3>

<p>把原始的数据源包装一层</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>@Bean("tradePlatformDataSource")
</span><span class='line'>public DataSource dataSource(@Qualifier("druidDataSource") DataSource ds) throws SQLException {
</span><span class='line'>    return EncryptDataSourceFactory.createDataSource(ds, getEncryptRuleConfiguration(), new Properties());
</span><span class='line'>
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<h2>脱敏配置Quick Start——Spring Boot版：</h2>

<p>以下步骤使用Spring Boot管理，可以仅用配置文件解决：</p>

<p>1.引入依赖</p>

<pre><code>&lt;!-- for spring boot --&gt;

&lt;dependency&gt;
&lt;groupId&gt;org.apache.shardingsphere&lt;/groupId&gt;
    &lt;artifactId&gt;sharding-jdbc-spring-boot-starter&lt;/artifactId&gt;
    &lt;version&gt;${sharding-sphere.version}&lt;/version&gt;
&lt;/dependency&gt;

&lt;!-- for spring namespace --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;org.apache.shardingsphere&lt;/groupId&gt;
    &lt;artifactId&gt;sharding-jdbc-spring-namespace&lt;/artifactId&gt;
    &lt;version&gt;${sharding-sphere.version}&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>

<ol>
<li>Spring 配置文件</li>
</ol>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>
</span><span class='line'>spring.shardingsphere.datasource.name=ds
</span><span class='line'>spring.shardingsphere.datasource.ds.type=com.alibaba.druid.pool.DruidDataSource
</span><span class='line'>spring.shardingsphere.datasource.ds.driver-class-name=com.mysql.jdbc.Driver
</span><span class='line'>spring.shardingsphere.datasource.ds.url=xxxxxxxxxxxxx
</span><span class='line'>spring.shardingsphere.datasource.ds.username=xxxxxxx
</span><span class='line'>spring.shardingsphere.datasource.ds.password=xxxxxxxxxxxx
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'># 默认的AES加密器
</span><span class='line'>spring.shardingsphere.encrypt.encryptors.encryptor_aes.type=aes
</span><span class='line'>spring.shardingsphere.encrypt.encryptors.encryptor_aes.props.aes.key.value=hkiqAXU6Ur5fixGHaO4Lb2V2ggausYwW
</span><span class='line'>
</span><span class='line'># card_info 姓名 AES加密
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.name.cipherColumn=name
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.name.encryptor=encryptor_aes
</span><span class='line'>
</span><span class='line'># card_info 身份证 AES加密
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.id_no.cipherColumn=id_no
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.id_no.encryptor=encryptor_aes
</span><span class='line'>
</span><span class='line'># card_info 银行卡号 AES加密
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.finshell_card_no.cipherColumn=finshell_card_no
</span><span class='line'>spring.shardingsphere.encrypt.tables.card_info.columns.finshell_card_no.encryptor=encryptor_aes
</span><span class='line'>
</span><span class='line'># pay_order 银行卡号 AES加密
</span><span class='line'>spring.shardingsphere.encrypt.tables.pay_order.columns.card_no.cipherColumn=card_no
</span><span class='line'>spring.shardingsphere.encrypt.tables.pay_order.columns.card_no.encryptor=encryptor_aes</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[SpringBoot+Dubbo优雅退出分析及方案]]></title>
    <link href="https://Jaskey.github.io/blog/2019/09/30/spring-boot-dubbo-graceful-shutdown/"/>
    <updated>2019-09-30T14:56:56+08:00</updated>
    <id>https://Jaskey.github.io/blog/2019/09/30/spring-boot-dubbo-graceful-shutdown</id>
    <content type="html"><![CDATA[<p>背景：</p>

<p>当我们使用SpringBoot+D做微服务的时候，可能再服务停机的过程，发现在一瞬间出现一些报错，最典型的如比如拿到的数据库连接已经关闭等问题，如下图所示：</p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/dubbo-shutdown-problem.png" alt="img" /></p>

<p>从日志错误可以看到，停机时还存在正在处理的请求，而此请求需要访问数据源，但数据源的资源被 Spring 容器关闭了，导致获取不到而报错。</p>

<p>但是实际上，无论Dubbo和Spring其实都实现了优雅退出，为什么最后退出还是不那么优雅呢？</p>

<p>要分析这个问题，首先得分析它们两者的优雅退出实现。</p>

<h1>Dubbo优雅退出</h1>

<p>dubbo框架本身基于ShutdownHook注册了一个优雅退出的钩子，背后会调用其destroyAll来实现自身的优雅关闭。</p>

<p>以下是Dubbo 2.6.2的源码：</p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/dubbo-shutdown-sourcecode-1.png" alt="img" /></p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/dubbo-shutdown-sourcecode-2.png" alt="img" /></p>

<p>Dubbo发现程序退出的时候，钩子方法会通知注册中心取消自身的注册——以便告知消费者不要调用自己了，然后关闭自身的端口连接——在关闭自身连接的时候还会sleep自旋的方法等待已有的处理请求先完成）</p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/dubbo-shutdown-sourcecode-3.png" alt="img" /></p>

<p>但是，Dubbo服务的优雅退出，不代表服务背后的代码是优雅的，也就是说在Dubbo优雅退出的完成前，我们的服务能否能保证可用——背后的资源/服务是否仍然可用。</p>

<p>本文一开始截图的错误，原因就是服务停机的时候，依赖的数据库资源因为某些原因已经回收了，这时候正在处理的请求自然报错而显得不优雅了。</p>

<p>而回收的人并不是别人，就是Spring的优雅退出。</p>

<h1>Spring的优雅退出</h1>

<p>Spring回收资源也是基于ShutdownHook实现的，Spring在启动的时候会调用<code>refreshContext</code>接口，这个接口默认会帮我们注册优雅退出的钩子方法。</p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/spring-shutdown-hook-sourcecode-1.png" alt="img" /></p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/spring-shutdown-hook-sourcecode-2.png" alt="img" /></p>

<p>这个钩子方法最后会销毁Spring容器，其中自然包括其背后的依赖的资源。</p>

<p>因为大部分情况下，我们的Dubbo服务是依赖于Spring的资源的，要真正实现优雅退出，除了双方本身退出的过程是优雅的，还需要保证Dubbo退出的过程中Spring的资源是可用的——也就是退出应该要是有顺序的：Dubbo退出→Spring退出。</p>

<p>但是Java的ShutdownHook背后的退出是并发执行而没有顺序依赖的，这是背后表现不优雅的原因。以下是JDK文档的描述：</p>

<p><img src="http://jaskey.github.io/images/dubbo-shutdown-hook/jdk-shudownhook-coments.png" alt="img" /></p>

<p>正是由于本身应该有顺序关系的退出逻辑，在并行的处理，导致部分的流量正在处理过程中，依赖的资源已经释放了，最终导致退出的不优雅。</p>

<p>要解决这个问题，可简单可行的思路是：给Dubbo退出一定的时间去处理，然后再执行Spring容器的关闭。但由于钩子方法的时机并不能程序员控制，那么怎么样才能做到呢——禁用原生Spring的钩子方法，在合适的时机手动销毁Spring容器。</p>

<h1>优雅退出方案（简版）——给予固定睡眠时间后才关闭Spring容器：</h1>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">SpringApplication</span> <span class="n">application</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">SpringApplication</span><span class="o">(</span><span class="n">Main</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'><span class="n">application</span><span class="o">.</span><span class="na">setRegisterShutdownHook</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span><span class="c1">//关闭spring的shutdown hook，后续手动触发</span>
</span><span class='line'><span class="kd">final</span> <span class="n">ConfigurableApplicationContext</span> <span class="n">context</span> <span class="o">=</span> <span class="n">application</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="n">args</span><span class="o">);</span>
</span><span class='line'><span class="n">Runtime</span><span class="o">.</span><span class="na">getRuntime</span><span class="o">().</span><span class="na">addShutdownHook</span><span class="o">(</span><span class="k">new</span> <span class="nf">Thread</span><span class="o">(</span><span class="s">&quot;T_SHUTDOWN_HOOK&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;”====================shutdown App====================“。&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="c1">//....这里可以做其他优雅退出处理，例如回收本地线程池、关闭定时调度器等的操作</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">2000</span><span class="o">);</span><span class="c1">//等待一段时间，这里给时间dubbo的shutdownhook执行，</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>            <span class="n">log</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">&quot;&quot;</span><span class="o">,</span><span class="n">e</span><span class="o">);</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">//关闭spring容器</span>
</span><span class='line'>        <span class="n">context</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">});</span>
</span></code></pre></td></tr></table></div></figure>


<h1>优雅退出方案（升级版）——动态地等待消费者及生产者连接关闭后才关闭Spring容器：</h1>

<p>上面的方案正常情况下也够用，因为大部分时间我们只需要估算一个退出时间，让dubbo处理销毁的工作即可，但是对于一些退出时间相对变化较大（如有动态的消费者），表现出来的结果就是dubbo的退出时间有时候较短，有时候缺比较长。如果直接给一个较大的睡眠时间，可能使得每次程序退出都等很久，就显得不太优雅了。</p>

<p>那么我们就可以使用一些底层的dubbo api去确认消费者和生产者的连接已经关闭，以下是一个方法用以取代上面代码片段中的sleep的语句：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="cm">/**</span>
</span><span class='line'><span class="cm"> * 等待Dubbo退出，优雅退出的shutdown hook可使用</span>
</span><span class='line'><span class="cm"> * @param sleepMillis 每次发现Dubbo没退出完就睡眠等待的毫秒数</span>
</span><span class='line'><span class="cm"> * @param sleepMaxTimes 最多睡眠的次数，避免一直dubbo退出太久卡住程序的退出，达到此次数后会不再等待</span>
</span><span class='line'><span class="cm"> */</span>
</span><span class='line'><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">waitDubboShutdown</span><span class="o">(</span><span class="kt">long</span> <span class="n">sleepMillis</span><span class="o">,</span> <span class="kt">int</span> <span class="n">sleepMaxTimes</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">sleepWaitTimes</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">sleepWaitTimes</span> <span class="o">&lt;</span><span class="n">sleepMaxTimes</span><span class="o">;</span> <span class="n">sleepWaitTimes</span><span class="o">++){</span><span class="c1">//如果dubbo的server没有关闭完成，会睡眠等待，最多等待三次</span>
</span><span class='line'>        <span class="n">Collection</span> <span class="n">existingDubboServers</span> <span class="o">=</span> <span class="n">DubboProtocol</span><span class="o">.</span><span class="na">getDubboProtocol</span><span class="o">().</span><span class="na">getServers</span><span class="o">();</span>
</span><span class='line'>        <span class="n">Collection</span> <span class="n">existingDubboExporters</span>  <span class="o">=</span> <span class="n">DubboProtocol</span><span class="o">.</span><span class="na">getDubboProtocol</span><span class="o">().</span><span class="na">getExporters</span><span class="o">();</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;existing dubbo servers : {}, existing dubbo expoerters {} ,  sleepWaitTimes : {}&quot;</span><span class="o">,</span> <span class="n">existingDubboServers</span><span class="o">,</span> <span class="n">existingDubboExporters</span><span class="o">,</span> <span class="n">sleepWaitTimes</span><span class="o">);</span>
</span><span class='line'>        <span class="k">if</span> <span class="o">(!</span><span class="n">existingDubboServers</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()</span> <span class="o">||</span> <span class="o">!</span><span class="n">existingDubboExporters</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>            <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>                <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="n">sleepMillis</span><span class="o">);</span>
</span><span class='line'>            <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>                <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
</span><span class='line'>            <span class="o">}</span>
</span><span class='line'>        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class='line'>            <span class="k">break</span><span class="o">;</span>
</span><span class='line'>        <span class="o">}</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">//优雅退出失败，打印日志</span>
</span><span class='line'>    <span class="n">Collection</span> <span class="n">existingDubboServers</span> <span class="o">=</span> <span class="n">DubboProtocol</span><span class="o">.</span><span class="na">getDubboProtocol</span><span class="o">().</span><span class="na">getServers</span><span class="o">();</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(!</span><span class="n">existingDubboServers</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">&quot;DUBBO服务Server依然存在，不再等待其销毁，可能会导致优雅退出失败 {}&quot;</span><span class="o">,</span><span class="n">existingDubboServers</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">Collection</span> <span class="n">existingDubboExporters</span>  <span class="o">=</span> <span class="n">DubboProtocol</span><span class="o">.</span><span class="na">getDubboProtocol</span><span class="o">().</span><span class="na">getExporters</span><span class="o">();</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(!</span><span class="n">existingDubboExporters</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">&quot;DUBBO服务Exporters依然存在，不再等待其销毁，可能会导致优雅退出失败 {}&quot;</span><span class="o">,</span><span class="n">existingDubboExporters</span><span class="o">);</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>注：这个方法用到了DubboProtocol的底层API，所以如果你的协议不是使用&#8221;dubbo&#8221;而是如HTTP协议、redis协议，则此方法不可用。关于协议的部分，可以参考官方文档：<a href="http://dubbo.apache.org/zh-cn/docs/user/references/protocol/introduction.html">http://dubbo.apache.org/zh-cn/docs/user/references/protocol/introduction.html</a></p>

<p>那么最后，升级版的优雅退出代码则如下所示：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">SpringApplication</span> <span class="n">application</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">SpringApplication</span><span class="o">(</span><span class="n">Main</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'><span class="n">application</span><span class="o">.</span><span class="na">setRegisterShutdownHook</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span><span class="c1">//关闭spring的shutdown hook，后续手动触发</span>
</span><span class='line'><span class="kd">final</span> <span class="n">ConfigurableApplicationContext</span> <span class="n">context</span> <span class="o">=</span> <span class="n">application</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="n">args</span><span class="o">);</span>
</span><span class='line'><span class="n">Runtime</span><span class="o">.</span><span class="na">getRuntime</span><span class="o">().</span><span class="na">addShutdownHook</span><span class="o">(</span><span class="k">new</span> <span class="nf">Thread</span><span class="o">(</span><span class="s">&quot;T_SHUTDOWN_HOOK&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&quot;”====================shutdown App====================“。&quot;</span><span class="o">);</span>
</span><span class='line'>        <span class="c1">//....这里可以做其他优雅退出处理，例如回收本地线程池、关闭定时调度器等的操作</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">waitDubboShutdown</span><span class="o">(</span><span class="mi">1000</span><span class="o">,</span><span class="mi">5</span><span class="o">);</span><span class="c1">//每次等1000ms，最多等5次；优雅退出时间是动态的（可能1秒就能优雅退出完毕）；但如果退出时间大于5秒，那么则放弃优雅退出，直接退出。</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">//关闭spring容器</span>
</span><span class='line'>        <span class="n">context</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">});</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[监控Spring Boot中的Tomcat性能数据]]></title>
    <link href="https://Jaskey.github.io/blog/2019/09/23/spring-boot-tomcat-mertic/"/>
    <updated>2019-09-23T14:56:56+08:00</updated>
    <id>https://Jaskey.github.io/blog/2019/09/23/spring-boot-tomcat-mertic</id>
    <content type="html"><![CDATA[<p>现在，我们经常会使用Spring Boot以开发Web服务，其内嵌容器的方法的确使得开发效率大大提升。</p>

<p>由于网关层通常是直接面对用户请求的一层，也是微服务里面最上游的一个服务，其请求量通常是所有服务中最大的，如果服务出现了性能上的问题，网关层通常都会出现阻塞、超时等现象，这时候就很可能需要性能的调优，其中最常见的则是参数调优。但如何知道哪些性能参数成为了瓶颈（如容器线程数是否不足等），则是调优的前提条件。</p>

<p>本文总结介绍如何在使用了Spring  Boot的前提下，获取运行时的Tomcat性能运行情况。</p>

<p>Spring Boot中有一个Spring Boot actuator的模块，用来监控和管理应用的运行状态，例如健康状况，线程运行情况等。</p>

<p>Maven 依赖：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&lt;dependencies&gt;
</span><span class='line'>    &lt;dependency&gt;
</span><span class='line'>    &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
</span><span class='line'>    &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;
</span><span class='line'>    &lt;/dependency&gt;
</span><span class='line'>&lt;/dependencies&gt;</span></code></pre></td></tr></table></div></figure>


<p>然后当Spring Boot运行之后，Spring Boot会有很多服务暴露在http服务中，这些服务叫EndPoints， 通过 <a href="http://">http://</a>{应用路径}/actuator 这个 url 即可访问，例如  <a href="http://">http://</a>{应用路径}/actuator/info， <a href="http://">http://</a>{应用路径}/actuator/health 这两个endpoints是默认开启的。</p>

<p>其中actuator这个路径可以通过配置修改：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>management.endpoints.web.base-path=/mypath</span></code></pre></td></tr></table></div></figure>


<p>以下是获取健康状态的一个例子：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ curl 'http://localhost:8080/actuator/health' -i -X GET</span></code></pre></td></tr></table></div></figure>


<p>可能会得到类似这样的结果：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>{
</span><span class='line'>    "status" : "UP"
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>比较简陋，如果希望这个接口有更多数据，可以尝试这样的配置：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>management.endpoint.health.show-details=always</span></code></pre></td></tr></table></div></figure>


<p>结果就会丰富了（我的应用用了Redis）：类似</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>{
</span><span class='line'>  "status": "UP",
</span><span class='line'>  "details": {
</span><span class='line'>      "diskSpace": {
</span><span class='line'>          "status": "UP",
</span><span class='line'>          "details": {
</span><span class='line'>              "total": 214745214976,
</span><span class='line'>              "free": 174805827584,
</span><span class='line'>              "threshold": 10485760
</span><span class='line'>          }
</span><span class='line'>      },
</span><span class='line'>      "redis": {
</span><span class='line'>          "status": "UP",
</span><span class='line'>          "details": {
</span><span class='line'>              "cluster_size": 3,
</span><span class='line'>              "slots_up": 16384,
</span><span class='line'>              "slots_fail": 0
</span><span class='line'>          }
</span><span class='line'>      }
</span><span class='line'>  }
</span><span class='line'>}
</span></code></pre></td></tr></table></div></figure>


<p>但是这还不够，我们需要详细的容器数据。监控状况只是一部分。而这些我们想要的数据，是在一个叫metric的EndPoint下面。 但是此endpoint 默认没有暴露到http接口的的，需要添加配置：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>#默认只开启info, health 的http暴露，在此增加metric endpoint
</span><span class='line'>management.endpoints.web.exposure.include=info, health,metric</span></code></pre></td></tr></table></div></figure>


<p>之后我们就能访问这个metric有哪些数据了</p>

<p>$ curl &lsquo;<a href="http://localhost:8080/actuator/metric">http://localhost:8080/actuator/metric</a>&rsquo; -i -X GET</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>{
</span><span class='line'>    "names": [
</span><span class='line'>        "jvm.memory.max",
</span><span class='line'>        "jvm.threads.states",
</span><span class='line'>        "process.files.max",
</span><span class='line'>        "jvm.gc.memory.promoted",
</span><span class='line'>        "tomcat.cache.hit",
</span><span class='line'>        "tomcat.servlet.error",
</span><span class='line'>        "system.load.average.1m",
</span><span class='line'>        "tomcat.cache.access",
</span><span class='line'>        "jvm.memory.used",
</span><span class='line'>        "jvm.gc.max.data.size",
</span><span class='line'>        "jvm.gc.pause",
</span><span class='line'>        "jvm.memory.committed",
</span><span class='line'>        "system.cpu.count",
</span><span class='line'>        "logback.events",
</span><span class='line'>        "tomcat.global.sent",
</span><span class='line'>        "jvm.buffer.memory.used",
</span><span class='line'>        "tomcat.sessions.created",
</span><span class='line'>        "jvm.threads.daemon",
</span><span class='line'>        "system.cpu.usage",
</span><span class='line'>        "jvm.gc.memory.allocated",
</span><span class='line'>        "tomcat.global.request.max",
</span><span class='line'>        "tomcat.global.request",
</span><span class='line'>        "tomcat.sessions.expired",
</span><span class='line'>        "jvm.threads.live",
</span><span class='line'>        "jvm.threads.peak",
</span><span class='line'>        "tomcat.global.received",
</span><span class='line'>        "process.uptime",
</span><span class='line'>        "http.client.requests",
</span><span class='line'>        "tomcat.sessions.rejected",
</span><span class='line'>        "process.cpu.usage",
</span><span class='line'>        "tomcat.threads.config.max",
</span><span class='line'>        "jvm.classes.loaded",
</span><span class='line'>        "http.server.requests",
</span><span class='line'>        "jvm.classes.unloaded",
</span><span class='line'>        "tomcat.global.error",
</span><span class='line'>        "tomcat.sessions.active.current",
</span><span class='line'>        "tomcat.sessions.alive.max",
</span><span class='line'>        "jvm.gc.live.data.size",
</span><span class='line'>        "tomcat.servlet.request.max",
</span><span class='line'>        "tomcat.threads.current",
</span><span class='line'>        "tomcat.servlet.request",
</span><span class='line'>        "process.files.open",
</span><span class='line'>        "jvm.buffer.count",
</span><span class='line'>        "jvm.buffer.total.capacity",
</span><span class='line'>        "tomcat.sessions.active.max",
</span><span class='line'>        "tomcat.threads.busy",
</span><span class='line'>        "process.start.time"
</span><span class='line'>    ]
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>其中列出的是所有可以获取的监控数据，在其中我们发现了我们想要的</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>"tomcat.threads.config.max"
</span><span class='line'>"tomcat.threads.current"
</span><span class='line'>"tomcat.threads.busy"
</span></code></pre></td></tr></table></div></figure>


<p>那么如何获取其中的值呢？只需要在metric路径下加上希望获取的指标即可： curl &lsquo;<a href="http://localhost:8080/actuator/metric/tomcat.threads.busy">http://localhost:8080/actuator/metric/tomcat.threads.busy</a>&rsquo;</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>{
</span><span class='line'>  "name": "tomcat.threads.busy",
</span><span class='line'>  "description": null,
</span><span class='line'>  "baseUnit": "threads",
</span><span class='line'>  "measurements": [{
</span><span class='line'>      "statistic": "VALUE",
</span><span class='line'>      "value": 1.0
</span><span class='line'>  }],
</span><span class='line'>  "availableTags": [{
</span><span class='line'>      "tag": "name",
</span><span class='line'>      "values": ["http-nio-10610"]
</span><span class='line'>  }]
</span><span class='line'>}
</span><span class='line'>
</span><span class='line'>
</span></code></pre></td></tr></table></div></figure>


<p>在此，基本我们想要的数据都能实时的通过http服务接口的方式获取了，那么在流量峰值的时候，一些实时的状态便可获取到了。</p>

<h2>监控数据</h2>

<p>但是我们面对的情况是这样的，半个小时前，一个push活动带来了很大的量，但现在流量已经过去了，需要定位当时的性能问题意味着需要采集到过去的数据。所以我们可能需要一个服务定期去监控这些数据。Spring Boot已经考虑到了这种情况，所以其中有一个prometheus的模块，他是一个独立的服务去采集其中的监控数据并可视化，具体的介绍可以参考：<a href="https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/">https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/</a></p>

<h2>以日志形式定期输出监控数据</h2>

<p>很多时候，如果有日志的方法去定期输出监控的数据这样已经足够我们分析了。在Spring Boot 2.x里，只需要配置一个Bean</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>@Configuration
</span><span class='line'>class MetricsConfig {
</span><span class='line'>    @Bean
</span><span class='line'>    LoggingMeterRegistry loggingMeterRegistry() {
</span><span class='line'>        return new LoggingMeterRegistry();
</span><span class='line'>    }
</span><span class='line'>}
</span></code></pre></td></tr></table></div></figure>


<p>之所以需要Spring Boot版本2.x，LoggingMeterRegistry是因为是micrometer-core里面的1.10以上才引入的，而Spring Boot 1.x都低于这个版本，如果不想升级Spring Boot版本，可以尝试显示变更此版本：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&lt;dependency&gt;
</span><span class='line'>    &lt;groupId&gt;io.micrometer&lt;/groupId&gt;
</span><span class='line'>    &lt;artifactId&gt;micrometer-core&lt;/artifactId&gt;
</span><span class='line'>    &lt;version&gt;1.1.3&lt;/version&gt;
</span><span class='line'>&lt;/dependency&gt;
</span></code></pre></td></tr></table></div></figure>


<p>最后日志的内容就会每一分钟的打印出来：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>jvm.buffer.count{id=direct} value=26 buffers
</span><span class='line'>jvm.buffer.count{id=mapped} value=0 buffers
</span><span class='line'>jvm.buffer.memory.used{id=direct} value=632.600586 KiB
</span><span class='line'>jvm.buffer.memory.used{id=mapped} value=0 B
</span><span class='line'>jvm.buffer.total.capacity{id=direct} value=632.599609 KiB
</span><span class='line'>jvm.buffer.total.capacity{id=mapped} value=0 B
</span><span class='line'>jvm.classes.loaded{} value=12306 classes
</span><span class='line'>jvm.gc.live.data.size{} value=39.339607 MiB
</span><span class='line'>jvm.gc.max.data.size{} value=2.666992 GiB
</span><span class='line'>jvm.memory.committed{area=nonheap,id=Compressed Class Space} value=8.539062 MiB
</span><span class='line'>jvm.memory.committed{area=nonheap,id=Code Cache} value=26.8125 MiB
</span><span class='line'>jvm.memory.committed{area=heap,id=PS Survivor Space} value=30 MiB
</span><span class='line'>jvm.memory.committed{area=heap,id=PS Eden Space} value=416.5 MiB
</span><span class='line'>jvm.memory.committed{area=heap,id=PS Old Gen} value=242 MiB
</span><span class='line'>jvm.memory.committed{area=nonheap,id=Metaspace} value=66.773438 MiB
</span><span class='line'>jvm.memory.max{area=heap,id=PS Survivor Space} value=30 MiB
</span><span class='line'>jvm.memory.max{area=heap,id=PS Eden Space} value=1.272949 GiB
</span><span class='line'>jvm.memory.max{area=heap,id=PS Old Gen} value=2.666992 GiB
</span><span class='line'>jvm.memory.max{area=nonheap,id=Metaspace} value=-1 B
</span><span class='line'>jvm.memory.max{area=nonheap,id=Compressed Class Space} value=1 GiB
</span><span class='line'>jvm.memory.max{area=nonheap,id=Code Cache} value=240 MiB
</span><span class='line'>jvm.memory.used{area=nonheap,id=Code Cache} value=26.635071 MiB
</span><span class='line'>jvm.memory.used{area=heap,id=PS Survivor Space} value=25.214882 MiB
</span><span class='line'>jvm.memory.used{area=heap,id=PS Eden Space} value=46.910545 MiB
</span><span class='line'>jvm.memory.used{area=heap,id=PS Old Gen} value=39.34742 MiB
</span><span class='line'>jvm.memory.used{area=nonheap,id=Metaspace} value=63.333778 MiB
</span><span class='line'>jvm.memory.used{area=nonheap,id=Compressed Class Space} value=7.947166 MiB
</span><span class='line'>jvm.threads.daemon{} value=52 threads
</span><span class='line'>jvm.threads.live{} value=54 threads
</span><span class='line'>jvm.threads.peak{} value=67 threads
</span><span class='line'>jvm.threads.states{state=terminated} value=0 threads
</span><span class='line'>jvm.threads.states{state=blocked} value=0 threads
</span><span class='line'>jvm.threads.states{state=new} value=0 threads
</span><span class='line'>jvm.threads.states{state=runnable} value=20 threads
</span><span class='line'>jvm.threads.states{state=waiting} value=19 threads
</span><span class='line'>jvm.threads.states{state=timed-waiting} value=15 threads
</span><span class='line'>process.cpu.usage{} value=-1
</span><span class='line'>process.start.time{} value=435900h 48m 53.344s
</span><span class='line'>process.uptime{} value=56m 6.709s
</span><span class='line'>system.cpu.count{} value=8
</span><span class='line'>system.cpu.usage{} value=-1
</span><span class='line'>tomcat.global.request.max{name=http-nio-10610} value=0.597s
</span><span class='line'>tomcat.servlet.request.max{name=dispatcherServlet} value=0.567s
</span><span class='line'>tomcat.sessions.active.current{} value=0 sessions
</span><span class='line'>tomcat.sessions.active.max{} value=0 sessions
</span><span class='line'>tomcat.threads.busy{name=http-nio-10610} value=0 threads
</span><span class='line'>tomcat.threads.config.max{name=http-nio-10610} value=200 threads
</span><span class='line'>tomcat.threads.current{name=http-nio-10610} value=10 threads
</span></code></pre></td></tr></table></div></figure>


<p>如果需要修改打印的频率，可修改LoggingRegistryConfig以更改其打印频率</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>  //下面是单独的配置实现的参考，当需要修改配置时候可以使用
</span><span class='line'>  return new LoggingMeterRegistry(new LoggingRegistryConfig() {
</span><span class='line'>       @Override
</span><span class='line'>     public Duration step() {
</span><span class='line'>         return Duration.ofSeconds(10);//10秒输出一次
</span><span class='line'>       }
</span><span class='line'>
</span><span class='line'>       @Override
</span><span class='line'>       public String get(String key) {
</span><span class='line'>            return null;
</span><span class='line'>       }
</span><span class='line'>   }, Clock.SYSTEM);
</span><span class='line'>}
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RabbitMQ实现延迟队列]]></title>
    <link href="https://Jaskey.github.io/blog/2018/08/15/rabbitmq-delay-queue/"/>
    <updated>2018-08-15T14:56:56+08:00</updated>
    <id>https://Jaskey.github.io/blog/2018/08/15/rabbitmq-delay-queue</id>
    <content type="html"><![CDATA[<p>RabbitMQ本身没有延迟队列的支持，但是基于其本身的一些特性，可以做到类似延迟队列的效果：基于死信交换器+TTL。</p>

<p>以下介绍下相关概念及方法</p>

<h2>Dead Letter Exchanges</h2>

<p>消息在队列满足达到一定的条件，会被认为是死信消息（dead-lettered），这时候，RabbitMQ会重新把这类消息发到另外一个的exchange，这个exchange称为Dead Letter Exchanges.</p>

<p>以下任一条件满足，即可认为是死信：</p>

<ul>
<li>消息被拒绝消费(basic.reject or basic.nack)并且设置了requeue=fasle</li>
<li>消息的TTL到了（消息过期）</li>
<li>达到了队列的长度限制</li>
</ul>


<p>需要注意的是，Dead letter exchanges (DLXs) 其实就是普通的exchange，可以和正常的exchange一样的声明或者使用。</p>

<h2>死信消息路由</h2>

<p>队列中可以设置两个属性：</p>

<ul>
<li>x-dead-letter-exchange</li>
<li>x-dead-letter-routing-key</li>
</ul>


<p>当这个队列里面的消息成为死信之后，就会投递到x-dead-letter-exchange指定的exchange中，其中带着的routing key就是中指定的值x-dead-letter-routing-key。</p>

<p>而如果使用默认的exchange(routing key就是希望指定的队列)，则只需要把x-dead-letter-exchange设置为空（不能不设置），类似下面</p>

<p><img src="http://jaskey.github.io/images/rabbitmq/delay-queue-param.png" title="rabbitmq 延迟队列的配置" alt="rabbitmq 延迟队列的配置" /></p>

<p>死信消息的路由则会根据x-dead-letter-routing-key所指定的进行路由，如果这个值没有指定，则会按照消息一开始发送的时候指定的routing key进行路由</p>

<blockquote><p>Dead-lettered messages are routed to their dead letter exchange either:</p>

<p>with the routing key specified for the queue they were on; or, if this was not set,
with the same routing keys they were originally published with.</p></blockquote>

<p>例如，如果一开始你对exchange X发送消息，带着routing key &ldquo;foo&#8221;，进入了队列 Q然后消息变死信后，他会被重新发送到 dead letter exchange ，其中发给dead letter exchange带着的routing key 还是foo。 但如果这个队列Q本身是设置了x-dead-letter-routing-key  bar， 那么他发送到 dead letter exchange的时候，带着的routing key 就是bar。</p>

<p>需要注意的是，当死信消息重新路由到新的队列的时候，在死信目标队列确认收到这条死信消息之前，原来队列的消息是不会删除的，也就是说在某些异常场景下例如broker突然shutdown，是有机会存在说一个消息既存在于原队列，又存在于死信目标队列。具体可参考官方说明：</p>

<blockquote><p>Dead-lettered messages are re-published with publisher confirms turned on internally so, the &ldquo;dead-letter queues&rdquo; (DLX routing targets) the messages eventually land on must confirm the messages before they are removed from the original queue. In other words, the &ldquo;publishing&rdquo; (the one in which messages expired) queue will not remove messages before the dead-letter queues acknowledge receiving them (see Confirms for details on the guarantees made). Note that, in the event of an unclean broker shutdown, the same message may be duplicated on both the original queue and on the dead-lettering destination queues.</p></blockquote>

<h2>Time-To-Live（TTL）</h2>

<p>开头我们说过，实现延迟队列除了用死信消息外，还需要利用消息过期的TTL机制，因为只要消息过期了，就会触发死信。</p>

<p>RabbitMQ有两种方法让设置消息的TTL：</p>

<h3>直接在消息上设置</h3>

<pre><code>byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.expiration("60000")
.build();
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
</code></pre>

<h3>为队列设置消息过期TTL</h3>

<p><img src="http://jaskey.github.io/images/rabbitmq/x-message-ttl.png" title="rabbitmq x-message-ttl" alt="rabbitmq x-message-ttl" /></p>

<p>注意，队列还有一个队列TTL，x-expires，这个的意思是队列空置经过一段时间（没有消费者，没有被重新声明，没有人在上面获取消息（basic.get））后，整个队列便会过期删除，不要混淆</p>

<p><strong>如果同时设置了消息的过期和队列消息过期属性，则取两个较小值。</strong></p>

<h2>设计延迟队列：</h2>

<p>例如，我们需要触发一个推送新闻，30分钟后统计这个新闻的下发情况，我们就需要一个延迟队列，新闻推送后，往延迟队列发送一个消息，这个队列的消息在30分钟后被消费，这时候触发即可统计30分钟的下发情况。我们可以这样设计：</p>

<p>定义一个正常的队列： ARRIVAL_STAT，统计程序监听此队列，进行消费。</p>

<p>定义一个“延迟队列”（RabbitMQ没有这样的队列，这里只是人为的制造一个这样的队列）：DELAY_ARRIVAL_STAT，其中设置好对应的x-dead-letter-exchange，x-dead-letter-routing-key。为了简单说明，我使用默认的exchange，那么配置如下：</p>

<pre><code>x-dead-letter-exchange=“”
x-dead-letter-routing-key=“ARRIVAL_STAT”
</code></pre>

<p>意思是，消息当这个队列DELAY_ARRIVAL_STAT的消息变死信之后，就会带着routing key &ldquo;ARRIVAL_STAT&#8221;发送默认的空exchange，即队列ARRIVAL_STAT。</p>

<p>并且这个队列不能有消费者消费消息。</p>

<p>这样我们就实现了消息的死信转发。下一步，只需要让消息在这个DELAY_ARRIVAL_STAT在30分钟后过期变死信即可。按照上文所说，有两种方法，我们可以为队列的消息设置30分钟TTL，或者发送消息的时候指定消息的TTL为30分钟即可。</p>

<p>示例如下：</p>

<p><img src="http://jaskey.github.io/images/rabbitmq/delay-queue-demo.png" title="rabbitmq 延迟队列示意" alt="rabbitmq 延迟队列示意" /></p>

<h2>“延迟队列”的堵塞缺陷</h2>

<p>由于设置了x-dead-letter-exchange的队列本身也是普通队列，其过期的顺序是按照队列头部顺序的过期的。也就是说，如果你队列头的消息A过期时间是5分钟，后面对这个队列发送消息B的带着过期时间1分钟，那么后面的队列B要等队列A过期了才会触发过期：</p>

<blockquote><p>Queues that had a per-message TTL applied to them retroactively (when they already had messages) will discard the messages when specific events occur. Only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered).</p></blockquote>

<p>所以，对于此类多延迟时间的，可以考虑设置多级延迟队列。例如1分钟，5分钟，10分钟，20分钟这样多级的延迟队列，使得延迟相近的尽量放到同一个队列中减少拥堵的最坏情况。</p>

<p><img src="http://jaskey.github.io/images/rabbitmq/multi-delay-queue.png" title="rabbitmq 多级延迟队列" alt="rabbitmq 多级延迟队列" /></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RabbitMQ常用命令与配置]]></title>
    <link href="https://Jaskey.github.io/blog/2018/01/15/rabbitmq-note/"/>
    <updated>2018-01-15T14:56:56+08:00</updated>
    <id>https://Jaskey.github.io/blog/2018/01/15/rabbitmq-note</id>
    <content type="html"><![CDATA[<p>以下记录RabbitMQ常用的运维命令和配置</p>

<hr />

<h1>常用命令</h1>

<h3>启动进程：</h3>

<pre><code>sbin/rabbitmq-server -detached
</code></pre>

<h3>关闭进程：</h3>

<pre><code>sbin/rabbitmqctl stop
</code></pre>

<h3>创建账号：</h3>

<pre><code>sbin/rabbitmqctl add_user admin ${mq_password}
sbin/rabbitmqctl set_user_tags admin administrator
sbin/rabbitmqctl set_permissions -p '/' admin '.*' '.*' '.*'
</code></pre>

<h3>启动监控：</h3>

<pre><code>#启动监控后，可以用http访问控制台 ip:监控端口（默认原端口+10000）
sbin/rabbitmq-plugins enable rabbitmq_management
</code></pre>

<h3>加入集群：</h3>

<pre><code>#要先停止应用
sbin/rabbitmqctl stop_app
#加入集群，cluster_name为之前启动的那个集群名称，通常为环境变量文件中配置的RABBITMQ_NODE_IP_ADDRESS
sbin/rabbitmqctl join_cluster ${cluster_name}
#再次启动应用    
sbin/rabbitmqctl start_app
</code></pre>

<p>命令文档：<a href="https://www.rabbitmq.com/rabbitmqctl.8.html">https://www.rabbitmq.com/rabbitmqctl.8.html</a></p>

<hr />

<h1>配置文件</h1>

<p>rabbitmq-env.conf</p>

<pre><code>RABBITMQ_NODE_IP_ADDRESS= //IP地址，空串bind所有地址，指定地址bind指定网络接口
RABBITMQ_NODE_PORT=       //TCP端口号，默认是5672
RABBITMQ_NODENAME=        //节点名称。默认是rabbit
RABBITMQ_CONFIG_FILE= //配置文件路径 ，即rabbitmq.config文件路径
RABBITMQ_MNESIA_BASE=     //mnesia所在路径
RABBITMQ_LOG_BASE=        //日志所在路径
RABBITMQ_PLUGINS_DIR=     //插件所在路径
</code></pre>

<p>rabbitmq.config</p>

<pre><code>tcp_listerners    #设置rabbimq的监听端口，默认为[5672]。
disk_free_limit     #磁盘低水位线，若磁盘容量低于指定值则停止接收数据，默认值为{mem_relative, 1.0},即与内存相关联1：1，也可定制为多少byte.
vm_memory_high_watermark    #设置内存低水位线，若低于该水位线，则开启流控机制，默认值是0.4，即内存总量的40%。
hipe_compile     #将部分rabbimq代码用High Performance Erlang compiler编译，可提升性能，该参数是实验性，若出现erlang vm segfaults，应关掉。
force_fine_statistics    #该参数属于rabbimq_management，若为true则进行精细化的统计，但会影响性能。
frame_max     #包大小，若包小则低延迟，若包则高吞吐，默认是131072=128K。
heartbeat     #客户端与服务端心跳间隔，设置为0则关闭心跳，默认是600秒。
</code></pre>

<h1>一台机器启动多个实例</h1>

<p>以上如果希望一个机器中启动多个实例，简单需要配置的地方仅有</p>

<p>rabbitmq-env.conf：</p>

<pre><code>#改个名字
RABBITMQ_NODENAME=your_new_node_name
#改个端口    
RABBITMQ_NODE_PORT=5673
</code></pre>

<p>rabbitmq.config:</p>

<pre><code>%tcp 监听端口对应修改%
{tcp_listeners, [5673]},
</code></pre>

<p>rabbitmq_management下面的监听端口对应修改，建议原端口加10000保持与原来默认的统一</p>

<pre><code>{listener, [{port,     15673}]}
</code></pre>

<p>文档：<a href="http://www.rabbitmq.com/configure.html#configuration-file">http://www.rabbitmq.com/configure.html#configuration-file</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RocketMQ 客户端配置]]></title>
    <link href="https://Jaskey.github.io/blog/2017/06/13/rocketmq-client-config/"/>
    <updated>2017-06-13T14:56:56+08:00</updated>
    <id>https://Jaskey.github.io/blog/2017/06/13/rocketmq-client-config</id>
    <content type="html"><![CDATA[<p>RocketMQ的客户端和服务端采取完全不一样的配置机制，客户端没有配置文件，所有的配置选项需要开发者使用对应的配置的setter进行设置。</p>

<p>注： 以下带 * 的，表示为重要参数。</p>

<hr />

<h1>ClientConfig</h1>

<p>RocketMQ的Producer（<code>DefaultMQProducer</code>）和Consumer(<code>DefaultMQPushConsumer</code>，<code>DefaultMQPullConsumer</code>)，甚至运维相关的的admin类（<code>DefaultMQAdminExt</code>）都继承自ClientConfig。这意味着，其中的配置无论Producer还是Consumer都可以进行设置，其中大部分都是公用的配置（但由于设计的问题，有些配置只会对消费或生产生效）。</p>

<h3>namesrvAddr*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>NameServer的地址列表，若是集群，用<code>;</code>作为地址的分隔符。 </td>
<td>-D系统参数<code>rocketmq.namesrv.addr</code>或环境变量<code>NAMESRV_ADDR</code> </td>
</tr>
</tbody>
</table>


<p>无论生产者还是消费者，只要是客户端需要和服务器broker进行操作，就需要依赖Name Server进行服务发现。具体请看：<a href="http://jaskey.github.io/blog/2016/12/14/rocketmq-component/" title="RocketMQ——组件">RocketMQ——组件</a></p>

<h3>instanceName*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>NameServer的地址列表，若是集群，用<code>;</code>作为地址的分隔符。 </td>
<td>从-D系统参数<code>rocketmq.client.name</code>获取，否则就是<code>DEFAULT</code></td>
</tr>
</tbody>
</table>


<p>这个值虽然默认写是<code>DEFAULT</code>，但在启动的时候，如果我们没有显示修改还是维持其<code>DEFAULT</code>的话，RocketMQ会更新为当前的进程号：</p>

<pre><code>public void changeInstanceNameToPID() {
    if (this.instanceName.equals("DEFAULT")) {
        this.instanceName = String.valueOf(UtilAll.getPid());
    }
}
</code></pre>

<p>RocketMQ用一个叫<strong>ClientID</strong>的概念，来唯一标记一个客户端实例，一个客户端实例对于Broker而言会开辟一个Netty的客户端实例。 而ClientID是由ClientIP+InstanceName构成，故如果一个进程中多个实例（无论Producer还是Consumer）ClientIP和InstanceName都一样,他们将公用一个内部实例（同一套网络连接，线程资源等）</p>

<p>此外，此ClientID在对于Consumer负载均衡的时候起到唯一标识的作用，一旦多个实例（无论不同进程、不通机器、还是同一进程）的多个Consumer实例有一样的ClientID，负载均衡的时候必然RocketMQ任然会把两个实例当作一个client（因为同样一个clientID）。 故为了避免不必要的问题，ClientIP+instance Name的组合建议唯一，除非有意需要共用连接、资源。</p>

<h3>clientIP</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>客户端IP</td>
<td><code>RemotingUtil.getLocalAddress()</code> </td>
</tr>
</tbody>
</table>


<p>这个值有两个用处：
1. 对于默认的<code>instanceName</code>（后面说明），如果没有显示设置，会使用ip+进程号，其中的ip便是这里的配置值
2. 对于Producer发送消息的时候，消息本身会存储本值到<code>bornHost</code>，用于标记消息从哪台机器产生的</p>

<h3>clientCallbackExecutorThreads</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>客户端通信层接收到网络请求的时候，处理器的核数</td>
<td><code>Runtime.getRuntime().availableProcessors()</code></td>
</tr>
</tbody>
</table>


<p>虽然大部分指令的发起方是客户端而处理方是broker/NameServer端，但客户端有时候也需要处理远端对发送给自己的命令，最常见的是一些运维指令如<code>GET_CONSUMER_RUNNING_INFO</code>，或者消费实例上线/下线的推送指令<code>NOTIFY_CONSUMER_IDS_CHANGED</code>，这些指令的处理都在一个线程池处理，<code>clientCallbackExecutorThreads</code>控制这个线程池的核数。</p>

<h3>pollNameServerInterval*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>轮询从NameServer获取路由信息的时间间隔</td>
<td>30000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>客户端依靠NameServer做服务发现（具体请看：<a href="http://jaskey.github.io/blog/2016/12/14/rocketmq-component/" title="RocketMQ——组件">RocketMQ——组件</a>），这个间隔决定了新服务上线/下线，客户端最长多久能探测得到。默认是30秒，就是说如果做broker扩容，最长需要30秒客户端才能感知得到新broker的存在。</p>

<h3>heartbeatBrokerInterval*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>定期发送注册心跳到broker的间隔</td>
<td>30000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>客户端依靠心跳告诉broker“我是谁（clientID，ConsumerGroup/ProducerGroup）”，“自己是订阅了什么topic&#8221;，&#8221;要发送什么topic&#8221;。以此，broker会记录并维护这些信息。客户端如果动态更新这些信息，最长则需要这个心跳周期才能告诉broker。</p>

<h3>persistConsumerOffsetInterval*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>作用于Consumer，持久化消费进度的间隔</td>
<td>5000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>RocketMQ采取的是定期批量ack的机制以持久化消费进度。也就是说每次消费消息结束后，并不会立刻ack，而是定期的集中的更新进度。 由于持久化不是立刻持久化的，所以如果消费实例突然退出（如断点）、或者触发了负载均衡分consue queue重排，有可能会有已经消费过的消费进度没有及时更新而导致重新投递。故本配置值越小，重复的概率越低，但同时也会增加网络通信的负担。</p>

<h3>vipChannelEnabled</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>是否启用vip netty通道以发送消息</td>
<td>-D com.rocketmq.sendMessageWithVIPChannel参数的值，若无则是true</td>
</tr>
</tbody>
</table>


<p>broker的netty server会起两个通信服务。两个服务除了服务的端口号不一样，其他都一样。其中一个的端口（配置端口-2）作为vip通道，客户端可以启用本设置项把发送消息此vip通道。</p>

<hr />

<h2>DefaultMQProducer</h2>

<p>所有的消息发送都通过DefaultMQProducer作为入口，以下介绍一下单独属于DefaultMQProducer的一些配置项。</p>

<h3>producerGroup*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>生产组的名称，一类Producer的标识</td>
<td>DEFAULT_PRODUCER</td>
</tr>
</tbody>
</table>


<p>详见 <a href="http://jaskey.github.io/blog/2016/12/15/rocketmq-concept/" title="RocketMQ——角色与术语详解">RocketMQ——角色与术语详解</a></p>

<h3>createTopicKey</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>发送消息的时候，如果没有找到topic，若想自动创建该topic，需要一个key topic，这个值即是key topic的值</td>
<td>TBW102</td>
</tr>
</tbody>
</table>


<p>这是RocketMQ设计非常晦涩的一个概念，整体的逻辑是这样的：</p>

<ul>
<li>生产者正常的发送消息，都是需要topic<strong>预先</strong>创建好的</li>
<li>但是RocketMQ服务端是支持，发送消息的时候，如果topic不存在，在发送的同时自动创建该topic</li>
<li>支持的前提是broker 的配置打开<code>autoCreateTopicEnable=true</code></li>
<li><code>autoCreateTopicEnable=true</code>后，broker会创建一个<code>TBW102</code>的topic，这个就是我们讲的默认的key topic</li>
</ul>


<p>自动构建topic（以下成为T）的过程：</p>

<ol>
<li>Producer发送的时候如果发现该T不存在，就会配置有Producer配置的key topic的那个broker发送消息</li>
<li>broker校验客户端的topic key是否在broker存在，且校验其权限最后一位是否是1（topic权限总共有3位，按位存储，分别是读、写、支持自动创建）</li>
<li>若权限校验通过，先在该broker把T创建，并且权限就是key topic除去最后一位的权限。</li>
</ol>


<p>为了方便理解，以下贴出broker的具体源码并加入部分注释：</p>

<pre><code>                TopicConfig defaultTopicConfig = this.topicConfigTable.get(defaultTopic);//key topic的配置信息
                if (defaultTopicConfig != null) {//key topic 存在
                    if (defaultTopic.equals(MixAll.DEFAULT_TOPIC)) {
                        if (!this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
                            defaultTopicConfig.setPerm(PermName.PERM_READ | PermName.PERM_WRITE);
                        }
                    }

                    if (PermName.isInherited(defaultTopicConfig.getPerm())) {//检验权限，如果允许自动创建
                        topicConfig = new TopicConfig(topic);//创建topic

                        int queueNums =
                            clientDefaultTopicQueueNums &gt; defaultTopicConfig.getWriteQueueNums() ? defaultTopicConfig
                                .getWriteQueueNums() : clientDefaultTopicQueueNums;

                        if (queueNums &lt; 0) {
                            queueNums = 0;
                        }

                        topicConfig.setReadQueueNums(queueNums);
                        topicConfig.setWriteQueueNums(queueNums);
                        int perm = defaultTopicConfig.getPerm();
                        perm &amp;= ~PermName.PERM_INHERIT;//权限按照key topic的来
                        topicConfig.setPerm(perm);
                        topicConfig.setTopicSysFlag(topicSysFlag);
                        topicConfig.setTopicFilterType(defaultTopicConfig.getTopicFilterType());
                    } else {//权限校验不过，自动创建失败
                        LOG.warn("Create new topic failed, because the default topic[{}] has no perm [{}] producer:[{}]",
                                defaultTopic, defaultTopicConfig.getPerm(), remoteAddress);
                    }
                } else {//key topic不存在，创建失败
                    LOG.warn("Create new topic failed, because the default topic[{}] not exist. producer:[{}]", defaultTopic, remoteAddress);
                }

             ...//把创建的topic维护起来
</code></pre>

<p>总的来说，这个功能设计出来比较晦涩，而从运维的角度上看，topic在大部分场景下也应该预创建，故本特性没有必要的话，也不会用到，这个配置也没有必要特殊的设置。</p>

<p>关于这个TBW102非常不直观的问题，我已经提了issue ：<a href="https://issues.apache.org/jira/browse/ROCKETMQ-223">https://issues.apache.org/jira/browse/ROCKETMQ-223</a></p>

<h3>defaultTopicQueueNums</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>自动创建topic的话，默认queue数量是多少</td>
<td>4</td>
</tr>
</tbody>
</table>


<h3>sendMsgTimeout</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>默认的发送超时时间</td>
<td>3000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>若发送的时候不显示指定timeout，则使用此设置的值作为超时时间。</p>

<p>对于异步发送，超时后会进入回调的<code>onException</code>，对于同步发送，超时则会得到一个<code>RemotingTimeoutException</code>。</p>

<h3>compressMsgBodyOverHowmuch</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消息body需要压缩的阈值</td>
<td>1024 * 4，4K</td>
</tr>
</tbody>
</table>


<h3>retryTimesWhenSendFailed</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>同步发送失败的话，rocketmq内部重试多少次</td>
<td>2</td>
</tr>
</tbody>
</table>


<h3>retryTimesWhenSendAsyncFailed</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>异步发送失败的话，rocketmq内部重试多少次</td>
<td>2</td>
</tr>
</tbody>
</table>


<h3>retryAnotherBrokerWhenNotStoreOK</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>发送的结果如果不是SEND_OK状态，是否当作失败处理而尝试重发</td>
<td>false</td>
</tr>
</tbody>
</table>


<p>发送结果总共有4钟：</p>

<pre><code>SEND_OK, //状态成功，无论同步还是存储
FLUSH_DISK_TIMEOUT, // broker刷盘策略为同步刷盘（SYNC_FLUSH）的话时候，等待刷盘的时候超时
FLUSH_SLAVE_TIMEOUT, // master role采取同步复制策略（SYNC_MASTER）的时候，消息尝试同步到slave超时
SLAVE_NOT_AVAILABLE, //slave不可用
</code></pre>

<p>注：从源码上看，此配置项只对同步发送有效，异步、oneway（由于无法获取结果，肯定无效）均无效</p>

<h3>retryAnotherBrokerWhenNotStoreOK</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>客户端验证，允许发送的最大消息体大小</td>
<td>1024 * 1024 * 4，4M</td>
</tr>
</tbody>
</table>


<p>若超过此大小，会得到一个响应码13（MESSAGE_ILLEGAL）的<code>MQClientException</code>异常</p>

<hr />

<h2>TransactionMQProducer</h2>

<p>事务生产者，截至至4.1，由于暂时事务回查功能缺失，整体并不完全可用，配置暂时忽略，等后面功能完善后补上。</p>

<p><a href="https://issues.apache.org/jira/browse/ROCKETMQ-123">https://issues.apache.org/jira/browse/ROCKETMQ-123</a></p>

<hr />

<h2>DefaultMQPushConsumer</h2>

<p>最常用的消费者，使用push模式（长轮询），封装了各种拉取的方法和返回结果的判断。下面介绍其配置。</p>

<h3>consumerGroup*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费组的名称，用于标识一类消费者</td>
<td>无默认值，必设</td>
</tr>
</tbody>
</table>


<p>详见 <a href="http://jaskey.github.io/blog/2016/12/15/rocketmq-concept/" title="RocketMQ——角色与术语详解">RocketMQ——角色与术语详解</a></p>

<h3>messageModel*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费模式</td>
<td>MessageModel.CLUSTERING</td>
</tr>
</tbody>
</table>


<p>可选值有两个：</p>

<ol>
<li>CLUSTERING //集群消费模式</li>
<li>BROADCASTING //广播消费模式</li>
</ol>


<p>两种模式的区别详见：<a href="http://jaskey.github.io/blog/2016/12/15/rocketmq-concept/" title="RocketMQ——角色与术语详解">RocketMQ——角色与术语详解</a></p>

<h3>consumeFromWhere*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费点策略</td>
<td>ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET</td>
</tr>
</tbody>
</table>


<p>可选值有两个：</p>

<ol>
<li>CONSUME_FROM_LAST_OFFSET //队列尾消费</li>
<li>CONSUME_FROM_FIRST_OFFSET //队列头消费</li>
<li>CONSUME_FROM_TIMESTAMP //按照日期选择某个位置消费</li>
</ol>


<p>注：此策略只生效于新在线测consumer group，如果是老的已存在的consumer group，都降按照已经持久化的consume offset进行消费</p>

<p>具体说明祥见： <a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management/" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a></p>

<h3>consumeTimestamp:</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>CONSUME_FROM_LAST_OFFSET的时候使用，从哪个时间点开始消费</td>
<td>半小时前</td>
</tr>
</tbody>
</table>


<p>格式为yyyyMMddhhmmss 如 20131223171201</p>

<h3>allocateMessageQueueStrategy*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>负载均衡策略算法</td>
<td>AllocateMessageQueueAveragely（取模平均分配）</td>
</tr>
</tbody>
</table>


<p>这个算法可以自行扩展以使用自定义的算法，目前有以下算法可以使用</p>

<ul>
<li>AllocateMessageQueueAveragely  //取模平均</li>
<li>AllocateMessageQueueAveragelyByCircle //环形平均</li>
<li>AllocateMessageQueueByConfig // 按照配置，传入听死的messageQueueList</li>
<li>AllocateMessageQueueByMachineRoom //按机房，从源码上看，必须和阿里的某些broker命名一致才行</li>
<li>AllocateMessageQueueConsistentHash //一致性哈希算法，本人于4.1提交的特性。用于解决“惊群效应”。</li>
</ul>


<p>需要自行扩展的算法的，需要实现<code>org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueStrategy</code></p>

<p>具体分配consume queue的过程祥见： <a href="http://jaskey.github.io/blog/2016/12/19/rocketmq-rebalance/" title="RocketMQ——水平扩展及负载均衡详解">RocketMQ——水平扩展及负载均衡详解</a></p>

<h3>subscription</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>订阅关系（topic->sub expression）</td>
<td>{}</td>
</tr>
</tbody>
</table>


<p>不建议设置，订阅topic建议直接调用<code>subscribe</code>接口</p>

<h3>messageListener</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消息处理监听器（回调）</td>
<td>null</td>
</tr>
</tbody>
</table>


<p>不建议设置，注册监听的时候应调用<code>registerMessageListener</code></p>

<h3>offsetStore</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消息消费进度存储器 </td>
<td>null</td>
</tr>
</tbody>
</table>


<p>不建议设置，<code>offsetStore</code> 有两个策略：<code>LocalFileOffsetStore</code> 和 <code>RemoteBrokerOffsetStore</code>。</p>

<p>若没有显示设置的情况下，广播模式将使用<code>LocalFileOffsetStore</code>，集群模式将使用<code>RemoteBrokerOffsetStore</code>，不建议修改。</p>

<h3>consumeThreadMin*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费线程池的core size</td>
<td>20</td>
</tr>
</tbody>
</table>


<p>PushConsumer会内置一个消费线程池，这个配置控制此线程池的core size</p>

<h3>consumeThreadMax*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费线程池的max size</td>
<td>64</td>
</tr>
</tbody>
</table>


<p>PushConsumer会内置一个消费线程池，这个配置控制此线程池的max size</p>

<h3>adjustThreadPoolNumsThreshold</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>动态扩线程核数的消费堆积阈值</td>
<td>1000</td>
</tr>
</tbody>
</table>


<p>相关功能以废弃，不建议设置</p>

<h3>consumeConcurrentlyMaxSpan</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>并发消费下，单条consume queue队列允许的最大offset跨度，达到则触发流控</td>
<td>2000</td>
</tr>
</tbody>
</table>


<p>注：只对并发消费（<code>ConsumeMessageConcurrentlyService</code>）生效</p>

<p>更多分析祥见： <a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management/" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a></p>

<h3>pullThresholdForQueue</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>consume queue流控的阈值</td>
<td>1000</td>
</tr>
</tbody>
</table>


<p>每条consume queue的消息拉取下来后会缓存到本地，消费结束会删除。当累积达到一个阈值后，会触发该consume queue的流控。</p>

<p>更多分析祥见： <a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management/" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a></p>

<p>截至到4.1，流控级别只能针对consume queue级别，针对topic级别的流控已经提了issue: <a href="https://issues.apache.org/jira/browse/ROCKETMQ-106">https://issues.apache.org/jira/browse/ROCKETMQ-106</a></p>

<h3>pullInterval*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>拉取的间隔</td>
<td>0，单位毫秒</td>
</tr>
</tbody>
</table>


<p>由于RocketMQ采取的pull的方式进行消息投递，每此会发起一个异步pull请求，得到请求后会再发起下次请求，这个间隔默认是0，表示立刻再发起。在间隔为0的场景下，消息投递的及时性几乎等同用Push实现的机制。</p>

<h3>pullBatchSize*</h3>

<p>f
| 配置说明 | 默认值 |
| &mdash;&mdash;| &mdash;&mdash; |
|一次最大拉取的批量大小|32|</p>

<p>每次发起pull请求到broker，客户端需要指定一个最大batch size，表示这次拉取消息最多批量拉取多少条。</p>

<h3>consumeMessageBatchMaxSize</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>批量消费的最大消息条数</td>
<td>1</td>
</tr>
</tbody>
</table>


<p>你可能发现了，RocketMQ的注册监听器回调的回调方法签名是类似这样的：</p>

<pre><code>ConsumeConcurrentlyStatus consumeMessage(final List&lt;MessageExt&gt; msgs, final ConsumeConcurrentlyContext context);
</code></pre>

<p>里面的消息是一个集合List而不是单独的msg，这个<code>consumeMessageBatchMaxSize</code>就是控制这个集合的最大大小。</p>

<p>而由于拉取到的一批消息会立刻拆分成N（取决于consumeMessageBatchMaxSize）批消费任务，所以集合中msgs的最大大小是<code>consumeMessageBatchMaxSize</code>和<code>pullBatchSize</code>的较小值。</p>

<h3>postSubscriptionWhenPull</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>每次拉取的时候是否更新订阅关系</td>
<td>false</td>
</tr>
</tbody>
</table>


<p>从源码上看，这个值若是true,且不是class fliter模式，则每次拉取的时候会把subExpression带上到pull的指令中，broker发现这个指令会根据这个上传的表达式重新build出注册数据，而不是直接使用读取的缓存数据。</p>

<h3>maxReconsumeTimes</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>一个消息如果消费失败的话，最多重新消费多少次才投递到死信队列</td>
<td>-1</td>
</tr>
</tbody>
</table>


<p>注，这个值默认值虽然是-1，但是实际使用的时候默认并不是-1。按照消费是并行还是串行消费有所不同的默认值。</p>

<p>并行：默认16次</p>

<p>串行：默认无限大（Interge.MAX_VALUE）。由于顺序消费的特性必须等待前面的消息成功消费才能消费后面的，默认无限大即一直不断消费直到消费完成。</p>

<h3>suspendCurrentQueueTimeMillis</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>串行消费使用，如果返回<code>ROLLBACK</code>或者<code>SUSPEND_CURRENT_QUEUE_A_MOMENT</code>，再次消费的时间间隔</td>
<td>1000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>注：如果消费回调中对<code>ConsumeOrderlyContext</code>中的<code>suspendCurrentQueueTimeMillis</code>进行过设置，则使用用户设置的值作为消费间隔。</p>

<h3>consumeTimeout</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费的最长超时时间</td>
<td>15，<strong> 单位分钟 </strong></td>
</tr>
</tbody>
</table>


<p>如果消费超时，RocketMQ会等同于消费失败来处理，更多分析祥见： <a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management/" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a></p>

<hr />

<h2>DefaultMQPullConsumer</h2>

<p>采取主动调用Pull接口的模式的消费者，主动权更大，但是使用难度也相对更大。以下介绍其配置，部分配置和PushConsumer一致。</p>

<h3>consumerGroup*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费组的名称，用于标识一类消费者</td>
<td>无默认值，必设</td>
</tr>
</tbody>
</table>


<p>详见 <a href="http://jaskey.github.io/blog/2016/12/15/rocketmq-concept/" title="RocketMQ——角色与术语详解">RocketMQ——角色与术语详解</a></p>

<h3>registerTopics*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费者需要监听的topic</td>
<td>空集合</td>
</tr>
</tbody>
</table>


<p>由于没有subscribe接口，用户需要自己把想要监听的topic设置到此集合中，RocketMQ内部会依靠此来发送对应心跳数据。</p>

<h3>messageModel*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消费模式</td>
<td>MessageModel.CLUSTERING</td>
</tr>
</tbody>
</table>


<p>可选值有两个：</p>

<ol>
<li>CLUSTERING //集群消费模式</li>
<li>BROADCASTING //广播消费模式</li>
</ol>


<p>两种模式的区别详见：<a href="http://jaskey.github.io/blog/2016/12/15/rocketmq-concept/" title="RocketMQ——角色与术语详解">RocketMQ——角色与术语详解</a></p>

<h3>allocateMessageQueueStrategy*</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>负载均衡策略算法</td>
<td>AllocateMessageQueueAveragely（取模平均分配）</td>
</tr>
</tbody>
</table>


<p>见DefaultPushConsumer的说明</p>

<h3>offsetStore</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>消息消费进度存储器 </td>
<td>null</td>
</tr>
</tbody>
</table>


<p>不建议设置，<code>offsetStore</code> 有两个策略：<code>LocalFileOffsetStore</code> 和 <code>RemoteBrokerOffsetStore</code>。</p>

<p>若没有显示设置的情况下，广播模式将使用<code>LocalFileOffsetStore</code>，集群模式将使用<code>RemoteBrokerOffsetStore</code>，不建议修改。</p>

<h3>maxReconsumeTimes</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>调用<code>sendMessageBack</code>的时候，如果发现重新消费超过这个配置的值，则投递到死信队列</td>
<td>16</td>
</tr>
</tbody>
</table>


<p>由于PullConsumer没有管理消费的线程池和管理器，需要用户自己处理各种消费结果和拉取结果，故需要投递到重试队列或死信队列的时候需要显示调用<code>sendMessageBack</code>。</p>

<p>回传消息的时候会带上maxReconsumeTimes的值，broker发现此消息已经消费超过此值，则投递到死信队列，否则投递到重试队列。此逻辑和<code>DefaultPushConsumer</code>是一致的，只是PushConsumer无需用户显示调用。</p>

<h3>brokerSuspendMaxTimeMillis</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>broker在长轮询下，连接最长挂起的时间</td>
<td>20*1000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>长轮询具体逻辑不在本文论述，且RocketMQ不建议修改此值。</p>

<h3>consumerTimeoutMillisWhenSuspend</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>broker在长轮询下，客户端等待broker响应的最长等待超时时间</td>
<td>30*1000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>长轮询具体逻辑不在本文论述，且RocketMQ不建议修改此值，此值一定要大于<code>brokerSuspendMaxTimeMillis</code></p>

<h3>consumerPullTimeoutMillis</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>pull的socket 超时时间</td>
<td>10*1000，单位毫秒</td>
</tr>
</tbody>
</table>


<p>虽然注释上说是socket超时时间，但是从源码上看，此值的设计是不启动长轮询也不指定timeout的情况下，拉取的超时时间。</p>

<h3>messageQueueListener</h3>

<table>
<thead>
<tr>
<th> 配置说明 </th>
<th> 默认值 </th>
</tr>
</thead>
<tbody>
<tr>
<td>负载均衡consume queue分配变化的通知监听器</td>
<td>null</td>
</tr>
</tbody>
</table>


<p>由于pull操作需要用户自己去触发，故如果负载均衡发生变化，要有方法告知用户现在分到的新consume queue是什么。使用方可以实现此接口以达到此目的：</p>

<pre><code>/**
 * A MessageQueueListener is implemented by the application and may be specified when a message queue changed
 */
public interface MessageQueueListener {
/**
 * @param topic message topic
 * @param mqAll all queues in this message topic
 * @param mqDivided collection of queues,assigned to the current consumer
 */
void messageQueueChanged(final String topic, final Set&lt;MessageQueue&gt; mqAll,final Set&lt;MessageQueue&gt; mqDivided);
}
</code></pre>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RocketMQ——消息文件过期原理]]></title>
    <link href="https://Jaskey.github.io/blog/2017/02/16/rocketmq-clean-commitlog/"/>
    <updated>2017-02-16T11:49:23+08:00</updated>
    <id>https://Jaskey.github.io/blog/2017/02/16/rocketmq-clean-commitlog</id>
    <content type="html"><![CDATA[<p><a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management//" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a> 文中提过，所有的消费均是客户端发起Pull请求的，告诉消息的offset位置，broker去查询并返回。但是有一点需要非常明确的是，消息消费后，消息其实<strong>并没有</strong>物理地被清除，这是一个非常特殊的设计。本文来探索此设计的一些细节。</p>

<h2>消费完后的消息去哪里了？</h2>

<p>消息的存储是一直存在于CommitLog中的。而由于CommitLog是以文件为单位（而非消息）存在的，CommitLog的设计是只允许顺序写的，且每个消息大小不定长，所以这决定了消息文件几乎不可能按照消息为单位删除（否则性能会极具下降，逻辑也非常复杂）。所以消息被消费了，消息所占据的物理空间并不会立刻被回收。</p>

<p>但消息既然一直没有删除，那RocketMQ怎么知道应该投递过的消息就不再投递？——答案是客户端自身维护——客户端拉取完消息之后，在响应体中，broker会返回下一次应该拉取的位置，PushConsumer通过这一个位置，更新自己下一次的pull请求。这样就保证了正常情况下，消息只会被投递一次。</p>

<h2>什么时候清理物理消息文件？</h2>

<p>那消息文件到底删不删，什么时候删？</p>

<p>消息存储在CommitLog之后，的确是会被清理的，但是这个清理只会在以下任一条件成立才会批量删除消息文件（CommitLog）：</p>

<ol>
<li>消息文件过期（默认72小时），且到达清理时点（默认是凌晨4点），删除过期文件。</li>
<li>消息文件过期（默认72小时），且磁盘空间达到了水位线（默认75%），删除过期文件。</li>
<li>磁盘已经达到必须释放的上限（85%水位线）的时候，则开始批量清理文件（无论是否过期），直到空间充足。</li>
</ol>


<p>注：若磁盘空间达到危险水位线（默认90%），出于保护自身的目的，broker会拒绝写入服务。</p>

<h2>这样设计带来的好处</h2>

<p>消息的物理文件一直存在，消费逻辑只是听客户端的决定而搜索出对应消息进行，这样做，笔者认为，有以下几个好处：</p>

<ol>
<li><p>一个消息很可能需要被N个消费组（设计上很可能就是系统）消费，但消息只需要存储一份，消费进度单独记录即可。这给强大的消息堆积能力提供了很好的支持——一个消息无需复制N份，就可服务N个消费组。</p></li>
<li><p>由于消费从哪里消费的决定权一直都是客户端决定，所以只要消息还在，就可以消费到，这使得RocketMQ可以支持其他传统消息中间件不支持的回溯消费。即我可以通过设置消费进度回溯，就可以让我的消费组重新像放快照一样消费历史消息；或者我需要另一个系统也复制历史的数据，只需要另起一个消费组从头消费即可（前提是消息文件还存在）。</p></li>
<li><p>消息索引服务。只要消息还存在就能被搜索出来。所以可以依靠消息的索引搜索出消息的各种原信息，方便事后排查问题。</p></li>
</ol>


<p>注：在消息清理的时候，由于消息文件默认是1GB，所以在清理的时候其实是在删除一个大文件操作，这对于IO的压力是非常大的，这时候如果有消息写入，写入的耗时会明显变高。这个现象可以在凌晨4点（默认删时间时点）后的附近观察得到。</p>

<p>RocketMQ官方建议Linux下文件系统改为Ext4，对于文件删除操作相比Ext3有非常明显的提升。</p>

<h2>跳过历史消息的处理</h2>

<p>由于消息本身是没有过期的概念，只有文件才有过期的概念。那么对于很多业务场景——一个消息如果太老，是无需要被消费的，是不合适的。</p>

<p>这种需要跳过历史消息的场景，在RocketMQ要怎么实现呢？</p>

<p>对于一个全新的消费组，PushConsumer默认就是跳过以前的消息而从最尾开始消费的，解析请参看<a href="http://jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management//" title="RocketMQ——消息ACK机制及消费进度管理">RocketMQ——消息ACK机制及消费进度管理</a>相关章节。</p>

<p>但对于已存在的消费组，RocketMQ没有内置的跳过历史消息的实现，但有以下手段可以解决：</p>

<ol>
<li><p>自身的消费代码按照日期过滤，太老的消息直接过滤。如：</p>

<pre><code>     @Override
     public ConsumeConcurrentlyStatus consumeMessage(List&lt;MessageExt&gt; msgs, ConsumeConcurrentlyContext context) {
         for(MessageExt msg: msgs){
             if(System.currentTimeMillis()-msg.getBornTimestamp()&gt;60*1000) {//一分钟之前的认为过期
                 continue;//过期消息跳过
             }

             //do consume here

         }
         return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
     }
</code></pre></li>
<li><p>自身的消费代码代码判断消息的offset和MAX_OFFSET相差很远，认为是积压了很多，直接return CONSUME_SUCCESS过滤。</p>

<pre><code>     @Override
     public ConsumeConcurrentlyStatus consumeMessage(//
         List&lt;MessageExt&gt; msgs, //
         ConsumeConcurrentlyContext context) {
         long offset = msgs.get(0).getQueueOffset();
         String maxOffset = msgs.get(0).getProperty(MessageConst.PROPERTY_MAX_OFFSET);
         long diff = Long. parseLong(maxOffset) - offset;
         if (diff &gt; 100000) { //消息堆积了10W情况的特殊处理
             return ConsumeConcurrentlyStatus. CONSUME_SUCCESS;
         }
         //do consume here
         return ConsumeConcurrentlyStatus. CONSUME_SUCCESS;
     }
</code></pre></li>
<li><p>消费者启动前，先调整该消费组的消费进度，再开始消费。可以人工使用控制台命令resetOffsetByTime把消费进度调整到后面，再启动消费。</p></li>
<li>原理同3，但使用代码来控制。代码中调用内部的运维接口，具体代码实例祥见<code>ResetOffsetByTimeCommand.java</code>.</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RocketMQ——消息ACK机制及消费进度管理]]></title>
    <link href="https://Jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management/"/>
    <updated>2017-01-25T20:49:23+08:00</updated>
    <id>https://Jaskey.github.io/blog/2017/01/25/rocketmq-consume-offset-management</id>
    <content type="html"><![CDATA[<p><a href="http://jaskey.github.io/blog/2016/12/19/rocketmq-rebalance/" title="RokectMQ——水平扩展及负载均衡详解">RokectMQ——水平扩展及负载均衡详解</a> 中剖析过，consumer的每个实例是靠队列分配来决定如何消费消息的。那么消费进度具体是如何管理的，又是如何保证消息成功消费的?（RocketMQ有保证消息肯定消费成功的特性,失败则重试）？</p>

<p>本文将详细解析消息具体是如何ack的，又是如何保证消费肯定成功的。</p>

<p>由于以上工作所有的机制都实现在PushConsumer中，所以本文的原理均只适用于RocketMQ中的PushConsumer即Java客户端中的<code>DefaultPushConsumer</code>。 若使用了PullConsumer模式，类似的工作如何ack，如何保证消费等均需要使用方自己实现。</p>

<p>注：广播消费和集群消费的处理有部分区别，以下均特指集群消费（CLSUTER），广播（BROADCASTING）下部分可能不适用。</p>

<h2>保证消费成功</h2>

<p>PushConsumer为了保证消息肯定消费成功，只有使用方明确表示消费成功，RocketMQ才会认为消息消费成功。中途断电，抛出异常等都不会认为成功——即都会重新投递。</p>

<p>消费的时候，我们需要注入一个消费回调，具体sample代码如下：</p>

<pre><code>    consumer.registerMessageListener(new MessageListenerConcurrently() {
        @Override
        public ConsumeConcurrentlyStatus consumeMessage(List&lt;MessageExt&gt; msgs, ConsumeConcurrentlyContext context) {
            System.out.println(Thread.currentThread().getName() + " Receive New Messages: " + msgs);
            doMyJob();//执行真正消费
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        }
    });
</code></pre>

<p>业务实现消费回调的时候，当且仅当此回调函数返回<code>ConsumeConcurrentlyStatus.CONSUME_SUCCESS</code>，RocketMQ才会认为这批消息（默认是1条）是消费完成的。（具体如何ACK见后面章节）</p>

<p>如果这时候消息消费失败，例如数据库异常，余额不足扣款失败等一切业务认为消息需要重试的场景，只要返回<code>ConsumeConcurrentlyStatus.RECONSUME_LATER</code>，RocketMQ就会认为这批消息消费失败了。</p>

<p>为了保证消息是肯定被至少消费成功一次，RocketMQ会把这批消息重发回Broker（topic不是原topic而是这个消费租的RETRY topic），在延迟的某个时间点（默认是10秒，业务可设置）后，再次投递到这个ConsumerGroup。而如果一直这样重复消费都持续失败到一定次数（默认16次），就会投递到DLQ死信队列。应用可以监控死信队列来做人工干预。</p>

<p>注：</p>

<ol>
<li>如果业务的回调没有处理好而抛出异常，会认为是消费失败当<code>ConsumeConcurrentlyStatus.RECONSUME_LATER</code>处理。</li>
<li>当使用顺序消费的回调<code>MessageListenerOrderly</code>时，由于顺序消费是要前者消费成功才能继续消费，所以没有<code>RECONSUME_LATER</code>的这个状态，只有<code>SUSPEND_CURRENT_QUEUE_A_MOMENT</code>来暂停队列的其余消费，直到原消息不断重试成功为止才能继续消费。</li>
</ol>


<h2>启动的时候从哪里消费</h2>

<p>当新实例启动的时候，PushConsumer会拿到本消费组broker已经记录好的消费进度（consumer offset），按照这个进度发起自己的第一次Pull请求。</p>

<p>如果这个消费进度在Broker并没有存储起来，证明这个是一个全新的消费组，这时候客户端有几个策略可以选择：</p>

<pre><code>CONSUME_FROM_LAST_OFFSET //默认策略，从该队列最尾开始消费，即跳过历史消息
CONSUME_FROM_FIRST_OFFSET //从队列最开始开始消费，即历史消息（还储存在broker的）全部消费一遍
CONSUME_FROM_TIMESTAMP//从某个时间点开始消费，和setConsumeTimestamp()配合使用，默认是半个小时以前
</code></pre>

<p>所以，社区中经常有人问：“为什么我设了<code>CONSUME_FROM_LAST_OFFSET</code>，历史的消息还是被消费了”？ 原因就在于只有全新的消费组才会使用到这些策略，老的消费组都是按已经存储过的消费进度继续消费。</p>

<p>对于老消费组想跳过历史消息需要自身做过滤，或者使用先修改消费进度。示例代码请参看：<a href="http://jaskey.github.io/blog/2017/02/16/rocketmq-clean-commitlog/" title="RocketMQ——消息文件过期原理">RocketMQ——消息文件过期原理</a></p>

<h2>消息ACK机制</h2>

<p>RocketMQ是以consumer group+queue为单位是管理消费进度的，以一个consumer offset标记这个这个消费组在这条queue上的消费进度。</p>

<p>如果某已存在的消费组出现了新消费实例的时候，依靠这个组的消费进度，就可以判断第一次是从哪里开始拉取的。</p>

<p>每次消息成功后，本地的消费进度会被更新，然后由定时器定时同步到broker，以此持久化消费进度。</p>

<p>但是每次记录消费进度的时候，只会把一批消息中最小的offset值为消费进度值，如下图：</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/rocketmq-ack.png" title="message ack" alt="message ack" /></p>

<p>这钟方式和传统的一条message单独ack的方式有本质的区别。性能上提升的同时，会带来一个潜在的重复问题——由于消费进度只是记录了一个下标，就可能出现拉取了100条消息如 2101-2200的消息，后面99条都消费结束了，只有2101消费一直没有结束的情况。</p>

<p>在这种情况下，RocketMQ为了保证消息肯定被消费成功，消费进度职能维持在2101，直到2101也消费结束了，本地的消费进度才能标记2200消费结束了（注：consumerOffset=2201）。</p>

<p>在这种设计下，就有消费大量重复的风险。如2101在还没有消费完成的时候消费实例突然退出（机器断电，或者被kill）。这条queue的消费进度还是维持在2101，当queue重新分配给新的实例的时候，新的实例从broker上拿到的消费进度还是维持在2101，这时候就会又从2101开始消费，2102-2200这批消息实际上已经被消费过还是会投递一次。</p>

<p>对于这个场景，RocketMQ暂时无能为力，所以业务必须要保证消息消费的幂等性，这也是RocketMQ官方多次强调的态度。</p>

<p>实际上，从源码的角度上看，RocketMQ可能是考虑过这个问题的，截止到3.2.6的版本的源码中，可以看到为了缓解这个问题的影响面，<code>DefaultMQPushConsumer</code>中有个配置<code>consumeConcurrentlyMaxSpan</code></p>

<pre><code>/**
 * Concurrently max span offset.it has no effect on sequential consumption
 */
private int consumeConcurrentlyMaxSpan = 2000;
</code></pre>

<p>这个值默认是2000，当RocketMQ发现本地缓存的消息的最大值-最小值差距大于这个值（2000）的时候，会触发流控——也就是说如果头尾都卡住了部分消息，达到了这个阈值就不再拉取消息。</p>

<p>但作用实际很有限，像刚刚这个例子，2101的消费是死循环，其他消费非常正常的话，是无能为力的。一旦退出，在不人工干预的情况下，2101后所有消息全部重复!</p>

<h3>Ack卡进度解决方案</h3>

<p>实际上对于卡住进度的场景，可以选择弃车保帅的方案：把消息卡住那些消息，先ack掉，让进度前移。但要保证这条消息不会因此丢失，ack之前要把消息sendBack回去，这样这条卡住的消息就会必然重复，但会解决潜在的大量重复的场景。 这也是我们公司<strong>自己定制</strong>的解决方案。</p>

<p>   部分源码如下：</p>

<pre><code>class ConsumeRequestWithUnAck implements Runnable {
    final ConsumeRequest consumeRequest;
    final long resendAfterIfStillUnAck;//n毫秒没有消费完，就重发

    ConsumeRequestWithUnAck(ConsumeRequest consumeRequest,long resendAfterIfStillUnAck) {
        this.consumeRequest = consumeRequest;
        this.resendAfterIfStillUnAck = resendAfterIfStillUnAck;
    }

    @Override
    public void run() {
        //每次消费前，计划延时任务，超时则ack并重发
        final WeakReference&lt;ConsumeRequest&gt; crReff = new WeakReference&lt;&gt;(this.consumeRequest);
        ScheduledFuture scheduledFuture=null;
        if(!ConsumeDispatcher.this.ackAndResendScheduler.isShutdown()) {
            scheduledFuture= ConsumeDispatcher.this.ackAndResendScheduler.schedule(new ConsumeTooLongChecker(crReff),resendAfterIfStillUnAck,TimeUnit.MILLISECONDS);
        }
        try{
            this.consumeRequest.run();//正常执行并更新offset
        }
        finally {
            if (scheduledFuture != null) scheduledFuture.cancel(false);//消费结束后,取消任务
        }
    }

}
</code></pre>

<ol>
<li>定义了一个装饰器，把原来的ConsumeRequest对象包了一层。</li>
<li>装饰器中，每条消息消费前都会调度一个调度器，定时触发，触发的时候如果发现消息还存在，就执行sendback并ack的操作。</li>
</ol>


<p>后来RocketMQ显然也发现了这个问题，RocketMQ在3.5.8之后也是采用这样的方案去解决这个问题。只是实现方式上有所不同（事实上我认为RocketMQ的方案还不够完善）</p>

<ol>
<li>在pushConsumer中 有一个<code>consumeTimeout</code>字段（默认15分钟），用于设置最大的消费超时时间。消费前会记录一个消费的开始时间，后面用于比对。</li>
<li>消费者启动的时候，会定期扫描所有消费的消息，达到这个timeout的那些消息，就会触发sendBack并ack的操作。这里扫描的间隔也是consumeTimeout（单位分钟）的间隔。</li>
</ol>


<p>核心源码如下：</p>

<pre><code>//ConsumeMessageConcurrentlyService.java
public void start() {
    this.CleanExpireMsgExecutors.scheduleAtFixedRate(new Runnable() {

        @Override
        public void run() {
            cleanExpireMsg();
        }

    }, this.defaultMQPushConsumer.getConsumeTimeout(), this.defaultMQPushConsumer.getConsumeTimeout(), TimeUnit.MINUTES);
}
//ConsumeMessageConcurrentlyService.java
private void cleanExpireMsg() {
    Iterator&lt;Map.Entry&lt;MessageQueue, ProcessQueue&gt;&gt; it =
            this.defaultMQPushConsumerImpl.getRebalanceImpl().getProcessQueueTable().entrySet().iterator();
    while (it.hasNext()) {
        Map.Entry&lt;MessageQueue, ProcessQueue&gt; next = it.next();
        ProcessQueue pq = next.getValue();
        pq.cleanExpiredMsg(this.defaultMQPushConsumer);
    }
}

//ProcessQueue.java
public void cleanExpiredMsg(DefaultMQPushConsumer pushConsumer) {
    if (pushConsumer.getDefaultMQPushConsumerImpl().isConsumeOrderly()) {
        return;
    }

    int loop = msgTreeMap.size() &lt; 16 ? msgTreeMap.size() : 16;
    for (int i = 0; i &lt; loop; i++) {
        MessageExt msg = null;
        try {
            this.lockTreeMap.readLock().lockInterruptibly();
            try {
                if (!msgTreeMap.isEmpty() &amp;&amp; System.currentTimeMillis() - Long.parseLong(MessageAccessor.getConsumeStartTimeStamp(msgTreeMap.firstEntry().getValue())) &gt; pushConsumer.getConsumeTimeout() * 60 * 1000) {
                    msg = msgTreeMap.firstEntry().getValue();
                } else {

                    break;
                }
            } finally {
                this.lockTreeMap.readLock().unlock();
            }
        } catch (InterruptedException e) {
            log.error("getExpiredMsg exception", e);
        }

        try {

            pushConsumer.sendMessageBack(msg, 3);
            log.info("send expire msg back. topic={}, msgId={}, storeHost={}, queueId={}, queueOffset={}", msg.getTopic(), msg.getMsgId(), msg.getStoreHost(), msg.getQueueId(), msg.getQueueOffset());
            try {
                this.lockTreeMap.writeLock().lockInterruptibly();
                try {
                    if (!msgTreeMap.isEmpty() &amp;&amp; msg.getQueueOffset() == msgTreeMap.firstKey()) {
                        try {
                            msgTreeMap.remove(msgTreeMap.firstKey());
                        } catch (Exception e) {
                            log.error("send expired msg exception", e);
                        }
                    }
                } finally {
                    this.lockTreeMap.writeLock().unlock();
                }
            } catch (InterruptedException e) {
                log.error("getExpiredMsg exception", e);
            }
        } catch (Exception e) {
            log.error("send expired msg exception", e);
        }
    }
}
</code></pre>

<p>通过这个逻辑对比我定制的时间，可以看出有几个不太完善的问题：</p>

<ol>
<li>消费timeout的时间非常不精确。由于扫描的间隔是15分钟，所以实际上触发的时候，消息是有可能卡住了接近30分钟（15*2）才被清理。</li>
<li>由于定时器一启动就开始调度了，中途这个consumeTimeout再更新也不会生效。</li>
</ol>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RocketMQ——水平扩展及负载均衡详解]]></title>
    <link href="https://Jaskey.github.io/blog/2016/12/19/rocketmq-rebalance/"/>
    <updated>2016-12-19T20:49:23+08:00</updated>
    <id>https://Jaskey.github.io/blog/2016/12/19/rocketmq-rebalance</id>
    <content type="html"><![CDATA[<p>RocketMQ是一个分布式具有高度可扩展性的消息中间件。本文旨在探索在broker端，生产端，以及消费端是如何做到横向扩展以及负载均衡的。</p>

<h2>Broker端水平扩展</h2>

<h3>Broker负载均衡</h3>

<p>Broker是以group为单位提供服务。一个group里面分master和slave,master和slave存储的数据一样，slave从master同步数据（同步双写或异步复制看配置）。</p>

<p>通过nameserver暴露给客户端后，只是客户端关心（注册或发送）一个个的topic路由信息。路由信息中会细化为message queue的路由信息。而message queue会分布在不同的broker group。所以对于客户端来说，分布在不同broker group的message queue为成为一个服务集群，但客户端会把请求分摊到不同的queue。</p>

<p>而由于压力分摊到了不同的queue,不同的queue实际上分布在不同的Broker group，也就是说压力会分摊到不同的broker进程，这样消息的存储和转发均起到了负载均衡的作用。</p>

<p>Broker一旦需要横向扩展，只需要增加broker group，然后把对应的topic建上，客户端的message queue集合即会变大，这样对于broker的负载则由更多的broker group来进行分担。</p>

<p>并且由于每个group下面的topic的配置都是独立的，也就说可以让group1下面的那个topic的queue数量是4，其他group下的topic queue数量是2，这样group1则得到更大的负载。</p>

<h3>commit log</h3>

<p>虽然每个topic下面有很多message queue，但是message queue本身并不存储消息。真正的消息存储会写在CommitLog的文件，message queue只是存储CommitLog中对应的位置信息，方便通过message queue找到对应存储在CommitLog的消息。</p>

<p>不同的topic，message queue都是写到相同的CommitLog 文件，也就是说CommitLog完全的顺序写。</p>

<p>具体如下图：</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/broker-loadbalance.png" title="broker负载均衡" alt="broker负载均衡" /></p>

<h2>Producer</h2>

<p>Producer端，每个实例在发消息的时候，默认会轮询所有的message queue发送，以达到让消息平均落在不同的queue上。而由于queue可以散落在不同的broker，所以消息就发送到不同的broker下，如下图：</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/producer-loadbalance.png" title="生产者负载均衡" alt="生产者负载均衡" /></p>

<h2>Consumer负载均衡</h2>

<h3>集群模式</h3>

<p>在集群消费模式下，每条消息只需要投递到订阅这个topic的Consumer Group下的一个实例即可。RocketMQ采用主动拉取的方式拉取并消费消息，在拉取的时候需要明确指定拉取哪一条message queue。</p>

<p>而每当实例的数量有变更，都会触发一次所有实例的负载均衡，这时候会按照queue的数量和实例的数量平均分配queue给每个实例。</p>

<p>默认的分配算法是AllocateMessageQueueAveragely，如下图：</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/consumer-loadbalance1.png" title="消费者负载均衡1" alt="消费者负载均衡1" /></p>

<p>还有另外一种平均的算法是AllocateMessageQueueAveragelyByCircle，也是平均分摊每一条queue，只是以环状轮流分queue的形式，如下图：</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/consumer-loadbalance2.png" title="消费者负载均衡2" alt="消费者负载均衡2" /></p>

<p>需要注意的是，集群模式下，queue都是只允许分配只一个实例，这是由于如果多个实例同时消费一个queue的消息，由于拉取哪些消息是consumer主动控制的，那样会导致同一个消息在不同的实例下被消费多次，所以算法上都是一个queue只分给一个consumer实例，一个consumer实例可以允许同时分到不同的queue。</p>

<p>通过增加consumer实例去分摊queue的消费，可以起到水平扩展的消费能力的作用。而有实例下线的时候，会重新触发负载均衡，这时候原来分配到的queue将分配到其他实例上继续消费。</p>

<p>但是如果consumer实例的数量比message queue的总数量还多的话，多出来的consumer实例将无法分到queue，也就无法消费到消息，也就无法起到分摊负载的作用了。所以需要控制让queue的总数量大于等于consumer的数量。</p>

<h3>广播模式</h3>

<p>由于广播模式下要求一条消息需要投递到一个消费组下面所有的消费者实例，所以也就没有消息被分摊消费的说法。</p>

<p>在实现上，其中一个不同就是在consumer分配queue的时候，会所有consumer都分到所有的queue。</p>

<p><img src="https://Jaskey.github.io/images/rocketmq/consumer-broadcast.png" title="消费者广播模式" alt="消费者广播模式" /></p>
]]></content>
  </entry>
  
</feed>