[Bug/Feature] Weibo keyword search only crawls deduplicated results; unable to crawl "all search results" after similar results are omitted

[Sicheng-Zeng]

Description

When using the Weibo keyword search crawler, the current implementation appears to crawl only the default search result set returned by the mobile Weibo search API. However, Weibo search may omit similar results by default.

For example, when searching the keyword:

镇干部钓鱼溺亡

the crawler stops at page 17 with the following API response:

{'ok': 0, 'msg': '这里还没有内容', 'data': {'cards': []}}

The crawler then retries several times and finally exits with:

tenacity.RetryError: RetryError[<Future ... raised DataFetchError>]

At first this looks like the crawler has reached the real end of search results. But after manually opening the Weibo web search page in the browser, the bottom of the page shows:

找到166条结果，部分相似结果已省略，您可以点击查看全部搜索结果

This exactly corresponds to around 16 pages of results. In other words, the crawler is currently collecting only the default deduplicated / similar-results-omitted result set, not the complete result set available after clicking "查看全部搜索结果".

Current behavior

The crawler uses the mobile Weibo search API:

https://m.weibo.cn/api/container/getIndex

with params similar to:

containerid=100103type=1&q={keyword}
page_type=searchall
page={page}

When Weibo returns:

{"ok": 0, "msg": "这里还没有内容", "data": {"cards": []}}

the crawler treats it as an error and exits after retrying.

Expected behavior

It would be helpful if the Weibo crawler could support crawling the full search result set, equivalent to manually clicking:

查看全部搜索结果

on the Weibo web search page.

Possible expected behavior:

1. Add a configuration option such as:

WEIBO_SEARCH_FULL_RESULT = True

2. When enabled, use the full-result search mode, probably by adding / supporting the nodup parameter or by using the corresponding web search endpoint.
3. When the search API returns "这里还没有内容", treat it as "no more results" instead of a fatal error, so that the crawler can stop the current keyword gracefully and continue with the next keyword.

Reproduction steps

1. Set platform to Weibo.
2. Use keyword search mode.
3. Set keyword:

KEYWORDS = "镇干部钓鱼溺亡"

4. Set a large enough max crawl count, e.g. more than 200.
5. Run the crawler.
6. The crawler successfully crawls pages 1-16, but page 17 returns "这里还没有内容" and exits with RetryError.
7. Manually opening the Weibo web search page shows that only similar-results-omitted results are displayed by default, and there is an option to view all search results.

Relevant log

[WeiboCrawler.search] search weibo keyword: 镇干部钓鱼溺亡, page: 17

[WeiboClient.request] request GET:
https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%E9%95%87%E5%B9%B2%E9%83%A8%E9%92%93%E9%B1%BC%E6%BA%BA%E4%BA%A1&page_type=searchall&page=17
err, res:{'ok': 0, 'msg': '这里还没有内容', 'data': {'cards': []}}

media_platform.weibo.exception.DataFetchError: 这里还没有内容

tenacity.RetryError: RetryError[<Future ... raised DataFetchError>]

Suggested fix

Possible directions:

1. Add a Weibo search config option to enable full search results.
2. Try adding the nodup parameter to Weibo search requests.
3. If the mobile API does not support this parameter reliably, add a fallback mode using the PC web search endpoint, such as s.weibo.com/weibo, and parse result mids before fetching post details through the existing detail method.
4. Gracefully handle "这里还没有内容" as end-of-results instead of throwing a fatal exception.

Environment

• Platform: Weibo
• Crawler type: search
• Python version: 3.11
• Running on Windows
• Login mode: existing browser / CDP mode

[dosubot]

Hi @Sicheng-Zeng! I'm Dosu and I'm helping the Relakkes's Org team.

Great bug report — I can confirm the behavior you described by looking at the code. The root cause is in the request() method in media_platform/weibo/client.py, which treats any response with ok == 0 as a fatal error and raises DataFetchError [1]. Combined with the @retry(stop=stop_after_attempt(5), wait=wait_fixed(3)) decorator [2], this means the "这里还没有内容" response triggers 5 retries then a tenacity.RetryError, crashing the crawler.

The search API call in get_note_by_keyword currently constructs the request with only containerid, page_type, and page — there is no nodup parameter or any mechanism to request full (non-deduplicated) results [3].

Regarding your suggested fixes:

1. Graceful end-of-results handling — This is the most straightforward fix. The request() method could check whether ok == 0 and the message is "这里还没有内容", and instead of raising DataFetchError, return an empty result (or a sentinel value) so the search loop in WeiboCrawler.search() can break cleanly and move to the next keyword.

2. nodup=1 parameter — On the Weibo web search (s.weibo.com), appending &nodup=1 to the URL disables duplicate filtering and returns all results. It's worth testing whether adding "nodup": 1 to the params dict in get_note_by_keyword works on the mobile API as well. If it does, this would be the simplest path to full results.

3. Fallback to PC web endpoint — If the mobile API doesn't honor nodup, using s.weibo.com/weibo?q={keyword}&nodup=1 with HTML parsing would be more complex but reliable.

A minimal patch addressing both issues might look like:

# In client.py - get_note_by_keyword
async def get_note_by_keyword(self, keyword, page=1, search_type=SearchType.DEFAULT):
    uri = "/api/container/getIndex"
    containerid = f"100103type={search_type.value}&q={keyword}"
    params = {
        "containerid": containerid,
        "page_type": "searchall",
        "page": page,
        "nodup": 1,  # Request full results without deduplication
    }
    return await self.get(uri, params)

# In client.py - request method, replace the ok_code == 0 block
if ok_code == 0:
    msg = data.get("msg", "")
    if msg == "这里还没有内容":
        # End of search results - return empty data instead of raising
        return data.get("data", {})
    utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
    raise DataFetchError(msg or "response error")

The search loop in WeiboCrawler.search() would also need a check for empty cards to break cleanly [4].

Hope this helps clarify the issue and potential solutions!

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.