Error only happening with `swift test` and not happening with Xcode

I have a Swift project including a test target, on GitHub: https://github.com/mshibanami/GitHubTrendingRSS

When I ran swift test for that, then I got errors like this:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:20:10: error: could not build module 'Dispatch'
#include <dispatch/dispatch.h>

This is the actual log: https://github.com/mshibanami/GitHubTrendingRSS/runs/431904961?check_suite_focus=true

However, this never happens on Xcode with generated xcodeproj by swift package generate-xcodeproj.
Also, when I went through these steps, then swift test tests all cases precisely:

  1. Remove the dependencies from testTarget in Package.swift
  2. Run swift test and get ld: symbol(s) not found for architecture x86_64 error
  3. Restore the removed dependencies
  4. Run swift test again

But since the errors are raised again when I ran swift clean, it doesn't completely resolve this problem.

What is the problem and how can I fix this?

The first error in the log is actually this:

/Users/runner/runners/2.164.0/work/GitHubTrendingRSS/GitHubTrendingRSS/.build/checkouts/Down/Source/cmark/include/module.modulemap:1:8: error: redefinition of module 'libcmark'
module libcmark [system][extern_c] {
       ^
/Users/runner/runners/2.164.0/work/GitHubTrendingRSS/GitHubTrendingRSS/.build/x86_64-apple-macosx/debug/libcmark.build/module.modulemap:1:8: note: previously defined here
module libcmark {
       ^

The following seemed to work:

swift package edit Down
rm Packages/Down/Source/cmark/include/module.modulemap
swift build --build-tests