Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

加速镜像可用性和稳定性 #71

Open
xuchaoxin1375 opened this issue Oct 25, 2024 · 2 comments
Open

加速镜像可用性和稳定性 #71

xuchaoxin1375 opened this issue Oct 25, 2024 · 2 comments

Comments

@xuchaoxin1375
Copy link

在镜像加速项目下许多人贡献了加速镜像,建议本项目能够将加速镜像的选择上做更灵活的处理

因为我发现ghp.ci的可靠性不好,很多情况下都下不动,下面的issue提供了许多筛选可用加速站点的方法,包括powershell实现的方案

例如

$GithubMirrors = @(
    # 注意,如果你的浏览器使用了代理,那么部分镜像站会和代理冲突,所以可能命令行中测试可以访问的链接在浏览器中确打不开镜像站,可以关闭代理或换个浏览器后重新验证该镜像站的可用性
    #搜集到的链接可以用gpt进行修改,将链接包裹引号(指令:为这些链接分别添加引号);或者自己粘贴到文件文件中然后编写脚本转换为数组格式
    'https://github.moeyy.xyz', # Moeyy - 提供 GitHub 文件的加速下载功能
    'https://ghproxy.cc',
    'https://ghproxy.net', # GitHub Proxy by GitHub99 - 提供 GitHub 文件的加速下载和克隆服务 #和用户自己的代理可能发生冲突
    'https://mirror.ghproxy.com',
    
    # 'https://ghproxy.com/bad/demo', #虚假镜像,用来测试代码是否能够正确处理不可用的镜像链接
    'https://ghproxy.homeboyc.cn/',
    'https://gh-proxy.com/',
    
    'https://gh.ddlc.top',
    'https://github.ur1.fun/',

    '' #空字符串收尾
)
$GithubMirrorsTest = @(
    'https://gh-proxy.com/',
    'https://ghps.cc/',
    'https://ghproxy.net/',
    'https://github.moeyy.xyz/',
    'https://gh.ddlc.top/',
    'https://slink.ltd/',
    'https://gh.con.sh/',
    'https://hub.gitmirror.com/',
    'https://sciproxy.com/',
    'https://cf.ghproxy.cc/',
    'https://gh.noki.icu/',

    ''#收尾
)
#如果你懒得添加引号,那么将镜像链接逐个添加到下面的多行字符串中,即便包含了引号或者双引号逗号也都能够正确处理

$GithubMirrorsInString = @'
https://sciproxy.com/
https://cf.ghproxy.cc/, # comment demo
'https://gh.noki.icu/',
""https://sciproxy.com/""
https://demo.testNew.com/
'https://slink.ltd/'

'@

$GithubMirrorsInString = $GithubMirrorsInString -replace '#.*', ' ' -replace '[",;\n\r]', ' ' -replace "'" , ' ' -replace '\s+', ' '

$GithubMirrorsInString = $GithubMirrorsInString -split ' ' #去重等操作留到后面一起处理

$GithubMirrors = $GithubMirrors + $GithubMirrorsTest + $GithubMirrorsInString
$GithubMirrors = $GithubMirrors | Where-Object { $_ }#移除空串
$GithubMirrors = $GithubMirrors | ForEach-Object { $_.trim('/') } | Sort-Object #统一链接风格(去除末尾的`/`如果有的话)
$GithubMirrors = $GithubMirrors | Select-Object -Unique # Get-Unique #移除重复条目(注意get-Unique要求被处理数组是有序的,否则无效,可以用select -unique更通用)

function Test-LinksLinearly
{
    <# 
    .SYNOPSIS
    线性地(串行地)测试链接是否能够在指定时间内响应,为powershell5 设计
    .NOTES
    链接数量多的话会造成测试时间很长,尽量使用并行方案(pwsh7),或者考虑设置小的$timeoutSec=1
    #>
    [cmdletbinding(DefaultParameterSetName = 'First')]
    param (
        $Mirrors = $GithubMirrors,
        $TimeOutSec = 4,
        [parameter(ParameterSetName = 'First')]
        $First = 5,
        [parameter(ParameterSetName = 'All')]
        [Alias('Full')]
        [switch]
        $All
    )
    $availableMirrors = @()
    
    foreach ($mirror in $Mirrors)
    {
        # $Mirrors | ForEach-Object {
        # $mirror = $_

        # Write-Verbose "Testing $mirror..."
        if (Test-MirrorAvailability -Url $mirror -TimeoutSec $TimeOutSec)
        {
            Write-Verbose "$mirror is available "
            Write-Host "`t $mirror" -ForegroundColor Green
            # 插入到数组中(这里如果foreach用了-parallel,就会导致无法访问外部的$availableMirros)
            $availableMirrors += $mirror
        }
        else
        {
            Write-Verbose "$mirror is not available "
            Write-Host "`t $mirror " -ForegroundColor Red
        }

        if ($pscmdlet.ParameterSetName -eq 'First')
        {

            if (($availableMirrors.Count -ge $First))
            {
                break #在foreach-object中会直接停止函数的运行,而使用传统foreach则是正常的
            }
        }
    } 
    if ($availableMirrors.Count -eq 0)
    {
        throw 'No mirrors are available!'
    }
    return $availableMirrors
}
function Test-LinksParallel
{
    <# 
    .SYNOPSIS
    为powershell 7+设计的并行测试链接是否能够在指定时间内响应
    #>
    [CmdletBinding()]
    param (
        $Mirrors = $GithubMirrors,
        $TimeOutSec = 2,
        $ThrottleLimits = 16
        # $First = 5
    )
    # 如果不是powershell 7报错
    if ($host.Version.Major -lt 7)
    {
        Throw 'PowerShell 7 or higher is required to run parallel foreach!'
        # return 
    }
    $availableMirrors = @()
    # 为了能够让$TimeOutSec能够被传递到子进程,这里使用了$env:来扩大其作用域
    # $env:TimeOutSec = $TimeOutSec
    # powershell提供了更好的方式访问并行scriptblock外的变量,使用$using: 这个关键字
    #然而这个关键字引用的变量无法更改(只读),可以考虑用.Net线程安全容器,或者用$env:来实现共享局部环境变量
    # $Envbak = $env:StopLoop
    # $env:StopLoop = 0
    # 创建线程安全容器(队列)
    $mirs = [System.Collections.Concurrent.ConcurrentQueue[Object]]::new()
    # $mirs.Enqueue('First_Demo')
    # Write-Host $mirs
    # 并行执行链接测试
    $Mirrors | ForEach-Object -Parallel {
        # if ([int]$env:StopLoop)
        # {
        #     return
        # }
        # Write-verbose $_
        #引用外部变量,并且赋值给简化的临时变量,方便后续引用(直接在-Parallel中引用外部变量是不合期望的)
        $mirs = $using:mirs
        $TimeOutSec = $using:TimeOutSec
        # $First = $using:First
        #  并行方案里用First参数指定前n个意义不大,而且会让代码变得复杂
        # Write-Verbose "mirs.cout=$($mirs.Count)" -Verbose
        # if ($mirs.Count -ge $First)
        # {
        #     # Write-Host $First
        #     Write-Verbose "The available links enough the $First !" -Verbose
        #     return
        # }
         

        $mirror = $_
        # Write-Debug "`$TimeOutSec=$env:TimeOutSec" -Debug #parallel 参数$DebugPreference无法起作用
        # if (Test-MirrorAvailability -Url $mirror -TimeoutSec $env:TimeOutSec)
        # 测试链接是否可用
        if (Test-MirrorAvailability -Url $mirror -TimeoutSec $TimeOutSec)
        {
            Write-Host "`t $_" -ForegroundColor Green
            # Write-Output $mirror

            #写入队列
            $mirs.Enqueue($mirror)
            # 查看$mirs队列长度
            # $mirs.Count, $mirs

        }
        else
        {
            Write-Verbose "$mirror is not available "
            Write-Host "`t $mirror." -ForegroundColor Red
        }

    } -ThrottleLimit $ThrottleLimits 

    $availableMirrors = $mirs #.ToArray()
    if ($availableMirrors.Count -eq 0)
    {
        throw 'No mirrors are available!'
    }
    return $availableMirrors
}
function Test-MirrorAvailability
{
    <# 
    .SYNOPSIS
    测试指定链接是否在规定时间内相应
    .NOTES
    此函数主要用来辅助Test-LinksLinearly和Test-LinksParallel调用
    .DESCRIPTION
    如果及时正确相应,将链接打印为绿色,否则打印为红色
    #>
    [CmdletBinding()]
    param (
        [string]$Url,
        $TimeoutSec = 3
    )

    try
    {
        # 使用 Invoke-WebRequest 检查可用性
        $response = Invoke-WebRequest -Uri $Url -Method Head -TimeoutSec $TimeOutSec -ErrorAction Stop
        
        $availability = $response.StatusCode -eq 200
    }
    catch
    {
        $availability = $false
    }
    if ($VerbosePreference)
    {

        if ($availability)
        {

            Write-Host "Mirror $Url is available" -ForegroundColor Green
        }
        else
        {

            Write-Host "Mirror $Url is not available" -ForegroundColor Red
        }
    }
    return   $availability
}
function Get-AvailableGithubMirrors
{
    <#
    .SYNOPSIS
    列出流行的或可能可用的 GitHub 加速镜像站。
    列表中的镜像站可能会过期,可用性不做稳定性和可用性保证。

    .DESCRIPTION
    这里采用了多线程的方式来加速对不同镜像链接的可用性进行检查
    并且更容易获取其中相应最快的可用的镜像站,这是通过串行检查无法直接达到的效果
    .EXAMPLE
    

 .NOTES
    推荐使用 aria2 等多线程下载工具来加速下载,让镜像加速效果更加明显。
 
    .LINK
    # 镜像站搜集和参考
    https://github-mirror.us.kg/
    https://github.com/hunshcn/gh-proxy/issues/116
    #>
    [CmdletBinding()]
    param(
        $Mirrors = $GithubMirrors,
        $ThrottleLimits = 16,
        $TimeOutSec = 3,
        [switch]$ListView,
        [switch]$PassThru,
        [switch]$SkipCheckAvailability,
        # 是否启用串行地试探镜像可访问性(默认是并行试探)
        [switch]
        # [parameter(ParameterSetName = 'Serial')]
        [Alias('Serial')]$Linearly,
        # [parameter(ParameterSetName = 'Serial')]
        $First = 5
    )
    
    # 检查镜像站的可用性
   

    Write-Host 'Checking available Mirrors...'
    $availableMirrors = $Mirrors
    # 检查可用的镜像列表
    if (!$SkipCheckAvailability)
    {
        $psVersion = $host.Version.Major 
        # 默认尝试并行测试
        if ($psVersion -lt 7 -and !$Linearly)
        {

            Write-Host 'PowerShell 7 or higher is required to run parallel foreach!' -ForegroundColor Red
            Write-Host 'Testing Links Linearly...'
            $Linearly = $true
        }
        if ($Linearly ) #-or $PSVersion -lt 7
        {
            #简单起见,这里仅简单调用 Test-LinksLinearly的Frist参数集语法,而不做分支判断
            $availableMirrors = Test-LinksLinearly -Mirrors $Mirrors -TimeOutSec $TimeOutSec -First $First -Verbose:$VerbosePreference
        }
        else
        {
        
            $availableMirrors = Test-LinksParallel -Mirrors $Mirrors -TimeOutSec $TimeOutSec -ThrottleLimits $ThrottleLimits -Verbose:$VerbosePreference 
        }
    } 

    # Start-Sleep $TimeOutSec
    # 显示可用的镜像
    Write-Host "`nAvailable Mirrors:"
    # 空白镜像保留(作为返回值)
    $availableMirrors = @('') + $availableMirrors

    # 按序号列举展示
    Write-Host ' 0: Use No Mirror' -NoNewline
    $index = 1
    $availableMirrors | ForEach-Object {
        # $index = [array]::IndexOf($availableMirrors, $_)
        # if($availableMirrors[$index] -eq ""){continue}
        if ($_.Trim())
        {

            Write-Host " ${index}: $_" -NoNewline
            $index += 1
        }
   
        
        Write-Host ''
    }

    if ($PassThru)
    {
        return $availableMirrors
    }
}


function Get-SelectedMirror
{
    <# 
    .SYNOPSIS
    让用户选择可用的镜像连接,允许选择多个
    .NOTES
    包含单个字符串的数组被返回时会被自动解包,这种情况下会是一个字符串
    如果却是需要外部接受数组,那么可以在外部使用@()来包装返回结果即可

    #>
    [CmdletBinding()]
    param (
        
        $Default = 1, # 默认选择第一个(可能是响应最快的)
        [switch]$Linearly,
        [switch]$Silent # 是否静默模式,不询问用户,返回第$Default个链接($Default默认为1)
    )
    $Mirrors = Get-AvailableGithubMirrors -PassThru -Linearly:$Linearly

    $res = @()
    if (!$Silent)
    {
        # 交互模式
        $numOfMirrors = $Mirrors.Count
        $range = "[0~$($numOfMirrors-1)]"
        $num = Read-Host -Prompt "Select the number(s) of the mirror you want to use $range ?(default: $default)"
        # $mirror = 'https://mirror.ghproxy.com'
        # if($num.ToCharArray() -contains ','){
        # }

        $numStrs = $num -split ',' | Where-Object { $_.Trim() } | Get-Unique #转换为数组(自动去除空白字符)
        # 如果$num是一个空字符串(Read-Host遇到直接回车的情况),那么$numStrs会是$null
        if (!$numStrs )
        {
            Write-Host 'choose the Default 1'
            # $n = $default
            $res += $Default
        }
        else
        {
            foreach ($num in $numStrs)
            {
                $n = $num -as [int] #可能是数字或者空$null
                if ($VerbosePreference)
                {
            
                    Write-Verbose "`$n=$n"
                    Write-Verbose "`$num=$num"
                    Write-Verbose "`$numOfMirrors=$numOfMirrors"
                }
   
                #  如果输入的是空白字符,则默认设置为0
                # if ( $num.trim().Length -eq 0)
       
                if ($n -notin 0..($numOfMirrors - 1))
                {
                    Throw " Input a number within the range! $range"
                }
                else
                {
                    # 合法的序号输入,插入到$res
                    $res += $n
                }
            }
        }
    }
    elseif ($Silent)
    {
        # Silent模式下默认选择第1个镜像
        $res += $default
    }
    # 抽取镜像
    $mirrors = $Mirrors[$res] #利用pwsh的数组高级特性
    # Write-Host $mirrors -ForegroundColor Blue
    $mirrors = @($mirrors) #确保其为数组
    
    # 用户选择了一个合法的镜像代号(0表示不使用镜像)
    Write-Host 'Selected mirror:[ ' # -NoNewline
    foreach ($mir in $mirrors)
    {
        Write-Host "`t$mir" -BackgroundColor Gray -NoNewline
        Write-Host ''

    }
    # Write-Host "$($Mirrors[$n])" -BackgroundColor Gray -NoNewline
    Write-Host ']'#打印一个空行

    # 包含单个字符串的数组被返回时会被自动解包,这种情况下会是一个字符串
    #如果却是需要外部接受数组,那么可以在外部使用@()来包装返回结果即可
    return $mirrors
    # return [array]$Mirrors
    # return $res

}


代码的最新版本和相关命令:[PS/Deploy/deploy.psm1 · xuchaoxin1375/scripts - Gitee.com](https://gitee.com/xuchaoxin1375/scripts/blob/main/PS/Deploy/deploy.psm1)

@xuchaoxin1375
Copy link
Author

xuchaoxin1375 commented Oct 25, 2024

或者考虑适配此项目:[Scoop: scoop国内镜像优化库,能够加速scoop安装及bucket源文件,无需用户设置代理。](https://gitee.com/scoop-installer/scoop)

用户可以选择分支,从而得到非加速得到原始github连接,用户借由gitee上的加速方案来加速

@renxia
Copy link
Member

renxia commented Oct 28, 2024

作为一个应用仓库可做的比较有限。
https://gitee.com/scoop-installer/scoop 是个不错的方案,可以考虑去提 issue 增加如下支持:

  • 支持自选/环境变量配置自定义代理地址;
  • 支持检测已存在的代理前缀,并替换为自定义地址。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants