curl CloudFlare在哪里检测Web和终端请求？！在平等的条件下

bxgwgixi  于 2023-08-06  发布在  其他
关注(0)|答案(2)|浏览(84)

我的问题是在CLI中打开CloudFlare上的站点。
我并不是说当有挑战，我不想解决挑战。
请以这个网站为例：https://pegaxy.io
在新安装的任何Web浏览器上首次打开时。打开时没有任何问题。接收到代码200。
x1c 0d1x的数据
但是当我点击Copy as cURL并在终端中得到403错误时。


的
curl 代码：

curl 'https://pegaxy.io/' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.5' \
  -H 'Connection: keep-alive' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: none' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Pragma: no-cache' \
  -H 'Cache-Control: no-cache' \
  --compressed --verbose

字符串
记录档：

$ curl 'https://pegaxy.io/' \
>   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
>   -H 'Upgrade-Insecure-Requests: 1' \
>   -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
>   -H 'Accept-Language: en-US,en;q=0.5' \
>   -H 'Connection: keep-alive' \
>   -H 'Upgrade-Insecure-Requests: 1' \
>   -H 'Sec-Fetch-Dest: document' \
>   -H 'Sec-Fetch-Mode: navigate' \
>   -H 'Sec-Fetch-Site: none' \
>   -H 'Sec-Fetch-User: ?1' \
>   -H 'Pragma: no-cache' \
>   -H 'Cache-Control: no-cache' \
>   --compressed --verbose
*   Trying 172.67.10.157:443...
* Connected to pegaxy.io (172.67.10.157) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.pegaxy.io
*  start date: Mar  3 05:22:24 2022 GMT
*  expire date: Jun  1 05:22:23 2022 GMT
*  subjectAltName: host "pegaxy.io" matched cert's "pegaxy.io"
*  issuer: C=US; O=Let's Encrypt; CN=E1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x671500)
> GET / HTTP/2
> Host: pegaxy.io
> accept-encoding: deflate, gzip
> user-agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> accept-language: en-US,en;q=0.5
> connection: keep-alive
> upgrade-insecure-requests: 1
> sec-fetch-dest: document
> sec-fetch-mode: navigate
> sec-fetch-site: none
> sec-fetch-user: ?1
> pragma: no-cache
> cache-control: no-cache
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Fri, 18 Mar 2022 13:03:04 GMT
< content-type: text/html; charset=UTF-8
< cache-control: max-age=15
< expires: Fri, 18 Mar 2022 13:03:19 GMT
< x-frame-options: SAMEORIGIN
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=rlU7vb3eTQzw02vcpzOo6gweMMadJXkNxsft3MqPLSY-1647608584-0-AXk3yx3EOmlDZ+tIGWB3S+1ud6hWmykBwT7IwKtO+e+eCdY36JjTgyM3SkdIyBeWvtZphzvnBZLCVE4R6YogbxI=; path=/; expires=Fri, 18-Mar-22 13:33:04 GMT; domain=.pegaxy.io; HttpOnly; Secure; SameSite=None
< vary: Accept-Encoding
< server: cloudflare
< cf-ray: 6ede2a920d7392c5-FRA
< content-encoding: gzip
<
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->

</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>
        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> pegaxy.io</h2>
      </div><!-- /.header -->

      <div class="cf-section cf-highlight">
        <div class="cf-wrapper">
          <div class="cf-screenshot-container cf-screenshot-full">

              <span class="cf-no-screenshot error"></span>

          </div>
        </div>
      </div><!-- /.captcha-container -->

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting
a certain word or phrase, a SQL command or malformed data.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>

            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
          </div>
        </div>
      </div><!-- /.section -->

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6ede2a920d7392c5</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 46.62.217.20</span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->

    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};

</script>

</body>
</html>
* Connection #0 to host pegaxy.io left intact

型
已在windows和linux上进行测试。

$ curl -V
curl 7.70.0 (x86_64-w64-mingw32) libcurl/7.70.0 OpenSSL/1.1.1g (Schannel) zlib/1.2.11 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP

型
注意，谈论的恰恰是第一个请求。因此，这不能与cookie或一段检查浏览器状态的JavaScript代码有关。
此外，我甚至意识到CloudFlare对IP和请求率的敏感性，但这些都是针对请求是连续的和令人困惑的，并最终显示的挑战。但在我的测试中，Web上的相同IP不是问题，只会发生第一个请求。
在Chrome、Firefox、Edge、Brave、Tor网页浏览器上没有问题。
但是CURL、wget、nghttp和lynx在命令行上存在问题。我还测试了几个有问题的nodejs包。

问题：在所有条件都相同的情况下，CloudFlare如何发现请求不是来自浏览器，如何在不使用Web浏览器的情况下在命令行上模拟或绕过请求？

请记住，我知道Selenium、Flaresolverr、pupflare等。我不打算使用浏览器，因为它们会呈现页面并降低操作速度。
事情我做了又没有得到答案：

  • 我想问题可能是在CURL命令行中缺少push server http2支持。所以我用PHP写了它，并在里面实现了push server，但是问题没有解决。
  • 我以为问题出在证书上，所以下载了浏览器证书，并将其转换为pem文件，并在CURL中使用，但问题并没有得到解决。
    如果我在Web浏览器上的第一个请求中获得代码200，我只想在第一种情况下在终端上获得相同的200响应代码！
curl

2条答案

按热度按时间
k2arahey

k2arahey1#

Cloudflare使用各种技术来确定用户代理是否是真实的的浏览器。而且，网站所有者还可以通过Cloudflare平台确定他们可以允许的风险级别。
让我们讨论一下Cloudflare使用的一些技术（我知道）：

  1. TLS指纹识别这是Cloudflare使用的著名技术之一。这也是原生代理等工具受欢迎的原因。链接：https://github.com/klzgrad/naiveproxy
  2. Cookies Cloudflare过去有一些cf_相关的cookie，用于区分真实的用户或非真实用户。
    而且，这些只是一些技术。Cloudflare还有更多。
    而且，这个问题不仅限于Cloudflare，中国防火墙也因使用这种做法来区分各种事物而臭名昭着。
赞(0）分享  回复(0）举报  2023-08-06
ttvkxqim

ttvkxqim2#

由于这个问题并没有建议用户想要绕过cloudflare保护的语言，我将提供node.js的代码：
库：

npm i puppeteer-extra puppeteer-extra-plugin-stealth puppeteer

字符串
nodejs：

const puppeteer = require('puppeteer-extra')
const pluginStealth = require('puppeteer-extra-plugin-stealth')
const { executablePath } = require('puppeteer')

const link = 'https://pegaxy.io/'

const getHtmlThoughCloudflare = async (url) => {
  puppeteer.use(pluginStealth())
  const result = await puppeteer
    .launch({ headless: true })
    .then(async (browser) => {
      const page = await browser.newPage()
      await page.goto(url)
      const html = await page.content()
      await browser.close()
      return html
    })

  console.log(` HTML: ${result}`)
  return result // html
}

getHtmlThoughCloudflare(link)

型

赞(0）分享  回复(0）举报  2023-08-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com