Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Few remarks on the implement of GM_xmlhttpRequest #104

Closed
WhiteSevs opened this issue Sep 15, 2023 · 75 comments
Closed

Few remarks on the implement of GM_xmlhttpRequest #104

WhiteSevs opened this issue Sep 15, 2023 · 75 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@WhiteSevs
Copy link

  1. 希望不使用fetch来实现GM_xmlhttpRequest,因为有时候同源请求需要设置headersUser-Agent时,也是属于跨域了,使用fetch的话不会生效该User-Agent
  2. 提高兼容性,当methodGET时,如果details中存在data键,那么设置把methodPOST,不然https://github.com/JingMatrix/ChromeXt/blob/master/app/src/main/assets/GM.js的第381行会报错
request = new Request(details.url, {
        cache: "force-cache",
        body: details.data, // 会报错
        ...details,
});
  1. GM_cookie未实现,希望可以在脚本环境中删除该API,在·TamperMonkey·中的GM_cookieojbect类型;
  2. 是否考虑在iframe内注入?webview可以hook接口shouldInterceptRequest,参数WebResourceRequest 对它返回的html进行修改加入js;
  3. GM_xmlhttpRequest似乎并没有对请求自动加入Cookie,比如我对一个api进行请求登录,返回的headers里有Cookie信息,后续的同域名请求并没有把Cookie加进去;
@JingMatrix
Copy link
Owner

Thanks for your interest in ChromeXt and valuable suggestions given here.

Your first two points are now implemented in 90ceb88. However, I shall keep using the fetch API since it can save the network-traffic when a UserScript is trying to get response data of current page resources. Moreover, it is faster than the Java native implement since there is a data conversion layer between Java and JavaScript.

For the third point, I shall add it to the dev plan and implement it soon.

For the fourth point, it is not an easy task for chromium based browsers, so it has lower priority in my plan. A possible implement requires CDP. Everyone is welcome to contribute to this part. Please refer to the following codes if you are interested:

fun bypassCSP(bypass: Boolean) {
if (cspBypassed == bypass) return
command(null, "Page.enable", JSONObject())
command(null, "Page.setBypassCSP", JSONObject().put("enabled", bypass))
cspBypassed = bypass
if (bypass) DevSessions.add(this)
}

For the fifth point, I personally think that it is the responsibility of the UserScript to handle the relevant Cookie headers. This is simply because that my implement of GM_xmlhttpRequest is stateless: no data of previous responses are stored anywhere.

@JingMatrix JingMatrix added bug Something isn't working enhancement New feature or request labels Sep 15, 2023
@WhiteSevs
Copy link
Author

关于第一点,不光User-Agent,还有设置RefererHostOrigin有时候也很重要的,不知道使用fetch能不能设置
我曾在代码中使用jQuery的$.ajax来代替过GM_xmlhttpRequest

let headers_options_key = [
          "Accept-Charset",
          "Accept-Encoding",
          "Access-Control-Request-Headers",
          "Access-Control-Request-Method",
          "Connection",
          "Content-Length",
          "Cookie",
          "Cookie2",
          "Date",
          "DNT",
          "Expect",
          "Host",
          "Keep-Alive",
          "Origin",
          "Referer",
          "TE",
          "Trailer",
          "Transfer-Encoding",
          "Upgrade",
          "User-Agent",
          "Via",
        ]

@WhiteSevs
Copy link
Author

Thanks for your interest in ChromeXt and valuable suggestions given here.

Your first two points are now implemented in 90ceb88. However, I shall keep using the fetch API since it can save the network-traffic when a UserScript is trying to get response data of current page resources. Moreover, it is faster than the Java native implement since there is a data conversion layer between Java and JavaScript.

For the third point, I shall add it to the dev plan and implement it soon.

For the fourth point, it is not an easy task for chromium based browsers, so it has lower priority in my plan. A possible implement requires CDP. Everyone is welcome to contribute to this part. Please refer to the following codes if you are interested:

fun bypassCSP(bypass: Boolean) {
if (cspBypassed == bypass) return
command(null, "Page.enable", JSONObject())
command(null, "Page.setBypassCSP", JSONObject().put("enabled", bypass))
cspBypassed = bypass
if (bypass) DevSessions.add(this)
}

For the fifth point, I personally think that it is the responsibility of the UserScript to handle the relevant Cookie headers. This is simply because that my implement of GM_xmlhttpRequest is stateless: no data of previous responses are stored anywhere.

感谢修复,待会儿会去试一下最新版

@JingMatrix
Copy link
Owner

There is a chroimum bug tracker for the User-Agent issue.
A priori, we are not sure which headers are not effective in the fetch API of chromium. Hence, I'd suggest that we change the condition for fetch API when new related bugs are reported.
Currently, in your UserScript, are there other headers must be changed?

@WhiteSevs
Copy link
Author

There is a chroimum bug tracker for the issue. A priori, we are not sure which headers are not effective in the API of chromium. Hence, I'd suggest that we change the condition for API when new related bugs are reported. Currently, in your UserScript, are there other headers must be changed?User-Agent``fetch``fetch

搜集了一下,我的脚本目前使用的headers

Accept
Authorization
Accept-Encoding
Accept-Language
Content-Type
Host
Origin
Pragma
Referer
x-csrf-token
X-Requested-With

另外关于这个有些建议90ceb88237行,对details.headers中的key进行小写/大写转换判断,因为可能会有不规范写法,如user-agent或者user-Agent又或者uSer-aGent

@JingMatrix
Copy link
Owner

JingMatrix commented Sep 15, 2023

Now ChromeXt will take all forbidden headers listed on MDN into consideration.
Moreover, you can specify the forceCORS option, see b7b023e .

No worry about the upper or lower spelling cases of the header keys, the Headers API of JavaScript can take care of them automatically.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 15, 2023

刚刚发现一个新bug,当用户的details.responseType期望为json时,此刻的返回的response.responseText内其实是html,这时候会解析失败
image

@JingMatrix
Copy link
Owner

JingMatrix commented Sep 15, 2023

Could you give an exmaplar URL for the above bug? Maybe you missed a content-type header.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 15, 2023

https://up.woozooo.com/mlogin.php
用于蓝奏云网盘登录
👇是details结构
image

@JingMatrix
Copy link
Owner

I cannot reproduce it using cURL.
Even without any specific headers, the response is still a JSON string:

curl -v 'https://up.woozooo.com/mlogin.php' --data-raw 'task=3&uid=jingmatrix%40gmail.com&pwd=test&setSessionId=&setSig=&setScene=&setToken=&formhash=ab2489d6'

The Java implement of GM_xmlhttpRequest should be the same as cURL.
Could you also show the response data? Maybe you can expand the promise, and find the response data.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 15, 2023

我使用TamperMonkey的请求复现了一下,reponseText并不是json而是html,它的response直接为undefined

image

@JingMatrix
Copy link
Owner

The finalUrl indicates that your request didn't follow the redirect correctly. You may retry with the URL https://up.woozooo.com/mlogin.php.

If TamperMonkey cannot response JSON data, then it seems not to be a bug of script manager.

@WhiteSevs
Copy link
Author

该请求为302重定向跳转到了https://up.woozooo.com/myfile.php,导致了response.responseText内容应该是json变成了html
·TamperMonkey·对response.reponse做的处理是不进行JSON.parseChromeXt也需要在上面进行兼容性处理,不然ChromeXtdata.response = JSON.parse(data.responseText);就会执行失败导致无法调用脚本的onloadCallBack

刚刚发现一个新bug,当用户的期望为时,此刻的返回的内其实是,这时候会解析失败details.responseType``json``response.responseText``html image

@JingMatrix
Copy link
Owner

It is wired about the redirection, because cURL tells me that https://up.woozooo.com/myfile.php redirects to ./mlogin.php, but you claimed the reverse.

I think when you specify responseType, you are expecting a JSON reponse. And if such response is not available, an error should be thrown.
Hence, I will reject the Promise (after catching it in JSON.parse) with unparsed response. Do you think it is more reasonable?

@WhiteSevs
Copy link
Author

很合理,但是如果resolve的话兼容性会更好一些

It is wired about the redirection, because cURL tells me that https://up.woozooo.com/myfile.php redirects to ./mlogin.php, but you claimed the reverse.

I think when you specify responseType, you are expecting a JSON reponse. And if such response is not available, an error should be thrown. Hence, I will reject the Promise (after catching it in JSON.parse) with unparsed response. Do you think it is more reasonable?

@JingMatrix
Copy link
Owner

In the commit aba8e9d, I throw the parsing error out. Please tell me if this solution is acceptable for you. I think that unless you are using the async version GM.xmlHttpRequest, your code won't stop when there is a promise error.

Also, I implemented the basic APIs of GM_cookie in 567dcdd, but I am not very familar with the usage case of it. Could you please give some usage scenarios? So that I am more aware of what kind of functionalities should be included.

@JingMatrix JingMatrix changed the title about GM_xmlhttpRequest some suggestion Few remarks on the implement of GM_xmlhttpRequest Sep 15, 2023
@JingMatrix
Copy link
Owner

JingMatrix commented Sep 15, 2023

For your fifth point, I found a way to implement it in Java.

Now the anonymous options fully control if the HTTP requests are stateless.

@WhiteSevs
Copy link
Author

In the commit aba8e9d, I throw the parsing error out. Please tell me if this solution is acceptable for you. I think that unless you are using the async version GM.xmlHttpRequest, your code won't stop when there is a promise error.

Also, I implemented the basic APIs of GM_cookie in 567dcdd, but I am not very familar with the usage case of it. Could you please give some usage scenarios? So that I am more aware of what kind of functionalities should be included.

  1. 试了一下相较之前的并没太大区别,虽然在ChromeXt内抛出了JSON.parse错误,但是脚本的onload并未触发,我删除了自己脚本的responseType: "json",自己在onload返回内对response.responseText进行了处理。
  2. 下面是一些我使用的GM_cookie的例子
// 用来获取用户是否登录的Cookie,该Cookie是HttpOnly,document.cookie无法获取到
function getCookie(cookieName) {
        return new Promise((resolve) => {
          GM_cookie.list({ name: cookieName }, function (cookies, error) {
            if (error) {
              resolve(null);
            } else {
              if (cookies.length == 0) {
                resolve(null);
              } else {
                resolve(cookies[0].value);
              }
            }
          });
        });
      }

await getCookie("userLogin")

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 16, 2023

我又找到一个关于response.responseText编码问题

GM_xmlhttpRequest({
    url:"https://tieba.baidu.com/f/search/res?isnew=1&kw=%C4%E6%CB%AE%BA%AE%CA%D6%D3%CE&qw=%CE%E8%D1%F4%B3%C7&un=&rn=10&pn=0&sd=&ed=&sm=1",
    method:"get",
    headers: {
    Referer: "https://tieba.baidu.com/f?ie=utf-8&kw=%E9%80%86%E6%B0%B4%E5%AF%92%E6%89%8B%E6%B8%B8",
    Host: "tieba.baidu.com",
    Accept:
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
  },
  responseType: "html",
    onload:(resp)=>{console.log(resp)}
})

TamperMonkey中正常解码👇

image

ChromeXt中乱码👇

image

猜测是content-type: text/html; charset=GBK的缘故?

@JingMatrix
Copy link
Owner

Thanks for the feedbacks!

  1. Now parsing error won't block onload.
  2. To get httpOnly cookie, I should use CDP. This can be done later.
  3. You are right, I was assuming UTF-8 encoding. Should be fixed soon.

@WhiteSevs
Copy link
Author

Thanks for the feedbacks!

  1. Now parsing error won't block onload.
  2. To get httpOnly cookie, I should use CDP. This can be done later.
  3. You are right, I was assuming UTF-8 encoding. Should be fixed soon.

关于第2点,我在TamperMonkey中测试了一下,当我在headers里设置了{cookie:"xxxxxxx"},它会自动将属于该domain的为HttpOnly的Cookie自动添加到Cookie前面
例如,当前domain下的cookie

key value domain path expires ... HttpOnly
github_test 1 github.com / ....
github_test2 2 github.com / ....
// 我设置的cookie

headers: {
  "user-agent":".....",
   cookie: "github_test=1;",
}

发出去的Cookie会变成
Cookie: github_test2=2;github_test=1
也就是,Cookie前面的是当前的,后面的属于用户设置的。
当我尝试通过抓包通过lspatch打包的ChromeXt时,发现请求头存在多个Cookie键,且我设置的Cookie是小写的cookie

image

@JingMatrix
Copy link
Owner

httpOnly cookie is not supported yet, but will be done soon. Once it is implemented, httpOnly cookie headers will be appended.
As for the given screenshot, is it a CORS request?
Could you please describe what should be the expected behavior?
If I am correct, in the API, cookie should be set as a property of details instead of headers.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 16, 2023

httpOnly cookie is not supported yet, but will be done soon. Once it is implemented, httpOnly cookie headers will be appended. As for the given screenshot, is it a CORS request? Could you please describe what should be the expected behavior? If I am correct, in the API, cookie should be set as a property of details instead of headers.

对,是跨域请求,修改了user-agent,该请求的作用是发送一个签到请求,需要Cookie验证当前身份是否和url中的formhash匹配,cookie确实应该是放在details中而非headers内,等后面实现HttpOnly我再试试,现在的话我发送签到请求返回内容会是验证身份失败

@JingMatrix
Copy link
Owner

Now httpOnly support is implemented. Please test it and share your feedbacks! 😃

@WhiteSevs
Copy link
Author

Now httpOnly support is implemented. Please test it and share your feedbacks! 😃

我尝试了一下最新的https://github.com/JingMatrix/ChromeXt/actions/runs/6228644262,似乎有严重的问题,GM_xmlhttpRequest的请求不触发任何回调
image

@JingMatrix
Copy link
Owner

Cannot reproduce your issue, in my devices,

GM_xmlhttpRequest({
    url: "https://bbs.binmt.cc/k_misign-sign.html?operation=qiandao&format=button&formhash=TESTHASHVALUE&inajax=1&ajaxtarget=midaben_sign",
    onload: (r)=>console.log(r.response),
    onerror: (r)=>console.log(r),
    timeout: 5000,
    forceCORS: true
})

returns a response with 您当前的访问请求当中含有非法字符,已经被系统拒绝.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 19, 2023

Cannot reproduce your issue, in my devices,

GM_xmlhttpRequest({
    url: "https://bbs.binmt.cc/k_misign-sign.html?operation=qiandao&format=button&formhash=TESTHASHVALUE&inajax=1&ajaxtarget=midaben_sign",
    onload: (r)=>console.log(r.response),
    onerror: (r)=>console.log(r),
    timeout: 5000,
    forceCORS: true
})

returns a response with 您当前的访问请求当中含有非法字符,已经被系统拒绝.

使用这个仍是未触发,GM_bridge.GM_xmlhttpRequest使用的是这个调试https://greasyfork.org/zh-CN/scripts/475424-%E8%B0%83%E8%AF%95

QQ2023919-162632-HD.mp4

@JingMatrix
Copy link
Owner

The re-implement of GM_cookie is done.
Now, it should work in the same way as TamperMonkey.

@JingMatrix
Copy link
Owner

@WhiteSevs Did you succeed to run your script with the latest commits of ChromeXt? I am planning to release a new version of ChromeXt this weekend. If you find no errors, please drop a message here. Also, this issue will be closed after that.

@WhiteSevs
Copy link
Author

@WhiteSevs Did you succeed to run your script with the latest commits of ChromeXt? I am planning to release a new version of ChromeXt this weekend. If you find no errors, please drop a message here. Also, this issue will be closed after that.

ok,现在基本上除了那些需要跨域登录的操作都没问题了

@JingMatrix
Copy link
Owner

CORS login should be working now. Did you still fail to do that?

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 22, 2023

CORS login should be working now. Did you still fail to do that?

Yes,抓包看登录请求成功后301重定向,该重定向请求并没有携带Cookie👉KEEP_LOGIN
9e4f966eabe3db71ce9a5473d306655d

image

@JingMatrix
Copy link
Owner

If you receive the 301 redirection, the cookie won't be sent according to this explanation. You should change your request URL to the redirected one and try again.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 22, 2023

If you receive the 301 redirection, the cookie won't be sent according to this explanation. You should change your request URL to the redirected one and try again.

ChromeXt中设置redirect: "manual"可以正常,但是在TamperMonkey中得需要继续请求https://www.z4a.net/来判断是已经否登录,这样的话,我就不考虑增加redirect: "manual"了。
在上传图片时出现错误👇
image

@JingMatrix
Copy link
Owner

Currently in ChromeXt, redirect: "manual" is the same as redirect: "follow", there should be no difference.
Could you give the details of you URL request? I may try it on my side.
500 error means the error is not in the client side.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 22, 2023

登录👇

// token可以在www.z4a.net页面HTML中搜索到 => PF.obj.config.auth_token
GM_xmlhttpRequest({
    url:"https://www.z4a.net/login",
    method:"POST",
    redirect: "manual",
    data: `login-subject=${user}&password=${pwd}&auth_token=${xxxxx}`,
    headers: {
              "User-Agent": GM_bridge.Utils.getRandomPCUA(),
            },
    onload:(r)=>{console.log(r)},
    onerror:(r)=>{console.log(r)}
})

上传图片👇,imageFileinput内的图片文件File

let form = new FormData();
form.append("type", "file");
form.append("action", "upload");
form.append("timestamp", new Date().getTime());
form.append("auth_token", auth_token);
form.append("nsfw", 0);
form.append("source", imageFile);
GM_xmlhttpRequest({
    url: `https://www.z4a.net/json`,
    method: "POST",
    data: form,
    async: false,
    headers: {
      Accept: "application/json",
      "User-Agent": utils.getRandomPCUA(),
      Referer: `https://www.z4a.net/`,
      Origin: "https://www.z4a.net",
    },
    onload:(r)=>{console.log(r)},
    onerror:(r)=>{console.log(r)}
})

@JingMatrix
Copy link
Owner

Okay, I see. That is because I didn't process FormData in GM_xmlhttpRequest yet. This will be done soon. It seems not to be a problem of cookie.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 22, 2023

对,但是上传的前一个步骤是先登录,那个步骤有重定向Cookie问题

@JingMatrix

This comment was marked as outdated.

@JingMatrix

This comment was marked as outdated.

@JingMatrix
Copy link
Owner

@WhiteSevs I have tested another website https://imgse.com/, the uploding is working well with the latest commit of ChromeXt.

@JingMatrix
Copy link
Owner

@WhiteSevs Did you try the cookie login again?
I am about to publish a new release of ChromeXt today. Currently, at least on my devices, it is working.

@WhiteSevs
Copy link
Author

@WhiteSevs Did you try the cookie login again? I am about to publish a new release of ChromeXt today. Currently, at least on my devices, it is working.

貌似有点问题,在登录时返回的responseText内没有内容
image

@JingMatrix
Copy link
Owner

That is because the default value for redirect is manual, and the status is 301.
You will see some response if you set it to follow.
This redirection is not important regarding to login.

@WhiteSevs
Copy link
Author

WhiteSevs commented Sep 24, 2023

That is because the default value for redirect is manual, and the status is 301. You will see some response if you set it to follow. This redirection is not important regarding to login.

TamperMonkey中,redirect默认为follow,我刚刚试了下,在ChromeXt中设置redirect: follow有这么个错误
设置manual的话和TamperMonkey中没啥大区别,可惜是用不到manual,基本使用默认的follow且它的返回statue为200,responseText内有内容
image

@JingMatrix
Copy link
Owner

Do you need the response for your script? If not, you can safely ignore it for a while.
I will change the code for redirection later.
How about the upload? Did it work as expected?

@WhiteSevs
Copy link
Author

是的,我脚本中有段是根据这个responseText内容来判断是否已成功登录,这个报错很奇怪,我是用的登录,并没有使用上传

@JingMatrix

This comment was marked as outdated.

@JingMatrix

This comment was marked as outdated.

@JingMatrix
Copy link
Owner

JingMatrix commented Sep 24, 2023

I have changed the default value of redirect to be follow.
Note that in ChromeXt, you can use the onredirect parameter to listen for the redirect event.

@WhiteSevs
Copy link
Author

奇怪,最新的版本无法进行图床登录,上个版本我测试了下还可以进行登录且上传图片

@JingMatrix
Copy link
Owner

Cannot reproduce on my phone. I tested with the following code, and it worked as expected.

let url = "https://imgse.com/";

async function get_token() {
  const html = await GM_bridge.GM_xmlhttpRequest({ url, forceCORS: true });
  return html.match(/PF.obj.config.auth_token = "(\w+)"/)[1];
}

function login(
  auth_token = PF.obj.config.auth_token,
  redirect = "follow",
  nocache = true
) {
  const form = new FormData();
  form.append("login-subject", user);
  form.append("password", pwd);
  form.append("auth_token", auth_token);
  return GM_bridge.GM_xmlhttpRequest({
    url: url + "login",
    method: "POST",
    redirect,
    nocache,
    onredirect: (xhr) => console.log(xhr),
    data: form,
    onload: (xhr) => console.log(xhr.status),
  });
}

function createForm() {
  if (document.querySelector("#chromext")) return;
  const input = document.createElement("input");
  input.setAttribute("type", "file");
  input.setAttribute("accept", "image/*");
  input.id = "chromext";
  document.body.prepend(input);
  return input;
}

function upload(auth_token = PF.obj.config.auth_token, putFile = true) {
  input = document.querySelector("#chromext");
  if (putFile && typeof input.files[0] == "undefined") {
    console.warn("No file chosen yet");
  }
  const form = new FormData();
  form.append("type", "file");
  form.append("action", "upload");
  form.append("timestamp", new Date().getTime());
  if (putFile) form.append("source", input.files[0]);
  form.append("auth_token", auth_token);
  form.append("nsfw", 0);
  GM_bridge.GM_xmlhttpRequest({
    responseType: "json",
    url: url + "json",
    method: "POST",
    data: form,
    timeout: 30000,
    onload: (xhr) => console.log(xhr.response),
  });
}

function test() {
  const promise = get_token().then((token) => {
    console.log("Auth_token:", token);
    login(token);
  });
  createForm();
  return promise;
}

You run it with test() and then upload(token) with the token given in the console.

@WhiteSevs
Copy link
Author

imgse.com倒是可以,在测试z4a.net登录时,发现没加redirect: "follow"登录会失败,然后加上后,就登录成功了,再刷新DevTools,上传功能就正常了

@JingMatrix
Copy link
Owner

Even with z4a.net, it is not reproducible on my device.
It is most likely to be a caching issue, or your traffic is somehow blocked by z4a.net.
To avoid such mistakes, please ensure to restart your browser each time when you test it.

This will not be considered as a bug. If you are Okay with that, please close the issue.

@WhiteSevs
Copy link
Author

OK,删除脚本后重新安装了下,第一次登录失败,第二次登录成功且上传图片完毕,应该是这个图床网站的问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants