error when using the "http request" node, with the uploaded file larger than 900KB

[Luminous-seed]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Problem

It seems that when dify's HTTP request uploads a file, with the file larger than about 800KB, it will not be able to send the file successfully.
I have already modified several parameters, such as:

MAX_VARIABLE_SIZE=2048000 not 204800,
WORKFLOW_FILE_UPLOAD_LIMIT=50 not 10,

# HTTP request node in workflow configuration
HTTP_REQUEST_NODE_MAX_BINARY_SIZE=10485760000 not 10485760
HTTP_REQUEST_NODE_MAX_TEXT_SIZE=1048576000 not 1048576

NGINX_CLIENT_MAX_BODY_SIZE=200M not 15M

However, These modifications are not helpful

preparation

I use the workflow to deal with pdf, whose first step is OCR.
I have completed the OCR backend and all services run normally.
I tested the OCR backend service with some files with size from 100KB to 25MB. Some records are shown below.

SUCCESS with 927KB

SUCCESS with myfilename_test.pdf 5566KB

work in DIFY

BUT In DIFY

1. the file is small, like 136KB.
   the success requests is like this:
   

2. the file is larger, like 900KB. ERROR OCCURS.
   but when the file is larger than 900KB, the error happened, like Picture ERROR01

Picture ERROR01 (927KB)

but when the file is 5566KB, the error happened, like Picture ERROR02

Picture ERROR02 (5566KB)

The errors log is like ：

10.14.121.217 - - [30/Dec/2024 16:18:53] "POST /local_ocr HTTP/1.1" 400 -
10.14.121.217 - - [30/Dec/2024 16:18:53] "POST /local_ocr HTTP/1.1" 400 -

and:
I debug the backend with the files for Picture ERROR01 and Picture ERROR02. The errors occur at :

@app.route('/local_ocr', methods=['POST'])
def local_ocr():

    file_data = request.get_data()
    if not file_data:
        flash('No file data provided')
        return redirect(request.url)
        ......

error occurs at file_data = request.get_data(),with request.get_data() log:

# request.get_data()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/skl_code/anaconda3/envs/minerU/lib/python3.10/site-packages/werkzeug/wrappers/request.py", line 419, in get_data
    rv = self.stream.read()
  File "/home/skl_code/anaconda3/envs/minerU/lib/python3.10/site-packages/werkzeug/wsgi.py", line 577, in readall
    data = self.read(1024 * 64)
  File "/home/skl_code/anaconda3/envs/minerU/lib/python3.10/site-packages/werkzeug/wsgi.py", line 562, in readinto
    self.on_disconnect()
  File "/home/skl_code/anaconda3/envs/minerU/lib/python3.10/site-packages/werkzeug/wsgi.py", line 499, in on_disconnect
    raise ClientDisconnected()
werkzeug.exceptions.ClientDisconnected: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.

The log indicates that the backend did not receive the file at all！！！

✔️ Expected Behavior

the https request should be successful with files larger than 800KB

❌ Actual Behavior

the https request failed with Server disconnected without sending a response. Because the server did not receive the correct file at all. I can conclude that this is a problem with dify, because the same https request can be successful when the file is small.

[dosubot]

I found a similar open issue where users experienced problems with file uploads in Dify when the file size exceeded certain limits. The suggested solution involves ensuring that both the UPLOAD_FILE_SIZE_LIMIT environment variable and the client_max_body_size parameter in the Nginx configuration are set correctly. Here are the steps to resolve the issue:

1. Set UPLOAD_FILE_SIZE_LIMIT: Ensure that the UPLOAD_FILE_SIZE_LIMIT in your .env file is set to a value larger than the file you are trying to upload.

2. Update Nginx Configuration: Modify the client_max_body_size parameter in the Nginx configuration to match the desired upload limit. This can be done by setting the NGINX_CLIENT_MAX_BODY_SIZE environment variable in your Docker setup.

   Example:

   NGINX_CLIENT_MAX_BODY_SIZE=50M

   Update the docker-compose.yaml file:

   environment:
     NGINX_CLIENT_MAX_BODY_SIZE: ${NGINX_CLIENT_MAX_BODY_SIZE:-50M}

   Ensure the Nginx configuration template reflects this change:

   client_max_body_size ${NGINX_CLIENT_MAX_BODY_SIZE};

3. Rebuild and Restart Docker Containers: After making these changes, rebuild and restart your Docker containers:

   docker-compose down
   docker-compose up --build -d

These steps should help resolve the "Server disconnected without sending a response" error and allow you to upload files successfully ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Luminous-seed]

@dosu. But I have changed the NGINX_CLIENT_MAX_BODY_SIZE=200M and

# Upload image file size limit, default 10M.
UPLOAD_IMAGE_FILE_SIZE_LIMIT=100
# Upload video file size limit, default 100M.
UPLOAD_VIDEO_FILE_SIZE_LIMIT=1000
# Upload audio file size limit, default 50M.
UPLOAD_AUDIO_FILE_SIZE_LIMIT=500```

[Luminous-seed]

@dosu I have changed the NGINX_CLIENT_MAX_BODY_SIZE=200M and UPLOAD_FILE_SIZE_LIMIT= 150MB in .env and run

docker-compose down
docker-compose up --build -d

But nothing changed. This suggestion doesn't seem to work.

[dosubot]

If you've already adjusted the NGINX_CLIENT_MAX_BODY_SIZE and UPLOAD_FILE_SIZE_LIMIT settings and the issue persists, consider checking the following:

1. Verify Environment Variables: Ensure that the environment variables are correctly set in your Docker environment. You can check this by running docker-compose exec <service_name> env to see if the variables are correctly loaded.

2. Nginx Configuration: Double-check the Nginx configuration file to ensure that the client_max_body_size directive is correctly set. Sometimes, the configuration might not be applied if there are syntax errors or if the configuration file is not being used by the running Nginx instance.

3. Application Logs: Check the application logs for any additional error messages that might provide more context on why the upload is failing. This can help identify if the issue is with the application logic or the server configuration.

4. Network and Proxy Settings: Ensure there are no network or proxy settings that might be interfering with the file upload process. Sometimes, intermediate proxies can impose their own size limits.

5. File Upload Code: Review the code handling file uploads to ensure there are no hardcoded limits or logic that might be causing the issue. The file upload process in Dify involves several components, and a limit might be enforced at a different layer ^[1].

If these steps do not resolve the issue, it might be helpful to isolate the problem by testing with different file sizes and types to see if the issue is consistent across all uploads or specific to certain conditions.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Luminous-seed]

@dosu I check the env in container docker-nginx-1 ,the log is shown below:

user@Ubuntu:/dify/docker$ docker exec -it docker-nginx-1 env
...Omit some unimportant parts...
HOSTNAME=0282579a2c57
TERM=xterm
NGINX_PORT=80
NGINX_SSL_PORT=443
NGINX_PROXY_READ_TIMEOUT=3600s
NGINX_SSL_CERT_KEY_FILENAME=dify.key
NGINX_ENABLE_CERTBOT_CHALLENGE=false
NGINX_PROXY_SEND_TIMEOUT=3600s
NGINX_CLIENT_MAX_BODY_SIZE=200M

It is clear that the NGINX_CLIENT_MAX_BODY_SIZE has been changed from 50MB to 200MB
but the error I encoutered with larger file still exists.( the file here is a new file that could be successfully with postman request, but failed with dify's https request )

and when the file is smaller, it(a new file with size at 526KB) can be successful

So in conclusion, nothing happen. This suggestion doesn't seem to work.
Also, as you mentioned, there may be restrictions on the size of uploaded files in the 'network and proxy' or other parts of the code, which is exactly what I am looking for. If possible, please provide more assistance.

[dosubot]

Since you've confirmed that the NGINX_CLIENT_MAX_BODY_SIZE is set correctly and the issue persists, let's explore other potential restrictions within the Dify codebase that might affect file uploads:

1. HTTP Request Node Limits: The Dify codebase has default limits for HTTP requests, which might be affecting your uploads. The HTTP_REQUEST_NODE_MAX_BINARY_SIZE is set to 10 MB by default. Ensure this is increased in your environment settings to accommodate larger files ^[1].

2. File Size Limits in Code: Check the file size limits defined in the Dify codebase. The web/app/components/base/file-uploader/constants.ts file specifies limits for different file types. Ensure these are set appropriately for your use case ^[2].

3. Middleware Restrictions: The api/services/file_service.py file includes methods that enforce file size limits based on file extensions. Verify that these limits are configured correctly in your environment ^[3].

4. Timeout Settings: The default timeout for HTTP requests in Dify is 15 seconds. If your uploads take longer, this could cause a timeout. Consider increasing this timeout if necessary ^[4].

5. Network and Proxy Settings: Double-check any network or proxy configurations that might impose additional restrictions on file sizes or timeouts.

By reviewing these areas, you can identify and adjust any additional restrictions that might be causing the issue with larger file uploads.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Luminous-seed]

@dosu Here is more information:
For the 1: I attached shell with container docker-api-1 ,obtaining the log.

root@9358d17301f6:/app/api# echo $HTTP_REQUEST_NODE_MAX_BINARY_SIZE 
10485760000

HTTP_REQUEST_NODE_MAX_BINARY_SIZE = 10485760000 is larager for a file with size at 5MB.

For the 2:
I set the limitation in docker/.env like belew, and run docker compose up -d ,so I think the the content of web/app/components/base/file-uploader/constants.ts is not effective and has no impact on the current error message.

# Upload file size limit, default 15M.
UPLOAD_FILE_SIZE_LIMIT=250

for the 3:
I attached shell with container docker-api-1 ,obtaining the log.

root@9358d17301f6:/app/api# echo $UPLOAD_FILE_SIZE_LIMIT
250

and in api/services/file_service.py, with code:

    def is_file_size_within_limit(*, extension: str, file_size: int) -> bool:
        if extension in IMAGE_EXTENSIONS:
            file_size_limit = dify_config.UPLOAD_IMAGE_FILE_SIZE_LIMIT * 1024 * 1024
        elif extension in VIDEO_EXTENSIONS:
            file_size_limit = dify_config.UPLOAD_VIDEO_FILE_SIZE_LIMIT * 1024 * 1024
        elif extension in AUDIO_EXTENSIONS:
            file_size_limit = dify_config.UPLOAD_AUDIO_FILE_SIZE_LIMIT * 1024 * 1024
        else:
            file_size_limit = dify_config.UPLOAD_FILE_SIZE_LIMIT * 1024 * 1024

A 5MB or 927KB PDF file will not exceed the limit. So I don't think the problem lies here

for the 4:
maybe what you said could be right , but the fact is the process fail when the file is with size at 927KB. I upload the file in LAN. I don't think the problem lies here, because the process of uploading files won't take more than five seconds in my opinion,

for the 5: I don't know how to check. But I don't think the problem lies here.

[dosubot]

Given the information you've provided, it seems you've thoroughly checked the configuration settings related to file size limits. Since the issue persists, here are a few additional areas to investigate:

1. Nginx Logs: Check the Nginx logs for any errors or warnings that might provide more insight into why the uploads are failing. This can help identify if the issue is occurring at the Nginx level.

2. Application Logs: Review the application logs in more detail to see if there are any specific error messages or stack traces that occur when the upload fails. This might provide clues about where the process is breaking down.

3. Network Configuration: Although you mentioned that the upload is happening over a LAN, ensure there are no network-level restrictions or firewalls that might be interfering with larger uploads.

4. Middleware and Decorators: Double-check any middleware or decorators applied to the file upload endpoints. These might impose additional checks or restrictions that aren't immediately obvious.

5. Testing with Different File Types: Try uploading different file types or slightly different sizes to see if the issue is specific to certain files or sizes. This can help narrow down the problem.

If none of these steps reveal the issue, consider isolating the problem by creating a minimal test case that replicates the upload process outside of Dify to see if the problem persists. This can help determine if the issue is with Dify specifically or with the underlying infrastructure.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[azhou125]

@Luminous-seed Did you solve the problem? I encountered the same issue.

[Luminous-seed]

@Luminous-seed Did you solve the problem? I encountered the same issue.

After conducting so many tests, I believe that the design of the HTTPS request node is a problem with this issue(maybe my opinion is wrong, but faor me, I cannot solve this problem ). So I changed my strategy. HTTPS requests themselves can be completed through Python code, so I created a code node to accomplish the task.

If you want something about my ecxperience, you can contact me, because some of the code involves privacy.

[azhou125]

@Luminous-seed Did you solve the problem? I encountered the same issue.

After conducting so many tests, I believe that the design of the HTTPS request node is a problem with this issue(maybe my opinion is wrong, but faor me, I cannot solve this problem ). So I changed my strategy. HTTPS requests themselves can be completed through Python code, so I created a code node to accomplish the task.

If you want something about my ecxperience, you can contact me, because some of the code involves privacy.

Absolutely. How did you pass file variable to python code node? In my version of dify, "Code" node doesn't support passing "File" object as an argument in python function.

[Luminous-seed]

@azhou125 Here is the key process.

[azhou125]

@Luminous-seed As far as I know, this "File->url" variable just pass an empty string "" in version 14.2 (and 15.0).

Is that the same case on your side?

[liliang1234]

@Luminous-seed Did you solve the problem? I encountered the same issue.

[liliang1234]

@Luminous-seed As far as I know, this "File->url" variable just pass an empty string "" in version 14.2 (and 15.0).

Is that the same case on your side?

I have the same problem.

[Luminous-seed]

@azhou125 @liliang1234
I also failed to run this module today because the 'File ->URL' variable is empty. Specifically, when a file is uploaded, the 'File ->URL' variable itself has a value, but it cannot be passed during transmission.
I also have this situation now. But when I tested a few days ago, it ran normally and the variable was passed correctly. I don't know how to fix. @laipz8200