custom tags added to external object coming in content tag after upload

[bhavanibrk]

I am new to fscrawler.
I want to associate custom fields like zipcode , city and state with the uploaded document and wanted to
search documents based on the custom fields along with the keyword search on the resume content.

###I have started fscrawler as a REST service as follows

 bin\fscrawler --rest --loop 0  --config_dir c:/FSCONFIGDIR  resume_job

###Trying to upload document with custom fields as follows

echo "This is my java resume" > test3.txt
echo "{ \"external\":{ \"zipcode\":\"08512\",  \"city\":\"cranbury\",  \"state\":\"NJ\"}}" > tags.txt
curl -F "[email protected]" -F "[email protected]" -F "index=resume_job" "http://127.0.0.1:8080/fscrawler/_upload"

{"ok":true,"filename":"test3.txt","url":"http://127.0.0.1:9200/resume_job/_doc/1be3ad6175e27573e78b7ed0d6d718c8"}

###custom tags added to content tag and overwriting the actual content.
Please help me where I am doing wrong

curl http://127.0.0.1:9200/resume_job/_doc/1be3ad6175e27573e78b7ed0d6d718c8?pretty

{
  "_index" : "resume_job",
  "_type" : "_doc",
  "_id" : "1be3ad6175e27573e78b7ed0d6d718c8",
  "_version" : 1,
  "_seq_no" : 25,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "content" : "{ \"external\":{ \"zipcode\":\"08512\",  \"city\":\"cranbury\",  \"state\":\"NJ\"}}",
    "meta" : { },
    "file" : { },
    "path" : { }
  }
}

[dadoonet]

This is probably a problem with windows and curl...
Using the same script on my laptop, with FSCrawler 2.7-SNAPSHOT, I'm getting at the end:

{
  "_index" : "rest",
  "_type" : "_doc",
  "_id" : "1be3ad6175e27573e78b7ed0d6d718c8",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "content" : "This is my java resume\n\n",
    "meta" : { },
    "file" : {
      "extension" : "txt",
      "content_type" : "text/plain; charset=ISO-8859-1",
      "indexing_date" : "2021-07-20T09:24:55.883+00:00",
      "filename" : "test3.txt"
    },
    "path" : {
      "virtual" : "test3.txt",
      "real" : "test3.txt"
    },
    "external" : {
      "zipcode" : "08512",
      "city" : "cranbury",
      "state" : "NJ"
    }
  }
}

Could you check the exact content of:

• file test3.txt
• file tags.txt

Did you by mistake submitted something like:

curl -F "[email protected]" -F "[email protected]" -F "index=resume_job" "http://127.0.0.1:8080/fscrawler/_upload"

Instead of [email protected]?

[bhavanibrk]

curl command syntax is exactly as follows

curl -F "[email protected]" -F "[email protected]" -F "index=resume_job" "http://127.0.0.1:8080/fscrawler/_upload"

I have attached fscrawler logs , commands and output from powershell,_settings.json and _settings.yaml content for your reference.

If we upload only file through curl command (test3.txt from powershell output ) ,it is indexed properly and can able to search through kibana also properly.

If we upload file and tags through curl command ,it is not working as expected.

Do I need to check on different OS other than windows?
Please let me know if there is any configuration issue.

---
name: "resume_job"
fs:
  url: "C:\\ElasticSearchData"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: true
  xml_support: false
  index_folders: true
  lang_detect: true
  continue_on_error: true
  ocr:
    language: "eng"
    enabled: true
   pdf_strategy: "auto"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "http://127.0.0.1:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: true

• fscrawler.log
• powershell output.txt
• _settingsdotjson.txt

[dadoonet]

Could you upload the file tags1.txt so I can look at it?

[bhavanibrk]

tags1.txt
Is there any other way I can check this upload functionality without curl?

[dadoonet]

Use this one instead tags1.txt

[bhavanibrk]

Thank you. I have tried like that and it worked and can also be able to search using kibana.
I am attaching files also for reference.

• tags4.txt
• test4.txt

curl -F "[email protected]" -F "[email protected]" "http://127.0.0.1:8080/fscrawler/_upload"

{
  "ok" : true,
  "filename" : "test4.txt",
  "url" : "http://127.0.0.1:9200/resume_job/_doc/576c93bf1b12c6a7252e8107c6716b5",
  "doc" : {
    "content" : "This is my new java resume\n",
    "meta" : {
      "language" : "en",
      "raw" : {
        "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
        "Content-Encoding" : "ISO-8859-1",
        "resourceName" : "test4.txt",
        "Content-Type" : "text/plain; charset=ISO-8859-1"
      }
    },
    "file" : {
      "extension" : "txt",
      "content_type" : "text/plain; charset=ISO-8859-1",
      "indexing_date" : "2021-07-21T15:08:53.450+00:00",
      "filename" : "test4.txt"
    },
    "path" : {
      "virtual" : "test4.txt",
      "real" : "test4.txt"
    }
  }
}

curl http://127.0.0.1:9200/resume_job/_doc/576c93bf1b12c6a7252e8107c6716b5?pretty

{
  "_index" : "resume_job",
  "_type" : "_doc",
  "_id" : "576c93bf1b12c6a7252e8107c6716b5",
  "_version" : 1,
  "_seq_no" : 19,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "content" : "This is my new java resume\n",
    "meta" : {
      "language" : "en",
      "raw" : {
        "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
        "Content-Encoding" : "ISO-8859-1",
        "resourceName" : "test4.txt",
        "Content-Type" : "text/plain; charset=ISO-8859-1"
      }
    },
    "file" : {
      "extension" : "txt",
      "content_type" : "text/plain; charset=ISO-8859-1",
      "indexing_date" : "2021-07-21T15:08:53.450+00:00",
      "filename" : "test4.txt"
    },
    "path" : {
      "virtual" : "test4.txt",
      "real" : "test4.txt"
    },
    "external" : {
      "zipcode" : "08512",
      "city" : "cranbury",
      "state" : "NJ"
    }
  }
}