fscrawler jobs with Elastics pipeline setting errors Can't find stored field name to check existing filenames in path [/]. Please set store: true on field [file.filename]

[rcheatham1]

When we run the fscrawler jobs with an Elasticsearch pipeline added to the _settings. yaml file we get errors like the one below.

16:25:25,859 WARN [f.p.e.c.f.FsParserAbstract] Can't find stored field name to check existing filenames in path [/mnt/folder1/document/files/qp]. Please set store: true on field [file.filename]
16:25:25,859 WARN [f.p.e.c.f.FsParserAbstract] Error while crawling /mnt/folder1/document/files/qp: Mapping is incorrect: please set stored: true on field [file.filename].**

We need the pipeline to replace part of the file.url field's content. . We need to the search results to display the file.url content differently. When we add the pipeline setting to the jobs file the index is created but sometime the document count total doesn't match the contents in the folder. When crawling smaller content it finishes but when crawler larger content the crawl error out and stops crawling and stops injecting documents into the elasticseach index. We notice if the crawl completes , when we add new documents to the crawl location or remove document the crawl doesn't update the index.

FYI: The folder locations, the path where the content being crawled is a mounted shares on a windows operating system, I'm not sure if that's causing an issue. The fscrawler job is running on a REHL8 server.

Here's the elastic search pipeline script

PUT _ingest/pipeline/fscrawler
{
"processors":
[
{
"script": {
"source": "ctx.file.url = ctx.file.url.replace('file:///mnt/folder1/document/','http://testsite/document/')",
"if": "ctx?.file?.url != null"
}
}
]
}

We added the pipeline settings to the jobs:
elasticsearch:
pipeline: "fscrawler"

Here’s the Index field mapping generated by fscrawler.

{
"qp" : {
"mappings" : {
"properties" : {
"content" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"file" : {
"properties" : {
"content_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"created" : {
"type" : "date"
},
"extension" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"filename" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"filesize" : {
"type" : "long"
},
"indexing_date" : {
"type" : "date"
},
"last_accessed" : {
"type" : "date"
},
"last_modified" : {
"type" : "date"
},
"url" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"meta" : {
"properties" : {
"author" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"created" : {
"type" : "date"
},
"creator_tool" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"date" : {
"type" : "date"
},
"format" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"language" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"path" : {
"properties" : {
"real" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"root" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"virtual" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}

Here's the _setting.yaml file

name: "qp"
fs:
url: "/mnt/folder1/document/files/qp"
update_rate: "1m"
excludes:

• "/~"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false

ocr:
language: "eng"
enabled: true
pdf_strategy: "ocr_and_text"
follow_symlinks: false
elasticsearch:
username: "elastic"
password: "password"
pipeline: "fscrawler"
ssl_verification: false
nodes:

• url: "https://servername:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"

[dadoonet]

Duplicate of #1240