PPS List

[seadog007]

Filter (Validator): Filter out a record

Processor (Remover): Doing something to the record

Stage 1 (Line Filter/Processor)

• Question Mark Filter (question-mark.s1.js)
  If there is a question mark in the line, which means something missing in the line, remove that line.

• CSV Comma Remover
  Parsing CSV for helping stage converter.

• TSV Tab Remover
  Parsing TSV for helping stage converter.

• Space Remover
  Remove all kind of wired space, highly change appear between name where which is only 2 chinese character.

Stage 2 ([name, id] Filter/Processor)

• 10 Digit ID to 6 digit (10d26d.s2.js)
  Coverter 10 digit id to last 6 digit for hashing.

• A123456789 Filter (dummy-id.s2.js)
  Too many testing data using this....

• Name English Filter (name-english-filter.s2.js)
  If there is English character in the name, then filter it out.

• ID Validator (id-validate.s2.js)
  Check if ID is vaild.

• Unifier
  Unify the data, be sure the data will not repeat.

Stage Converter

• Stage 1 to Stage 2 Converter (s1-to-s2.js)
  Convert line to [name, id]

• Stage 2 to Final
  By hashing the name + id form stage 2

[seadog007]

#76
ID Validator
Name English filter

[seadog007]

Example of Trash TSV (Copy and paste from Excel)

Need some filter for this