Allow user to deactivate Khiops memory dimensioning

[lucaurelien]

Khiops sometimes prevents the execution of a data mining workflow, citing a lack of memory based on a rough estimate of the required memory.

After using a machine with significantly more available RAM, we often notice that far less memory was actually used during the process.

Allowing Khiops to run the task regardless of the available resources would be more convenient.
After all, if I know my data well, Khiops simply prevents me from analyzing it in a too conservative manner.
And if the program crashes, well, that’s what can happen with all alternative methods, such as Scikit-Learn: why bother?

[marcboulle]

Khiops Policy Regarding Resource Management

Main Objective: Never Crash

• Khiops is an AutoML solution designed to be as simple and robust as possible to use.
  • The only user parameters concern the specification of the data analysis problem or business constraints, such as modeling speed, accuracy, or interpretability.
  • Technical parameters are never exposed to the user.
• The tool is intended for use in an industrial context.
  • It should never crash, regardless of the scale or complexity of the data to analyze, or the available resources.
  • Business constraints and deadlines for delivering scores are of key importance.
• Importance
  • A crash is likely to be highly costly for an operational team.
  • It is impossible to determine whether the crash originates from a bug or a lack of resources.
  • Testing and trials by running a task under different resource configurations (number of machines, cores, available disk, RAM) are not compatible with industrial constraints.

Khiops Policy Regarding Resources

• Khiops exploits available resources as much as possible to maximize the speed of task execution.
• How:
  • Each internal algorithm evaluates the minimum necessary resources to run the task.
  • Given that, Khiops uses as many cores as possible, provided sufficient RAM is globally available.
  • If all data cannot fit into memory, Khiops splits the data:
    • Into chunks of instances for algorithms that can be decomposed on instances (e.g., deployment).
    • Into chunks of variables for algorithms that can be decomposed on variables (e.g., preprocessing).
  • If there is not enough memory even after applying the splitting strategy:
    • The user is provided with an estimate of the minimum additional memory required. -
• This allows for out-of-core processing:
  • For example, analyzing a 9 GB dataset using one core and 512 MB RAM.
  • See Khiops Hardware Adaptation.

Why It Sometimes Fails

• Estimating the necessary resources in advance is challenging due to many unknown data characteristics:
  • Number of selected instances in case of a selection criterion.
  • Number of missing values.
  • Data sparsity (e.g., texts).
  • Number of distinct values for numerical or categorical variables.
  • Length of categorical values.
  • Number of informative variables.
  • Number of secondary records per main record.
  • Etc.
• The principle for these evaluations is to iteratively gather information about the data after each analysis pass:
  • Basic statistics for the number of main instances and target variable values.
  • Descriptive statistics.
  • Preprocessing metrics.
  • Etc.
• Although Khiops can manage very large datasets using its out-of-core strategy, the number of main instances may still be too large.
  • Example: If there are 100 million instances in the main table:
    • Needs 100 million values simultaneously for univariate preprocessing: at least 800 MB.
    • Requires provisioning enough RAM for complex algorithms used in data preprocessing.
      • Example: around 100 bytes per value for double-linked lists, including value, criterion contribution, merging structures.
      • Total requirement: approximately 10 GB.
• If a numerical variable is actually a boolean with only two distinct values, the resource requirement may be excessively high.
  • Conversely, if a variable is truly continuous with as many distinct values as instances, the resource needs are different.
  • This can happen with a single variable added to a Khiops dictionary (e.g., using the Random rule) or generated automatically (e.g., variance of a secondary variable).

How to Improve This

• Estimating required resources remains a challenge; please report extreme cases where this estimation is clearly too conservative.
• This issue is most likely to occur when a huge number of instances are kept for analysis.
  • Use sampling or instance selection to reduce the data volume.
  • Remember that increasing the number of instances may improve results but with diminishing returns:
    • 10,000 to 100,000 instances: likely beneficial.
    • 100,000 to 1,000,000: may be relevant in some cases.
    • 1,000,000 to 10,000,000: will significantly increase processing time and cost, with unlikely improvements.