CSV rules wishes (overview)

[simonmichael]

Here's an overview issue to collect wishes - things people sometimes want - for CSV rules.
All of these are at least worth considering, and research and development is welcome.
Some of these have their own issue numbers, not yet linked.

• add text to hledger fields (particularly comment fields)
• save the original csv record in a comment field
• comparison operators for dates
• comparison operators for amounts
• nested ifs
• comment lines anywhere
• end of line comments anywhere (?)
• text transforms: substring, capitalisation, regex replace..
• amount transforms: arithmetic operations
• more control over archive file names, archiving policies
• generate multiple journal entries from one record
• generate one journal entry from multiple records
• a better story for when and how to switch between or combine hledger csv rules and a real programming language.
• include csv files in a journal file
• include rules files in a journal file
• better docs / examples / getting started experience

About CSV rules

hledger's CSV rules format started out as a minimal config format that non-programmers could use to describe a CSV file's fields and make it readable. This goal remains in effect, but by now rules files are also a domain specific language not just for describing the CSV but also

• how to fetch/produce/archive the CSV
• how to interpret complex, partial, possibly wrong CSV records
• how to assign accounting categories based on simple or complex conditions
• how to generate custom, sometimes complex, valid journal entries.

They are intentionally not a turing-complete programming language. When there's a need they don't cover, we recommend to replace or complement them with one of those. Eg as a pre-filter before rules processing. (Because some things are easier to do in a real programming language, and some other things are easier and more robust to do in the rules file.) Nevertheless, advanced csv rules usage is a kind of programming.

[Anachron]

First off: Very nice list.

I have a general question about comparisments.

Do you plan to support things like string substitution, so that we can modify values in a way to use amount operators, for example?

Eg:

if Subst(%Purpose, /TaxID: /, "") < 123456789

This is for example useful when we have one field that contains additional information that are seperated and need to be split before being processed.

Also would it be possible to skip remaining rules? That would make rule order much easier for me.

Something like apply or finish?

Thanks!

[kalupa]

a better story for when and how to switch between or combine hledger csv rules and a real programming language.

this probably could go a long way. I am very new to hledger but I think it's limitations are one of it's strongest features, if you can call it that. I don't want to dissuade non-programmers from using it but sometimes that input CSV is just a nightmare (speaking from experience ... description fields are not a substitute for real fields) and a "real" language is not just a better choice but even a requirement, even with this list of features.

What might be good is finding a way to help non-programmer deal with trashy csvs.

or maybe just some more details on using an ofx2csv tool for the more kindly institutions that offer it and not just PDF and terrible CSVs

[adept]

The way things go, it might be faster and easier to provide a good prompt that describes the requirements for a "good hledger-friendly output" or even hledger journal syntax that could be fed into LLM together with anonymized csv to produce a (python?) script that does CSV convesion :)