feat: Added BOM capability for output files (#1267)

[alvaro-osvaldo-tm]

• Added the '--add-bom' parameter for almost utilities

[alvaro-osvaldo-tm]

Implementation

Implemented the feature to optionality add UTF-8 Byte Order Mark (BOM) into output content in all utilities,
except csvpy and sql2csv

Solution

• The UTF-8 BOM only will be added if the parameter '--add-bom' is specified, otherwise is ignored.
• The parameter configuration and execution was implemented in the file csvkit/features/AddBOM.py ,
  I used a 'feature' pattern to avoid 'spaghetti code', no problem if the code need to be put into CSVKitUtility class.
  • The advantage of this approach is the code is more clear.
  • But the CSVToolKit is not prepared for it as seen in 'argument' method. Also, a few more CPU cycles will be perceived, if the user process a HUGE amount of files.

Tests

• A attached a end-to-end test script

  • test-1267.txt

• No unit test was made because the tests use 'StringIO' as 'input file', but the BOM need to be added as bytes using 'TextIOWrapper'.

  • If you want, I can implement a conversion in 'CSVToolkit' and 'LazyFile' to enable the tests.

• All PyTests and end-to-end tests passed in the following versions:

  • Python 3.8.20
  • Python 3.9.21
  • Python 3.10.16
  • Python 3.11.11
  • Python 3.12.8
    • Except csvgrep and csvcut due CSVToolkitbug.

Checklist

• Unit Testing
• End-to-end Testing

Considerations

• I found some bugs related to CSVToolkit that affect this implementation, I reported each in csvgrep: Catastrophic failure in Python 3.12 #1275 , csvcut: Catastrophic failure in Python 3.12 #1276, and Ignoring PythonIOEnconing #1277

• The CSVToolkit architecture always output in UTF-8, even if PYTHONIOENCONDING environment variable is set,
  therefore, only UTF-8 BOM is supported.

References

• Feature Pattern.https://martinfowler.com/articles/feature-toggles.html

[jpmckinney]

Please conform to the existing code, instead of introducing an entirely different code organization pattern.

[jpmckinney]

Just inline the 2 lines of code here, instead of creating a new 54-line file:

if getattr(self.args, 'add_bom', False):
    self.output_file.buffer.write(BOM_UTF8)

And of course do from codecs import BOM_UTF8 with the other imports.