v0.34.0

v0.34.0 (2023-12-05)

Hightlights

• Date-time parsing in target sinks is now significantly faster, thanks to @BuzzCutNorman!
• Experimental support for Parquet encoding in BATCH messages, thanks to @jamielxcarter!

✨ New

• #2044 Add Parquet as a batch encoding option -- Thanks @jamielxcarter!
• #768 Better error messages when config validation fails
• #1854 Make stream logger a child of the tap logger

🐛 Fixes

• #2076 Make a explicit dependency on python-dateutil

⚙️ Under the Hood

• #2070 Parse dates with datetime.fromisoformat/backports.datetime_fromisoformat -- Thanks @BuzzCutNorman!
• #2056 Break up TapTestClassFactory._annotate_test_class into simpler methods
• #2058 Implement a SingerWriter class in singer_sdk.io_base and use it to emit Singer messages -- Thanks @BuzzCutNorman!
• #2061 Simplify target template file names with post_gen_project.py hook -- Thanks @vicmattos!
• #2060 Simplify tap template file names with post_gen_project.py hook -- Thanks @vicmattos!

📚 Documentation Improvements

• #2039 Add 404 page with sphinx-notfound-page
• #2037 Add flattening configuration examples

Full Changelog: v0.33.1...v0.34.0

v0.33.1

v0.33.1 (2023-11-08)

🐛 Fixes

• #2035 Retry all 5xx status codes -- Thanks @asamasoma!

v0.33.0

v0.33.0 (2023-10-12)

Hightlights

• Support for simple comprehension expression in stream maps: https://sdk.meltano.com/en/v0.33.0/stream_maps.html#compound-expressions
• Performance improvements for stream maps: expressions are now parsed once and evaluated multiple times.

---

✨ New

• #1999 Log JSONPath match count at the INFO level
• #1779 Cache SQL columns and schemas
• #2003 Add ability to do list comprehensions in stream map expressions -- Thanks @haleemur!
• #2018 Drop Python 3.7 support in cookiecutter templates -- Thanks @visch!

🐛 Fixes

• #2006 Parse record time_extracted into datetime.datetime instance
• #1996 Respect nullability of leaf properties when flattening schema
• #1844 Safely skip parsing record field as date-time if it is missing in schema
• #1885 Map record field to a JSON object type
• #2015 Ensure default property is passed to SCHEMA messages -- Thanks @prakharcode!

📚 Documentation Improvements

• #2017 Document support for comprehensions in stream maps

v0.32.0

v0.32.0 (2023-09-22)

Highlights

• Initial experimental support for different loading methods in targets. Currently, the overwrite method works by dropping and recreating tables in SQL targets. Methods upsert and append are still expected to be implemented by the target.

✨ New

• #1893 Standard configurable load methods
• #1861 SQLTap connector instance shared with streams -- Thanks @BuzzCutNorman!

🐛 Fixes

• #1954 Missing begin()s related to SQLAlchemy 2.0 -- Thanks @andyoneal!
• #1951 Ensure SQL streams are sorted when a replication key is set
• #1949 Retry SQLAlchemy engine creation for adapters without JSON SerDe support
• #1939 Handle decimal.Decimal instances in flattening
• #1927 Handle replication key not found in stream schema -- Thanks @mjsqu!
• #1977 Fix hanging downstream tests in tap-postgres
• #1970 Warn instead of crashing when schema helpers cannot append null to types

⚡ Performance Improvements

• #1962 Ensure raw_schema in stream mapper is immutable

📚 Documentation Improvements

• #1925 Add viztracer command for testing targets -- Thanks @mjsqu!

v0.31.1

v0.31.1 (2023-08-17)

✨ New

• #1905 Add email field and use human-readable questions in templates

🐛 Fixes

• #1913 Fix tap tests for multiple test classes with different input catalogs

v0.31.0

v0.31.0 (2023-08-07)

Highlights

• Improved connection handling in SQL targets by sharing a connector instance among stream sinks
• Expose builtin add_record_metadata and batch_config target settings

✨ New

• #1892 Add a mapper cookiecutter template
• #1864 SQLTarget connector instance shared with sinks -- Thanks @BuzzCutNorman!
• #1878 Add _sdc_sync_started_at metadata column to indicate the start of the target process
• #1484 Bump latest supported sqlalchemy from 1.* to 2.*

🐛 Fixes

• #1898 Correctly serialize decimal.Decimal in JSON fields of SQL targets
• #1881 Expose add_record_metadata as a builtin target setting
• #1880 Append batch config if target supports the batch capability
• #1865 Handle missing record properties in SQL sinks
• #1838 Add deprecation warning when importing legacy testing helpers
• #1842 Ensure all expected tap parameters are passed to SQLTap initializer
• #1853 Check against the unconformed key properties when validating record keys
• #1843 Target template should not reference tap_id
• #1708 Finalize and write last state message with dedupe
• #1835 Avoid setting up mapper in discovery mode

⚙️ Under the Hood

• #1877 Use importlib.resources instead of __file__ to retrieve sample Singer output files

📚 Documentation Improvements

• #1852 Fix stale pip_url example that uses shell script workaround for editable installation

v0.30.0

v0.30.0 (2023-07-10)

✨ New

• #1815 Support optional headers for OAuth request -- Thanks @s7clarke10!
• #1800 Publish supported python versions in --about

🐛 Fixes

• #1829 Update cookiecutter copyright assignment to cookiecutter user -- Thanks @riordan!
• #1826 Serialization of decimal.Decimal
• #1827 Add explicit dependency on packaging library
• #1820 Include SCHEMA message count in target logs

📚 Documentation Improvements

• #1824 Document RESTStream.rest_method
• #1818 Update testing.md

v0.29.0

v0.29.0 (2023-07-06)

Highlights

• A bunch of target improvements, including better batch message handling and more extensive tests.
• Union schemas are now supported. These are specially useful for tap or target configurations that accept mutually exclusive settings (e.g. both OAuth and API key auth methods). Read the documentation for more details.

✨ New

• #1769 Validate parsed/transformed record against schema message
• #1525 Support union schemas

🐛 Fixes

• #1809 Deserialize floats as decimal.Decimal
• #1770 Check schema has arrived before record
• #1796 Create batch directory if missing
• #1688 Incremental where clause generation from triggering TypeError -- Thanks @BuzzCutNorman!
• #1778 Sink schema comparison before adding metadata columns
• #1698 Force stream selection in tests
• #1775 Add tests for SQL type conversion from JSON schemas
• #1771 Add descriptions for batch_config properties
• #1752 Change runner scope to function for target tests
• #1753 Always emit a STATE message at the start of the sync process

⚙️ Under the Hood

• #1745 Change SQLStream.schema into a cached property -- Thanks @mjsqu!

📚 Documentation Improvements

• #1756 Fix invalid JSON in Stream Maps page and add meltano.yml tabs -- Thanks @mjsqu!
• #1763 Add Cloud banner

v0.28.0

v0.28.0 (2023-06-05)

✨ New

• #1728 Add an optional Dependabot file to projects generated from templates
• #1572 Add batch_config handling in append_builtin_config() -- Thanks @aaronsteers!
• #1686 Log stream errors
• #1711 Validate records against stream schema in standard tap tests
• #1709 Add a default Apache 2.0 license to tap and target templates

🐛 Fixes

• #1742 Recommend meltano run in target cookiecutter README

⚙️ Under the Hood

• #936 Use inheritance to construct plugin CLI

📚 Documentation Improvements

• #1721 Remove unsupported previous_token from HATEOAS example
• #1703 Fix broken docs link for record_metadata page -- Thanks @menzenski!

v0.27.0

v0.27.0 (2023-05-11)

✨ New

• #1681 Allow SQL tap developers to leverage post_process -- Thanks @BuzzCutNorman!
• #1672 Support deselecting streams by default
• #1648 Use Ruff to lint projects generated with Cookiecutter templates

🐛 Fixes

• #1680 Pin urllib3 to <2 to avoid incompatibility issues with botocore
• #1646 Use get_new_paginator in REST tap cookiecutter template

⚙️ Under the Hood

• #1668 Break out default batch file writer into a separate class

📚 Documentation Improvements

• #1685 Add PyCharm debugging tips to docs
• #1673 Fix docs build by specifying OS in RTD config file