Skip to content

Releases: dimajix/flowman

0.16.0

26 Apr 05:41
Compare
Choose a tag to compare
  • Add logo to Flowman Shell
  • Fix name of config option flowman.execution.executor.class
  • Add new groupedAggregate mapping
  • Reimplement target ordering, configurable via flowman.execution.scheduler.class
  • Implement new assertions columns and expression

0.15.0

23 Mar 13:56
Compare
Choose a tag to compare
  • New configuration variable floman.default.target.rebalance
  • New configuration variable floman.default.target.parallelism
  • Changed behaviour: The mergeFile target now does not assume any more that the target is local. If you already
    use mergeFiles with a local file, you need to prefix the target file name with file://.
  • Add new -t argument for selectively building a subset of targets
  • Remove example-plugin
  • Add quickstart guide
  • Add new "flowman-parent" BOM for projects using Flowman
  • Move com.dimajix.flowman.annotations package to com.dimajix.flowman.spec.annotations
  • Add new log redaction
  • Integrate Scala scode coverage analysis
  • assemble will fail when trying to use non-existing columns
  • Move swagger and json schema support into separate plugins
  • Change default build to Spark 3.0 and Hadoop 3.2
  • Update Spark to 3.0.2
  • Rename class Executor to Execution - watch your plugins!
  • Implement new configurable Executor class for executing build targets.
  • Add build profile for Spark 3.1.x
  • Update ScalaTest to 3.2.5 - watch your unittests for changed ScalaTest API!
  • Add new case mapping
  • Add new --dry-run command line option
  • Add new mock and null mapping types
  • Add new mock relation
  • Add new values mapping
  • Add new values dataset
  • Implement new testing capabilities
  • Rename update mapping to upsert mapping, which better describes its functionality
  • Introduce new VALIDATE phase, which is executed even before CREATE phase
  • Implement new validate and verify targets
  • Implement new deptree command in Flowman shell

0.14.2

12 Oct 10:04
Compare
Choose a tag to compare
  • Upgrade to Spark 2.4.7 and Spark 3.0.1
  • Clean up dependencies
  • Disable build of Docker image
  • Update examples

0.14.1

03 Oct 14:26
Compare
Choose a tag to compare
  • Fix dropping of partitions which could cause issues on CDH6

0.14.0

14 Dec 11:25
Compare
Choose a tag to compare
  • Fix AWS plugin for Hadoop 3.x
  • Improve setup of logging
  • Shade Velocity for better interoperability with Spark 3
  • Add new web hook facility in namespaces and jobs
  • Existing targets will not be overwritten anymore by default. Either use the --force command line option, or set
    the configuration property flowman.execution.target.forceDirty to true for the old behaviour.
  • Add new command line option --keep-going
  • Implement new com.dimajix.spark.io.DeferredFileCommitProtocol which can be used by setting the Spark configuration
    parameter spark.sql.sources.commitProtocolClass
  • Add new flowshell application

0.13.1

14 Dec 11:25
Compare
Choose a tag to compare
  • Code improvements
  • Do not implicitly set SPARK_MASTER in configuration
  • Add support for CDH6
  • Add support for Spark 3.0
  • Improve support for Hadoop 3.x

0.13.0

14 Dec 11:26
Compare
Choose a tag to compare
  • Refactor Maven module structure
  • Implement new Scala DSL for creating projects
  • Fix ordering bug in target execution
  • Merge 'migrate' phase into 'create' phase
  • Rename 'input' field to 'mapping' in most targets
  • Lots of minor code improvements