-
Notifications
You must be signed in to change notification settings - Fork 18
/
Copy pathCHANGELOG
231 lines (149 loc) · 7.32 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
unreleased
* Bug fixes
* When :ignore_error is true and there is an error, set value to nil instead of true
3.0.0 / 2014-02-04
* Breaking changes
* (alpha) :dictionary is now just a plain Hash-like object (responds to []) - and no longer attempts to refresh between runs
* (alpha) no more unit conversions
* (alpha) always nullifies blank strings
* (alpha) always overwrites columns whether there was a non-null value there before or not (no more :overwrite option)
* (alpha) don't use synthesize option, just pass a block
* (alpha) synthesized values get the same whitespace compression and stripping as other values
* (alpha) not tested against the Earth library, which has particular requirements and makes testing too complicated
* (alpha) doesn't keep DataMiner::Run or DataMiner::Run::ColumnStatistic records
* (alpha) no more :matcher option
* Enhancements
* :date_format option takes format for Date.strptime
* :ignore_error options recovers from some value conversion errors
* (rc2) (Re-)Start logging the beginning, erroring, and ending of steps
* (rc2) Raise errors if a step fails
* (rc1) :field_name can be passed as an Array like [ :first_name, :last_name ], which will join them with a space
* (beta) test steps with after: and every: options, referring to row counts in the previous step
* (beta) :limit option for import steps - oh yeah
* (beta) :random_skip option - goes nicely with :limit if you're debugging
* (alpha) if you have a postgres hstore column called "foo", you can do store 'foo.bar'
* add MIT license
2.5.2 / 2013-07-05
* Bug fixes
* Get connection from the right connection pool (namely, the model's, not AR::Base's)
2.5.1 / 2013-03-15
* Enhancements
* Just suggest that people use roo > 1.10.3
2.5.0 / 2013-03-15
* Breaking changes
* Lock to roo 1.10.1 and activerecord > 3
* Enhancements
* If you don't define a `key` in a data_miner block, then just keep appending records - thanks @towerhe
* Got rid of ridiculous "install unicode-utils" warning
* Bug fixes
* Don't die because an Alchemist::NumericConversion is passed to a function that expects a plain Float - thanks @towerhe
2.4.1 / 2012-07-26
* Enhancements
* Support sqlite in sql steps
* More accurately detect if a table has been created during a run
2.4.0 / 2012-07-26
* Breaking changes
* Entirely remove taps support - it doesn't preserve foreign key constraints and adds a lot of complexity
* Enhancements
* New "sql" step - executes SQL provided as a string OR a local/remote SQL file provided as a URL
* Die with a useful error message if a column specified in an import step doesn't exist - thanks @chrisle https://github.com/seamusabshere/data_miner/issues/17
* Allow setting :validate => true on import steps - thanks @chrisle https://github.com/seamusabshere/data_miner/issues/18
2.3.4 / 2012-07-06
* Bug fixes
* Fix silly deadlock
2.3.3 / 2012-07-06
* Bug fixes
* Always register data miner scripts with DataMiner.model_names or else DataMiner.run won't work as expected - just bad logic before
2.3.2 / 2012-06-25
* Bug fixes
* Make sure 't', 'true', 1, '1', 'on', 'yes', etc. are treated as true for boolean columns (and 'f', 'false', 'FALSE', etc.)
2.3.1 / 2012-06-22
* Bug fixes
* Avoid duplicate rows by correctly identifying auto-increment versus non-auto-increment integer primary keys
* Properly sort per-row column statistics
2.3.0 / 2012-06-21
* Enhancements
* Using https://github.com/seamusabshere/upsert to speed up import steps when possible.
2.2.0 / 2012-06-11
* Breaking changes
* You must specify DataMiner.units_convert = {:alchemist,:conversions}
* Enhancements
* Swappable unit conversion libraries [@dkastner]
* Intelligent parsing of commas and periods in number fields [@ihough]
2.1.2 / 2012-05-22
* Breaking changes
* DataMiner::Run#column_statistics_for(column_name, period) -> {initial,final}_column_statistics(column_name)
* Remove std dev from statistics kept
* Enhancements
* Added zero count and null count
2.1.1 / 2012-05-16
* Enhancements
* Keep row counts from before and after runs.
* If DataMiner.per_column_statistics?, keep per-column statistics from before and after runs.
2.1.0 / 2012-05-10
* Enhancements
* Don't use "perform" as a class or instance method name - it can interfere with Resque or Sidekiq.
* Don't lock DataMiner::Run#perform. Provide instructions for how people can do this manually.
* Bug fixes
* Don't blow up trying to get current database name on SQLite3
2.0.3 / 2012-05-07
* Enhancements
* Rename :nullify to :nullify_blank_strings to clarify that it only affects textual columns and means "treat blank input as null".
* Don't try to set units for a column that is currently nil (thanks @ihough)
* Bug fixes
* Don't blow up if mass-assignment rules are strict.
* Don't accidentally set a numeric column to 0.0 when the input is blank or nil.
2.0.2 / 2012-05-04
* Breaking changes
* Import descriptions are no longer optional
* Import options are no longer optional (but then, they never were)
* Enhancements
* Real documentation!
* Make it easier to clear locks with DataMiner::Run.clear_locks
* Replace class-level mutexes with simple Thread.exclusive calls
* Simplified DataMiner::Dictionary
2.0.1 / 2012-04-18
* Enhancements
* DataMiner.run -> DataMiner.perform
* Some basic tests that don't rely on Earth
* Bug fixes
* Fix the "call stack" - the thing that keeps infinite loops from occurring
* Make sure sources get refreshed every time you re-run data miner
* Make sure dictionaries " " " " " " "
2.0.0 / 2012-04-17
* Breaking changes
* Renamed data_miner_config to data_miner_script (etc. for class/method naming)
* Simplify DataMiner.run arguments
was: DataMiner.run(:resource_names => ['Country'])
now: DataMiner.run(['Country'])
* Rename "resources" to "models"
was: DataMiner.resource_names
now: DataMiner.model_names
* Expect procs instead of lambdas (because they are just instance-eval'ed now, Blockenspiel is no longer guessing where to find methods)
was: :synthesize => lambda { class_method }
now: :synthesize => proc { Klass.class_method }
* Use UnicodeUtils to correctly upcase and downcase
* Use throw/catch instead of exceptions to signal to force a step to stop successfully
was: DataMiner::Succeed
now: throw :data_miner_succeed
* Import steps no longer accept deprecated :table => RemoteTable or :errata => Errata options
* DataMiner::Run structure has changed (in addition to other internals)
* Enhancements
* Easy to modify data miner scripts using DataMiner::Script#{append|prepend|append_once|prepend_once}
* DRYer codebase
* No longer depends on Blockenspiel
* Uses UnixUtils instead of its own spawning code
* Should be threadsafe (no more autoload, has mutexes, more careful/fewer singletons, etc.)
1.1.0
* fixed dependency issues
1.0.0
* bundler and gemspec instead of jeweler
* clear up memory leaks and destructive argument borking
* mostly backwards compatible (but no add_hints! in remote_table transforms, for example)
0.2.6
* Upgrade to remote_table 0.1.6 to handle UTF-8 CSVs and long urls.
0.3.0
* Removed association code... now data_miner focuses on just importing.
* New, simpler DSL
* Upgrade to remote_table 0.2.1 for row_hashes and better blank row handling
* Remove all association-related code