This repository has been archived by the owner on Jul 12, 2024. It is now read-only.
forked from pothiers/tada
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathRELEASE-NOTES.txt
227 lines (191 loc) · 8.32 KB
/
RELEASE-NOTES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
* Release 2.7.1
- TADA-2: add TARGET param to direct_post (explicit relative
directory for ingest file)
- Ingest svc timeout can be set from hiera
+ e.g. "arch_timeout: 9" (defaults to 10 seconds)
* Release 2.6.1
- fix TADA-21; Now removes HDR when ingest fails due to PROPID missing
from Archive DB
* Release 2.5.16
- hotfix: On mtn fpack from dropbox rename X.fits to X.fits.fz
- feature: use fitsverify (MAY CAUSE INGESTS THAT PREVIOUSLY WORKED TO FAIL)
+ mtn (against dropbox)
+ val (on file to ingest)
+ REQUIRES PROVISIONING CHANGE
- install "fitsverify" into /usr/local/bin
- simplify Auditor class init
- highest level exception protect on submit
- tic/toc timings in some places (iput, archive svc)
- replace uses of urllib with "requests" package
- addition errcode coding for audit
- smokes for dropbox can now be done file by file (with pass/fail expects)
- change in error reporting (in logs)
- use different RPM creation mechanism; now store as python-tada-*.rpm
+ REQUIRES CHANGE to ~yum update~
- REMOVED "pre_action" functionality from TADA mountain processing.
This was no longer used (mostly replaced by inline fpack processing)
and was a feature that could screw things up badly.
- Contains known bug (jira DEVEL-675)
* Release 2.5.15 <2016-08-04 Thu>
- fix bug causing ingestion of corrupted files
* Release 2.5.8 <2016-06-29 Wed>
- Add instrument: ccd_imager
- change personality location to /var/tada
- add jobid_type=obsmicro for additional field in Archive filename
using DATE-OBS microseconds (for arcoiris)
- add IRODS prefix to HDR path for call to Archive Ingest service
* Release 2.5.0 <2016-06-17 Fri>
Always trust schedule. After gaining experience using the TAC
schedule, we have come to believe it should be trusted more than the
FITS header for Propid. However, when the schedule indicates a split
night (multiple proposals in one observing night) the schedule
necessary but not sufficient. For split nights, we use the Propid
from the header as long is it is in the schedule for the night. We've
added a "default" for each instrument to the schedule so that every
day does not have be entered for every instrument. All "*trust*"
functions used in personalities ("calchdr" section) are deprecated
(they do nothing except emit warning).
Because of the strong reliance on the TAC schedule for Propid, this
release is tightly tied to the MARS schedule service. We no longer
view that service as optional. If the schedule service is down,
Archive Ingest will be rejected. Changes to the MARS schedule service
were made to support this release.
* Release 2.4.0 <2016-04-14 Thu 07:50>
This updates the version numbers to match Major.Minor version of TADA puppet
code. Previous RPM version was 0.2.17
Config now read from TWO files. One of them (/etc/tada/hiera.yaml) is
generated from puppet. This allows hiera to be used to specify host
specific values such as the Valley that a Mountain should transfer
files to.
* Release 0.2.13 <2016-03-24 Thu 14:49>
- TADA
+ lossless on-the-fly fpack for floating point images
- If two indentical file names put in dropbox except one has fz,
fz is submited, other ignored.
- If one fz file put in dropbox, is submitted as is (no recompress)
- If one non-fiz file put in dropbox, compressed lossless, submitted
+ reform all YAML files (better human readibility)
+ regen personality table from YAMLs
+ drop fail gracefully on bad directory format
+ drop working to Mtn and Valley; name changes to reflect ("watch_drop")
- dataque
+ added function for direct programatic push of file
+ added configurable logging
* Release 0.2.13 <2016-03-24 Thu 14:49>
- TADA
+ lossless on-the-fly fpack for floating point images
- If two indentical file names put in dropbox except one has fz,
fz is submited, other ignored.
- If one fz file put in dropbox, is submitted as is (no recompress)
- If one non-fiz file put in dropbox, compressed lossless, submitted
+ reform all YAML files (better human readibility)
+ regen personality table from YAMLs
+ drop fail gracefully on bad directory format
+ drop working to Mtn and Valley; name changes to reflect ("watch_drop")
- dataque
+ added function for direct programatic push of file
+ added configurable logging
* Release 0.2.11 <2016-03-18 Fri 12:28>
- Create HDR from concatenation of all HDUs (in order)
Previously used only Primary. For some cases of fpacked files
this resulted in the HDR file not containing RA & DEC (etc.)
* Release 0.2.10 <2016-03-17 Thu 14:37>
- prefix table name change (triplespec -> arcoiris) and new prefix (c4ai)
- add mapping_function source fields to list of "USED_FIELDS" that may
be updated
- add on the fly fpack for dropbox (NOT ADEQUATELY TESTED)
- improved format of YAML output from personaltiy(bash)-to-yaml
- more smoke tests (uncompressed mosaic3, arcoiris)
- removed legacy mosaic_1_1 smoke
* Release 0.2.4 <2016-02-03 Wed 07:50>
Don't modify DTACQNAM if it already has a value.
Includes replacement for LPR front-end to TADA. Now you can submit a set
of files to TADA (on the mountain host) from anywhere with something like:
rsync -avz --password-file ~/rsync.pwd $srcdir tada@$mtnhost::dropbox
In the above, the rsync.pwd file should have the same contents as
mtnhost:/etc/tada/rsync.pwd, should be owned by you, and should have
permissions of 0400. Your $srcdir directory structure should match:
<YYYYMMDD>/<instrument>/.../myfile.fits.fz
Where YYYYMMDD is the observation day (DTCALDAT) corressponding to the
FITS files underneath. And <instrument> is currently one of:
bok23m-90prime
ct13m-andicam
ct15m-echelle
ct4m-cosmos
ct4m-decam
kp09m-hdi
kp4m-kosmos
kp4m-mosaic_1_1
kp4m-mosaic3
kp4m-newfirm
soar-goodman
soar-osiris
soar-sami
soar-soi
soar-spartan
wiyn-bench
wiyn-whirc
You can tell if your files have been successfully submitted (to the
mountain TADA queue) with:
rsync -a --password-file ~/.tada/rsync.pwd tada@$mtnhost::statusbox $mydir
find $mydir -type f
* Release 0.2.2 <2016-01-27 Wed 07:50>
Includes replacement for LPR front-end to TADA. Now you can submit a set
of files to TADA (on the mountain host) from anywhere with something like:
rsync -avz --password-file ~/rsync.pwd $srcdir tada@$mtnhost::dropbox
In the above, the rsync.pwd file should have the same contents as
mtnhost:/etc/tada/rsync.pwd, should be owned by you, and should have
permissions of 0400. Your $srcdir directory structure should match:
<YYYYMMDD>/<instrument>/.../myfile.fits.fz
Where YYYYMMDD is the observation day (DTCALDAT) corressponding to the
FITS files underneath. And <instrument> is currently one of:
bok23m-90prime
ct13m-andicam
ct15m-echelle
ct4m-cosmos
ct4m-decam
kp09m-hdi
kp4m-kosmos
kp4m-mosaic_1_1
kp4m-mosaic3
kp4m-newfirm
soar-goodman
soar-osiris
soar-sami
soar-soi
soar-spartan
wiyn-bench
wiyn-whirc
You can tell if your files have been successfully submitted (to the
mountain TADA queue) with:
rsync -a --password-file ~/.tada/rsync.pwd tada@$mtnhost::statusbox $mydir
find $mydir -type f
* Release 0.1.29 <2016-01-20 Wed 14:37>
Added service that watches for any FITS files added below
mountain:/var/tada/dropbox/. Submits an new (or modified) files to
TADA and adds a corresponding entry in /var/tada/statusbox/ if the
submit succeeded. Currently that info isn't used but could be used to
determine which files under /var/tada/dropbox should be touch to cause
them to be submitting again.
Provisioning changes made to make Mountain and Valley machines more
nearly identical in terms of provisioning.
* Release 0.1.25 <2016-01-08 Fri 14:37>
- Obstype entry added for mosaic3
- smoke overhaul
* Release 0.1.22 <2015-12-17 Thu 14:37>
- Support dry-run (no ingest) using direct_submit; Use " -o
__dry_run=1" in personality
- Fix to temporary file copy on direct_submit. Previously, would be non-writable
under some circumstances.
* Release 0.1.21 <2015-12-16 Wed 14:37>
Hotfix in support of BOK batch ingests.
- Abort ingest if *.hdr file exists under iRODS. To disable abort (but
still warn), add "__test_resubmit=1" to personality.
- Fixed duplicate EXTNAME in bok data
- All header mods now done to PrimaryHDU and include only fields in
defined list (try: "fitsheader baz.fits.fz | grep MODIFIED:")
* Release 0.1.18 <2015-12-07 Mon 14:37>
nothing here
// Local variables:
// mode: org
// end: