-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathTODO.html
387 lines (385 loc) · 17 KB
/
TODO.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
<!DOCTYPE html>
<html>
<head>
<title>Dataset Project</title>
<link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="https://caltechlibrary.github.io/css/site.css">
</head>
<body>
<header>
<a href="http://library.caltech.edu" title="link to Caltech Library Homepage"><img src="https://caltechlibrary.github.io/assets/liblogo.gif" alt="Caltech Library logo"></a>
</header>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="index.html">README</a></li>
<li><a href="LICENSE">LICENSE</a></li>
<li><a href="install.html">INSTALL</a></li>
<li><a href="user_manual.html">User Manual</a></li>
<li><a href="about.html">About</a></li>
<li><a href="search.html">Search</a></li>
<li><a href="https://github.com/caltechlibrary/dataset">GitHub</a></li>
</ul>
</nav>
<section>
<h1 id="action-items">Action Items</h1>
<dl>
<dt>X</dt>
<dd>
completed
</dd>
<dt>D</dt>
<dd>
declined, decided not to implement
</dd>
</dl>
<h2 id="bugs">Bugs</h2>
<h2 id="next-prep-for-v2.1.23">Next (prep for v2.1.23)</h2>
<ul>
<li><label><input type="checkbox" />Update datasetd to support
urlencoded data submissions in additional to application/json</label>
<ul>
<li>this would allow a simple data entry system to be build directly
from HTML without the need for JavaScript in the browser</li>
<li>the urlencoded data should support embedded YAML in text areas for
extrapolating more complex data structures</li>
<li><label><input type="checkbox" checked="" />Handle POST with
‘application/x-www-form-urlencoded’</label>
<ul class="task-list">
<li><label><input type="checkbox" checked="" />Handle the submissions
mapping create/update to POST</label></li>
<li><label><input type="checkbox" checked="" />Handle the success or
failure of the create/update of POST (using text page for result, will
replace later)</label></li>
<li><label><input type="checkbox" />Figure out what to do success or
failure pages after a POST</label></li>
</ul></li>
<li>[D] Handle PUT with ‘application/x-www-form-urlencoded’
<ul>
<li>Browsers only honor GET, POST and DIALOG in 2024. Frustrating.</li>
<li>Modified POST to work for both Create and Update actions, delete
will get handled like GET but I need to decide on symantics
(e.g. <code>?delete=true</code>)</li>
</ul></li>
<li><label><input type="checkbox" />Integrate models package into
dataset cli and datasetd</label></li>
<li><label><input type="checkbox" />Modify Create and Update in datasetd
to use the models package</label></li>
<li><label><input type="checkbox" />Handle generated elements on Create
and Update</label></li>
<li><label><input type="checkbox" />For forms submited with URL encoding
reply I currently reply with JSON to indicate success or failure, should
return HTML</label>
<ul>
<li><label><input type="checkbox" />Success response should provide
record view of submitted content</label></li>
<li><label><input type="checkbox" />Failure should redirect back to the
form that was submitted</label></li>
<li><label><input type="checkbox" />It’d be nice to style/theme the HTML
for better integration with website</label></li>
<li>Is this a configuration chioice (e.g. success, fail pages in model
attributes?)</li>
</ul></li>
<li><label><input type="checkbox" />Can I can skip the handlebars
templates and just support HTML?</label>
<ul>
<li>Skipping the templates has several advantages
<ul>
<li>one less thing to document</li>
<li>less dependency for datasetd</li>
</ul></li>
<li>If I only support HTML generation then I need to include JavaScript
in the generated code
<ul>
<li>If I do the then PUT and DELETE would work</li>
<li>Downside is it requires JavaScript to update records and submit
them</li>
</ul></li>
</ul></li>
<li><label><input type="checkbox" />Decide if it is exceptible to render
HTML with JavaScript to adjust form behavior between create and
update</label></li>
</ul></li>
<li>[D] Update datasetd to handle YAML submission for create and update:
This didn’t work in my experiments, not sure why.
<ul>
<li>Form handling in browser restricts the mime types submitted, I’d
have to use text/plain to submit YAML then check server side to make
sure I had YAML</li>
</ul></li>
</ul>
<h2 id="someday-maybe">Someday, Maybe</h2>
<ul class="task-list">
<li><label><input type="checkbox" />Should the project be renamed
“collections”?</label></li>
<li><label><input type="checkbox" />Update datasetd to allow multipart
form subission treating file(s) upload as an attachment
request</label></li>
<li><label><input type="checkbox" />Rewrite py_dataset, drop support for
libdataset</label>
<ul class="task-list">
<li><label><input type="checkbox" />Figure out correct approach</label>
<ul class="task-list">
<li><label><input type="checkbox" />Generate WASM module for
libdataset</label></li>
<li><label><input type="checkbox" />Use ts_dataset approach and required
datasetd for Python support</label></li>
<li><label><input type="checkbox" />Rewrite dataset, datasetd and
libdataset in Rust and continue shared library support without built in
GC</label></li>
</ul></li>
</ul></li>
<li><label><input type="checkbox" />create a cli named <code>ds</code>
that wraps all the cli except datasetd similar to how the Go command or
Git works</label></li>
<li><label><input type="checkbox" />My current approach to versioning is
too confusing, causing issues in implementing py_dataset, versioning
needs to be automatic with a minimum set of methods explicitly
supporting it otherwise versioning should just happen in the back ground
and only be supported at the package and libdataset levels.</label>
<ul class="task-list">
<li><label><input type="checkbox" />create, read, update, list
operations should always reflect the “current” version (objects or
attachments), delete should delete all versions of objects as should
prune for attachments, this is because versioning suggests things never
really get deleted, just replaced.</label></li>
</ul></li>
<li><label><input type="checkbox" />Common dataset verbs
(dataset/datasetd)</label>
<ul class="task-list">
<li><label><input type="checkbox" checked="" />keys</label>
<ul>
<li>list the keys in a collection</li>
<li>at the package level keys returns a list of keys and an error
value</li>
</ul></li>
<li><label><input type="checkbox" checked="" />has_key</label>
<ul>
<li>return “true”(w/OS exit 0 in CLI) if key is in collection, “false”
otherwise (w/OS exit 1 in CLI)</li>
</ul></li>
<li><label><input type="checkbox" />sample</label>
<ul>
<li>return a sample of keys from a collection</li>
<li><label><input type="checkbox" />the newly create collections should
have versioning disabled by default</label></li>
</ul></li>
<li><label><input type="checkbox" />create</label>
<ul>
<li>add an new object to the collection if key does not exist, return
false if object already exists or unable to create the new object</li>
<li>if versioning is enabled set the semver appropriately</li>
</ul></li>
<li><label><input type="checkbox" />read</label>
<ul>
<li>return the object with nil error in the collection with the provided
key, nil object and error value if not found</li>
<li>read always returns to the “current” object version</li>
<li><label><input type="checkbox" /><code>read_versions()</code>, list
the versions available for JSON object</label></li>
<li><label><input type="checkbox" /><code>read_version()</code> list an
JSON object for a specific version</label>
<ul>
<li>return the object with nil error in the collection with the provided
key and version, nil object and error value if not found</li>
</ul></li>
</ul></li>
<li><label><input type="checkbox" />update</label>
<ul>
<li>replace the object in the collection for given key, return false is
object does not to replace or replacement fails
<ul>
<li>if collection has versioning turned on then version the object</li>
</ul></li>
<li><label><input type="checkbox" /><code>update()</code> update the
current record respecting the version settings for
collection</label></li>
<li><label><input type="checkbox" checked="" />delete</label>
<ul>
<li>delete the object in the collection for given key, return true if
deletion was successful, false if the object was not deleted (e.g. key
not found or the collection is read only)</li>
<li>if collection has versioning turned on then delete <strong>all
objects</strong>, if you want to revert you just update the object with
the revised object values</li>
<li><label><input type="checkbox" /><code>delete()</code> delete all
versions of an object</label></li>
<li>If versioning is enabled the idea of “deleting” an object or
attachment doesn’t make sense, you only need to support Create, Read,
Update and List, possibly with the ability to read versions available
and retrieve the specific version, is this worth implementing in the
CLI? Or is this just a lib dataset/package “feature”?</li>
</ul></li>
<li><label><input type="checkbox" />versioning, versioning is now set
for the whole collection and effects JSON objects and their attachments
(you’re versioning both or neither), versioning will auto-increment for
patch, minir and major semvere values if set</label>
<ul class="task-list">
<li><label><input type="checkbox" /><code>set_versioning()</code>, set
the versioning on a collection, the following strings enable versioning
“major”, “minor”, “patch”. Any other value disables versioning on the
collection</label></li>
<li><label><input type="checkbox" /><code>get_versioning()</code> on a
colleciton (should return “major”, “minor”, “patch” or ““)</label></li>
</ul></li>
<li><label><input type="checkbox" />Attachment support</label>
<ul class="task-list">
<li><label><input type="checkbox" /><code>attach()</code> will add a
basename file to the JSON object record, if versioning is enabled then
it needs to handle the appropraite versioning setting</label></li>
<li><label><input type="checkbox" /><code>attachments()</code> lists the
attachments for a JSON object record</label></li>
<li><label><input type="checkbox" /><code>attachment_versions()</code>
list versions of a specific attachment</label></li>
<li><label><input type="checkbox" /><code>detach()</code> retrieve
“current” version of attachment</label></li>
<li><label><input type="checkbox" /><code>detach_version()</code>
retrieve a specific version of attachment</label></li>
<li><label><input type="checkbox" /><code>prune()</code> remove all
versions of attachments</label></li>
</ul></li>
<li><label><input type="checkbox" />Data Frame Support</label>
<ul class="task-list">
<li><label><input type="checkbox" />frame_names</label>
<ul>
<li>list the names of the frames currently defined in the
collection</li>
</ul></li>
<li><label><input type="checkbox" />frame</label>
<ul>
<li>define a new frame in the collection, if frame exists replace
it</li>
</ul></li>
<li><label><input type="checkbox" />frame_meta</label>
<ul>
<li>return the frame definition and metadata about the frame (e.g. how
many objects and attributes)</li>
</ul></li>
<li><label><input type="checkbox" />frame_objects</label>
<ul>
<li>return the frame’s list of objects</li>
</ul></li>
<li><label><input type="checkbox" />refresh</label>
<ul>
<li>update all the objects in the frame based on current state of the
collection</li>
</ul></li>
<li><label><input type="checkbox" />reframe</label>
<ul>
<li>replace the frame definition but using the existing frame’s keys
refresh the frame with the new object describe</li>
</ul></li>
<li><label><input type="checkbox" />delete_frame</label></li>
<li><label><input type="checkbox" />has_frame</label></li>
</ul></li>
</ul></li>
</ul></li>
<li><label><input type="checkbox" />Verbs supported by cli only</label>
<ul class="task-list">
<li><label><input type="checkbox" />set_versioning (accepts ““,”patch”,
“minor”, or “major” as values)</label></li>
<li><label><input type="checkbox" />get_versioning (returns collection’s
version setting)</label></li>
<li><label><input type="checkbox" />keys</label></li>
<li><label><input type="checkbox" />create (if versioning is enable then
handle versioning)</label></li>
<li><label><input type="checkbox" />read (if versioning is enabled
return the current version of an object)</label></li>
<li><label><input type="checkbox" />update (if versioning is enable then
handle versioning)</label></li>
<li><label><input type="checkbox" />delete (if versioning is enable,
delete all versions of object and attachments)</label></li>
<li><label><input type="checkbox" />sample</label></li>
<li><label><input type="checkbox" />clone</label></li>
<li><label><input type="checkbox" />clone-sample</label></li>
<li><label><input type="checkbox" />check</label></li>
<li><label><input type="checkbox" />repair</label></li>
<li><label><input type="checkbox" />frames (return a list of frames
defined for collection)</label></li>
<li><label><input type="checkbox" />frame, define a new frame in a
collection</label></li>
<li><label><input type="checkbox" />frame_objects, return the object
list from a frame</label></li>
<li><label><input type="checkbox" />refresh, refresh all objects in a
frame based on the current state of the collection</label></li>
<li><label><input type="checkbox" />reframe, replace the frame
definition but using the existing frame’s keys for the object listed in
frame</label></li>
<li><label><input type="checkbox" />delete_frame, remove a frame from
the collection</label></li>
<li><label><input type="checkbox" />has_frame return true if frame
exists or false otherwise</label></li>
<li><label><input type="checkbox" />attachments, list the attachments
for a JSON object in the collection</label></li>
<li><label><input type="checkbox" />attach, add an attachment to a JSON
object in the collection, respect versioning if enabled</label></li>
<li><label><input type="checkbox" />detach, retrieve an attachment from
the JSON object in the collection</label></li>
<li><label><input type="checkbox" />prune, remove attachments (including
all versions) from an JSON object in the collection</label></li>
</ul></li>
<li><label><input type="checkbox" />Add support for segmented key maps
(to limit memory consumption for very large collections) settings in
collection.json using keywords of patch, minor, major</label></li>
<li><label><input type="checkbox" />Auto-version attachments by patch,
minor or major release per</label></li>
<li><label><input type="checkbox" />Need to add getting updated Man
pages using the <code>dataset help ...</code> command</label></li>
<li><label><input type="checkbox" />Allow a WASM module to be used to
validate objects in the collection. It needs to me integrate such that
it “travels” will the dataset collection</label>
<ul>
<li>this would let our JSON collections support explicit JSON structures
as well as ad-hoc JSON objects</li>
<li>could use the YAML model approach in Newt to define the
structures</li>
</ul></li>
<li><label><input type="checkbox" />Document an example Python 3 http
client support for web API implementing a drop in replacement for
py_dataset using the web service or cli</label></li>
<li><label><input type="checkbox" />Implement a wrapping logger that
takes a verboseness level for output (e.g. 0 - quiet, 1 progress
messages, 2 warnings, errors should always show)</label></li>
<li><label><input type="checkbox" />Add support for https:// based
datasets (in addition to local disc and s3://)</label></li>
<li><label><input type="checkbox" />dsbagit would generate a “BagIt” bag
for preservation of collection objects</label></li>
<li><label><input type="checkbox" />dsgen would take a model described
in YAML and generate HTML and browser side ES6 for quick prototyping
with datasetd</label></li>
<li><label><input type="checkbox" />OAI-PMH importer to prototype iiif
service based on Islandora content driven by a dataset
collection</label></li>
<li><label><input type="checkbox" />Implement version support in the web
service</label></li>
<li><label><input type="checkbox" />Implement an integrated a web UI for
managing dataset collections and their data structures</label>
<ul class="task-list">
<li><label><input type="checkbox" />Form pages could be expressed in
Markdown+YAML for forms and embedded in the datasetd settings YAML
file</label>
<ul>
<li>See my notes on my text oreinted web experiment,
yaml2webform.go</li>
<li>Forms could be render into the htdocs auto-magically saving
development effort</li>
<li>The same forms could then be used server side for validation based
on descriptors and JavaScript converted to WASM code</li>
</ul></li>
<li><label><input type="checkbox" />A standard JavaScript library could
be used to knit the forms to the datasetd web service (sort of a
mini-newt) It would be nice if citesearch was defined by the
citesearch.yaml file and some markdown documents taking a text oriented
web approach to embedding forms in Markdown combined with some JS glue
code to knit the two together</label></li>
</ul></li>
</ul>
</section>
<footer>
<span>© 2022 <a href="https://www.library.caltech.edu/copyright">Caltech Library</a></span>
<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address>
<span><a href="mailto:[email protected]">Email Us</a></span>
<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
</footer>
</body>
</html>