forked from zeek/spicy
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES
1877 lines (1213 loc) · 66.9 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
0.4.0-1112 | 2020-12-10 10:33:31 +0000
* GH-25: Add support for manual backtracking to Spicy. If a field is
marked with ``&try``, a later call to ``self.backtrack()``
anywhere down in the parse tree will return to that position and
continue there. See the documentation for more, including an
example. (Robin Sommer, Corelight).
* Support &eod for vector parsing. (Robin Sommer, Corelight)
* Fix Sphinx helper script that emits command lines. (Robin Sommer,
Corelight)
0.4.0-1106 | 2020-12-10 10:10:16 +0000
* Make capture groups available inside hooks of regular expression fields.
(Robin Sommer, Corelight)
Example:
x: /(a.c)(de*f)(h.j)/ {
print $1, $2, $3;
}
This will print the pieces corresponding to the three groups inside the
expression.
Couple notes:
- We don't make $0 available as that would be the same as `$$`.
- Matching with capture groups is more expensive that without. If
you don't use any groups in the regexp, the overhead is
automatically removed. If you use groups, but don't need them to
capture, add `&nosub` to the field.
* Make new captures() method available on regexp matching state in
HILTI. (Robin Sommer, Corelight)
* Fix argument mismatches between Spicy generated C++ prototypes and
Zeek plugin runtime code. (Robin Sommer, Corelight)
* Fix compile error on non-JIT version of plugin, and fix unsafe use
of string_view. (Robin Sommer, Corelight)
* Fix warnings reported by Apple's compiler. (Robin Sommer,
Corelight)
0.4.0-1093 | 2020-12-10 08:39:26 +0100
* Switch runtime library to use "fiber" library instead of libtask.
We now leverage https://github.com/simonfxr/fiber. (Benjamin
Bannier, Corelight)
* GH-599: Fix construction of `Bytes`. (Benjamin Bannier, Corelight)
* GH-603: Disable precompiled headers if we are building with GCC.
(Benjamin Bannier, Corelight)
* GH-94: Explicitly enable leak checking on tests relying on it being
active. (Benjamin Bannier, Corelight)
* GH-492: Add unit tests for IntrusivePtr. (Benjamin Bannier,
Corelight)
* Make run-zeek script also work with ASAN on macOS. (Benjamin
Bannier, Corelight)
* Remove unused Zeek compiler driver variable. (Benjamin Bannier,
Corelight)
* Add missing runtime cleanups in a few places. (Benjamin Bannier,
Corelight)
* Fix some documentation links and link checking. (Benjamin Bannier)
* Fix GitHub link. (Robin Sommer, Corelight)
0.4.0-1071 | 2020-11-30 18:16:35 +0000
* Series of speed-ups to the Spicy compiler. (Robin Sommer, Corelight)
- Add caching of derived information to some AST nodes.
- Use optional_ref<T> for returning references of AST node attributes instead of copies. (Robin Sommer, Corelight)
- Cache location instances.
- Returning const references for lots of accessor methods. (Robin Sommer, Corelight)
- Mark two AST node classes as singletons.
- Switch AST infrastructure from shared_ptr to IntrusivePtr.
- Use forward declaration for nlohmann:json where we can.
* Relabel profiling collector names in JIT code. (Robin Sommer, Corelight)
* Fix for type erasure profiling. (Robin Sommer, Corelight)
0.4.0-1057 | 2020-11-23 08:35:04 +0000
* GH-584: Add new vector methods resize/sub/at. (Robin Sommer, Corelight)
* Revert accidental unbumping of tinyformat. (Benjamin Bannier, Corelight)
0.4.0-1053 | 2020-11-18 08:33:32 +0000
* GH-568: Change parsing of look-ahead vectors to use a loop instead
of recursion. (Robin Sommer, Corelight)
* GH-583: Fix `foreach` inside a unit hook. The `foreach` attribute
wasn't passed on, and the '$$' lookup didn't work inside a
non-inline hook. (Robin Sommer, Corelight)
* GH-574: Implement and document hook priorities. Robin Sommer,
Corelight)
* GH-389: Add additional tuple constructor expression (`tuple(...)`).
(Robin Sommer, Corelight)
* GH-544: Let Zeek plugin search for HLTO files at startup. By
default, the plugin now loads all *.hlto files found in
`$prefix/lib/spicy/Zeek_Spicy/modules/`. This can be changed by
setting the environment variable SPICY_MODULE_PATH to a
colon-separated list of directories to search. (Robin Sommer,
Corelight)
* Force upgrade to well-defined BTest version in our Docker
containers. (Robin Sommer, Corelight)
* Forward Zeek plugin's debug messages to Zeek's debug logger. That
means that 'zeek -B plugin-_Zeek-Spicy' will send them into Zeek's
debug.log. (Robin Sommer, Corelight)
0.4.0-1031 | 2020-11-17 14:05:32 +0100
* GH-571: Fix padding handling when formatting nan and inf values.
This patch bumps tinyformat to a version containing the upstream fix for
https://github.com/c42f/tinyformat/issues/76 (and the follow up fix
https://github.com/c42f/tinyformat/pull/80).
We also add a regression test.
Closes #571. (Benjamin Bannier, Corelight)
* GH-589: Fix "'_debug’ was not declared in this scope" build error (Robin Sommer, Corelight)
* Mention header precompilation in installation docs. (Benjamin Bannier, Corelight)
* Disable precompiled headers for gcc CI job.
While mixing GCC and Clang with precompiled headers is an issue for JIT
(JIT always uses Clang to compile modules while precompiled headers are
created with the compiler used to compile Spicy itself), we disable
precompiled headers for the non-JIT, GCC job since GCC emits a
non-suppressible warning, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89808,
build/hilti/runtime/include/hilti/rt/libhilti.h:3:9: error: #pragma once in main file [-Werror]
3 | #pragma once
| ^~~~
cc1plus: all warnings being treated as errors
Instead of not treating warnings as errors anymore we disable precompiled
headers in this build since it does not use them anyway. (Benjamin Bannier, Corelight)
* Add missing include. (Benjamin Bannier, Corelight)
* Optionally precompile libhilti.h.
This patch adds precompilation of libhilti.h for developer setups. This
header is included in all files during JIT, and precompiling it can for
individual files reduce JIT time by up to 30%. This behavior can be
disabled by setting the CMake option `HILTI_DEV_PRECOMPILE_HEADERS=OFF`.
Specifically, we add a cmake target which creates debug and release
versions of a precompiled libhilti.h. We modify HILTI's built-in
CXXFLAGS to check for the presence of the needed precompiled header in
the expected location. We expect the header either in a specific
location in the build directory, or optionally in a version directory in
`$HOME/.cache/spicy`.
In this patch we enable use of the precompiled header only during JIT,
but e.g., not when building Spicy targets. This allows us to profit from
faster JIT while still making efficient use of e.g., ccache.
Additionally, this approach allows us to punt on fixing a number of
issues in header files which get exposed during JIT (JIT artifacts are
not affected at the moment).
The original goal was to precompile lihilti.h before installation which
is not reached. In general, it seems that precompiled headers are not
intended to be moved after creation. This poses a challenge when the
header gets installed: while we could in principle generate the
precompiled header at the final location during install time, this seems
not reliably possible when generating packages which can be expanded in
arbitrary locations.
Additionally, we also install a script for users to take advantage of
precompiled headers once Spicy has been installed. (Benjamin Bannier, Corelight)
0.4.0-1020 | 2020-11-12 11:04:57 +0000
* Fix performance regression by removing wasteful formatting of debug message
in release mode in Spicy runtime driver. (Robin Sommer, Corelight)
* Fix fiber result handling on abort. (Robin Sommer, Corelight)
* GH-578: Fix glib decompression. Robin Sommer, Corelight)
* Add helper tool 'spicy-batch-extract' that extracts a single session out
of a Spicy batch file. (Robin Sommer, Corelight)
* Add preliminary benchmark script `scripts/run-benchmarks`. See developer
docs for more. (Robin Sommer, Corelight)
0.4.0-1011 | 2020-11-05 17:09:53 +0100
* GH-451: Support building on centos7 (Benjamin Bannier, Corelight)
0.4.0-1006 | 2020-11-05 12:56:56 +0000
* GH-497: Add support for packet analyzers to Zeek plugin. (Robin Sommer,
Corelight)
Zeek's master branch received support for adding low-level "packet
analyzers" through plugins. This extends the Spicy plugin to support that.
The documentation gained a new section explaining how to sue that.
Note: This feature will not "officially" be supported until Zeek 4.0
comes out (because we don't have Zeek master in CI).
* Add --zeek-version argument to spicy-config. (Robin Sommer, Corelight)
* Compile fix for GCC on Ubuntu. (Robin Sommer, Corelight)
0.4.0-1001 | 2020-11-05 08:47:23 +0000
* Use stable names for hook implementations, plus a text fix.
0.4.0-999 | 2020-11-04 15:18:45 +0100
* Use a lightweight lambda class to callbacks executing on fibers. (Benjamin Bannier, Corelight)
* Require all callbacks executing on fibers to return something. (Benjamin Bannier, Corelight)
* Remove unused type_info::overload. (Benjamin Bannier, Corelight)
* GH-546: Model `TypeInfo` with a union instead of a `std::variant`. (Benjamin Bannier, Corelight)
0.4.0-994 | 2020-11-03 09:22:40 +0000
* Update test baseline. (Robin Sommer, Corelight)
0.4.0-993 | 2020-11-02 16:09:13 +0000
* GH-159: Add some missing operator<< to runtime library. This now lets
"-X trace" work for Spicy parsers. (Robin Sommer, Corelight)
* GH-531: Make generated grammar symbols globally unique during Spicy
codegen. (Robin Sommer, Corelight)
* GH-176: Avoid infinite recursion when printing ASTs with self-recursive
types. (Robin Sommer, Corelight)
* Fix premature library shutdown in spicy-dump, which could lead to crashes
during cleanup of the parsed unit. (Robin Sommer, Corelight)
* Fix error message in validator. (Robin Sommer, Corelight)
* Aggressively compress package artifact for static build. (Benjamin Bannier,
Corelight)
* Remove packaging tasks for static CI build. Static builds tend to produce
very large binaries which cannot reliably be compressed to be inside Cirrus
CI's 1GB artifact size limit. (Benjamin Bannier, Corelight)
* Fix documentation source link. (Benjamin Bannier, Corelight)
* Consistently use `hilti::rt::filesystem` instead of `std::filesystem`.
(Benjamin Bannier, Corelight)
0.4.0-979 | 2020-10-23 14:11:27 +0000
* GH-95: Zeek plugin: Respect %port and %mime-type properties in
units. (Robin Sommer, Corelight)
* GH-513: Catch if an interval's number of seconds is out of range.
(Robin Sommer, Corelight)
* Clean up parsers' bounds checking of signed integer values. (Robin
Sommer, Corelight)
* GH-515: Change {time,interval}::nanoseconds() to return integers
instead of real. (Robin Sommer, Corelight)
* Fix bashism in test helper script. (Robin Sommer, Corelight)
* GH-490, GH-507: Provide error message when attempting to have
spicy-driver compile inputs when JIT is not available. (Robin
Sommer, Corelight)
* Reject regexp unit fields that aren't constant. (Robin Sommer, Corelight)
* GH-524: Fix spicy-driver usage message. (Robin Sommer, Corelight)
* GH-526: Show executable's name in usage/error messages of binaries
replying on the spicy-driver host code. (Robin Sommer, Corelight)
* GH-527: Updating usage output in docs. (Robin Sommer, Corelight)
* GH-528: Clarify spicyc's error message on needing HLTO filenames
with -o. (Robin Sommer, Corelight)
* GH-529: Clarifying libhilti's usage() fallback message. (Robin
Sommer, Corelight)
* GH-540: Fix bug in regular expression parsing. (Robin Sommer,
Corelight)
* GH-270: Fix for unit filters. (Robin Sommer, Corelight)
* GH-500: Remove --include-linker option from driver. The better way
to do all this is using "-l" to generate *just* the linker code,
then compile the two together as two compilation units. (Robin
Sommer, Corelight)
* GH-499: Provide has_label() method for enum values. Returns true
if the enum maps to a known type value other than Undef. (Robin
Sommer, Corelight)
* GH-556: Fix coercion of optional. (Robin Sommer, Corelight)
* GH-481: Correctly set locale in centos8 image. (Benjamin Bannier,
Corelight)
* Use working and stable link in host application documentation.
(Benjamin Bannier, Corelight)
* Fix documentation to reference latest support alpine version.
(Benjamin Bannier, Corelight)
* GH-546: Use plain function pointers for type info accessors.
(Benjamin Bannier, Corelight)
0.4.0-946 | 2020-10-23 09:35:04 +0200
* GH-522: Switch filesystem impl over to ghc::filesystem.
Closes #522. (Benjamin Bannier, Corelight)
* Fix use of uninitialized value when numbering unit fields.
When were using a non-const member function to call a delegated base
class constructor. Since at the point where the member function was
called the derived class was not fully constructed, the members it
accessed and mutated were uninitialized, we were reading from an unset
variable, attempting to set it, and then overwrote its values when the
member initializer was triggered (after the base class was constructed).
This lead to unit field indices set to large value which were not usable
down the line.
We now delegate maintaining the field counter to a mixin class which can
be constructed before the base class needing its state. (Benjamin Bannier, Corelight)
0.4.0-939 | 2020-10-21 14:35:30 +0200
* GH-521: Do not create copies when loading HLTO libraries. (Benjamin Bannier, Corelight)
0.4.0-935 | 2020-10-12 07:13:46 +0000
* Add support for new %print hook to Spicy units. (Robin Sommer,
Corelight)
This allows a unit to customize its appearance when printed (or
more generally: rendered into a string).
Example:
# cat test.spicy
public type Test = unit {
on %init { print self; }
f1: uint32;
f2: uint8;
on %done { print self; }
on %print {
local f1 = "-";
local f2 = "-";
if ( self?.f1 )
f1 = "%s" % self.f1;
if ( self?.f2 )
f2 = "%s" % self.f2;
return "%s|%s" % (f1, f2);
}
};
# printf '\x00\x00\x00\x01\x02' | spicy-driver test.spicy
-|-
1|2
* Allow customizing the string rendering of HILTI structs. If a
struct provide a hook __str__(), that will be used instead of the
built-in rendering of all its fields. (Robin Sommer, Corelight)
* Make ID getter method constant. (Robin Sommer, Corelight)
0.4.0-930 | 2020-10-09 18:10:13 +0200
* Change `Stream::data` and `stream::View::data` to return a `Bytes`
instead of a `string`. (Benjamin Bannier, Corelight)
* Test & CI updates. (Benjamin Bannier, Corelight)
- Add a CI master step to validate Homebrew formula works.
- Add missing Cirrus CI instance to Homebrew task.
- Build compiler toolchain by default when configuring directly from CMake.
- Bump default Zeek package in Docker images to more recent version.
- Fix removal of build directory in CentOS 8 Docker container.
- Increase sleep amount in test to make time measurable in CI.
- Speed up test `spicy.types.unit.trimming`.
* Help clang-tidy understand control flow in a couple of places.
(Benjamin Bannier, Corelight)
0.4.0-918 | 2020-10-06 12:23:08 +0200
* Introduce the notion of Spicy-specific compiler options.
So far, we only had codegen options for the HILTI compiler. This
change allows Spicy to track options as well that control specifics of
how the Spicy-to-HILTI code generations operates. Includes a couple of
things:
- As there's no appropiate global state on the Spicy side
where we could story options easily, we extend the
HILTI-side options struct with the notion of "auxiliary
options", described by key/value pairs. Spicy can then store
its options in there.
- We derived a separate spicy::Driver from hilti::Driver to
customize behaviour.
- We introduce hooks into the hilti::Driver to add support for
additional command line options. This remains a bit ad-hoc
for now. At some point, we'll clean up of command-line
handling generally works across the various tools, we have
quite a bit of code duplication there right now.
- This introduces a Spicy-side `-Q` option to record stream
offsets during parsing. Right now, the option isn't having
any effect yet, the implementation is going to come in a
subsequent commit. spicy-dump's previous `-o` is renamed to
`-Q` as well. (Robin Sommer, Corelight)
* Add an option to emit data offsets from spicy-dump.
This patch adds an option `-o` to `spicy-dump` which can be used to
trigger outputting of the stream offsets where unit fields were parsed
from. (Benjamin Bannier, Corelight)
* Make parser struct `__offset` member accessible from Values. (Benjamin Bannier, Corelight)
* Store field offsets during parsing. (Benjamin Bannier, Corelight)
* Make unit fields store their unit-wide field counter.
Initially the parser will create unit fields as `UnresolvedField`s. This
patch adds an `_index` field and accessor for this field which will
be used to number all unit fields in the final, fully constructed and
resolved unit so offsets can be computed. (Benjamin Bannier, Corelight)
* Store whether a struct field is internal on type level. (Benjamin Bannier, Corelight)
* Fix `type::struct_::Field::is*` methods to return booleans.
These functions were previously returning optional field values which
carry no information beyond being set or not. This patch changes these
methods to return booleans instead. (Benjamin Bannier, Corelight)
* Fully type constructions of empty optional values. (Benjamin Bannier, Corelight)
* Make `rt::Map::contains` `const`. (Benjamin Bannier, Corelight)
0.4.0-906 | 2020-10-06 08:11:53 +0200
* GH-511: Fix order of flags in spicyc usage message.
* GH-512: Fix typo in error message.
0.4.0-903 | 2020-09-28 18:42:38 +0000
* Reorganize source code to allow for building the runtime library
standalone. This adds a corresponding configure option:
--build-toolchain=yes/no (default: yes)
* Add configure option --disable-zeek-plugin-install to disable
installing the Zeek plugin into Zeek's plugin path. (Robin Sommer,
Corelight)
* Add CMake module that enables Zeek plugins to build Spicy parsers.
* Rename Zeek plugin to "_Zeek::Spicy" to work-around a loading
order issue in Zeek. (Robin Sommer, Corelight)
* Turn all 3rdparty code into proper git submodules. (Robin Sommer,
Corelight)
* Make using Gold linker default in CMake config. This is no change
from when using "configure", but it changes the default when
running "cmake" directly. (Robin Sommer, Corelight)
* Fix configuration of Gold linker to apply to cmake modules as
well. (Robin Sommer, Corelight)
* Extend hilti-config and spicy-config with options to print the
runtime's C++ include directories. (Robin Sommer, Corelight)
* Avoid needing "which" in configure. (Robin Sommer, Corelight)
* Disable a couple unit tests when running as root. (Robin Sommer,
Corelight)
* Fix compiler warning. (Robin Sommer, Corelight)
0.4.0-890 | 2020-09-28 18:41:08 +0000
* Port Zeek plugin to current Zeek master. (Robin Sommer, Corelight)
* GH-508: Prevent loss of precision when codgening Time
constructors. (Benjamin Bannier, Corelight)
* GH-503: Fix constructor `interval_ns` to accept unsigneds like
documented. (Benjamin Bannier, Corelight)
* Include `netinet/in.h` for `in[6]_addr` in `rt/address.h`.
(Benjamin Bannier, Corelight)
* Fix whitespace in spicy-driver batch usage string. (Benjamin
Bannier, Corelight)
0.4.0-884 | 2020-09-08 07:39:00 +0000
* Reimplement HILTI's stream type to substantially improve
performance. (Robin Sommer, Corelight)
* Move resolving of exceptions' backtrace symbols to point-of-use
for improving performance. (Robin Sommer, Corelight)
* Fix problem with fiber reuse. (Robin Sommer, Corelight)
* Fix a unit test that's using multiple filters. (Robin Sommer,
Corelight)
* Add macOS work-around for problem with doctests' platform
detection. (Robin Sommer, Corelight)
* When configured for debug mode, build tests without -O and link
them against {hilti,spicy}-rt-debug. (Robin Sommer, Corelight)
0.4.0-874 | 2020-08-27 14:21:57 +0200
* Update parser lookup tables exclusively from `spicy::rt::init`.
We previously were updating `globalState()->parsers_by_mime_type` from
`registerParser` while `globalState()->parsers_by_name` was updated by
`init`. This patch makes `init` the only function updating the parser
lookup tables `parsers_by_name` and `parsers_by_mime_type`. (Benjamin Bannier, Corelight)
* Remove unintentional `return` in `rt::haveEod`. (Benjamin Bannier, Corelight)
* Avoid creating vain global state in `spicy::rt::{done,isInitialized}`. (Benjamin Bannier, Corelight)
* Reinitialize default parser when reinitializing spicy::rt library. (Benjamin Bannier, Corelight)
0.4.0-861 | 2020-08-24 13:04:33 +0200
* Diagnose invalid `zlib::Stream`s.
When a `zlib::Stream` enters a failed state it cannot be reused
anymore. Previously we would return empty data whenever such a stream
was used again nevertheless. Since empty data is a valid result of
decompressing non-empty input this leaves users with no good way to
catch such scenarios after the fact.
This patch modifies `zlib::Stream::decompress` to now throw an exception
instead of returning valid but empty data. (Benjamin Bannier, Corelight)
0.4.0-855 | 2020-08-24 10:56:17 +0000
* Fix Cirrus CI file for Alpine 3.12. (Robin Sommer, Corelight)
0.4.0-854 | 2020-08-24 08:41:23 +0000
* Switch Dockerfile for Alpine from 3.11 to 3.12.
* Compile code with debug information also when in release mode.
(Robin Sommer, Corelight)
* Compile the toolchain with -O3 instead of -O2. (Robin Sommer,
Corelight)
* Always link against thread library. The performance of some of our
runtime code depends on whether the executable is linked against
pthread or not. (Robin Sommer, Corelight)
0.4.0-849 | 2020-08-24 08:39:14 +0000
* Fiber runtime library updates. (Robin Sommer, Corelight)
- Fix state tracking in fibers. When taking a fiber from the
cache, we would reset its state ot INIT, which then led to
restarting it from scratch instead of reusing the already
running loop.
- Add a function to fill the fiber cache with preinitialized
instances.
- Track number of fiber initialization in statistics.
- Unify fiber debug logging.
0.4.0-843 | 2020-08-21 13:10:42 +0200
* Consistently deal with `std::filesystem` still being experimental in hilti-rt. (Benjamin Bannier, Corelight)
0.4.0-840 | 2020-08-21 10:04:02 +0200
* Prevent use after free in `rt::base64::finish`. (Benjamin Bannier, Corelight)
0.4.0-833 | 2020-08-19 11:06:44 +0200
* Add assertion catching out-of-bounds HILTI module access. (Benjamin Bannier, Corelight)
* Add check to prevent nullptr dereference in `hilti::rt::done`. (Benjamin Bannier, Corelight)
* Add missing context cleanup in `hilti::rt::done`.
We configure a `Context` owned by `GlobalState` in `hilti::rt::init`.
Since we destruct that context in `hilti::rt::done` we also need to
reset the context as we would otherwise leave a dangling refernence
behind. (Benjamin Bannier, Corelight)
* Call out gitlint in commit message style guidelines. (Benjamin Bannier, Corelight)
* Remove guidance on not rebasing development branches. (Benjamin Bannier, Corelight)
* Add developer manual section on pre-commit linters. (Benjamin Bannier, Corelight)
* Add pre-commit linter for commit messages. (Benjamin Bannier, Corelight)
0.4.0-819 | 2020-08-19 11:05:55 +0200
* Explicitly set locale in centos Docker container.
`hilti::rt::init` needs to be able to set the locale on the host system
and will abort with a fatal error if it cannot do so. This patch makes
sure this is possible in the centos-8 Docker container provided by us.
Note: This already worked in BTests since its config file enforces the
needed environment variables. This patch fixes it also for standalone
usage of Spicy (and now e.g., the `locale` command emits no more
errors). (Benjamin Bannier, Corelight)
0.4.0-817 | 2020-08-18 11:24:52 +0200
* Add `hilti::rt::Library::symbol`.
This patch adds a new member function `Library::symbol` which can be
used to obtained a symbol from a specific library.
While currently `Library` users just call `::dlsym` globally to obtain a
symbol, this becomes ambigous if multiple libraries in the process image
provide similarly named symbols. Instead they could hold handles to
all interesting libraries and obtain symbols with the added function.
With that we could also `::dclose` libraries in the `Library`
destructor; we'll leave that change for a later point. (Benjamin Bannier, Corelight)
* Fix warning potentially emitted from `rt::Library` destructor. (Benjamin Bannier, Corelight)
* Fix format string in `rt::createTemporaryFile`. (Benjamin Bannier, Corelight)
* Extend error message when `rt::Library::save` fails. (Benjamin Bannier, Corelight)
* Change CI job to use non-priviledged user.
At least on ubuntu `root` seems to be able to modify even read-only
directories which makes it hard to write tests checking handling of
permission errors.
Change the image used in CI to use a non-priviledged user instead. (Benjamin Bannier, Corelight)
* Change directory used for ccache store in CI.
We were originally using `/var/spool/ccache` which could require
extended privileges to create. Switch over to use `/tmp/ccache` which
should be easier to create even for unprivileged users. (Benjamin Bannier, Corelight)
* Prevent `rt::Resumable::get` from leaking non-HILTI exceptions. (Benjamin Bannier, Corelight)
* Make `hilti::rt::Resumable`'s operator bool explicit. (Benjamin Bannier, Corelight)
* Consistently provide stringification for all exceptions. (Benjamin Bannier, Corelight)
* Add comparision operators to `rt::Backtrace`. (Benjamin Bannier, Corelight)
* Fix `to_string_for_print` for `DeferredExpression`.
Since `DeferredExpression` is a template itself we cannot specialize the
master template of the `detail` namespace; instead we need to provide an
overload. (Benjamin Bannier, Corelight)
0.4.0-798 | 2020-08-14 07:25:00 +0000
* CI: Move Zeek package specs back into tasks to work around Cirrus
issue, per https://github.com/cirruslabs/cirrus-ci-docs/issues/692.
(Robin Sommer, Corelight)
0.4.0-796 | 2020-08-14 07:20:31 +0000
* Embed version information into HLTO files that encodes information
about the HILTI version/config they were built with. (Robin
Sommer, Corelight)
We use this for detecting (1) mismatches in the HITLI version used
for building vs running, and (2) running an optimized HLTO file
with a debug runtime. In both cases, the runtime will print a
warning to stderr, but still proceed processing the library.
Note that this is breaking change that will prevent HLTOs built
with earlier versions from loading now.
* Add option -U to tools that summarizes runtime resource
consumption at termination. The measurement excludes time spent
for startup/termination. (Robin Sommer, Corelight)
* Use cout/cerr in debug logger instead of reopening them
separately. Output can get mixed up otherwise if other places
write to cout/cerr directly. (Robin Sommer, Corelight)
* Update nlohmann-json and move to top-level 3rdparty directory.
(Robin Sommer, Corelight)
0.4.0-789 | 2020-08-11 07:47:30 +0000
* Lots of new test coverage for HILTI runtime library: references,
debug logger, configuration, safe-int, iterator, backtrace; along
with releated fixed and cleanup. (Benjamin Bannier, Corelight)
0.4.0-737 | 2020-08-11 07:39:31 +0000
* Fix exception handling across library boundaries. (Robin Sommer,
Corelight)
* Allow limiting the "%port" of a unit to a specific direction.
(Robin Sommer, Corelight)
We now allow to add attributes `&originator`/`&responder` to a `%port`
unit property to limit it to the corresponding direction of a session.
When selecting a parser through `--parser <port>` in spicy-driver, one
can now add `%orig` or `%resp` to the port to indicate the desired
directionality. If one leaves that off, only parsers are selected that
have no directionlity defined (i.e., just like it used to be).
* Add batch input mode to spicy-dump. spicy-dump gains an option -F
to process multiple interleaved input streams pre-recorded in a
batch input file. The format of that file is described in the
manual, and there's a Zeek script generating batch files from PCAP
input traces. (Robin Sommer, Corelight)
* When naming a parser to use through "spicy-driver -p XXX", one can
now also specifiy a port ("80/tcp") or a MIME type
("application/text") if the desired Spicy unit defines a
corresponding property. See the documentation of spicy-driver for
more. (Robin Sommer, Corelight)
* Port Zeek plugin to new driver stateful matching API. (Robin
Sommer, Corelight)
0.4.0-730 | 2020-08-11 07:38:44 +0000
* Switch to Zeek 3.2.0 release packages for CI. (Robin Sommer,
Corelight)
* Add helper Makefile to build and run the CI docker image locallly.
(Robin Sommer, Corelight)
0.4.0-725 | 2020-08-05 12:44:26 +0200
* Remove conversion of `rt::Time` to `bool`. (Benjamin Bannier, Corelight)
* Fix assignment operator for `rt::Union` of non-copyable types. (Benjamin Bannier, Corelight)
* Fix format string error possibly triggered when adding Times and Intervals. (Benjamin Bannier, Corelight)
0.4.0-716 | 2020-08-03 22:24:56 +0200
* Make default-constructed `rt::Address` instances invalid. (Benjamin Bannier, Corelight)
* Make `rt::Address` remember its address family.
This class was previously unable to distinguish IPv4 and IPv6 addresses
with all bits sets to zero from each other. By explicitly storing its
address family we are able to distinguish them, and can also preserve
the address family across operations like masking. (Benjamin Bannier, Corelight)
* Bump doctest to 2.4.0. (Benjamin Bannier, Corelight)
* Validate prefix length inputs when constructing `rt::Network` values. (Benjamin Bannier, Corelight)
0.4.0-703 | 2020-08-03 07:41:51 +0000
* GH-444: Port Zeek plugin to Zeek 3.2. (Robin Sommer, Corelight)
* GH-323: Suppress bogus warning by Zeek when installing our plugin.
(Robin Sommer, Corelight)
* Extend EVT file syntax with @if/@else/@endif preprocessor directives.
For now, we support one type of expression for "@if": a Zeek version check. Example:
@if ZEEK_VERSION < 30200
....
@else
....
@endif (Robin Sommer, Corelight)
Also collecting all the Zeek includes into zeek-compat.h. (Robin Sommer, Corelight)
* Fix frequent off-by-one errors in EVT error message locations.
(Robin Sommer, Corelight)
0.4.0-691 | 2020-07-31 15:19:57 +0200
* Emitted fully typed initializer lists in codegen.
* Correctly handle flipping with zero valid bytes in `rt::integer::flip`. (Benjamin Bannier, Corelight)
* Add additional precondition checks to `rt::integer::bits`. (Benjamin Bannier, Corelight)
* Consistently use `ntohX` type functions for `rt::integer::ntohX`. (Benjamin Bannier, Corelight)
0.4.0-681 | 2020-07-23 14:55:25 +0000
* Fix potential invalid iterator deref in debug output. (Robin Sommer, Corelight)
* Add operator "bytes += uint8" for adding a single byte. (Robin Sommer, Corelight)
* Fix missing filter end-of-input signal. (Robin Sommer, Corelight)
* GH-423: Remove C{,XX}_SYSTEM_INCLUDE_DIRS cmake flags. (Robin
Sommer, Corelight)
If this functionality is needed, set CXX_FLAGS instead during
configure, e.g.:
CXXFLAGS="-isystem /opt/rh/devtoolset-9/root/usr/include/c++/9" ./configure ...
* GH-441: Fix generation of lists of reserved words. (Robin Sommer, Corelight)
* GH-449: Fix documentation example. (Robin Sommer, Corelight)
0.4.0-670 | 2020-07-20 15:25:04 +0000
* GH-415: Add missing type coercion for parameterized types. (Robin
Sommer, Corelight)
* GH-395: Fix tuple constructor expressions to not return a type
until all elements are fully resolved. (Robin Sommer, Corelight)
* GH-402: Allow switch statements without any "case". (Robin Sommer,
Corelight)
* GH-387: Extend a test's life time. (Robin Sommer, Corelight)
0.4.0-661 | 2020-07-16 07:06:14 +0000
* Fix build failures from building against Zeek 3.2 dev versions.
(Tim Wojtulewicz, Corelight)
0.4.0-659 | 2020-07-16 06:34:30 +0000
* GH-352: Add chapter on custom host applications to manual. (Robin
Sommer, Corelight)
* Series of smaller documentation updates. (Robin Sommer, Corelight)
* Add documentation for types time/interval/optional.
* Add list of reservered keywords.
* Update debugging sections.
* Add note about commutativity to operator documentation where applicable.
* Add note about checking if optional unit variables remains unset still.
* Fix typo in documentation for 'print'.
* Fix an operator's description.
* GH-408: Remove "time + time" operator. (Robin Sommer, Corelight)
* GH-404: Add `time_ns(X)` and `interval_ns(X)` expressions to
instantiate values with nanosecond resolution. (Robin Sommer,
Corelight)
* Extend `{hilti,spicy}-config` with a `--dynamic-loading` option.
This adapts `--ldflags` for applications that want to dynamically
load precompiled parsers. (Robin Sommer, Corelight)
* Make `&cxxname` usable for user code: The code generator now
automatically adds prototypes for the corresponding C++ functions.
To optionally disable that, a new attribute `&have_prototype` has
been added. We also no longer require explicit global scoping for
`&cxxname` IDs. (Robin Sommer, Corelight)
* Fix joint usage of options `spicyc` options `-cKo`. This would end
up saving only the linker code, not the combined parsing + linker
code, in the specific output file. (Robin Sommer, Corelight)
0.4.0-645 | 2020-07-16 06:33:23 +0000
* GH-423: Update configure's usage message. (Robin Sommer,
Corelight)
* GH-433: Add support for declaring constants with explicit types.
(Robin Sommer, Corelight)
0.4.0-641 | 2020-07-14 06:13:11 +0000
* GH-429: Fix generation of C++ prototypes for host applications.
(Robin Sommer, Corelight)
0.4.0-639 | 2020-07-13 10:01:31 +0000
* GH-400: Fix build with static libraries. (Robin Sommer, Corelight)
* Fix ill-defined order of internal HILTI plugin execution. (Robin
Sommer, Corelight)
* Remove JIT's {init,finish}Runtime() methods. They were no-ops
already. (Robin Sommer, Corelight)
0.4.0-634 | 2020-07-09 12:32:17 +0200
* Lots of new & updates unit tests for the HILTI runtime. (Benjamin Bannier, Corelight)
* Remove `AddressFamily::` prefix when stringifying values. (Benjamin Bannier, Corelight)
0.4.0-612 | 2020-07-08 15:21:20 +0000
* GH-39: Add `spicy-dump`, a new standalone command line application
that compiles and executes Spicy parsers on the fly and prints out
the parsed information at the end in either a readable, custom
ASCII format, or as JSON. (Robin Sommer, Corelight)
# echo "GET /index.html HTTP/1.0" | spicy-dump my-http.hlto
MyHTTP::RequestLine {
method: GET
uri: /index.html
version: MyHTTP::Version {
number: 1.0
}
}
# echo "GET /index.html HTTP/1.0" | spicy-dump -J my-http.hlto
{"method":"GET","uri":"/index.html","version":{"number":"1.0"}}
* GH-56: Add reflection API to libhilti for dynamic, runtime
inspection of parsed units. This includes:
* A "typeinfo(X)" expression for HILTI that, at runtime, wwill
yield a pointer to a new `hilti::rt::TypeInfo`` struct
describing the type of `X`, which can be a type or an
expression.
* `include/rt/type-info.h` builds out a comprehensive set of
classes to describe HILTI types, and to access their values
generically, at runtime.
* A 3rd kind of host application-side parse function for public
units. Spicy's driver's processInput() now uses that.
* Change string rendering of ports to skip "Protocol::" prefix.
(Robin Sommer, Corelight)
* Fix bug in move assignment operator of ValueReference. (Robin
Sommer, Corelight)
* Add stringification for exception type. (Robin Sommer, Corelight)
* Add stringification for "any" type. (Robin Sommer, Corelight)
* Add missing &internal attributes to some codegenerated struct
fields. This hides them from user visible output. (Robin Sommer,
Corelight)
* Add equality operator to regexp type. (Robin Sommer, Corelight)
* Add operator<< for regexp. (Robin Sommer, Corelight)
* Support anonymous namespaces in C++ output. (Robin Sommer,
Corelight)
* Add missing printer code for iterator<set> and iterator<map>.