Rename BEGIN-TLV and END-TLV to BEGIN and END, and make them work for…

… TLVs too

doc/antora/modules/reference/nav.adoc

@@ -104,8 +104,8 @@
 *** xref:dictionary/vendor.adoc[VENDOR]
 *** xref:dictionary/begin-protocol.adoc[BEGIN-PROTOCOL]
 *** xref:dictionary/end-protocol.adoc[END-PROTOCOL]
-*** xref:dictionary/begin-tlv.adoc[BEGIN-TLV]
-*** xref:dictionary/end-tlv.adoc[END-TLV]
+*** xref:dictionary/begin.adoc[BEGIN]
+*** xref:dictionary/end.adoc[END]
 *** xref:dictionary/begin-vendor.adoc[BEGIN-VENDOR]
 *** xref:dictionary/end-vendor.adoc[END-VENDOR]
 

doc/antora/modules/reference/pages/dictionary/begin-protocol.adoc

@@ -25,7 +25,7 @@ END-PROTOCOL RADIUS
 ----
 
 Note that unlike xref:dictionary/begin-vendor.adoc[END-VENDOR] and
-xref:dictionary/begin-tlv.adoc[END-TLV], it is not possible to omit
+xref:dictionary/begin.adoc[END], it is not possible to omit
 the `BEGIN-PROTOCOL` keyword.
 
 // Copyright (C) 2023 Network RADIUS SAS.  Licenced under CC-by-NC 4.0.

doc/antora/modules/reference/pages/dictionary/begin-tlv.adoc → doc/antora/modules/reference/pages/dictionary/begin.adoc

@@ -1,40 +1,40 @@
-= The BEGIN-TLV keyword
+= The BEGIN keyword
 
 .Syntax
 ----
-BEGIN-TLV <name>
+BEGIN <name>
 ----
 
 .Description
 
-The `BEGIN-TLV` keyword starts a "nested" set of
-xref:dictionary/attribute.adoc[ATTRIBUTE] definitions wwhich are all
-for a particular parent attribute of type `tlv`.
+The `BEGIN` keyword starts a "nested" set of
+xref:dictionary/attribute.adoc[ATTRIBUTE] definitions which are all
+for a particular parent attribute of type `tlv` or `struct`.
 
 <name>:: The name of the parent attribute.
 +
-The tlv must have previously been created in anref:dictionary/attribute.adoc[ATTRIBUTE] definition.
+The tlv or struct must have previously been created in anref:dictionary/attribute.adoc[ATTRIBUTE] definition.
 
-The `BEGIN-TLV` keyword must always be paired with a matching xref:dictionary/end-tlv.adoc[END-TLV] keyword.
+The `BEGIN` keyword must always be paired with a matching xref:dictionary/end.adoc[END] keyword.
 
-Within the context of a ``BEGIN-TLV` block, the numbers for each
+Within the context of a ``BEGIN` block, the numbers for each
 xref:dictionary/attribute.adoc[ATTRIBUTE] definition are relative to
-the parent TLV.
+the parent tlv or struct.
 
 .Example
 ----
 ATTRIBUTE Foo 2 tlv
-BEGIN-TLV Foo
+BEGIN Foo
 ATTRIBUTE Bar 1 string
-END-TLV Foo
+END Foo
 ----
 
 This example defines an attribute `Foo.Bar`, with OID `2.1`.
 
 == Purpose
 
-The `BEGIN-TLV` keyword exists for efficiency.  It is not strictly
-needed, but without a `BEGIN-TLV` keyword, every
+The `BEGIN` keyword exists for efficiency.  It is not strictly
+needed, but without a `BEGIN` keyword, every
 xref:dictionary/attribute.adoc[ATTRIBUTE] name would need to contain
 the parent attribute name, as in the following
 example.
@@ -46,19 +46,19 @@ ATTRIBUTE Foo.bar 2.1 string
 
 == Nesting
 
-It is possible to nest multiple `BEGIN-TLV` keywords, so long as each
-one is paired with a matching xref:dictionary/end-tlv.adoc[END-TLV] keyword.
+It is possible to nest multiple `BEGIN` keywords, so long as each
+one is paired with a matching xref:dictionary/end.adoc[END] keyword.
 
-.Example of nested BEGIN-TLV
+.Example of nested BEGIN
 ----
 ATTRIBUTE Foo 2 tlv
-BEGIN-TLV Foo
+BEGIN Foo
 ATTRIBUTE Bar 1 string
 ATTRIBUTE Baz 2 tlv
-BEGIN-TLV Baz
+BEGIN Baz
 ATTRIBUTE Such 4 ipaddr
-END-TLV Baz
-END-TLV Foo
+END Baz
+END Foo
 ----
 
 The above example is equivalent to the example below.
@@ -73,7 +73,7 @@ ATTRIBUTE Such .2.4 ipaddr
 
 For short entries, it can be simpler to use the full name an OID.
 However, for complex dictionaries, it is almost always clearer to use
-`BEGIN-TLV` and xref:dictionary/end-tlv.adoc[END-TLV].
+`BEGIN` and xref:dictionary/end.adoc[END].
 
 // Copyright (C) 2023 Network RADIUS SAS.  Licenced under CC-by-NC 4.0.
 // This documentation was developed by Network RADIUS SAS.

doc/antora/modules/reference/pages/dictionary/end.adoc

@@ -0,0 +1,21 @@
+= The END keyword
+
+.Syntax
+----
+END [<name>]
+----
+
+.Description
+
+The `END` keyword finished a "nested" set of
+xref:dictionary/attribute.adoc[ATTRIBUTE] definitions which are all
+for a particular `tlv` or `struct` parent attribute.
+
+[<name>]:: The name of the xref:dictionary/attribute.adoc[ATTRIBUTE].  This is useful
+for validation purposes, but may be omitted.
++
+The dictionary must have previously contained a matching
+xref:dictionary/begin.adoc[BEGIN].
+
+// Copyright (C) 2023 Network RADIUS SAS.  Licenced under CC-by-NC 4.0.
+// This documentation was developed by Network RADIUS SAS.

doc/antora/modules/reference/pages/dictionary/index.adoc

@@ -155,8 +155,8 @@ The following keywords still XXX
 |=====
 | xref:dictionary/begin-protocol.adoc[BEGIN-PROTOCOL]  | Begin defining a protocol dictionary
 | xref:dictionary/end-protocol.adoc[END-PROTOCOL]      | End a protocol dictionary
-| xref:dictionary/begin-tlv.adoc[BEGIN-TLV]            | Begin defining children of a `tlv` data type
-| xref:dictionary/end-tlv.adoc[END-TLV]                | End defining children of a `tlv` data type
+| xref:dictionary/begin.adoc[BEGIN]            | Begin defining children of a `tlv` data type
+| xref:dictionary/end.adoc[END]                | End defining children of a `tlv` data type
 | xref:dictionary/begin-vendor.adoc[BEGIN-VENDOR]      | Begin defining vendor-specific attributes
 | xref:dictionary/end-vendor.adoc[END-VENDOR]          | End defining vendor-specific attributes
 |=====

man/man5/dictionary.5

@@ -260,12 +260,12 @@ Include dictionary entries from the file \fIfilename\fP.  The
 \fIfilename\fP is taken as relative to the location of the file which
 is asking for the inclusion.
 .TP 0.5i
-.B BEGIN-TLV name
+.B BEGIN name
 This feature is supported for backwards compatibility with older
 dictionaries.  It should not be used.  The new "oid" form for defining
 the attribute number should be used instead.
 .TP 0.5i
-.B END-TLV name
+.B END name
 This feature is supported for backwards compatibility with older
 dictionaries.  It should not be used.  The new "oid" form for defining
 the attribute number should be used instead.

share/dictionary/eap/aka-sim/dictionary.rfc4187

@@ -51,19 +51,18 @@ ATTRIBUTE	Encr-Data				130	tlv encrypt=aes-cbc
 #
 #  These attributes are reversibly encrypted
 #
-BEGIN-TLV	Encr-Data
+BEGIN	Encr-Data
 ATTRIBUTE	Padding					6	octets		# 4, 8, 12 bytes
 ATTRIBUTE	Counter					19	short
 ATTRIBUTE	Counter-Too-Small			20	bool
 ATTRIBUTE	Nonce-S					21	octets[16]
 ATTRIBUTE	Next-Pseudonym				132	string
 ATTRIBUTE	Next-Reauth-ID				133	string
-END-TLV		Encr-Data
+END		Encr-Data
 
 ATTRIBUTE	Checkcode				134	octets		# 0, 20 or 32 bytes
 ATTRIBUTE	Result-Ind				135	bool
 ATTRIBUTE	Bidding					136	short
 
 VALUE	Bidding				Prefer-AKA		0
 VALUE	Bidding				Prefer-AKA-Prime	32768	# D Bit
-

share/dictionary/freeradius/dictionary.freeradius.internal

@@ -421,7 +421,7 @@ ATTRIBUTE	Log-Type				1896	integer
 ATTRIBUTE	WiMAX-MN-NAI				1900	string
 
 ATTRIBUTE	TLS-Certificate				1901	tlv
-BEGIN-TLV	TLS-Certificate
+BEGIN		TLS-Certificate
 ATTRIBUTE	Serial					1	octets
 ATTRIBUTE	Signature				2	octets
 ATTRIBUTE	Signature-Algorithm			3	string
@@ -439,7 +439,7 @@ ATTRIBUTE	X509v3-Extended-Key-Usage		14	string
 ATTRIBUTE	X509v3-Subject-Key-Identifier		15	string
 ATTRIBUTE	X509v3-Authority-Key-Identifier		16	string
 ATTRIBUTE	X509v3-Basic-Constraints		17	string
-END-TLV		TLS-Certificate
+END		TLS-Certificate
 
 ATTRIBUTE	TLS-PSK-Identity			1933	string
 ATTRIBUTE	TLS-Session-Certificate-File		1934	string

share/dictionary/freeradius/dictionary.freeradius.internal.ippool

@@ -15,12 +15,11 @@ FLAGS	internal
 
 ATTRIBUTE	IP-Pool					5100	tlv
 
-BEGIN-TLV 	IP-Pool
+BEGIN 	IP-Pool
 ATTRIBUTE	Name					1	string	# Generic identifier for the IP pool to allocate from
 ATTRIBUTE	Name-NA					2	string	# DHCPv6 - Non-Temporary association pool
 ATTRIBUTE	Name-PD					3	string	# DHCPv6 - Prefix-deligation pool
 ATTRIBUTE	Name-TA					4	string	# DHCPv6 - Temporary association pool
 
 ATTRIBUTE	Range					6	string
-END-TLV		IP-Pool
-
+END		IP-Pool

share/dictionary/freeradius/dictionary.freeradius.internal.password

@@ -17,7 +17,7 @@
 FLAGS	internal
 
 ATTRIBUTE	Password				2004	tlv
-BEGIN-TLV	Password
+BEGIN	Password
 ATTRIBUTE	With-Header				1	string
 
 ATTRIBUTE	Cleartext				2	string	secret
@@ -64,14 +64,13 @@ ATTRIBUTE	SSHA3-256				28	octets
 ATTRIBUTE	SSHA3-384				29	octets
 ATTRIBUTE	SSHA3-512				30	octets
 
-END-TLV		Password
+END		Password
 
 #  TOTP passwords and secrets
 ATTRIBUTE	TOTP					2005	tlv
 
-BEGIN-TLV	TOTP
+BEGIN	TOTP
 ATTRIBUTE	Secret					1	string	secret
 ATTRIBUTE	Key					2	octets	secret
 ATTRIBUTE	From-User				3	string
-END-TLV		TOTP
-
+END		TOTP

share/dictionary/ldap/dictionary.freeradius.internal

@@ -30,7 +30,7 @@ VALUE	Packet-Type			Do-Not-Respond		255
 ATTRIBUTE	Sync-Packet-ID				1001	integer
 
 ATTRIBUTE	LDAP-Sync				1193	tlv
-BEGIN-TLV	LDAP-Sync
+BEGIN	LDAP-Sync
 ATTRIBUTE	Cookie					1	octets
 ATTRIBUTE	DN					2	string
 ATTRIBUTE	Filter					3	string
@@ -45,4 +45,4 @@ ATTRIBUTE	Entry-DN				5	string
 ATTRIBUTE	Entry-UUID				6	octets
 ATTRIBUTE	Original-DN				7	string
 ATTRIBUTE	Directory-Root-DN			8	string
-END-TLV		LDAP-Sync
+END		LDAP-Sync

share/dictionary/snmp/dictionary.freeradius

@@ -48,14 +48,14 @@ ATTRIBUTE	FreeRADIUS-Mib-2			.3.6.1.2.1	tlv
 #
 #  Everything below is defined in the context of MIB-2
 #
-BEGIN-TLV	1.3.6.1.2.FreeRADIUS-Mib-2
+BEGIN		1.3.6.1.2.FreeRADIUS-Mib-2
 ATTRIBUTE	Radius-Mib				67	tlv
 ATTRIBUTE	Radius-Authentication			.1	tlv
 ATTRIBUTE	Radius-Auth-Serv-Mib			.1.1	tlv
 ATTRIBUTE	Radius-Auth-Serv-Mib-Objects		.1.1.1	tlv
 ATTRIBUTE	Radius-Auth-Serv			.1.1.1.1	tlv
 
-BEGIN-TLV	67.1.1.1.Radius-Auth-Serv
+BEGIN		67.1.1.1.Radius-Auth-Serv
 ATTRIBUTE	Radius-Auth-Serv-Ident			1	string
 ATTRIBUTE	Radius-Auth-Serv-Up-Time		2	integer
 ATTRIBUTE	Radius-Auth-Serv-Reset-Time		3	integer
@@ -82,7 +82,7 @@ ATTRIBUTE	Radius-Auth-Client-Table-Index		.0	integer
 ATTRIBUTE	Radius-Auth-Client-Entry		.1	tlv
 
 # Client statistics
-BEGIN-TLV	15.Radius-Auth-Client-Entry
+BEGIN	15.Radius-Auth-Client-Entry
 ATTRIBUTE	Radius-Auth-Client-Index		1	integer
 ATTRIBUTE	Radius-Auth-Client-Address		2	ipaddr
 ATTRIBUTE	Radius-Auth-Client-ID			3	string
@@ -95,13 +95,13 @@ ATTRIBUTE	Radius-Auth-Serv-Malformed-Access-Requests 9	integer
 ATTRIBUTE	Radius-Auth-Serv-Bad-Authenticators	10	integer
 ATTRIBUTE	Radius-Auth-Serv-Packets-Dropped	11	integer
 ATTRIBUTE	Radius-Auth-Serv-Unknown-Types		12	integer
-END-TLV		15.Radius-Auth-Client-Entry
+END		15.Radius-Auth-Client-Entry
 
 ATTRIBUTE	Radius-Auth-Client-Ext-Table		16	tlv
 ATTRIBUTE	Radius-Auth-Client-Ext-Table-Index	.0	integer
 ATTRIBUTE	Radius-Auth-Client-Ext-Entry		.1	tlv
 
-BEGIN-TLV	16.Radius-Auth-Client-Ext-Entry
+BEGIN	16.Radius-Auth-Client-Ext-Entry
 ATTRIBUTE	Radius-Auth-Client-Ext-Index		1	integer
 ATTRIBUTE	Radius-Auth-Client-Inet-Address-Type	2	integer
 ATTRIBUTE	Radius-Auth-Client-Inet-Address		3	ipaddr
@@ -116,7 +116,7 @@ ATTRIBUTE	Radius-Auth-Serv-Ext-Bad-Authenticators	11	integer
 ATTRIBUTE	Radius-Auth-Serv-Ext-Packet-Dropped	12	integer
 ATTRIBUTE	Radius-Auth-Serv-Ext-Unknown-Types	13	integer
 ATTRIBUTE	Radius-Auth-Serv-Counter-Discontinuity	14	integer
-END-TLV		16.Radius-Auth-Client-Ext-Entry
+END		16.Radius-Auth-Client-Ext-Entry
 
-END-TLV		67.1.1.1.Radius-Auth-Serv
-END-TLV		1.3.6.1.2.FreeRADIUS-Mib-2
+END		67.1.1.1.Radius-Auth-Serv
+END		1.3.6.1.2.FreeRADIUS-Mib-2

src/lib/util/dict.h

@@ -66,6 +66,7 @@ typedef struct value_box_s fr_value_box_t;
 #  define DA_VERIFY(_x)		fr_cond_assert(_x)
 #endif
 
+typedef struct dict_tokenize_ctx_s dict_tokenize_ctx_t;
 typedef struct fr_dict_autoload_talloc_s fr_dict_autoload_talloc_t;
 
 /** Values of the encryption flags
@@ -913,6 +914,8 @@ fr_dict_t const		*fr_dict_internal(void);
  *
  * @{
  */
+void			dict_dctx_debug(dict_tokenize_ctx_t *dctx);
+
 int			fr_dict_parse_str(fr_dict_t *dict, char *buf,
 					  fr_dict_attr_t const *parent);