Merge pull request #275 from sabracrolleton/master

New ability to pass parameters in binary (per connection basis)

CHANGELOG.md

@@ -1,3 +1,87 @@
+# Changelog v. 1.33.0
+This version of Postmodern now provides the ability to pass parameters to Postgresql in binary format IF that format is available for that datatype. Currently this means int2, int4, int8, float, double-float (except clisp) and boolean. Rational numbers continue to be passed as text.
+
+The flag is set in the database connection object. (Thank you Cyrus Harmon for suggesting that). This means it can be set either in the initial connection to the database or using the use-binary-parameters function to set it after the initial connection has been established. If you are using multiple connections, some can be set to use binary parameters, some not.
+
+If a query to Postgresql does not have a table column which would allow Postgresql to determine the correct datatype and you do not specify differently, Postgresql will treat the parameters passed with the query as text. The default text setting with results:
+
+    (query "select $1" 1 :single)
+    "1"
+    (query "select $1" 1.5 :single)
+    "1.5"
+    (query "select $1" T :single)
+    "true"
+    (query "select $1" nil :single)
+    "false"
+    (query "select $1" :NULL :single)
+    :NULL
+
+You can specify parameter type as so:
+
+    (query "select $1::integer" 1 :single)
+    1
+
+Setting the use-binary slot in the database connection object to t has the following results:
+
+    (query "select $1" 1 :single)
+    1
+    (query "select $1" 1.5 :single)
+    1.5
+    (query "select $1" T :single)
+    T
+    (query "select $1" nil :single)
+    NIL
+    (query "select $1" :NULL :single)
+    :NULL
+
+The default for cl-postgres/Postmodern is to continue to pass parameters to Postgresql as text (not in binary format) in order to avoid breaking existing user code. If you want to pass parameters to Postgresql in binary format and want to set that up when you are making the database connection, the following examples may help. We continue the difference in the signatures (cl-postgres using optional parameters and postmodern using keyword parameters) because of the expected downstream breakage if we shifted cl-postgres:open-database to using keyword parameters.
+
+The signature for opening a database in cl-postgres:
+
+    (defun open-database (database user password host
+                          &optional (port 5432) (use-ssl :no)
+                          (service "postgres") (application-name "")
+                          (use-binary nil))
+        ...)
+
+or your favorite macro.
+
+In postmodern you have the connect function or the with-connection macro:
+
+    (defun connect (database-name user-name password host
+                    &key (port 5432) pooled-p
+                    (use-ssl *default-use-ssl*)
+                    (use-binary nil)
+                    (service "postgres")
+                    (application-name ""))
+      ...)
+
+    (defmacro with-connection (spec &body body)
+      `(let ((*database* (apply #'connect ,spec)))
+         (unwind-protect (progn ,@body)
+           (disconnect *database*))))
+
+In any case, you can set the flag after the connection is established with the use-binary-parameters function:
+
+    (pomo:use-binary-parameters *database* t)
+
+    (cl-postgres:use-binary-parameters some-database-connection t)
+
+Using binary parameters does tighten type checking when using prepared queries. You will not be able to use prepared queries with varying formats. In other words, if you have a prepared query that you pass an integer as the first parameter and a string as the second parameter the first time it is used, any subsequent uses of that prepared query during that session will also have to pass an integer as the first parameter and a string as the second parameter.
+
+Benchmarking does indicate a slight speed and consing benefit to passing parameters as binary, but your mileage will vary depending on your use case.
+
+In addition, this version also adds the ability to have queries returned as vectors of vectors, using a vectors keyword.
+
+    (query "select id, some_int, some_text from tests_data :where id = 1" :vectors)
+    or
+    (query (:select 'id 'some-int 'some-text :from 'test-data)
+         :vectors)
+    #(#(1 2147483645 "text one")
+      #(2 0 "text two")
+      #(3 3 "text three"))
+
+Like :array-hash, if there is no result it will return an empty array, not nil.
 # Changelog v. 1.32.9
 Adds new utility functions
 

README.md

@@ -4,7 +4,7 @@ A Common Lisp PostgreSQL programming interface
 
 ---
 
-Version 1.38
+Version 1.33.0
 
 Postmodern is a Common Lisp library for interacting with [PostgreSQL](http://www.postgresql.org) databases. It is under active development. Features are:
 
@@ -58,8 +58,7 @@ Finally, Postmodern itself is a wrapper around these packages and provides highe
 
 Postmodern is released under a zlib-style license. Which approximately means you can use the code in whatever way you like, except for passing it off as your own or releasing a modified version without indication that it is not the original.
 
-The functions execute-file.lisp were ported from [[https://github.com/dimitri/pgloader][pgloader]] with grateful thanks to
-Dimitri Fontaine and are released under a BSD-3 license.
+The functions execute-file.lisp were ported from [pgloader](https://github.com/dimitri/pgloader) with grateful thanks to Dimitri Fontaine and are released under a BSD-3 license.
 
 ## Download and installation
 
@@ -83,7 +82,7 @@ Assuming you have already installed it, first load and use the system:
 ```
 
 If you have a PostgreSQL server running on localhost, with a database called 'testdb' on it, which is accessible for user 'foucault' with password 'surveiller', there are two basic ways to connect
-to a database. If your role/application/database(s) looks like a 1:1 relationship, you can connect like this:
+to a database. If your role/application/database(s) looks like a 1:1 relationship and you are not using threads, you can connect like this:
 
 ```lisp
 (connect-toplevel "testdb" "foucault" "surveiller" "localhost")
@@ -103,14 +102,14 @@ If the Postgresql server is running on a port other than 5432, you would also pa
 Ssl connections would similarly use the keyword parameter :use-ssl and pass :yes, :no or :try
 
 If you have multiple roles connecting to one or more databases, i.e. 1:many or
-many:1, (in other words, changing connections) then with-connection form which establishes a connection with a lexical scope is more appropriate.
+many:1, (in other words, changing connections) or you are using threads (each thread will need to have its own connection) then with-connection form which establishes a connection with a lexical scope is more appropriate.
 
 ```lisp
 (with-connection '("testdb" "foucault" "surveiller" "localhost")
     ...)
 ```
 
-For example, if you are creating a database, you need to have established a connection
+If you are creating a database, you need to have established a connection
 to a currently existing database (typically "postgres"). Assuming the foucault role
 is a superuser and you want to stay in a development connection with your new database
 afterwards, you would first use with-connection to connect to postgres, create the
@@ -129,11 +128,14 @@ Note: (create-database) functionality is new to postmodern v. 1.32. Setting the
 anyone who you have not explicitly given permission (except other superusers).
 
 A word about Postgresql connections. Postgresql connections are not lightweight
-threads. They actually consume about 10 MB of memory per connection and Postgresql
-can be tuned to limit the number of connections allowed at any one time. In
+threads. They actually consume about 10 MB of memory per connection.  In
 addition, any connections which require security (ssl or scram authentication)
 will take additiona time and create more overhead.
 
+Postgresql can be tuned to limit the number of connections allowed at any one time. It defaults to 100. The parameter is set in the postgresql.conf file. Depending on the size of your server and what you are doing, the sweet spot generally seems to be between 200-400 connections before you need to bring in connection pooling.
+
+If your application is threaded, each thread should use its own connection. Connections are stateful and attempts to use the same connection for multiple threads will
+
 If you have an application like a web app which will make many connections, you also
 generally do not want to create and drop connections for every query. The usual solution
 is to use connection pools so that the application is grabbing an already existing connection
@@ -524,6 +526,12 @@ values to that scale. For more detail, see <https://www.postgresql.org/docs/curr
 
 ---
 
+### Passing Parameters as Text or Binary
+
+See [index.html#passing-binary-parameters](https://marijnhaverbeke.nl/postmodern/index.html#passing-binary-parameters)
+
+---
+
 ### Arrays
 
 See [array-notes.html](https://marijnhaverbeke.nl/postmodern/array-notes.html)
@@ -633,7 +641,7 @@ It is highly suggested that you do not use words that are reserved by Postgresql
 ## Feature Requests
 
 Postmodern is under active development so issues and feature requests should
-be flagged on [[https://github.com/marijnh/Postmodern](Postmodern's site on github).
+be flagged on [https://github.com/marijnh/Postmodern](Postmodern's site on github).
 
 ## Resources
 

ROADMAP.md

@@ -49,7 +49,7 @@ No guarantee is given with respect to resolution or timing on any item.
 - [ ]   Named Prepared Statement explicit arglist
 - [ ]   SQL Read Table Review (comments requested on any work that should be done here)
 - [ ]   Row Reader Review (comments requested on any work that should be done here)
-- [ ]   Allow parameters to be passed as binary to postgresql
+- [X]   Allow parameters to be passed as binary to postgresql
 
 ## Connections/Reconnections and Transactions
 - [ ]   Ensure transactions can deal with reconnections/restarts
@@ -62,7 +62,7 @@ No guarantee is given with respect to resolution or timing on any item.
 - [ ]   Creation of new datatypes
 - [ ]   Pooling with external pooling applications
 
-## Armed Bear Isuses
+## Armed Bear Issues
 - [ ]   What is the problem with :null
 - [ ]   Armed Bear issues with unicode. See e.g. icelandic cities in test-s-sql
 

cl-postgres.asd

@@ -16,7 +16,7 @@
   :author "Marijn Haverbeke <[email protected]>"
   :maintainer "Sabra Crolleton <[email protected]>"
   :license "zlib"
-  :version "1.32.9"
+  :version "1.33.0"
   :depends-on ("md5" "split-sequence" "ironclad" "cl-base64" "uax-15"
                      (:feature (:or :sbcl :allegro :ccl :clisp :genera
                                 :armedbear :cmucl :lispworks)
@@ -26,28 +26,30 @@
   ((:module "cl-postgres"
     :components ((:file "package")
                  (:file "features")
+                 (:file "config")
                  (:file "errors" :depends-on ("package"))
-                 (:file "sql-string" :depends-on ("package"))
-                 (:file "trivial-utf-8" :depends-on ("package"))
+                 (:file "data-types" :depends-on ("package" "config"))
+                 (:file "sql-string" :depends-on ("package" "config" "data-types"))
+                 (:file "trivial-utf-8" :depends-on ("package" "config"))
                  (:file #.*string-file*
-                  :depends-on ("package" "trivial-utf-8"))
+                  :depends-on ("package" "trivial-utf-8" "config"))
                  (:file "communicate"
-                  :depends-on (#.*string-file* "sql-string"))
-                 (:file "messages" :depends-on ("communicate"))
-                 (:file "oid" :depends-on ("package"))
-                 (:file "ieee-floats")
+                  :depends-on (#.*string-file* "sql-string" "config"))
+                 (:file "messages" :depends-on ("communicate" "config"))
+                 (:file "oid" :depends-on ("package" "config"))
+                 (:file "ieee-floats" :depends-on ("config"))
                  (:file "interpret"
-                  :depends-on ("oid" "communicate" "ieee-floats"))
-                 (:file "saslprep")
+                  :depends-on ("oid" "communicate" "ieee-floats" "config"))
+                 (:file "saslprep" :depends-on ("package" "config"))
                  (:file "scram"
-                  :depends-on ("messages" "errors" "saslprep"
-                                          "trivial-utf-8"))
+                  :depends-on ("package" "messages" "errors" "saslprep"
+                                          "trivial-utf-8" "config"))
                  (:file "protocol"
-                  :depends-on ("interpret" "messages" "errors" "scram"
-                                           "saslprep" "trivial-utf-8"))
-                 (:file "public" :depends-on ("protocol" "features"))
+                  :depends-on ("package" "interpret" "messages" "errors" "scram"
+                                           "saslprep" "trivial-utf-8" "config"))
+                 (:file "public" :depends-on ("package" "protocol" "features" "config"))
                  (:file "bulk-copy"
-                  :depends-on ("public" "trivial-utf-8")))))
+                  :depends-on ("package" "public" "trivial-utf-8")))))
   :in-order-to ((test-op (test-op "cl-postgres/tests")
                          (test-op "cl-postgres/simple-date-tests"))))
 
@@ -56,13 +58,20 @@
   :components
   ((:module "cl-postgres/tests"
     :components ((:file "test-package")
-                 (:file "tests")
-                 (:file "tests-scram" :depends-on ("test-package" "tests")))))
+                 (:file "tests" :depends-on ("test-package"))
+                 (:file "test-binary-parameters" :depends-on ("test-package" "tests"))
+                 (:file "test-oids" :depends-on ("test-package" "tests"))
+                 (:file "test-ieee-float" :depends-on ("test-package" "tests"))
+                 (:file "test-clp-utf8" :depends-on ("test-package" "tests"))
+                 (:file "test-data-types" :depends-on ("test-package" "tests"))
+                 (:file "test-communicate" :depends-on ("test-package" "tests"))
+                 (:file "tests-scram" :depends-on ("test-package" "tests"))
+                 (:file "tests-saslprep" :depends-on ("test-package")))))
+
   :perform (test-op (o c)
                     (uiop:symbol-call :cl-postgres-tests '#:prompt-connection)
                     (uiop:symbol-call :fiveam '#:run! :cl-postgres)))
 
-
 (defsystem "cl-postgres/simple-date-tests"
   :depends-on ("cl-postgres" "cl-postgres/tests" "fiveam" "simple-date"
                              "simple-date/postgres-glue")

cl-postgres/communicate.lisp

@@ -88,6 +88,7 @@
 (integer-writer 1)
 (integer-writer 2)
 (integer-writer 4)
+(integer-writer 8)
 
 (defun write-bytes (socket bytes)
   "Write a byte-array to a stream."
@@ -113,7 +114,9 @@ support is enabled.)."
   (declare (type stream socket)
            (type fixnum length)
            #.*optimize*)
-  (let ((result (make-array length :element-type '(unsigned-byte 8))))
+  (let ((result (make-array length
+                            :element-type '(unsigned-byte 8)
+                            :initial-element 0)))
     (read-sequence result socket)
     result))
 

cl-postgres/config.lisp

@@ -0,0 +1,31 @@
+;;;; -*- Mode: LISP; Syntax: Ansi-Common-Lisp; Base: 10; Package: CL-POSTGRES; -*-
+(in-package :cl-postgres)
+
+(defparameter *silently-truncate-ratios* t "Given a ratio, a stream and a
+digital-length-limit, if *silently-truncate-ratios* is true,
+will return a potentially truncated ratio. If false and the digital-length-limit
+is reached, it will throw an error noting the loss of precision and offering to
+continue or reset *silently-truncate-ratios* to true. Code contributed by
+Attila Lendvai.")
+
+(defparameter *query-log* nil "When debugging, it can be helpful to inspect the
+queries that are being sent to the database. Set this variable to an output
+stream value (*standard-output*, for example) to have CL-postgres log every
+query it makes.")
+(defparameter *query-callback* 'log-query "When profiling or debugging, the
+*query-log* may not give enough information, or reparsing its output may not be
+feasible. This variable may be set to a designator of function taking two
+arguments. This function will be then called after every query, and receive
+query string and internal time units (as in (CL:GET-INTERNAL-REAL-TIME)) spent
+in query as its arguments.
+
+Default value of this variable is 'LOG-QUERY, which takes care of *QUERY-LOG*
+processing. If you provide custom query callback and wish to keep *QUERY-LOG*
+functionality, you will have to call LOG-QUERY from your callback function")
+
+(defvar *retry-connect-times* 5
+  "How many times do we try to connect again. Borrowed from pgloader")
+
+(defvar *retry-connect-delay* 0.5
+  "How many seconds to wait before trying to connect again. Borrowed from
+pgloader")