dexador
2024-10-12
Yet another HTTP client for Common Lisp
Dexador
Dexador is yet another HTTP client for Common Lisp with neat APIs and connection-pooling.
Warning
This software is still BETA quality. The APIs will be likely to change.
Differences from Drakma
- Fast, particularly when requesting to the same host (See Benchmark)
- Neat APIs
- Signal a condition when HTTP request failed
- OpenSSL isn't required for Windows
See also a presentation given at Lisp Meetup #31.
Usage
(dex:get "http://lisp.org/") (dex:post "https://example.com/login" :content '(("name" . "fukamachi") ("password" . "1ispa1ien")))
Posting a form-data
You can specify a form-data at :content
in an association list. The data will be sent in application/x-www-form-urlencoded
format.
(dex:post "http://example.com/entry/create" :content '(("title" . "The Truth About Lisp") ("body" . "In which the truth about lisp is revealed, and some alternatives are enumerated.")))
Auto-detects Multipart
If the association list contains a pathname, the data will be sent as multipart/form-data
.
(dex:post "http://example.com/entry/create" :content '(("photo" . #P"images/2015030201.jpg")))
Following redirects (GET or HEAD)
If the server reports that the requested page has moved to a different location (indicated with a Location header and a 3XX response code), Dexador will redo the request on the new place, the fourth return value shows.
(dex:head "http://lisp.org") ;=> "" ; 200 ; #<HASH-TABLE :TEST EQUAL :COUNT 7 {100D2A47A3}> ; #<QURI.URI.HTTP:URI-HTTP http://lisp.org/index.html> ; NIL
You can limit the count of redirection by specifying :max-redirects
with an integer. The default value is 5
.
Using cookies
Dexador adopts cl-cookie for its cookie management. All functions takes a cookie-jar instance at :cookie-jar
.
(defvar *cookie-jar* (cl-cookie:make-cookie-jar)) (dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0 ; Host: mixi.jp ; Accept: */* ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ; HTTP/1.1 200 OK ; Date: Tue, 10 Mar 2015 10:16:29 GMT ; Server: Apache ; X-Dealer: 152151 ; X-XRDS-Location: https://mixi.jp/xrds.pl ; Cache-Control: no-cache ; Pragma: no-cache ; Vary: User-Agent ; Content-Type: text/html; charset=EUC-JP ; Set-Cookie: _auid=9d47ca5a00ce4980c41511beb2626fd4; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:29 GMT ; Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:29 GMT ; X-Content-Type-Options: nosniff ; ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ;; Again (dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0 ; Host: mixi.jp ; Accept: */* ; Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; _lcp=8ee4121c9866435007fff2c90dc31a4d ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ; HTTP/1.1 200 OK ; Date: Tue, 10 Mar 2015 10:16:59 GMT ; Server: Apache ; X-Dealer: 152146 ; X-XRDS-Location: https://mixi.jp/xrds.pl ; Cache-Control: no-cache ; Pragma: no-cache ; Vary: User-Agent ; Content-Type: text/html; charset=EUC-JP ; Set-Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:59 GMT ; Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:59 GMT ; X-Content-Type-Options: nosniff ; ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Authorization
You can only supply either basic or bearer authorization.
Basic Authorization
(dex:head "http://www.hatena.ne.jp/" :basic-auth '("nitro_idiot" . "password") :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0 ; Host: www.hatena.ne.jp ; Accept: */* ; Authorization: Basic bml0cm9faWRpb3Q6cGFzc3dvcmQ= ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Bearer Authorization
(dex:head "http://www.hatena.ne.jp/" :bearer-auth "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9" :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Dexador/0.9.15 (SBCL 2.4.3); Linux; 6.7.0-20-amd64 ; Host: www.hatena.ne.jp ; Accept: */* ; Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Faking a User-Agent header
You can overwrite the default User-Agent header by simply specifying "User-Agent" in :headers
.
(dex:head "http://www.sbcl.org/" :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Dexador/0.1 (SBCL 1.2.6); Darwin; 14.1.0 ; Host: www.sbcl.org ; Accept: */* ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (dex:head "http://www.sbcl.org/" :headers '(("User-Agent" . "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18")) :verbose t) ;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ; HEAD / HTTP/1.1 ; User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18 ; Host: www.sbcl.org ; Accept: */* ; ; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Reusing a connection
Dexador reuses a connection by default. As it skips a TCP handshake, it would be much faster when you send requests to the same host continuously.
Handling unexpected HTTP status code
Dexador signals a condition http-request-failed
when the server returned 4xx or 5xx status code.
;; Handles 400 bad request (handler-case (dex:get "http://lisp.org") (dex:http-request-bad-request () ;; Runs when 400 bad request returned ) (dex:http-request-failed (e) ;; For other 4xx or 5xx (format *error-output* "The server returned ~D" (dex:response-status e)))) ;; Ignore 404 Not Found and continue (handler-bind ((dex:http-request-not-found #'dex:ignore-and-continue)) (dex:get "http://lisp.org")) ;; Retry (handler-bind ((dex:http-request-failed #'dex:retry-request)) (dex:get "http://lisp.org")) ;; Retry 5 times (let ((retry-request (dex:retry-request 5 :interval 3))) (handler-bind ((dex:http-request-failed retry-request)) (dex:get "http://lisp.org")))
Proxy
You can connect via proxy.
(dex:get "http://lisp.org/" :proxy "http://proxy.yourcompany.com:8080/")
You can connect via SOCKS5 proxy.
(dex:get "https://www.facebookcorewwwi.onion/" :proxy "socks5://127.0.0.1:9150")
You can set the default proxy by setting
dex:*default-proxy*
which defaults to the value of the environment variable HTTPS_PROXY or HTTP_PROXY
Functions
All functions take similar arguments.
uri
(string or quri:uri)method
(keyword)- The HTTP request method:
:GET
,:HEAD
,:OPTIONS
,:PUT
,:POST
, or:DELETE
. The default is:GET
.
- The HTTP request method:
version
(number)- The version of the HTTP protocol: typically
1.0
or1.1
. The default is1.1
.
- The version of the HTTP protocol: typically
content
(string, alist or pathname)- The body of the request. content may be an alist containing key value pairs, where the value can be a string, pathname, an (array (unsigned-byte 8) (*)), or a cons. If the value is
headers
(alist)- The headers of the request. If the value of a pair is
NIL
, the header won't be sent. You can overwrite the default headers (Host, User-Agent, Accept, Content-Type) by this with the same header name.
- The headers of the request. If the value of a pair is
basic-auth
(cons of username and password)- Username and password for basic authorization. This is a cons having username at car and password at cdr. (e.g.
'("foo" . "bar")
)
- Username and password for basic authorization. This is a cons having username at car and password at cdr. (e.g.
cookie-jar
(cookie-jar of cl-cookie)- A cookie jar object.
connect-timeout
(fixnum)- The seconds to timeout until the HTTP connection established. The default is
10
, the value of*default-connect-timeout*
.
- The seconds to timeout until the HTTP connection established. The default is
read-timeout
(fixnum)- The seconds to timeout until the whole HTTP body read. The default is
10
, the value of*default-read-timeout*
.
- The seconds to timeout until the whole HTTP body read. The default is
keep-alive
(boolean)- A flag if the connection keep connected even after the HTTP request. The default is
T
.
- A flag if the connection keep connected even after the HTTP request. The default is
use-connection-pool
(boolean)- When combined with
:keep-alive t
, will internally cache the socket connection to web servers to avoid having to open new ones. This is compatible with:want-stream t
(when you close the returned stream or it is garbage collected the connection will be returned to the pool). If you pass in a stream with:stream
then the connection pool is not used (unless there is a redirect to a new web server). This is not supported when using the WINHTTP backend. The default isT
.
- When combined with
max-redirects
(fixnum)- The limit of redirections. The default is
5
. If the redirection exceeds the limit, functions return the last response (not raise a condition).
- The limit of redirections. The default is
ssl-key-file
,ssl-cert-file
,ssl-key-password
- for HTTPS connection
stream
- The stream to write an HTTP request. This is a way to reuse a connection and commonly used with
:keep-alive T
. This allows the caller to do connection pooling, etc. It is easier to just use:use-connection-pool t
, which is the default, and let the dexador internals take care of this for you (only supported for usocket backends).
- The stream to write an HTTP request. This is a way to reuse a connection and commonly used with
verbose
(boolean)- This option is for debugging. When
T
, it dumps the HTTP request headers.
- This option is for debugging. When
force-binary
(boolean)- A flag for suppressing auto-decoding of the response body.
want-stream
(boolean)- A flag to get the response body as a stream.
proxy
(string)- for use proxy. defaults to the value of
dex:*default-proxy*
which defaults to the value of environment variables HTTPS_PROXY or HTTP_PROXY. Not supported on windows currently
- for use proxy. defaults to the value of
insecure
(boolean)- To bypass SSL certificate verification (use at your own risk). The default is
NIL
, the value of*not-verify-ssl*
.
- To bypass SSL certificate verification (use at your own risk). The default is
[Function] request
(dex:request uri &key (method get) (version 1.1) content headers basic-auth cookie-jar (connect-timeout *default-connect-timeout*) (read-timeout *default-read-timeout*) (keep-alive t) (use-connection-pool t) (max-redirects 5) ssl-key-file ssl-cert-file ssl-key-password stream (verbose *verbose*) force-binary force-string want-stream proxy (insecure *not-verify-ssl*) ca-path) ;=> body ; status ; response-headers ; uri ; stream
Send an HTTP request to uri
.
The body
is an octet vector or a string if the Content-Type
is text/*
. If you always want it to return an octet vector, specify :force-binary
as T
.
The status
is an integer which represents HTTP status code.
The response-headers
is a hash table which represents HTTP response headers. Note that all hash keys are downcased like "content-type". If there's duplicate HTTP headers, those values are concatenated with a comma.
The uri
is a QURI object which represents the last URI Dexador requested.
The stream
is a usocket stream to communicate with the HTTP server if the connection is still alive and can be reused. This value may be NIL
if :keep-alive
is NIL
or the server closed the connection with Connection: close
header or you are using :use-connection-pool t
which handles re-using the connections for you.
This function signals http-request-failed
when the HTTP status code is 4xx or 5xx.
[Function] get
(dex:get uri &key version headers basic-auth cookie-jar keep-alive use-connection-pool connect-timeout read-timeout max-redirects force-binary force-string want-stream ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] post
(dex:post uri &key version content headers basic-auth cookie-jar keep-alive use-connection-pool connect-timeout read-timeout force-binary force-string want-stream ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] head
(dex:head uri &key version headers basic-auth cookie-jar connect-timeout read-timeout max-redirects ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] put
(dex:put uri &key version content headers basic-auth cookie-jar keep-alive use-connection-pool connect-timeout read-timeout force-binary force-string want-stream ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] patch
(dex:patch uri &key version content headers basic-auth cookie-jar keep-alive use-connection-pool connect-timeout read-timeout force-binary force-string want-stream ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] delete
(dex:delete uri &key version headers basic-auth cookie-jar keep-alive use-connection-pool connect-timeout read-timeout force-binary force-string want-stream ssl-key-file ssl-cert-file ssl-key-password stream verbose proxy insecure ca-path)
[Function] fetch
Send a GET request to URI
and write the response body to the DESTINATION
.
(dex:fetch uri destination &key (if-exists error) verbose proxy insecure)
Benchmark
- Server
- Sakura VPS 1GB
- nginx 1.2.7, KeepAlive On
- Client
- MacBook Pro OS X Yosemite (CPU: 3GHz Intel Core i7, Memory: 8GB)
- SBCL 1.2.9
- Downloads an HTML file (181 bytes).
Drakma
(time (dotimes (i 30) (drakma:http-request "http://files.8arrow.org/181B.html")))
Evaluation took:
1.012 seconds of real time
0.174742 seconds of total run time (0.148141 user, 0.026601 system)
17.29% CPU
1,683 forms interpreted
500 lambdas converted
3,027,928,949 processor cycles
29,416,656 bytes consed
Dexador
(time (dotimes (i 30) (dex:get "http://files.8arrow.org/181B.html")))
Evaluation took:
0.499 seconds of real time
0.028057 seconds of total run time (0.019234 user, 0.008823 system)
5.61% CPU
56 forms interpreted
16 lambdas converted
1,494,851,690 processor cycles
1,472,992 bytes consed
See Also
Author
- Eitaro Fukamachi ([email protected])
Copyright
Copyright (c) 2015 Eitaro Fukamachi ([email protected])
License
Licensed under the MIT License.
Dependencies (23)
Dependents (56)
- aws-foundation
- aws-sdk-lisp
- caveman
- chirp
- ciao
- clack
- claxy
- cl-coinpayments
- cl-coveralls
- cl-covid19
- cl-gists
- cl-incognia
- clipper
- cl-jingle
- cl-jwk
- cl-kraken
- cl-selenium-webdriver
- cl-sendgrid
- cl-sentry-client
- cl-ses4
- cl-telebot
- cl-telegram-bot
- cl-textmagic
- cl-torrents
- cl-transit
- cl-weather-jp
- cl-webdriver-client
- dfio
- doc
- freesound
- glacier
- jsonrpc
- json-schema
- lack
- latter-day-paypal
- lemmy-api
- lisp-pay
- lisp-stat
- lunamech-matrix-api
- mailgun
- north
- openai-openapi-client
- openapi-generator
- openid-key
- openrpc
- oxenfurt
- qlot
- reblocks
- reblocks-auth
- simple-currency
- stepster
- stripe
- stripe-against-the-modern-world
- translate-client
- wallstreetflets
- webapi