Spark has built-in support for BSD sockets. It also have higher-level abstractions to directly deal with protocols like HTTP and FTP. To ease development of high-performance network servers without the overhead of one thread per client, it has an implementation of the Reactor Pattern. Each of these facilities are explained in the following sections.
A socket is an end-point of a two-way communication link between two programs running
on a network. The Berkeley sockets application programming interface (API) (also known
as the BSD Sockets API) comprises a library for developing applications in the C
programming language that perform inter-process communication, most commonly across
a computer network. Spark has a wrapper for this API in the net library.
We will try to grasp the basics of networking by writing a client for the World Wide Web. The browser you are using to view this page is an example of such a client. The one we are going to write is much simpler. You pass it the address of a web server, it will download the default page on that server and print the contents to the console. Code for our tiny HTTP client is listed below. The embedded comments explain what each piece of code does.
(import (net))
;; Get the URL from the command-line.
(define args (argv-list))
(if (= 0 (length args))
(begin
(display "Usage: spark http-client.ss ")
(newline)
(exit)))
;; The first argument is the name of a HTTP server,
;; for instance "spark.vmathew.in".
(define url (car args))
;; Create a new socket object.
(define client-socket (socket))
;; Initialize an address object with the url and the
;; default HTTP port.
(define host-address (address url 80))
;; Open the socket and connect to the web server.
(socket-open client-socket)
(socket-connect client-socket host-address)
;; Send a HTTP GET request.
(socket-send client-socket "GET / HTTP/1.0")
;; \r\n\r\n
(socket-send-bytes client-socket (list 13 10 13 10))
;; Get back the response into a buffer, 1024 bytes in length.
(define buffer-size 1024)
(define data (socket-recv client-socket buffer-size))
;; Read and print the whole data.
(while (not (null? data))
(printf "~a" data)
(set! data (socket-recv client-socket buffer-size)))
;; Finally, close the socket.
(socket-close client-socket)
Now let us turn our attention to writing server applications. Let us write our own tiny HTTP server using the socket library. This web server will accept client connections in a loop and respond with a simple HTML page.
(import (net))
;; Create the socket on which the server will
;; listen for incoming client connections.
(define server-socket (socket))
;; Initialize an address object.
;; By default, it points to localhost.
;; Only, set the port.
(define addr (address))
(address-port! addr 80)
;; Open the socket and bind it to the address.
(socket-open server-socket)
(socket-bind server-socket addr #t)
;; Listen for incoming client connections.
;; Our server will handle up to 10 clients.
(for i in (range 10)
(socket-listen server-socket)
;; Accept an incoming connection.
;; (socket-accept) returns a pair that consists of
;; a socket that represents the new connection
;; and an address objects that contains information
;; about the client. The values of this pair can
;; be extracted using
;; (connection-socket conn) and
;; (connection-address conn).
(let ((client-socket (connection-socket
(socket-accept server-socket))))
;; Print the request. As of now, we don't parse it.
(print (socket-recv client-socket 1024))
(newline)
(flush-output)
;; Send back the response ...
(socket-send client-socket "hello, world!
")
;; ... and close the connection.
(socket-close client-socket)))
;; Close the server socket.
(socket-close server-socket)
There it is! A web server is 15 lines of code (excluding comments). Open your web browser and go to http://localhost. You should see the message "hello, world!" in your browser window.
As you might have already noticed, our web server is not very scalable. It handles only one client at a time. It can be extended to serve more than one simultaneous connection by spawinig a new thread for each connection. There is another, more scalable method to write robust network servers. We will learn about this in the next section.
Spark comes with a networking framework which is based on the Reactor Pattern. The idea is to initialize an object that can watch a Socket for incoming or outgoing data. When data is available, a callback is executed. An advantage of this pattern is that it brings concurrency to network programming without the overhead of creating and managing Threads. These libraries should be preferred over plain Sockets for writing highperformance network servers.
An object called socket-acceptor is used to write servers that can handle client
connections asynchronously. Let us make our tiny web server more robust with the help of
a socket-acceptor:
(import (net) (reactor))
;; Callback for a new client connection.
(define (on-client-connect acceptor client-conn)
(let ((client-socket (connection-socket client-conn)))
;; Watch thi socket for incoming data.
(acceptor-add-watch acceptor client-socket 'for-read)))
;; Callback executed when data is available
;; for read on a connected client socket.
(define (on-client-read acceptor client-socket)
(socket-recv client-socket 1024)
(socket-send client-socket
"hello, world")
;; Remove the socket from being watched and close it.
(acceptor-remove-watch acceptor client-socket 'for-read)
(socket-close client-socket))
;; Callback for connection timeouts.
(define (on-timeout acceptor)
(printf "Timeout.~n")
(acceptor-close time-server)
(exit))
;; Initialize an acceptor to listen on port 80.
(define time-server (socket-acceptor))
(acceptor-port! time-server 80)
;; Provide the acceptor with callbacks.
(acceptor-on-client-connect! time-server on-client-connect)
(acceptor-on-client-read! time-server on-client-read)
(acceptor-on-server-timeout! time-server on-timeout)
;; Specify a 10 second timeout for the acceptor.
(acceptor-open time-server #t (list 10 0))
;; Wait for incoming connections.
(while #t
(acceptor-watch time-server))
;; Done. Close the acceptor.
(acceptor-close time-server)
Execute the above code and point your browser to http://localhost. You should see the message "hello, world". The new web server can handle thousands of concurrent client requests. It achieves this scalability as it does not block on a single connection. Instead connections are polled for incoming data. When data is available, a callback procedure is evaluated.
An object called acceptor wraps up the details of polling and callback evaluation. We
initialize an acceptor to listen on a port and pass it three callbacks. The first one is invoked
when a new client connection is accepted. To actually read from this connection, it should
be added to a list of watched sockets. This is achieved by a call to the acceptor-add-watch
procedure. When data arrives on a watched socket, the on-client-read
callback is evaluated. This callback can be invoked many times for a single connection,
based on the amount of data that needs to be read. The acceptor can be made to timeout
if a connection did not arrive within a specific period of time. The timeout value is passed
as the last argument of acceptor-open. This is a list of two integers that represent the
timeout period as seconds and microseconds. In most cases, timeout is specified as
seconds and the microseconds field is left as zero.
Though the reactor framework is more suitable for implementing servers, it can also be used to write clients that handle network events asynchronously. The socket-connector object is used for this purpose. A connector can be given four callback procedures - one that gets called when connection with the server is established, two to handle data read and write operations and the last one to deal with timeout situations. The following code shows our simple HTTP client re-written using a connector:
(import (net)
(reactor))
;; Get the URL from the command-line
(define args (argv-list))
(if (= 0 (length args))
(begin
(display "Usage: spark http-client.ss ")
(newline)
(exit)))
(define url (car args))
(define done #f)
;; Callback for connection event.
(define (on-connect connector client-socket)
;; Send the HTTP GET request
(socket-send client-socket "GET / HTTP/1.0")
(socket-send-bytes client-socket (list 13 10 13 10)))
;; Callback for data read events.
;; May get called many times...
(define (on-write connector client-socket)
(let* ((buffer-size 1024)
(data (socket-recv client-socket buffer-size)))
;; Read and print whatever we were able to read.
(if (not (null? data))
(printf "~a" data)
(set! done #t))))
;; Called when we timeout waiting for data.
(define (on-timeout connector)
(connector-remove-watch connector sock 'for-write)
(set! done #t))
;; Set up the connector and the callbacks.
(define http-client (socket-connector (list (address url
80))))
(connector-on-connect! http-client on-connect)
(connector-on-write! http-client on-write)
(connector-on-timeout! http-client on-timeout)
;; Open the connector with a timeout of 10 seconds.
(connector-open http-client (list 10 0))
;; Watch the connector till the data
;; we expect has arrived from the server.
(while (not done)
(connector-watch http-client))
(connector-close http-client)
The connector-watch procedure is constantly called to listen for data events on the
client Socket. Once we are done with a connection, its associated Socket can be removed
from being watched by calling the procedure connector-remove-watch. All resources
held by the connector must be released using connector-close to avoid possible resource
leaks.
Note that the procedure socket-connector takes a list of addresses. This is to facilitate
using a single connector to talk to several servers simultaneously.
Spark makes it easy to program with internet protocols like HTTP and FTP. For this we
have wrapper for the popular cURL library. The following code shows how easy it is to
download data from the internet using the net-curl library:
(import (net-curl))
;; Initialize the curl environment.
(curl-env)
;; Create a curl object and associate it with a URL.
(define c (curl "http://www.scheme.org" (lambda (s d)
(printf "~a~n" s))))
;; Execute the network operations.
(curl-perform c)
;; Finally, release the curl object and the curl environment.
(curl-dispose c)
(curl-env-dispose)
The second argument to curl is a procedure object. This gets called when data is written
to the curl object from the network. There can also be an optional third argument, which is
useful as a callback for uploading data. These procedures take two arguments, one is the
actual data and the other is a user defined object specified using the curl-callback-param!
procedure. For most cases this will be null. There are few other callbacks that can
be set using the curl-callback! procedure.
(Please see the source code of the net-curl library for more details.).