Skip to content

Commit 7d55e57

Browse files
gwynnegwynne
committed
First draft of MySQLNIO proposal
1 parent 7cfe7c6 commit 7d55e57

File tree

1 file changed

+357
-0
lines changed

1 file changed

+357
-0
lines changed

‎proposals/NNNN-mysql-nio.md

+357
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,357 @@
1+
# MySQLNIO: An NIO-based MySQL Driver
2+
3+
* Proposal: [SSWG-NNNN](https://github.com/swift-server/sswg/tree/mysql-nio-proposal/proposals/NNNN-mysql-nio.md)
4+
* Author(s): [Gwynne Raskind](https://github.com/gwynne)
5+
* Review Manager: TBD
6+
* Status: **Implemented**
7+
* Implementation: [vapor/mysql-nio](https://github.com/vapor/mysql-nio)
8+
* Forum Threads: Pitch (Pending), Discussion (Pending)
9+
10+
## Introduction
11+
12+
`MySQLNIO` is a client package for connecting to, authorizing, and querying a MySQL server. At the heart of this module are channel handlers for parsing and serializing messages in MySQL's proprietary wire protocol. These channel handlers are combined in a request / response style connection type that provides a convenient, client-like interface for performing queries. Support for both simple (text) and parameterized (binary) querying is provided out of the box alongside a `MySQLData` type that handles conversion between MySQL's wire format and native Swift types.
13+
14+
## Motiviation
15+
16+
Most Swift implementations of MySQL clients are based on the [libmysqlclient](https://dev.mysql.com/doc/c-api/8.0/en/) C library which handles transport internally. Building a library directly on top of MySQL's wire protocol using SwiftNIO should yield a more reliable, maintainable, and performant interface for MySQL databases.
17+
18+
## Dependencies
19+
20+
This package has four dependencies:
21+
22+
- `swift-nio` from `2.0.0`
23+
- `swift-nio-ssl` from `2.14.0`
24+
- `swift-log` from `1.0.0`
25+
- `swift-crypto` from `1.0.0 ..< 3.0.0`
26+
27+
This package has no additional system dependencies.
28+
29+
## Proposed Solution
30+
31+
This section goes into detail on a few distinct types from this module to give an idea of how they work together and what using the package looks like.
32+
33+
### MySQLConnection
34+
35+
The base connection type, `MySQLConnection`, is a wrapper around NIO's `ClientBootstrap` that initializes the pipeline to communicate via MySQL packets using a request / response pattern.
36+
37+
```swift
38+
import MySQLNIO
39+
40+
// create a new event loop group
41+
let elg = MultiThreadedEventLoopGroup(numberOfThreads: 1)
42+
defer { try! elg.syncShutdownGracefully() }
43+
44+
// create a new connection and authenticate using credentials
45+
let address = try SocketAddress(ipAddress: "127.0.0.1", port: 5432)
46+
let conn = try await MySQLConnection.connect(
47+
to: address,
48+
username: "username",
49+
password: "password",
50+
// optionally configure TLS
51+
tlsConfiguration: .makeClientConfiguration(),
52+
serverHostname: "127.0.0.1",
53+
on: elg.next()
54+
)
55+
defer { try! conn.close().wait() } // `await` is not available in `defer`
56+
57+
// ready to query
58+
print(conn) // MySQLConnection
59+
```
60+
61+
#### Closing
62+
63+
A connection _must_ be closed before it deinitializes. `MySQLConnection` ensures this by asserting that it has been closed in its `deinit` handler. This is meant to help developers implement proper graceful shutdown early and avoid leaking memory or sockets.
64+
65+
### Simple Query
66+
67+
Assuming we have an active, authenticated `MySQLConnection`, we can query the connected server using MySQL's simply query command:
68+
69+
```swift
70+
import MySQLNIO
71+
72+
let conn = try await MySQLConnection.connect(...)
73+
defer { try! conn.close().wait() }
74+
75+
// select the current version
76+
let rows = try await conn.simpleQuery("SELECT VERSION() AS version")
77+
print(rows) // [MySQLRow]
78+
79+
// fetch the version column from the first row by attempting to
80+
// read it as a Swift string
81+
let version = rows.first?.column("version")?.string
82+
print(version) // String?
83+
```
84+
85+
This format does not support parameterizing input and returns all data in string format. To provide input values, they must be directly inserted into the query:
86+
87+
```swift
88+
try await conn.simpleQuery("SELECT * FROM planets WHERE name = 'Earth'")
89+
```
90+
91+
### Query
92+
93+
We can also perform parameterized queries with an active `MySQLConnection`. These queries support binding input parameters and return data in a more compact binary format.
94+
95+
Input parameters are passed as an array of `MySQLData` following the SQL string. In the query string, input parameters are referenced by the `?` placeholder marker:
96+
97+
```swift
98+
import MySQLNIO
99+
100+
let conn = try await MySQLConnection.connect(...)
101+
defer { try! conn.close().wait() }
102+
103+
// selects all planets where name is equal to the first bound parameter
104+
let rows = try await conn.query("SELECT * FROM planets WHERE name=?", ["Earth"])
105+
106+
// fetch the "name" column from the first row as a string
107+
let foo = rows.first?.column("name")?.string
108+
print(foo) // Optional("Earth")
109+
```
110+
111+
### MySQLData
112+
113+
`MySQLData` represents data both going to and coming from MySQL.
114+
115+
#### Input
116+
117+
An array of `MySQLData` is supplied alongside parameterized queries, one for each parameter. There are many initializers for creating `MySQLData` instances from Swift's standard types. For example:
118+
119+
```swift
120+
import MySQLNIO
121+
122+
let string = MySQLData(string: "Hello")
123+
let double = MySQLData(double: 3.14)
124+
let date = MySQLData(date: Date(timeIntervalSince1970: 42))
125+
```
126+
127+
`MySQLData` also conforms to Swift's `ExpressibleBy...` protocols, allowing various kinds of literals to be used directly:
128+
129+
```swift
130+
import MySQLNIO
131+
132+
let inputs: [MySQLData] = ["hello", 3]
133+
```
134+
135+
#### Output
136+
137+
Likewise, `MySQLData` can be converted back to Swift types. This is useful for converting data returned by MySQL queries into meaningful types. There are many methods for Swift's standard types; for example:
138+
139+
```swift
140+
import MySQLNIO
141+
142+
let data: MySQLData
143+
print(data.string) // String?
144+
```
145+
146+
The full list of supported types at the time of this writing is:
147+
148+
- `Swift.String`
149+
- `Swift.Int`
150+
- `Swift.Int64`
151+
- `Swift.Int32`
152+
- `Swift.Int16`
153+
- `Swift.Int8`
154+
- `Swift.UInt`
155+
- `Swift.UInt64`
156+
- `Swift.UInt32`
157+
- `Swift.UInt16`
158+
- `Swift.UInt8`
159+
- `Swift.Float`
160+
- `Swift.Double`
161+
- `Swift.Bool`
162+
- `Foundation.Decimal`
163+
- `Foundation.Date`
164+
- `Foundation.Data`
165+
- `Foundation.UUID`
166+
- `MySQLNIO.MySQLTime`
167+
- Any `Swift.Encodable` value (automatically encoded as JSON)
168+
169+
### MySQLRow
170+
171+
Both `simpleQuery()` and `query()` return an array of `MySQLRow`. Each row can be thought of as a dictionary with column names as the key and data as the value. While the actual storage implementation is private, `MySQLRow` provides the `column(_:)` method for accessing column data:
172+
173+
```swift
174+
struct MySQLRow {
175+
func column(_ name: String, table: String? = nil) -> MySQLData?
176+
}
177+
```
178+
179+
If no column with the given name exists in the row, `nil` is returned. If a table name is provided, the column must have been selected from the table with that name to be returned. If no table name is provided (the default), the first matching column in the row from _any_ table is returned.
180+
181+
### MySQLError
182+
183+
The `MySQLError` type represents errors thrown from both the MySQL package itself (during encoding, for example) and errors returned by the server:
184+
185+
```swift
186+
public enum MySQLError: Error {
187+
case secureConnectionRequired
188+
case unsupportedAuthPlugin(name: String)
189+
case authPluginDataError(name: String)
190+
case missingOrInvalidAuthMoreDataStatusTag
191+
case missingOrInvalidAuthPluginInlineCommand(command: UInt8?)
192+
case missingAuthPluginInlineData
193+
case unsupportedServer(message: String)
194+
case protocolError
195+
case server(MySQLProtocol.ERR_Packet)
196+
case closed
197+
case duplicateEntry(String)
198+
case invalidSyntax(String)
199+
}
200+
```
201+
202+
### MySQLDatabase
203+
204+
While `MySQLConnection` is the primary type used for connecting, authorization, and TLS negotation, the `MySQLDatabase` protocol provides an abstract interface for issuing commands to the MySQL server and explicitly requesting a dedicated connection (intended for use cases where connection pooling must be avoided, such as leveraging transactions). `MySQLConnection` conforms to this protocol:
205+
206+
```swift
207+
protocol MySQLDatabase {
208+
var eventLoop: EventLoop { get }
209+
var logger: Logger { get }
210+
func send(_ command: MySQLCommand, logger: Logger) -> EventLoopFuture<Void>
211+
func withConnection<T>(_ closure: @escaping (MySQLConnection) -> EventLoopFuture<T>) -> EventLoopFuture<T>
212+
}
213+
```
214+
215+
It is expected that consumers of this package will adopt the `MySQLDatabase` protocol to provide, for example, automatic connection pooling. (See Vapor's [MySQLKit](https://github.com/vapor/mysql-kit) for an example of such an implementation.)
216+
217+
The `send(_:logger:)` method communicates with the server to perform the operation implemented by the provided `MySQLCommand` instance. `MySQLNIO` users should not call this method directly; it is an unfortunately exposed implementation detail which will be removed in a future revision of the API.
218+
219+
The `withConnection(_:)` method calls the provided closure with an instance of `MySQLConnection`. This instance is guaranteed, by the fact that is specifically a "connection" and not a generic "database", to represent a single, active database connection. This method exists so that users can require any `MySQLDatabase` to provide a singular connection, guaranteed to be the same connection (i.e., not newly opened or from a pool) for the duration of the closure's scope. Normally, a connection pool implementation of `MySQLDatabase` would be free to, for example, send every query on a different underlying connection; this behavior is incompatible with database-level transactions and other uses of per-connection state (such as MySQL's local variables).
220+
221+
The following additional methods are also available on `MySQLDatabase`:
222+
223+
```swift
224+
extension MySQLDatabase {
225+
public func query(
226+
_ sql: String,
227+
_ binds: [MySQLData] = [],
228+
onMetadata: @escaping (MySQLQueryMetadata) throws -> () = { _ in }
229+
) -> EventLoopFuture<[MySQLRow]>
230+
231+
public func query(
232+
_ sql: String,
233+
_ binds: [MySQLData] = [],
234+
onRow: @escaping (MySQLRow) throws -> (),
235+
onMetadata: @escaping (MySQLQueryMetadata) throws -> () = { _ in }
236+
) -> EventLoopFuture<Void>
237+
238+
public func simpleQuery(_ sql: String) -> EventLoopFuture<[MySQLRow]>
239+
240+
public func simpleQuery(_ sql: String, onRow: @escaping (MySQLRow) -> ()) -> EventLoopFuture<Void>
241+
242+
public func logging(to logger: Logger) -> MySQLDatabase
243+
}
244+
```
245+
246+
The `query()` and `simpleQuery()` methods call the base protocol's `send(_:logger:)` method; these are the only `MySQLCommand` types currently implemented at the time of this writing, and it should never be necessary to call `send(_:logger:)` directly.
247+
248+
The `logging(to:)` method returns a new "database" which uses the specified logger for all operations. The returned instance is a simple wrapper around the original, and the original remains usable, though calling methods on the original will not use the new logger. The wrapper works with any `MySQLDatabase`, regardless of type.
249+
250+
Finally, Concurrency-based convenience methods are also available:
251+
252+
```swift
253+
extension MySQLDatabase {
254+
public func send(
255+
_ command: MySQLCommand,
256+
logger: Logger
257+
) async throws -> Void
258+
259+
public func withConnection<T>(
260+
_ closure: @escaping @Sendable (MySQLConnection) async throws -> T
261+
) async throws -> T
262+
263+
public func query(
264+
_ sql: String,
265+
_ binds: [MySQLData] = [],
266+
onMetadata: @escaping (MySQLQueryMetadata) throws -> Void = { _ in }
267+
) async throws -> [MySQLRow]
268+
269+
public func simpleQuery(_ sql: String) async throws -> [MySQLRow]
270+
}
271+
```
272+
273+
#### Note on usage
274+
275+
By design, most of `MySQLNIO`'s useful methods are made available on `MySQLDatabase`, rather than only on `MySQLConnection`. It is intended that the `MySQLDatabase` type be used for all operations not directly related to opening and closing connections. For example, in a theoretical controller:
276+
277+
```swift
278+
final class UserController: Controller {
279+
let db: MySQLDatabase
280+
281+
init(db: MySQLDatabase) {
282+
self.db = db
283+
}
284+
285+
func names(_ req: HTTPRequest) async throws -> [String] {
286+
return try await self.db.query("SELECT name FROM users").compactMap {
287+
$0.column("name")?.string
288+
}
289+
}
290+
}
291+
```
292+
293+
Because this controller uses `MySQLDatabase`, any of the following could be supplied to it:
294+
295+
- A connected `MySQLConnection`
296+
- A pool of `MySQLConnection`s
297+
- A mock database for testing purposes
298+
299+
#### `MySQLCommand`
300+
301+
MySQL's wire protocol follows a stateful command/reply pattern, where a single command can involve anything from a simple request and reply, to an entire "secondary" protocol specific to the command in question. `MySQLCommand` provides an abstraction over the common behaviors defined by the underlying wire protocol, allowing a command to be defined in terms of a packet handler and a "state" encapsulating the various kinds of possible responses:
302+
303+
```swift
304+
public protocol MySQLCommand {
305+
func handle(
306+
packet: inout MySQLPacket,
307+
capabilities: MySQLProtocol.CapabilityFlags
308+
) throws -> MySQLCommandState
309+
310+
func activate(capabilities: MySQLProtocol.CapabilityFlags) throws -> MySQLCommandState
311+
}
312+
313+
public struct MySQLCommandState {
314+
static var noResponse: MySQLCommandState
315+
static var done: MySQLCommandState
316+
static func response(_ packets: [MySQLPacket]) -> MySQLCommandState
317+
318+
public init(
319+
response: [MySQLPacket] = [],
320+
done: Bool = false,
321+
resetSequence: Bool = false,
322+
error: Error? = nil
323+
)
324+
}
325+
```
326+
327+
A `MySQLCommand` is responsible for sending the initial command message and handling the server's responses. The command returns the `.done` command state when processing is completed, causing the client's send future to be fulfilled.
328+
329+
### Cryptographic requirements
330+
331+
The `mysql_native_password` authentication mechanism (the default up through MySQL 5.7, available by configuration in MySQL 8.0+) requires the use of SHA-1 hashing. The SHA-1 digest implementation is provided by the SwiftCrypto package.
332+
333+
The `caching_sha2_password` mechanism (the default starting in MySQL 8.0) requires SHA-256 hashing in all cases. SwiftCrypto also provides this implementation. If the connection to the server is unencrypted (e.g. not over TLS), an RSA encryption operation must also be performed to avoid sending credentials in plaintext. No version of MySQL offers an alternative encryption for this purpose. Once again, the implementation in SwiftCrypto is relied upon.
334+
335+
### Todo / Discussion
336+
337+
Here are some things that are still a work in progress:
338+
339+
- **Reuse of prepared statements**: In addition to providing binary data formats and safe parameterization for bound values in queries, MySQL's prepared statements are intended to speed up repetitions of the same query. This should ideally be leveraged.
340+
341+
### How to use
342+
343+
To try this package out at home, add the following dependency to your `Package.swift` file:
344+
345+
```swift
346+
.package(url: "https://github.com/vapor/mysql-nio.git", from: "1.0.0"),
347+
```
348+
349+
Then add `.product(name: "MySQLNIO", package: "mysql-nio")` to your module target's dependencies array.
350+
351+
### Seeking Feedback
352+
353+
* If anything, what does this proposal *not cover* that you will definitely need?
354+
* If anything, what could we remove from this and still be happy?
355+
* API-wise: what do you like, what don't you like?
356+
357+
Feel free to post feedback as response to this post and/or GitHub issues on [vapor/mysql-nio](https://github.com/vapor/mysql-nio).

0 commit comments

Comments
 (0)