-
Notifications
You must be signed in to change notification settings - Fork 223
HOW TO GIZMO: create a JSONService using MySQL
In December 2015, the gophers at The New York Times open sourced a new microservice toolkit for the Go programming language called Gizmo. This will be the first of several tutorials explaining how to to use the toolkit to its fullest capabilities. We’re going to walk through the process of using a Gizmo SimpleServer to create a very simple JSON service for managing My Saved Items for a single user on NYTimes.com.
We’ll cover the following:
- Configuring a Gizmo server with MySQL for persistence
- Creating the main.go file
- Creating the service.go file
- Creating a JSONEndpoint
- Testing a JSONEndpoint in a SimpleServer
- Creating a testable repository with sqliface
We’re going to use a MySQL database to act as a repository for a user’s “saved items.” We’ll assume this table has been created in our local database:
CREATE TABLE `saved_items` (
`user_id` INT(11) UNSIGNED NOT NULL,
`url` VARCHAR(255) NOT NULL,
`timestamp` DATETIME NOT NULL,
PRIMARY KEY (`user_id`, `url`)
);From here, let’s start by putting together a JSON file to configure our server and give it credentials for our MySQL server. In real life, this config file should probably be provisioned by a configuration management tool. In our case, we use Puppet and consul-template to dynamically populate our database server addresses. For now, let’s just hardcode it into our base directory. Here’s what our config.json file looks like:
{
"MySQL": {
"User": "db_user",
"Pw": "db_pw",
"DBName": "saveditems",
"Host": "localhost:3306"
},
"Server": {
"HTTPPort": 8080,
"HTTPAccessLog": "/var/log/nyt-saved-items-api-access.log",
"Log": "/var/log/nyt-saved-items-api.log"
}
}If you’re familiar with Gizmo, you may notice that this JSON object is the same structure as the gizmo/config.Config struct, which is meant to be a handy catchall struct for configuring Gizmo services like this one. It contains structs for MySQL and Gizmo’s server.Server, as well as MongoDB, Oracle, AWS’s S3, SNS and SQS, Kafka and Gorilla toolkit’s secure cookie. It also includes attributes for Graphite hosts and levelled logging. Having all these different types helps us reuse the struct for many services, which keeps developers from having to implement their own ‘config’ type structs for every project.
Since we’re using the default server type, health check type and log level, we do not need to include configuration values for them. If HTTPPort is not passed, Gizmo will default to passing “:” as the address to server for net.Listen and allow the standard library to pick a port number. Log will default to an empty string, which will tell default to stderr, but if set Gizmo will set up logrus’ output to be a File from NYTimes’ logrotate package. The logrotate File wraps an *os.File and listens for SIGHUPs from logrotated to signal that the current log file has been “rotated” and a new one is available. If HTTPAccessLog is left blank, no access logs will be created. By passing a value, an access log middleware will be added to the service and Apache style logs will be written to the given location with a logrotate wrapper.
In this example, we’re using a JSON file. However, with Gizmo’s helper functions in the config package, we could very well load this information from environment variables or Consul’s key/value store.
Once we have our configuration set up, we can start writing Go by creating our main.go file to manage initiating and running the SimpleServer. The main.go file (and the main package in general) tends to be very small as most of the basics of managing the server are done by Gizmo. Here’s the main.go function for this implementation:
package main
import (
"flag"
"github.com/NYTimes/gizmo/config"
"github.com/NYTimes/gizmo/server"
_ "github.com/go-sql-driver/mysql"
"github.com/NYTimes/gizmo/examples/servers/mysql-saved-items/service"
)
func main() {
// load from the local JSON file into a config.Config struct
cfg := config.NewConfig("./config.json")
flag.Parse()
// SetServerOverrides will allow us to override some of the values in
// the JSON file with CLI flags (i.e. -log)
config.SetServerOverrides(cfg.Server)
// initialize Gizmo’s server with given configs
server.Init("nyt-saved-items", cfg.Server)
// instantiate a new ‘saved items service’ with our MySQL credentials
svc, err := service.NewSavedItemsService(cfg.MySQL)
if err != nil {
server.Log.Fatal("unable to create saved items service: ", err)
}
// register our saved item service with the Gizmo server
err = server.Register(svc)
if err != nil {
server.Log.Fatal("unable to register saved items service: ", err)
}
// run the Gizmo server
err = server.Run()
if err != nil {
server.Log.Fatal("unable to run saved items service: ", err)
}
}For the rest of this tutorial, we are going to focus on what goes in the service package that contains the SavedItemsService referenced in our main.go. We’ll start by looking at the service.go, which contains all the methods required to implement and describe a Gizmo JSONService.
package service
import (
"net/http"
"strconv"
"github.com/NYTimes/gizmo/config"
"github.com/NYTimes/gizmo/server"
"github.com/NYTimes/gziphandler"
"github.com/gorilla/context"
)
// SavedItemsService will keep a handle on the saved items repository and implement
// the gizmo/server.JSONService interface.
type SavedItemsService struct {
repo SavedItemsRepo
}
// NewSavedItemsService will attempt to instantiate a new repository and service.
func NewSavedItemsService(cfg *config.MySQL) (*SavedItemsService, error) {
repo, err := NewSavedItemsRepo(cfg)
if err != nil {
return nil, err
}
return &SavedItemsService{repo}, nil
}SavedItemsService will be our struct to implement server.JSONService, which can be registered with Gizmo’s server.SimpleServer. The service will act similarly to a controller in a traditional MVC framework. It will provide all of the endpoints for a service, their routing information and methods for adding common functionality across all the endpoints, also known as ‘middleware.’
Beyond implementing Gizmo interfaces, the service struct also provides hooks to the ‘business’ or ‘model’ layers of our server. In this scenario, the business layer is encapsulated in a repository struct, which will handle interacting with the MySQL database. Our service will hold onto the SavedItemsRepo interface instead of the concrete implementation to make testing easier. We’ll include more on this later. For now, here’s the rest of the service.go file:
// Prefix is to implement gizmo/server.Service interface. The string will be prefixed to all endpoint
// routes.
func (s *SavedItemsService) Prefix() string {
return "/svc"
}
// Middleware provides a hook to add service-wide http.Handler middleware to the service.
// In this example we are using it to add GZIP compression to our responses.
// This method helps satisfy the server.Service interface.
func (s *SavedItemsService) Middleware(h http.Handler) http.Handler {
// wrap the response with our GZIP Middleware
return gziphandler.GzipHandler(h)
}
// JSONMiddleware provides a hook to add service-wide middleware for how JSONEndpoints
// should behave. In this example, we’re using the hook to check for a header to
// identify and authorize the user. This method helps satisfy the server.JSONService interface.
func (s *SavedItemsService) JSONMiddleware(j server.JSONEndpoint) server.JSONEndpoint {
return func(r *http.Request) (code int, res interface{}, err error) {
// wrap our endpoint with an auth check (see func declaration below)
// and call the endpoint
code, res, err = authCheck(j)(r)
// if the endpoint returns an unexpected error, return a generic message
// and log it.
if err != nil && code != http.StatusUnauthorized {
// LogWithFields will add all the request context values
// to a structured log entry along some other request info
server.LogWithFields(r).WithField("error", err).Error("unexpected service error")
return http.StatusServiceUnavailable, nil, ErrServiceUnavailable
}
return code, res, err
}
}
// idKey is a type to use as a key for storing data in the request context.
type idKey int
// userIDKey can be used to store/retrieve a user ID in a request context.
const userIDKey idKey = 0
// authCheck is a JSON middleware to check the request for a valid USER_ID
// header and set it into the request context. If the header is invalid
// or does not exist, a 401 response will be returned.
func authCheck(j server.JSONEndpoint) server.JSONEndpoint {
return func(r *http.Request) (code int, res interface{}, err error) {
// check for User ID header injected by API Gateway
idStr := r.Header.Get("USER_ID")
// verify it's an int
id, err := strconv.ParseUint(idStr, 10, 64)
// reject request if bad/no user ID
if err != nil || id == 0 {
return http.StatusUnauthorized, nil, ErrUnauth
}
// set the ID in context if we're good
context.Set(r, userIDKey, id)
return j(r)
}
}
// JSONEndpoints is the most important method of the Service implementation. It provides a
// listing of all endpoints available in the service with their routes and HTTP methods.
// This method helps satisfy the server.JSONService interface.
func (s *SavedItemsService) JSONEndpoints() map[string]map[string]server.JSONEndpoint {
return map[string]map[string]server.JSONEndpoint{
"/saved-items": map[string]server.JSONEndpoint{
"GET": s.Get,
"PUT": s.Put,
"DELETE": s.Delete,
},
}
}In the service/service.go file above, we’ve declared the basic layout of what our service looks like. It has three endpoints under the same URI to GET, PUT and DELETE saved items for a user, and there are two middleware layers to authorize users and gzip our responses. The authorization layer expects some sort of gateway to have injected a “USER_ID” header into our request header. It will grab the value, parse it and save it to the request’s context for use in the endpoint layer. With a bad or invalid header, the request will get rejected with a 401 status code and an error message.
Below, in service.go’s continuation, we’ve also declared a couple of small structs and errors that will be shared across endpoints to make the service highly cohesive.
type (
// jsonResponse is a generic struct for responding with a simple JSON message.
jsonResponse struct {
Message string `json:"message"`
}
// jsonErr is a tiny helper struct to make displaying errors in JSON better.
jsonErr struct {
Err string `json:"error"`
}
)
func (e *jsonErr) Error() string {
return e.Err
}
var (
// ErrServiceUnavailable is a global error that will get returned when we are experiencing
// technical issues.
ErrServiceUnavailable = &jsonErr{"sorry, this service is currently unavailable"}
// ErrUnauth is a global error returned when the user does not supply the proper
// authorization headers.
ErrUnauth = &jsonErr{"please include a valid USER_ID header in the request"}
)Next, we’ll take a look at the simple PUT endpoint that allows a user to add a new item to their “Saved Items” list. To hit the endpoint, clients can send an HTTP request with a PUT method and a URI of “/svc/saved-items?url={URL of item to save}” to the server with an appropriate “USER_ID” header.
package service
import (
"net/http"
"github.com/NYTimes/gizmo/server"
"github.com/gorilla/context"
)
// Put is a JSONEndpoint for adding a new saved item to a user's list.
func (s *SavedItemsService) Put(r *http.Request) (int, interface{}, error) {
// gather the inputs from the request
id := context.Get(r, userIDKey).(uint64)
url := r.URL.Query().Get("url")
// do work and respond
err := s.repo.Put(id, url)
if err != nil {
return http.StatusInternalServerError, nil, err
}
server.LogWithFields(r).Info("successfully saved item")
return http.StatusCreated, jsonResponse{"successfully saved item"}, nil
}The endpoint is a method on SavedItemsService that implements the JSONEndpoint type. It first gathers the user ID from the request context, which was put there by our “authCheck” middleware. The URL of the item to be saved is pulled from the request’s URL query next, and then both parameters are passed to our service’s repo to persist the data to the database (we’ll cover the repo later in this tutorial). From there, our endpoint returns with a 500 status code if there was an error; otherwise, it logs the successful save and responds with a 201 status code.
Since this is a JSONEndpoint and we’re using a JSONService, internally Gizmo wrapped the function with its server.JSONToHTTP function, which converts the JSONEndpoint to an http.Handler. The conversion simply deals with setting the proper JSON content-type header and marshalling the JSON of response struct or error to the http.ResponseWriter.
Now that we’ve got an endpoint, we need to write some tests to verify that everything acts as we expect it to. Before we get into the actual tests, we need to create a ‘test’ implementation of our SavedItemsRepo interface to inject into our service for testing.
// testSavedItemsRepo is a mock implementation of the SavedItemsRepo interface.
type testSavedItemsRepo struct {
MockGet func(uint64) ([]*SavedItem, error)
MockPut func(uint64, string) error
MockDelete func(uint64, string) error
}
// Get will call the MockGet function of the test repo.
func (r *testSavedItemsRepo) Get(userID uint64) ([]*SavedItem, error) {
return r.MockGet(userID)
}
// Put will call the MockPut function of the test repo.
func (r *testSavedItemsRepo) Put(userID uint64, url string) error {
return r.MockPut(userID, url)
}
// Delete will call the MockDelete function of the test repo.
func (r *testSavedItemsRepo) Delete(userID uint64, url string) error {
return r.MockDelete(userID, url)
}Our test implementation of the repository is very “dumb” in that you have to tell it how you want it to behave. This gives us a great hook to make the repository behave however we want for each test scenario. Let’s see it in action in the tests for the PUT endpoint.
func TestPut(t *testing.T) {
tests := []struct {
givenID string
givenURL string
givenRepoFunc func(uint64, string) error
wantCode int
wantError *jsonErr
wantResp *jsonResponse
}{As you can see, we’re using table-driven tests throughout this example. We’ll set up a few scenarios with a ‘given’ URL and ID to inject into our request object and a ‘repo’ function that will be set into a testSavedItemsRepo’s MockPut attribute before running the test.
{
"123456",
"http://nytimes.com/article",
func(id uint64, url string) error {
if id != 123456 {
t.Errorf("MockPut expected id of 123456; got %d", id)
}
if url != "http://nytimes.com/article" {
t.Errorf("MockPut expected url of `http://nytimes.com/article'; got %s", url)
}
return nil
},
http.StatusOK,
&jsonErr{},
&jsonResponse{"successfully saved item"},
},
{
"123456",
"http://nytimes.com/article",
func(id uint64, url string) error {
if id != 123456 {
t.Errorf("MockPut expected id of 123456; got %d", id)
}
if url != "http://nytimes.com/article" {
t.Errorf("MockPut expected url of `http://nytimes.com/article'; got %s", url)
}
return errors.New("nope")
},
http.StatusServiceUnavailable,
ErrServiceUnavailable,
&jsonResponse{""},
},
{
"",
"http://nytimes.com/article",
func(id uint64, url string) error {
t.Error("MockPut should not have been called in this scenario!")
return nil
},
http.StatusUnauthorized,
ErrUnauth,
&jsonResponse{""},
},
}
for _, test := range tests {
// create a new Gizmo simple server
ss := server.NewSimpleServer(nil)
// create our test repo implementation
testRepo := &testSavedItemsRepo{MockPut: test.givenRepoFunc}
// inject the test repo into a new SavedItemsService
sis := &SavedItemsService{testRepo}
// register the service with our simple server
ss.Register(sis)
// set up the w and r to pass into our server
w := httptest.NewRecorder()
r, _ := http.NewRequest("PUT", "/svc/saved-items?url="+test.givenURL, nil)
if test.givenID != "" {
r.Header.Set("USER_ID", test.givenID)
}
// run the test by passing a request we expect to hit our endpoint
// into the simple server's ServeHTTP method.
ss.ServeHTTP(w, r)
// first test validation: check the HTTP response code
if w.Code != test.wantCode {
t.Errorf("expected status code of %d; got %d", test.wantCode, w.Code)
}
// get the body of the response to inspect
bod := w.Body.Bytes()
// if we were expecting an error scenario, marshal the response
// JSON into an error to compare with what we want.
var gotErr *jsonErr
json.Unmarshal(bod, &gotErr)
if !reflect.DeepEqual(gotErr, test.wantError) {
t.Errorf("expected status response of '%#v'; got '%#v'", test.wantError, gotErr)
}
// if we expect a normal response, compare it to our wanted response struct
var got *jsonResponse
json.Unmarshal(bod, &got)
if !reflect.DeepEqual(got, test.wantResp) {
t.Errorf("expected response of \n%#v; got \n%#v", test.wantResp, got)
}
}
}The table tests have been set up for three different scenarios to check a normal success response, a response when encountering a database error and a response when no authorization header has been provided.
The code and tests for the GET and DELETE endpoints are very similar to what we’ve covered for the PUT endpoint, so we’ll skip them for the sake of brevity. You can see them (along with the rest of the code in this tutorial) in the examples/servers/mysql-saved-items directory on Github.
At this point, we’ve covered the basics of what is required to make a Gizmo server that hosts a JSON service. We have a set of cohesive endpoints that are encapsulated in our SavedItemsService and a set of tests to cover our service-level logic.
Beyond the endpoints we know of, Gizmo has also added an additional “/status.txt” endpoint to our server meant to allow external systems to check on our service. This handy endpoint can be used for several purposes. It can provide a health check for a load balancer like Amazon’s ELB, a reverse proxy like NGINX or HAProxy or even Consul to use with service discovery and monitoring.[c][d]
Gizmo has also wrapped all of our endpoints with timers and counters keeping track of which status codes each endpoint has been responding with (1XX, 2XX, 3XX. 4XX or 5XX). If we were to provide the config.Server struct with a “GraphiteHost,” the server would start emitting metrics to Graphite (PRs are welcome to add additional metrics systems).
The last subject we’d like to cover in this tutorial is testing the business layer, which is our repository of saved items. To help provide an example of how we write tests when interacting with Go’s database/sql package, we’re happy to announce that we’ve open sourced another Go package: sqliface. sqliface is a package that contains a set of interfaces and implementations to help make code using the database/sql package from the standard library more testable.
In this example, we’ll see how the MySQLSavedItemsRepo (the ‘repo’ in SavedItemsService) uses the sqliface package to test the logic involved with scanning query data from the database into a slice of our structs. Here’s the GET implementation of the MySQL repository:
// Get will attempt to query the underlying MySQL database for saved items for a single user
func (r *MySQLSavedItemsRepo) Get(userID uint64) ([]*SavedItem, error) {
query := `SELECT
user_id,
url,
timestamp
FROM saved_items
WHERE user_id = ?
ORDER BY timestamp DESC`
rows, err := r.db.Query(query, userID)
if err != nil {
return nil, err
}
defer rows.Close()
return scanItems(rows)
}
func scanItems(rows sqliface.Rows) ([]*SavedItem, error) {
var err error
// intializing so we return an empty array in case of 0
items := []*SavedItem{}
for rows.Next() {
item := &SavedItem{}
err = rows.Scan(&item.UserID, &item.URL, &item.Timestamp)
if err != nil {
return nil, err
}
items = append(items, item)
}
return items, nil
}Notice the logic for iterating over the “Rows” of our query has been abstracted to a separate “scanItems” function. Instead of accepting the normal “*sql.Rows”, which is returned by the “sql.DB. Query” function, our function accepts a new “Rows” interface from sqliface that provides the same behaviors, but allows us to swap in other implementations of the interface when we write unit tests. sqliface also happens to supply MockRow and MockRows implementations to use within your unit tests. Here’s the contents of saveditems_test.go with an example using sqliface’s mock implementations.
package service
import (
"reflect"
"testing"
"time"
"github.com/NYTimes/sqliface"
)
// TestScanItems will test our repo's logic for scanning data
// out of the DB and into structs.
func TestScanItems(t *testing.T) {
testTime := time.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC)
testTime2 := time.Date(2015, 1, 11, 12, 0, 0, 0, time.UTC)
tests := []struct {
given *sqliface.MockRows
want []*SavedItem
wantErr error
}{
// test to verify we get an empty slice when
// the DB has no data.
{
sqliface.NewMockRows(),
[]*SavedItem{},
nil,
},
// normal success test
{
sqliface.NewMockRows(
sqliface.MockRow{
uint64(123),
"http://nytimes.com/awesome-article",
testTime,
},
sqliface.MockRow{
uint64(456),
"http://nytimes.com/awesome-article-2",
testTime2,
},
),
[]*SavedItem{
&SavedItem{
uint64(123),
"http://nytimes.com/awesome-article",
testTime,
},
&SavedItem{
uint64(456),
"http://nytimes.com/awesome-article-2",
testTime2,
},
},
nil,
},
// testing with Scan returning unexpected error.
// using the wrong type in MockRow (a uint64 instead of a string in
// this case) to trigger the error.
{
sqliface.NewMockRows(
sqliface.MockRow{
uint64(123),
uint64(123),
testTime,
},
),
[]*SavedItem(nil),
sqliface.NewTypeError("string", uint64(123)),
},
}
for _, test := range tests {
// run the test, passing in the MockRows implementation.
got, err := scanItems(test.given)
// verify the test's results
if !reflect.DeepEqual(got, test.want) {
t.Errorf("expected \n%#v\ngot,\n%#v", test.want, got)
}
if !reflect.DeepEqual(err, test.wantErr) {
t.Errorf("expected error of \n%#v\ngot,\n%#v", test.wantErr, err)
}
}
}Like our other tests, this set of tests was using a table test format to supply three different test scenarios. All involve using the MockRows and MockRow structs to set up the data we’d expect to be returned from the database. In the error scenario, we use an invalid type for our SavedItem struct. This will trigger the Scan method to behave as though database problem has occurred to give us full test coverage on our database logic.
Hopefully this tutorial has helped you get a deeper understanding of how we use Gizmo servers at The New York Times. All the code in this example is available on GitHub. Stay tuned for our next tutorial covering the toolkits pubsub package.