• Introduction
• Screenshot
• tldr
• Build, Run, and Access
  • Prerequisites
  • Docker images used
  • Frameworks used
  • Managing application
    • Running application
    • Stopping application
    • Cleaning maven & gradle cache
  • Accessing the application
  • Customizing application deployment
    • Docker image location
    • Deployment Host name
• Implementation
  • Endpoints
    • (frontend)
    • query
    • author
    • book
    • user
    • user_book
    • tag
    • review
    • authorize
    • swagger
  • Databases
    • MySQL
    • Redis
      • Authentication
      • Cache
• Notes and Thoughts
  • Initial
  • Memory Usage
  • Framework Changes
  • Plans

This project has two goals; to create a book repository (like goodreads.com or librarything.com), and to experiment with various microservice frameworks, golang, java, docker, swagger, and javascript.

The frontend web application is a SPA written in Javascript (vue.js).

The backend is a set of micro-services written in Golang, backed by mysql and redis databases. There are various (micro-service) endpoints for interfacing with and querying the data, see endpoints below. Most endpoints requires authorization, which is obtained at the /authorize endpoint. See the /swagger endpoint for full information.

There is an initial user of admin with same password. The admin user can create a new user, or update the admin user via the /user endpoint or the main web application.

• build: ./books.sh build
• run: ./books.sh start
• swagger url: http://localhost:8080/swagger/
• frontend url: http://localhost:8080/
• stop: ./books.sh stop
• clean: ./books.sh clean

There is a top level script that builds, starts, and stops the application (and the corresponding microservices):

> ./books.sh build

Each microservice is built with either maven or gradle and uses a Docker container to do the complication. The containers use .m2 and .gradle directories that cache the corresponding repositories.

• bash
• Docker (17.09) (install here)
• docker-compose optional

These will be pulled automatically from dockerhub.com.

• go-kit
• swagger
• bulma css
• vue javascript

Deprecated

• graphite
• grafana
• dropwizard
• spring boot

The application is managed using docker-compose via a docker-compose.yml file. A convenience script is provided that runs docker-compose via a separate container.

> ./books.sh start

or

> docker-compose -up -d

> ./books.sh stop

or

> docker-compose down

> ./books.sh clean

The web application runs at localhost:8080/. It takes a few seconds for the MySQL database to come up.

Swagger API description is at localhost:8080/swagger/. This can be used to test each of the microservices

Docker compose utilizes an .env file where environment variables can be located. This project has the following variables.

This defaults to using the local Docker repository, not a private registry. If you want to tag and push images to a private repository (or dockerhub), you can run the docker compose with the environment variable BOOK_REPOSITORY.

If your repository were at mysuperrepo.com:5432, you add the following to the .env file:

BOOK_REPOSITORY=mysuperrepo.com:5432

And then docker compose will utilize that for the root image locations.

A utility script tagAndPushToRepo.sh is in the root directory and will tag all of the books images and push to the BOOK_REPOSITORY repository.

This application defaults to localhost for the Swagger UI definition. When set to localhost, the UI can have trouble running example REST calls. For instance, if you're running the application on your hosted box foo.com and you access the UI via browser from your laptop, the queries will not go through. If you update the .env file with your host box name things will go smoother via Swagger.

DEPLOY_HOST_NAME=foo.com

See the swagger-combine project for more information on this variable.

Notes on implementation.

The web application, each microservice endpoint, and backend database are contained in Docker containers. All containers are attached to the Docker network books_booknet.

All endpoints are exposed to the outside world via an API gateway which runs in a separate container and listens on port 8080.

The following endpoints are proxied via the API Gateway on port 8080 (configurable via the docker-compose.yml file).

Frontend web application at /. This uses the microservices to manage the books, authors, users, user lists, and tags. Utilizes the vue framework and bulma framework.

See frontend for more information.

REST microservice that queries openlibrary for new authors and book titles. Would be used by frontend to add new entries to application.

See query for more information.

Microservice to manage the complete set of Authors in the database. Operations include: add, list all, list single and delete.

See author.go for more information.

See deprecated author for more information.

Microservice to manage the complete set of Books in the database. Operations include; list, query, add, delete.

Sett book.go for more information.

See deprecated book for more information.

Microservice to manage users. A user is used to maintain a set of user books, which stores which books the user is cataloging, along with metadata, tags and a rating. In addition, a user is used to obtain an authorization token for authenticating against the various endpoints.

See user for more information.

See deprecated user for more information.

Microservice to manage a set of books for a user. Each user has a list of books they can catalog. Each user book has a link to the real Book and associated Author. In addition, a user book has user data and a set of tags.

See user_book for more information.

See deprecated user_book for more information.

Microservice to manage tags. Tags can be applied to a user's set of books via the user_books endpoint. Multiple tags may be applied to a single book, e.g. "e-book" and "sci-fi".

Note: This is now implemented in Go instead of Java.

See tag.go for more information.

See deprecated tag for more information.

Microservice to list reviews for a book.

See review.go for more information.

Microservice to authenticate a user. This creates a token of the form 'Bearer qwerty-1234-asdf-9876', which is then added to the headers of any calls to the other endpoints.

See authorize for more information.

See deprecated authorize for more information.

Swagger-ui that combines the swagger.yaml files from the REST endpoints. Uses swagger-combine image to grab the definitions.

This waits for the various endpoints to come up and then grabs the designated (in docker-compose.yml) yaml files, combines them and then serves up the endpoint via swagger-ui.

There are two databases used to manage the books and users. The data is stored in the database directory.

MySQL is used to store books, authors, users, tags and user books lists.

The database schema is stored in database/initial/mybooks.sql

See mysql for more information and method to update the admin user's password.

Redis is used to store key/value pairs for the services.

Authentication tokens are created by the authorize service and stored in Redis. Redis is then used by all services to verify authentication.

Redis is also used as a cache for the services. When a service makes a REST call to another service, the calling service stores the returned data in the cache. For example, the book service calls the author service to get the author's name for a book. The book service stores that in the cache for the next call. The author service will flush the cache when mutations to the author database have been made.

Map of services performing caching and their consumers

I created this project to experiment with writing multiple micro-services and wiring them together in a docker environment. I started using dropwizard for a Java based server. The ramp up time was fairly quick and I was able to easily do most tasks with little pain. I played with spring boot for the query endpoint, but stuck with dropwizard for the rest of the services.

With 6 services implemented, I ran everything on my hosted environment and saw that I was running out of memory. I checked my docker stats and my micro services were each using 250MB.

I lowered the Java memory usage (Xmx64m) for each service and the usage dropped to ~180MB. This was still an inordinate amount of RAM for a simple web service.

I came across a blog post detailing where the memory for Java (on Docker) was going. Unsurprisingly those results correlated with this projects usage.

Given the large memory usage, I started looking for other micro-service frameworks. I ended up using go for a test service and saw the memory usage was in the single MB range. I also looked at node but that seemed to use almost as much memory as Java and others.

I ported that tag endpoint to golang using the go-kit framework and saw the following results:

The golang service's image size is 5% of the corresponding Java image.

More note worthy is golang's memory usage, being 1% the size of the Java implementation.

Caveat: Individual image size is not always an accurate overall measurement due to image layer sharing. All of the java images share the same base image, so the accumulated size is closer to 140MB than 900MB (7 * 130MB).

• Add graphite metrics to golang services.
• Investigate istio.io for service discovery and metrics.