In this coursework we will consider the scenario of a network of "Smart Speed Cameras", equipped with Automatic Number Plate Recognition software. Newcastle City Council wish to scale the processing of this speed camera data into the Cloud. This coursework provides you with the opportunity to develop an entire Cloud-based system, connected using Azure’s Service Bus brokering system. As a result you will need to become familiar with a wide range of technologies. It is important that you start the work early and ask for help when needed. By completing this coursework, you will gain some useful experience and skills.
-
To make appropriate use of a variety of technologies to build a traffic management application in the Microsoft Azure cloud.
-
To gain familiarity, and reflect on, the use of Cloud technologies to tackle a real-world problem.
Building your work
For this project, if you use Java, you are advised to use a Maven build for your application. Maven will assist you in managing your 'dependencies' on the Azure SDK, and will simplify the production of a JAR file so you can run your application on the Cloud.
Storing your work
You may save your work using the Git version control system, regularly commiting changes and pushing to the provided private Github repository. If you would like a private repository created, please email Matt Forshaw.
You should push commits of your coursework regularly to your GitHub repo. It is a good idea to do so every time you get a small piece of functionality working. This ensures that you can move back to a working state if you get in a mess. It also ensures that you have a backup and that the demonstrators can view your code and track progress.
Task 1: You will first create a program in Java (or a programming language of your choice) to simulate a Smart Speed Camera.
Your Smart Speed Camera should have the following properties. These may either be passed to the command line as command line arguments, or read from a properties or configuration file, but must not be hard coded into your application.
-
Unique Identifier
-
Street Name
-
Town/City
-
Maximum speed limit for the area they monitor
Your Smart Speed Camera must provide the following functionality:
-
Every time your Smart Speed Camera starts up, it should inform (by adding a message to the Azure Service Bus Topic) the rest of the system of its Unique Identifer, Street Name, Town/City, Maximum speed limit for the area they monitor, and the date and time (timestamp) the Smart Speed Camera started up.
|
Note
|
In addition to Smart Speed Cameras located at fixed locations, cameras may be mounted within police vehicles and change location over time. Similarly, it is possible that the speed limit for an area monitored by a Smart Speed Camera may change over time (for example, due to poor weather conditions). You may assume in the case of a change in Maximum speed limit that the camera will be restarted. |
-
Your program should simulate vehicles passing the camera, and generate random (but realistic!) information. Each time a vehicle passes the camera, the following information is recorded.
-
Vehicle registration plate (e.g. 'BD51 SMR', 'LA51 ABC', 'LL51 JMB')
-
Vehicle type (Car, Truck, Motorcycle)
-
Current speed of travel
-
Details of the speed camera which made the sighting.
-
-
Your program should send these details to an Azure Service Bus Topic for later processing in Part 2.
Task 2 - Vehicle Rate: You should also provide into your application (as a command line argument, or from a configuration file) an expected rate of traffic (expressed as "number of vehicles per minute"). This should specify the rate that vehicle sightings are produced by your Smart Speed Camera; controlling the rate of messages produced.
Task 3 - Offline operation: It should be possible for your Smart Speed Camera application to operate when an internet connection is not available. You should develop your Smart Speed Camera application so it is capable of storing messages in-memory (e.g. in an array, Collection, or other data structure of your choice), to be sent when an internet connection is re-established.
|
Note
|
We expect you will run multiple instances of your Smart Speed Camera application on your local machine, to simulate a network of Smart Speed Cameras located throughout a town or city. |
You will then create a 'NoSQL Consumer' which retrieves messages from your queue and stores the details of in a NoSQL data store. In this coursework we will be using Azure’s Table Storage, a scalable datastore for the storage and querying of structured, non-relational data.
|
Note
|
A full description of the Azure Table Storage service, instructions for creating a storage account from the Azure Management Portal, and Java source code examples can be found in the practical material. |
Your NoSQL Consumer should provide the following functionality:
-
Your NoSQL Consumer should execute continuously, periodically checking the queue for incoming messages.
-
Sightings of vehicles, and registrations of Smart Speed Cameras, should be persisted to two separate tables within Azure’s Table Storage.
-
In Azure, as with many Cloud providers, you are charged based on the number of requests you make against a particular service. For example, it would be wasteful to continue polling the Service Bus Topic repeatedly, even when there are only few messages in the system. Consider an approach to handle this issue, and update your NoSQL Consumer to implement this. You will want to discuss your design decisions here, as part of your written report.
Now we have begun to persist data to our database (in Azure Table Storage), we are interested to be able to query this data to produce some interesting insights.
You should now create an application which queries your database to answer the following question:
-
Query One: Produce a list of all Smart Speed Camera registrations, including the details of their ID, Street Name, Town/City, and Speed Limit.
|
Note
|
You will likely have multiple registrations for a particular Smart Speed Camera, with an entry for every occasion where the Smart Speed Camera (your program) starts. |
As we progress in the coursework assignment, we will extend this program to perform additional queries.
The police are now interested in obtaining immediately the details of any sightings where a vehicle is travelling over the speed limit.
Task 1: Write another worker application which makes use of the Subscriptions with Filters feature of Azure Service Bus to retreive only sightings where a vehicle has exceeded the speed limit for a camera. Your program should print the details of any such sightings to the terminal.
|
Note
|
"Create subscriptions with filters" in https://azure.microsoft.com/en-gb/documentation/articles/service-bus-java-how-to-use-topics-subscriptions/#create-subscriptions |
Task 2: It is well understood that Smart Speed Cameras may provide inaccurate results by up to 10%. Update your Police Monitor to specify whether the observed speed is greater than the speed limit by 10%. Your application should indicate any sightings which exceed the speed limit by more than 10%, by adding "PRIORITY" to your terminal output for that entry.
Task 3: Update your Police Monitor to persist the details of any vehicles caught speeding (including whether they are a priority or non-priority sighting) into a new table "SpeedingVehicles" in Azure Table storage.
Task 4: Extend the Query application you began developing in Part 3, to provide an additional query to help the police track the movement of suspicious vehicles. Query Two: Produce a list of all historical sightings of vehicles caught speeding and considered a "PRIORITY" (taken from the table you create in Task 3).
|
Note
|
Unlike relational databases, NoSQL does not support JOIN statements, so you will need to execute multiple queries against your NoSQL data store and perform the JOIN between SpeedingVehicles and Sightings programmatically within your application logic. |
|
Note
|
It is not necessary to have completed this part before attempting the Part 6 Extra Credit section. |
So far in this assignment we have considered simple message consumers which are capable of processing messages (vehicle sightings) quickly.
We now wish to extend our application to perform checks on vehicles, a process which we can assume will take a number of seconds per vehicle. In this situation, a single Vehicle Check application may struggle to satisfy demand when there are a high volume of requests to satisfy.
In this section we will create a simple 'Vehicle Check' application, and use this to explore the use of Auto-Scaling in Azure to dynamically provision additional Vehicle Check applications to satisfy demand.
Task 1: Create a small program which simulates the time-consuming process of performing a vehicle check based on a vehicle registration.
This should look rather similar to your other applications which consume messages from the Azure Service Bus, developed in earlier sections. However, here you are expected to simulate a long-running process. For example, in Java, you could use Thread.sleep(…) to slow down the running of your application.
public static boolean isVehicleStolen(String vehicleRegistration)
{
Thread.sleep(5000);
return (Math.random() < 0.95)
}Test your application (running a single instance of the Vehicle Check), and demonstrate that due to the long-running nature of the task, it is unable to cope with large volumes of requests in a timely manner.
In this Extra Credit section of the assignment, we will experiment with the use of auto-scaling in Azure, to run your Vehicle Check application in the cloud and configure it to automatically respond to increased rate of requests.
At present, Azure Service Bus Topics do not support scaling based on queue length, so you will have to update your Police Monitor application to forward messages relating to speeding cars to an Azure Queue for further processing, and update your Vehicle Check application to retrieve messages from that queue.
|
Note
|
See "How to Autoscale an Application" and in particular the "Queue Message" section at https://azure.microsoft.com/en-gb/documentation/articles/cloud-services-how-to-scale/ |
|
Important
|
Based on the above guide, it is only possible to scale on queue length for an "Azure Queue" rather than Service Bus Topics we have used so far in this coursework. An intended learning outcome of this section is to adapt your solution to be able to make use of this additional type of queue to provide the scaling functionality. In future, Service Bus Topics will also allow for scaling, and will simplify this process. |
|
Note
|
Remember, you will want to configure your application to start when your Virtual Machine is created. Ubuntu upstart is a great place to start. |
Re-run your application and observe Azure auto-scaling your application in response to the number (and traffic rate) of Smart Speed Cameras active in your system.
You will demonstrate your working coursework assignment, during the final practical slot on Thursday 15th December 2016, 4-6pm. We will distribute sign-up sheets so you can book a slot during the second week of the course.
You must submit all work via the coursework submission system (NESS).
This should constitute a zip file containing the project source code and your Maven build scripts (if applicable). We will use this zip file to test your submission, so it should contain everything necessary to build and test your project.
In addition to the source code, it would be useful to our markers to see evidence (either in textual output, or screenshot form) demonstrating each area of functionality you have completed as part of the assignment. These could be included in your report document (with sufficient screen resolution so the images are legible), or included as images as part of your zip file and carefully named to suggest which area of functionality they pertain to.
You should also submit a short report via NESS (roughly three-five pages) summarising the work carried out on this project, and an evaluation of how much you achieved. We are particularly interested in any assumptions you made, and how they motivated particular design decisions. As part of your report, you are required to write a brief reflection of your experiences using Cloud technologies. Areas you may wish to discuss include:
-
how you found the process,
-
what you learned about testing,
-
what you learned about development,
-
what you disliked most,
-
how you felt during and after the process,
-
what you found most difficult/straightforward,
-
how confident your are that your production code works
Demonstrators will be available in your cluster rooms during all practical sessions. You should go and see them if you are having any difficulties. This includes understanding what you have to do.
A dedicated Slack channel is available for CSC8110 (https://newcastle-csc8110.slack.com/messages/). You may post any questions about the tutorial or coursework assignment here, and the channel will be monitored by Course Demonstrators. Before posting you should use Slack' search facilities to see if somebody has already encountered the same problem. Also frequently asked questions will be posted by demonstrators here.
|
Tip
|
If you see a question on the discussion boards you know how to answer, we strongly encourage you to assist your colleagues! |