asg/parallel-graph: Update assignment

content/assignments/parallel-graph/README.md

@@ -1,109 +1,183 @@
 # Parallel Graph
 
-For this assignment we will implement a generic thread pool, which we will then use to traverse a graph and compute the sum of the elements contained by the nodes.
+## Objectives
+
+- Learn how to design and implement parallel programs
+- Gain skills in using synchronization primitives for parallel programs
+- Get a better understanding of the POSIX threading and synchronization API
+- Gain insight on the differences between serial and parallel programs
+
+## Statement
+
+Implement a generic thread pool, then use it to traverse a graph and compute the sum of the elements contained by the nodes.
 You will be provided with a serial implementation of the graph traversal and with most of the data structures needed to implement the thread pool.
 Your job is to write the thread pool routines and then use the thread pool to traverse the graph.
 
-## Thread Pool Description
+## Support Code
+
+The support code consists of the directories:
+
+- `src/` is the skeleton parallel graph implementation.
+  You will have to implement missing parts marked as `TODO` items.
+
+- `utils/` utility files (used for debugging & logging)
+
+- `tests/` are tests used to validate (and grade) the assignment.
+
+## Implementation
+
+### Thread Pool Description
 
 A thread pool contains a given number of active threads that simply wait to be given specific tasks.
-The threads are created when the thread pool is created they poll a task queue until a task is available.
-Once tasks are put in the task queue, the threads start running the task.
-A thread pool creates N threads when the thread pool is created and does not destroy (join) them throughout the lifetime of the thread pool.
-That way, the penalty of creating and destroying threads ad hoc is avoided.
-As such, you must implement the following functions (marked with `TODO` in the provided skeleton):
-
-- `task_create`: creates an `os_task_t` that will be put in the task queue - a task consists of a function pointer and an argument.
-- `add_task_in_queue`: adds a given task in the thread pool's task queue.
-- `get_task`: get a task from the thread pool's task queue.
-- `threadpool_create`: allocate and initialize a new thread pool.
-- `thread_loop_function`: all the threads in the thread pool will execute this function - they all wait until a task is available in the task queue; once they grab a task they simply invoke the function that was provided to `task_create`.
-- `threadpool_stop`: stop all the threads from execution.
-
-Notice that the thread pool is completely independent from any given application.
+The threads are created when the thread pool is created.
+Each thread continuously polls the task queue for available tasks.
+Once tasks are put in the task queue, the threads poll tasks, and start running them.
+A thread pool creates **N** threads upon its creation and does not destroy (join) them throughout its lifetime.
+That way, the penalty of creating and destroying threads ad-hoc is avoided.
+As such, you must implement the following functions (marked with `TODO` in the provided skeleton, in `src/os_threadpool.c`):
+
+- `enqueue_task()`: Enqueue task to the shared task queue.
+  Use synchronization.
+- `dequeue_task()`: Dequeue task from the shared task queue.
+  Use synchronization.
+- `wait_for_completion()`: Wait for all worker threads.
+  Use synchronization.
+- `create_threadpool()`: Create a new thread pool.
+- `destroy_threadpool()`: Destroy a thread pool.
+  Assume all threads have been joined.
+
+You must also update the `os_threadpool_t` structure in `src/os_threadpool.h` with the required bits for synchronizing the parallel implementation.
+
+Notice that the thread pool is completely independent of any given application.
 Any function can be registered in the task queue.
 
-## Graph Traversal
+Since the threads are polling the task queue indefinitely, you need to define a condition for them to stop once the graph has been traversed completely.
+That is, the condition used by the `wait_for_completion()` function.
+The recommended way is to note when no threads have any more work to do.
+Since no thread is doing any work, no other task will be created.
 
-Once you have implemented the thread pool, you need to test it by using it for computing the sum of all the nodes of a graph.
-A serial implementation for this algorithm is provided in `skep/serial.c`
+### Graph Traversal
+
+Once you have implemented the thread pool, you need to test it by doing a parallel traversal of all connected nodes in a graph.
+A serial implementation for this algorithm is provided in `src/serial.c`.
 To make use of the thread pool, you will need to create tasks that will be put in the task queue.
 A task consists of 2 steps:
 
-- adding the current node value to the overall sum.
-- creating tasks and adding them to the task queue for the neighbouring nodes.
+1. Add the current node value to the overall sum.
+1. Create tasks and add them to the task queue for the neighbouring nodes.
 
-Since the threads are polling the task queue indefinitely, you need to find a condition for the threads to stop once the graph has been traversed completely.
-This condition should be implemented in a function that is passed to `threadpool_stop`.
-`threadpool_stop` then needs to wait for the condition to be satisfied and then joins all the threads.
+Implement this in the `src/parallel.c` (see the `TODO` items).
+You must implement the parallel and synchronized version of the `process_node()` function, also used in the serial implementation.
 
-## Synchronization
+### Synchronization
 
 For synchronization you can use mutexes, semaphores, spinlocks, condition variables - anything that grinds your gear.
-However, you are not allowed to use hacks such as `sleep`, `printf` synchronization or adding superfluous computation.
+However, you are not allowed to use hacks such as `sleep()`, `printf()` synchronization or adding superfluous computation.
 
-## Input Files
+### Input Files
 
 Reading the graphs from the input files is being taken care of the functions implemented in `src/os_graph.c`.
 A graph is represented in input files as follows:
 
-- first line contains 2 integers N and M: N - number of nodes, M - numbed or edges
-- second line contains N integer numbers - the values of the nodes
-- the next M lines contain each 2 integers that represent the source and the destination of an edge
+- First line contains 2 integers `N` and `M`: `N` - number of nodes, `M` - numbed or edges
+- Second line contains `N` integer numbers - the values of the nodes.
+- The next `M` lines contain each 2 integers that represent the source and the destination of an edge.
 
-## Data Structures
+### Data Structures
 
-### Graph
+#### Graph
 
-A graph is represented internally as an `os_graph_t` (see `src/os_graph.h`).
+A graph is represented internally by the `os_graph_t` structure (see `src/os_graph.h`).
 
-### List
+#### List
 
-A list is represented internally as an `os_queue_t` (see `src/os_list.h`).
+A list is represented internally by the `os_queue_t` structure (see `src/os_list.h`).
 You will use this list to implement the task queue.
 
-### Thread pool
+#### Thread Pool
 
-A thread pool is represented internally as an `os_threadpool_t` (see `src/os_threadpool.h`)
+A thread pool is represented internally by the `os_threadpool_t` structure (see `src/os_threadpool.h`).
 The thread pool contains information about the task queue and the threads.
 
-You are not allowed to modify these data structures.
-However, you can create other data structures that leverage these ones.
+### Requirements
+
+Your implementation needs to be contained in the `src/os_threadpool.c`, `src/os_threadpool.h` and `src/parallel.c` files.
+Any other files that you are using will not be taken into account.
+Any modifications that you are doing to the other files in the `src/` directory will not be taken into account.
 
-## Infrastructure
+## Operations
 
-### Compilation
+### Building
 
-To compile both the serial and the parallel version, enter the `src/` directory and run:
+To build both the serial and the parallel versions, run `make` in the `src/` directory:
 
 ```console
-make
+student@so:~/.../content/assignments/parallel-graph$ cd src/
+
+student@so:~/.../assignments/parallel-graph/src$ make
 ```
 
-That will create the `serial` and `parallel` binaries/
+That will create the `serial` and `parallel` binaries.
 
-### Testing
+## Testing and Grading
 
-Input tests cases are located in `tests/in/`.
-The parallel and the serial version should provide the same results for the same input test case.
+Testing is automated.
+Tests are located in the `tests/` directory.
+
+```console
+student@so:~/.../assignments/parallel-graph/tests$ ls -F
+Makefile  checker.py  grade.sh@  in/
+```
 
-If you want manually run a single test, use commands such as below while in the `src/` directory:
+To test and grade your assignment solution, enter the `tests/` directory and run `grade.sh`.
+Note that this requires linters being available.
+The easiest is to use a Docker-based setup with everything installed and configured.
+When using `grade.sh` you will get grades for checking correctness (maximum `90` points) and for coding style (maxim `10` points).
+A successful run will provide you an output ending with:
 
 ```console
-$./parallel ../tests/in/test5.in
--11
+### GRADE
 
-$ ./serial ../tests/in/test5.in
--11
+
+Checker:                                                         90/ 90
+Style:                                                           10/ 10
+Total:                                                          100/100
+
+
+### STYLE SUMMARY
+
+
+```
+
+### Running the Checker
+
+To run only the checker, use the `make check` command in the `tests/` directory:
+
+```console
+student@so:~/.../assignments/parallel-graph/tests$ make check
+[...]
+SRC_PATH=../src python checker.py
+make[1]: Entering directory '...'
+rm -f *~
+[...]
+TODO
+test1.in                         ....................... failed ...   0.0
+test2.in                         ....................... failed ...   0.0
+test3.in                         ....................... failed ...   0.0
+[...]
+
+Total:                                                             0/100
 ```
 
-### Checker
+Obviously, all tests will fail, as there is no implementation.
+
+Each test is worth a number of points.
+The maximum grade is `90`.
 
-The testing is automated and performed with the `checker.py` script from the `tests/` directory.
-It's easiest to use the `Makefile` to run the tests:
+A successful run will show the output:
 
 ```console
-$ make check
+student@so:~/.../assignments/parallel-graph/tests$ make check
 [...]
 SRC_PATH=../src python checker.py
 test1.in                         ....................... passed ...   4.5
@@ -130,27 +204,35 @@ test20.in                        ....................... passed ...   4.5
 Total:                                                              90/100
 ```
 
-It's recommended that you use the [local Docker-based checker](./README.checker.md#local-checker).
-You would use the command:
+### Running the Linters
+
+To run the linters, use the `make lint` command in the `tests/` directory:
 
 ```console
-./local.sh checker
+student@so:~/.../assignments/parallel-graph/tests$ make lint
+[...]
+cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
+[...]
+cd .. && cpplint --recursive src/ tests/ checker/
+[...]
+cd .. && shellcheck checker/*.sh tests/*.sh
 ```
 
-to run the checker in a Docker-based environment that is identical to the one used for the official assignment evaluation.
+Note that the linters have to be installed on your system: [`checkpatch.pl`](https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl), [`cpplint`](https://github.com/cpplint/cpplint), [`shellcheck`](https://www.shellcheck.net/).
+They also need to have certain configuration options.
+It's easiest to run them in a Docker-based setup with everything configured.
 
-## Grading
+### Fine-Grained Testing
 
-The grade that the checker outputs is not the final grade.
-Your homework will be manually inspected and may suffer from penalties ranging from 1 to 100 points depending on the severity of the hack, including, but not limited to:
+Input tests cases are located in `tests/in/`.
+If you want to run a single test, use commands such as below while in the `src/` directory:
 
-- using a single mutex at the beginning of the traversal
-- not using the thread pool to solve the homework
-- inefficient usage of synchronization
-- incorrect graph traversal
+```console
+$./parallel ../tests/in/test5.in
+-38
 
-## Deployment
+$ ./serial ../tests/in/test5.in
+-38
+```
 
-Your implementation needs to be contained in the `src/os_threadpool.c` and `src/os_parallel.c` files.
-Any other files that you are using will not be taken into account.
-Any modifications that you are doing to the other files in the `src/` directory will not be taken into account.
+Results provided by the serial and parallel implementation must be the same for the test to successfully pass.

content/assignments/parallel-graph/src/Makefile

@@ -1,36 +1,34 @@
 BUILD_DIR := build
-CC := gcc
-CFLAGS := -c -Wall -g
-LD := ld
-LDFLAGS :=
-LDLIBS := -lpthread
-
-SERIAL_SRCS := serial.c os_graph.c
-PARALLEL_SRCS:= parallel.c os_graph.c os_list.c os_threadpool.c
-SERIAL_OBJS := $(patsubst $(SRC)/%.c,$(BUILD_DIR)/%.o,$(SERIAL_SRCS))
-PARALLEL_OBJS := $(patsubst $(SRC)/%.c,$(BUILD_DIR)/%.o,$(PARALLEL_SRCS))
+UTILS_PATH ?= ../utils
+CPPFLAGS := -I$(UTILS_PATH)
+CFLAGS := -Wall -Wextra
+# Remove the line below to disable debugging support.
+CFLAGS += -g -O0
+PARALLEL_LDLIBS := -lpthread
+
+SERIAL_SRCS := serial.c os_graph.c $(UTILS_PATH)/log/log.c
+PARALLEL_SRCS:= parallel.c os_graph.c os_threadpool.c $(UTILS_PATH)/log/log.c
+SERIAL_OBJS := $(patsubst %.c,%.o,$(SERIAL_SRCS))
+PARALLEL_OBJS := $(patsubst %.c,%.o,$(PARALLEL_SRCS))
 
 .PHONY: all pack clean always
 
 all: serial parallel
 
-always:
-	mkdir -p build
-
-serial: always $(SERIAL_OBJS)
-	$(CC) $(LDFLAGS) -o serial $(SERIAL_OBJS)
+serial: $(SERIAL_OBJS)
+	$(CC) -o $@ $^
 
-parallel: always $(PARALLEL_OBJS)
-	$(CC) $(LDFLAGS) -o parallel $(PARALLEL_OBJS) $(LDLIBS)
+parallel: $(PARALLEL_OBJS)
+	$(CC) -o $@ $^ $(PARALLEL_LDLIBS)
 
-$(BUILD_DIR)/%.o: %.c
-	$(CC) $(CFLAGS) -o $@ $<
+$(UTILS_PATH)/log/log.o: $(UTILS_PATH)/log/log.c $(UTILS_PATH)/log/log.h
+	$(CC) $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
 
 pack: clean
 	-rm -f ../src.zip
 	zip -r ../src.zip *
 
 clean:
-	-rm -f ../src.zip
-	-rm -rf build
+	-rm -f $(SERIAL_OBJS) $(PARALLEL_OBJS)
 	-rm -f serial parallel
+	-rm -f *~