Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assignment 2.1: Build Automation Tool #60

Closed
wants to merge 51 commits into from
Closed
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
ca5e40c
Add Config Parser
ankurdubey521 Sep 9, 2019
a338501
Add Command Line Runner
ankurdubey521 Sep 9, 2019
42557d3
Implement Builder
ankurdubey521 Sep 9, 2019
1f28c4f
Add tests for Builder
ankurdubey521 Sep 9, 2019
f33e3f7
Fix tests and PEP8 violations
ankurdubey521 Sep 9, 2019
6270d50
Fix test and add docstrings
ankurdubey521 Sep 9, 2019
f0be0a3
General Syntax and Naming Fixes
ankurdubey521 Sep 10, 2019
1e5bd0d
Implement Command Class and add basic validation
ankurdubey521 Sep 10, 2019
cd94bc8
Fix Main.py
ankurdubey521 Sep 10, 2019
9bcb8cb
Add support for Relative Paths
ankurdubey521 Sep 10, 2019
c633b29
Add support for Root Referenced Commands
ankurdubey521 Sep 11, 2019
ac947b8
Implement Check for Circular Dependencies
ankurdubey521 Sep 11, 2019
d71d9c4
Move main function to it's own class
ankurdubey521 Sep 11, 2019
277b7d6
Pre-Generate all Command Objects while initializing BuildConfig Object
ankurdubey521 Oct 3, 2019
f1c42c3
Remove _hasFiles and _hasDependencies from BuildConfig
ankurdubey521 Oct 3, 2019
5f26c51
Print Logs when doing a dry run
ankurdubey521 Oct 3, 2019
22580e3
Move Builder Class to it's own file and Save root_dir_abs
ankurdubey521 Oct 3, 2019
a1b25bd
Move command runner to Builder Class
ankurdubey521 Oct 3, 2019
c0da0bc
Use unittest setup method instead of @classMethod
ankurdubey521 Oct 3, 2019
166c5db
Catch any exceptions that might be thrown in main.py by Builder
ankurdubey521 Oct 3, 2019
923ead1
Fixes in Builder Tests
ankurdubey521 Oct 3, 2019
5da1c4c
Add type hints
ankurdubey521 Oct 3, 2019
980e9e1
Switch to Argparse
ankurdubey521 Oct 3, 2019
c421aa6
Implement class for listening changes to files
ankurdubey521 Oct 3, 2019
eabddd1
Remove redundant for loop from file_watcher
ankurdubey521 Oct 3, 2019
7df0d4a
Integrate File watching in Build System
ankurdubey521 Oct 3, 2019
fc505ed
Make Root Directory Configurable
ankurdubey521 Oct 3, 2019
9c83a4b
Code Quality Fixes
ankurdubey521 Oct 5, 2019
4ba29bc
Add Topological Sort Code
ankurdubey521 Oct 7, 2019
0cf3e73
Implement execution of dependency rules in parallel
ankurdubey521 Oct 7, 2019
54a3e62
Cleanup
ankurdubey521 Oct 7, 2019
31226cc
Include file of dependencies when listening for changes. Also provide…
ankurdubey521 Oct 8, 2019
28a1c63
Document ParallelBuilder
ankurdubey521 Oct 8, 2019
60c414e
Document TopologicalSort
ankurdubey521 Oct 8, 2019
4e1cadf
Document buildconfig
ankurdubey521 Oct 8, 2019
86c116c
Add global constants
ankurdubey521 Oct 8, 2019
cfae026
Remove Redundant Exceptions
ankurdubey521 Oct 8, 2019
b77e20b
Run tests from Tempdir instead of running them from source
ankurdubey521 Oct 8, 2019
3658eb8
Remove Redundant path correction function from TestFileWatcher
ankurdubey521 Oct 8, 2019
ce7515e
Refactoring
ankurdubey521 Oct 9, 2019
2ec874f
Implement Logging
ankurdubey521 Oct 9, 2019
64f9272
Stop spawning more threads if build fails for dependency
ankurdubey521 Oct 9, 2019
18f8219
Move Test file contents to Testing Scripts
ankurdubey521 Oct 9, 2019
13b446d
Use os.path.join instead of manually joining paths
ankurdubey521 Oct 9, 2019
baf6e64
Optimize imports and use os.path.join in tests
ankurdubey521 Oct 10, 2019
f574672
Fix Logging
ankurdubey521 Oct 30, 2019
4f9bdfe
Refactoring
ankurdubey521 Oct 30, 2019
d9f55eb
Implement JSON Caching
ankurdubey521 Oct 30, 2019
1176bfa
Fix Tests
ankurdubey521 Oct 30, 2019
1f9d18b
Remove redundant paramaters from _execute_build_rule
ankurdubey521 Oct 30, 2019
f516ae0
Implement parallel execution using Popen polling
ankurdubey521 Oct 30, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions solutions/ankurdubey521/Build Automation Tool/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/venv/
.idea
.gitignore
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
"""
Global Constants
"""


class GlobalConstants:
CONFIG_FILE_NAME = "build.json"
FILE_WATCH_INTERVAL_SECONDS = 1

Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
from typing import List, Dict, TypeVar
from queue import Queue

T = TypeVar('T')


class TopologicalSort:
"""Topological Sort
Provides methods for generating Topological Sort of a DAG.
"""
class GraphContainsCycleException(Exception):
pass

@staticmethod
def sort(graph_adj_list: Dict[T, List[T]]) -> List[T]:
"""
Generates topological sort and raises exception if cycle in found
:param graph_adj_list: .
:return: topologically sort of graph
"""
toposort = []
visited = {}
queue = Queue()
in_degree = {}

# Calculate In-Degree of Each Vertex
for source_node in graph_adj_list:
visited[source_node] = False
for dest_node in graph_adj_list[source_node]:
visited[dest_node] = False
if dest_node in in_degree:
in_degree[dest_node] = in_degree[dest_node] + 1
else:
in_degree[dest_node] = 1

# Find all vertices that have 0 In-degree.
for source_node in graph_adj_list:
if source_node not in in_degree:
queue.put(source_node)

# BFS
while not queue.empty():
front = queue.get()
visited[front] = True
toposort.append(front)
if front in graph_adj_list:
for dest_node in graph_adj_list[front]:
if not visited[dest_node]:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about this line for a while ... trying to figure out when it will be false. Can you help me out?

in_degree[dest_node] = in_degree[dest_node] - 1
if in_degree[dest_node] == 0:
queue.put(dest_node)

# If any vertex exists whose In-Degree has not been exhausted, then graph must contain a cycle
for node in in_degree:
if in_degree[node] != 0:
raise TopologicalSort.GraphContainsCycleException()

return toposort

Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
"""" Sample build.json
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an fairly detailed and specific example. Better suited for a readme file?

[
{
"name": "clean",
"deps": ["algorithms/clean"],
"files": ["test.cpp"],
"command": "rm -f test.o && rm -f test.out"
},
{
"name": "test",
"files": ["test.cpp"],
"command": "g++ -std=c++11 -c test.cpp"
},
{
"name": "run",
"deps": ["test", "algorithms/sort_bubble", "algorithms/sort_merge", "algorithms/sort_quick"],
"command": "g++ algorithms/sort_bubble.o algorithms/sort_merge.o ..."
}
]
"""

import json
from typing import List
from Builder.global_constants import GlobalConstants


class BuildRule:
"""Stores Command Information"""
def __init__(self, *, name: str, command_string: str, deps: List[str] = [], files: List[str] = []) -> None:
self._name = name
self._command_string = command_string
self._files = files
self._dependencies = deps

def get_name(self) -> str:
return self._name

def get_files(self) -> List[str]:
return self._files

def get_command_string(self) -> str:
return self._command_string

def get_dependencies(self) -> List[str]:
return self._dependencies

class NoDependenciesException(Exception):
pass

class NoFilesException(Exception):
pass


class BuildConfig:
"""Parses and Stores build.config in the form of BuildRule objects"""
def __init__(self, json_containing_folder: str) -> None:
# Parse JSON
json_path = json_containing_folder + "/" + GlobalConstants.CONFIG_FILE_NAME
with open(json_path) as file_handle:
self._raw_json = json.load(file_handle)

self._name_to_command_object = {}
for command_entry in self._raw_json:
name = command_entry['name']
command_string = command_entry['command']
if 'deps' in command_entry:
deps = command_entry['deps']
else:
deps = []
if 'files' in command_entry:
files = command_entry['files']
else:
files = []
self._name_to_command_object[name] =\
BuildRule(name=name, command_string=command_string, deps=deps, files=files)

def get_command(self, command_name: str) -> BuildRule:
if command_name in self._name_to_command_object:
return self._name_to_command_object[command_name]
else:
raise BuildConfig.UnknownCommandException('No such command "{}" found.'.format(command_name))

class UnknownCommandException(Exception):
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from os import stat
from typing import List, Callable, Dict
from Builder.global_constants import GlobalConstants
from time import sleep


class FileWatcher:
FILE_WATCH_INTERVAL_SECONDS = GlobalConstants.FILE_WATCH_INTERVAL_SECONDS

@staticmethod
def _get_file_edit_times(file_list: List[str]) -> Dict[str, int]:
"""Populate a Dict of last edit times of files and return it"""
file_edit_times = {}
for file_path in file_list:
file_edit_times[file_path] = stat(file_path).st_mtime
return file_edit_times

@staticmethod
def watch_and_execute(file_list: List[str], function: Callable[[any], any], *args: any) -> None:
"""Execute a function whenever a file from file_list changes"""
print("Listening for changes on {}...".format(file_list))
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of printing to stdout directly, you can send these messages to a logger object, provided in the constructor, and shared across your codebase. That way, if I import your code as a library, I can just point the logger object to write to a file or something.

file_edit_times = FileWatcher._get_file_edit_times(file_list)
print("Executing for the first time...")
function(*args)
while True:
sleep(FileWatcher.FILE_WATCH_INTERVAL_SECONDS)
new_file_edit_times = FileWatcher._get_file_edit_times(file_list)
if new_file_edit_times != file_edit_times:
file_edit_times = new_file_edit_times
print("Detected change of file. Executing...")
function(*args)


Loading