Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added more query params #98

Merged
merged 10 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 76 additions & 4 deletions maap/maap.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from maap.AWS import AWS
from maap.dps.DpsHelper import DpsHelper
from maap.utils import endpoints
from maap.utils import job
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jobs (plural), but perhaps during the "cleanup" you mentioned


logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -256,13 +257,84 @@ def cancelJob(self, jobid):
job.id = jobid
return job.cancel_job()

def listJobs(self, username=None, page_size=None, offset=None):
if username==None and self.profile is not None and 'username' in self.profile.account_info().keys():
def listJobs(self, username=None, *,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this * for ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The * means that any parameters that appear after it in the parameter list must be supplied using keywords. In this case, username is the only parameter that can be supplied without a keyword, but everything else must be supplied with keywords. This is very helpful when there are many possible options, but you don't want to have to pass every argument in the correct order, supplying None value for args you don't care about.

algo_id=None,
end_time=None,
get_job_details=True,
offset=0,
page_size=10,
queue=None,
start_time=None,
status=None,
tag=None,
version=None):
"""
Returns a list of jobs for a given user that matches query params provided.

Args:
username (str, optional): Platform user. If no username is provided, the profile username will be used.
algo_id (str, optional): Algorithm type.
end_time (str, optional): Specifying this parameter will return all jobs that have completed from the provided end time to now. e.g. 2024-01-01 or 2024-01-01T00:00:00.000000Z.
get_job_details (bool, optional): Flag that determines whether to return a detailed job list or a compact list containing just the job ids and their associated job tags. Default is True.
offset (int, optional): Offset for pagination. Default is 0.
page_size (int, optional): Page size for pagination. Default is 10.
queue (str, optional): Job processing resource.
start_time (str, optional): Specifying this parameter will return all jobs that have started from the provided start time to now. e.g. 2024-01-01 or 2024-01-01T00:00:00.000000Z.
status (str, optional): Job status, e.g. job-completed, job-failed, job-started, job-queued.
tag (str, optional): User job tag/identifier.
version (str, optional): Algorithm version, e.g. GitHub branch or tag.

Returns:
list: List of jobs for a given user that matches query params provided.

Raises:
ValueError: If username is not provided and cannot be obtained from the user's profile.
ValueError: If either algo_id or version is provided, but not both.
"""
if username is None and self.profile is not None and 'username' in self.profile.account_info().keys():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the code is not able to determine a username, should it continue or throw an error ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more error handling to handle this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we have a function to determine username ?

Suggested change
if username is None and self.profile is not None and 'username' in self.profile.account_info().keys():
username = username if username else self.get_username()
if username is None:
raise SomeException

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably clean this up a bit. I'm opting to looking at this when we run through a cleanup of maap-py.

username = self.profile.account_info()['username']

url = os.path.join(self.config.dps_job, username, endpoints.DPS_JOB_LIST)
params = {k: v for k, v in (("page_size", page_size), ("offset", offset)) if v}
if username is None:
raise ValueError("Unable to determine username from profile. Please provide a username.")

url = "/".join(
segment.strip("/")
for segment in (self.config.dps_job, username, endpoints.DPS_JOB_LIST)
)

params = {
k: v
for k, v in (
("algo_id", algo_id),
("end_time", end_time),
("get_job_details", get_job_details),
("offset", offset),
("page_size", page_size),
("queue", queue),
("start_time", start_time),
("status", status),
("tag", tag),
("username", username),
("version", version),
)
if v is not None
}

if (not algo_id) != (not version):
# Either algo_id or version was supplied as a non-empty string, but not both.
# Either both must be non-empty strings or both must be None.
raise ValueError("Either supply non-empty strings for both algo_id and version, or supply neither.")

# DPS requests use 'job_type', which is a concatenation of 'algo_id' and 'version'
if algo_id and version:
params['job_type'] = f"{algo_id}:{version}"

algo_id = params.pop('algo_id', None)
version = params.pop('version', None)

if status is not None:
params['status'] = job.validate_job_status(status)

headers = self._get_api_header()
logger.debug('GET request sent to {}'.format(url))
logger.debug('headers:')
Expand Down
21 changes: 21 additions & 0 deletions maap/utils/job.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Valid job statuses (loosely based on OGC job status types)
JOB_STATUSES = {'Accepted', 'Running', 'Succeeded', 'Failed', 'Dismissed', 'Deduped', 'Offline'}

def validate_job_status(status):
'''
Validates job status

Args:
status (str): Job status. Accepted values are: 'Accepted', 'Running', 'Succeeded', 'Failed, 'Dismissed', 'Deduped', and 'Offline'.

Returns:
status (str): Returns unmodified job status if job status is valid.

Raises:
ValueError: If invalid job status is provided.
'''
if status not in JOB_STATUSES:
valid_statuses = ", ".join(str(status) for status in JOB_STATUSES)
raise ValueError("Invalid job status: '{}'. Job status must be one of the following: {}".format(status, valid_statuses))

return status