Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added more query params #98

Merged
merged 10 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 23 additions & 18 deletions maap/maap.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from maap.AWS import AWS
from maap.dps.DpsHelper import DpsHelper
from maap.utils import endpoints
from maap.utils import JobUtils

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -256,13 +257,12 @@ def cancelJob(self, jobid):
job.id = jobid
return job.cancel_job()

def listJobs(self, username=None,
def listJobs(self, username=None, *,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this * for ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The * means that any parameters that appear after it in the parameter list must be supplied using keywords. In this case, username is the only parameter that can be supplied without a keyword, but everything else must be supplied with keywords. This is very helpful when there are many possible options, but you don't want to have to pass every argument in the correct order, supplying None value for args you don't care about.

algo_id=None,
end_time=None,
get_job_details=True,
offset=None,
page_size=None,
priority=None,
offset=0,
page_size=10,
queue=None,
start_time=None,
status=None,
Expand All @@ -272,25 +272,27 @@ def listJobs(self, username=None,
Returns a list of jobs for a given user that matches query params provided.

Args:
username (str): Platform user.
algo_id (str): Algorithm type.
username (str, optional): Platform user. If no username is provided, the profile username will be used.
algo_id (str, optional): Algorithm type.
end_time (str, optional): Specifying this parameter will return all jobs that have completed from the provided end time to now. e.g. 2024-01-01 or 2024-01-01T00:00:00.000000Z.
get_job_details (bool, optional): Flag that determines whether to return a detailed job list or a compact list containing just the job ids and their associated job tags. Default is True.
offset (int, optional): Offset for pagination.
page_size (int, optional): Page size for pagination.
priority (int, optional): Job processing priority. Valid values are integers from 0-9.
offset (int, optional): Offset for pagination. Default is 0.
page_size (int, optional): Page size for pagination. Default is 10.
queue (str, optional): Job processing resource.
start_time (str, optional): Specifying this parameter will return all jobs that have started from the provided start time to now. e.g. 2024-01-01 or 2024-01-01T00:00:00.000000Z.
status (str, optional): Job status e.g. job-completed, job-failed, job-started, job-queued.
tag (str, optional): User job tag.
version (str, optional): Algorithm version i.e. GitHub branch.
status (str, optional): Job status, e.g. job-completed, job-failed, job-started, job-queued.
tag (str, optional): User job tag/identifier.
version (str, optional): Algorithm version, e.g. GitHub branch or tag.

Returns:
list: List of jobs for a given user that matches query params provided.
"""
if username is None and self.profile is not None and 'username' in self.profile.account_info().keys():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the code is not able to determine a username, should it continue or throw an error ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more error handling to handle this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we have a function to determine username ?

Suggested change
if username is None and self.profile is not None and 'username' in self.profile.account_info().keys():
username = username if username else self.get_username()
if username is None:
raise SomeException

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably clean this up a bit. I'm opting to looking at this when we run through a cleanup of maap-py.

username = self.profile.account_info()['username']

if username is None:
raise ValueError("Username must be supplied.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the username field is optional, I think this error should be more descriptive.
Something like, Unable to determine username from profile, please provide a username

Thoughts:
In the future we would need to add a method to explicitly do maap.login() by providing the PGT token if not found in the env vars, and then update this error message to be Unable to determine username from profile, please ensured you are logged in by calling maap.login() cc @bsatoriu


url = "/".join(
segment.strip("/")
for segment in (self.config.dps_job, username, endpoints.DPS_JOB_LIST)
Expand All @@ -304,7 +306,6 @@ def listJobs(self, username=None,
("get_job_details", get_job_details),
("offset", offset),
("page_size", page_size),
("priority", priority),
("queue", queue),
("start_time", start_time),
("status", status),
Expand All @@ -315,16 +316,20 @@ def listJobs(self, username=None,
if v is not None
}

# DPS requests use 'job_type', which is a concatenation of 'algo_id' and 'version'
algo_id = params.pop('algo_id', None)
version = params.pop('version', None)

if (not algo_id) != (not version):
# Either algo_id or version was supplied as a non-empty string, but not both.
# Either both must be non-empty strings or both must be None.
raise ValueError("Either supply non-empty strings for both algo_id and version, or supply neither.")

params['job_type'] = f"{algo_id}:{version}"
# DPS requests use 'job_type', which is a concatenation of 'algo_id' and 'version'
if algo_id and version:
params['job_type'] = f"{algo_id}:{version}"

algo_id = params.pop('algo_id', None)
version = params.pop('version', None)

if status is not None:
params['status'] = JobUtils.validate_job_status(status)

headers = self._get_api_header()
logger.debug('GET request sent to {}'.format(url))
Expand Down
18 changes: 18 additions & 0 deletions maap/utils/JobUtils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Valid job statuses (loosely based on OGC job status types)
marjo-luc marked this conversation as resolved.
Show resolved Hide resolved
JOB_STATUSES = ['Accepted', 'Running', 'Succeeded', 'Failed', 'Dismissed', 'Deduped', 'Offline']
marjo-luc marked this conversation as resolved.
Show resolved Hide resolved

def validate_job_status(status):
'''
Validates job status

Args:
status (str): Job status. Accepted values are: 'Accepted', 'Running', 'Succeeded', 'Failed, 'Dismissed', 'Deduped', and 'Offline'.

Returns:
status (str): Returns unmodified job status if job status is valid.
marjo-luc marked this conversation as resolved.
Show resolved Hide resolved
'''
if status not in JOB_STATUSES:
valid_statuses = ", ".join(str(status) for status in JOB_STATUSES)
raise ValueError("Invalid job status: '{}'. Job status must be one of the following: {}".format(status, valid_statuses))

return status