Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable data caching cross jobs #263

Merged
merged 6 commits into from
Sep 14, 2022

Conversation

zjchenn
Copy link
Contributor

@zjchenn zjchenn commented Jul 29, 2022

Signed-off-by: Zhijun Chen [email protected]

Ⅰ. Describe what this PR does

This PR decouples cacheBackend from jobs and makes cacheBackend can be reused in different jobs.

II. Does this pull request fix one issue?

#252

III. Special notes for reviewers if any.

In progress, update doc & unit testing later.

  • code
  • unit testing
  • document

@codecov-commenter
Copy link

codecov-commenter commented Jul 29, 2022

Codecov Report

Merging #263 (7f7769e) into master (7c2c969) will increase coverage by 0.64%.
The diff coverage is 34.40%.

@@            Coverage Diff             @@
##           master     #263      +/-   ##
==========================================
+ Coverage   29.43%   30.07%   +0.64%     
==========================================
  Files          89       89              
  Lines        6193     6260      +67     
==========================================
+ Hits         1823     1883      +60     
- Misses       4108     4115       +7     
  Partials      262      262              
Flag Coverage Δ
unittests 30.07% <34.40%> (+0.64%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/job_controller/job.go 24.84% <0.00%> (-0.32%) ⬇️
pkg/job_controller/job_controller.go 37.50% <35.48%> (+26.38%) ⬆️
controllers/cache/cachebackend_controller.go 41.30% <40.38%> (+3.20%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@SimonCqk
Copy link
Collaborator

@zjchenn hi zjchenn, thanks for your contribution! you may design & implement unit testing in next stage of work.

UsedBy []string `json:"usedBy,omitempty"`

// UsedNum equals to the size of UsedBy array
UsedNum int `json:"usedNum,omitempty"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since UsedNum equals to the size of UsedBy array, why we need maintain another filed?

UsedNum int `json:"usedNum,omitempty"`

// IdleTime means how long cacheBackend is not currently in use
IdleTime string `json:"idleTime,omitempty"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-type string -> time.Duration may be better

@zjchenn zjchenn changed the title [WIP] feat: enable data caching cross jobs feat: enable data caching cross jobs Aug 21, 2022
@zjchenn
Copy link
Contributor Author

zjchenn commented Sep 1, 2022

/cc @SimonCqk

@@ -17,6 +17,8 @@ limitations under the License.
package v1alpha1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the naming between CacheBackend and Dataset, CacheBackend couples with caching system semantics but Dataset only describes a set of data with higher-level abstraction, cache can be an add-on feature over dataset.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Simon, you are right. Maybe we can merge this PR first? I will refactor CacheBackend to Dataset in another PR.

@SimonCqk
Copy link
Collaborator

/LGTM

@SimonCqk SimonCqk merged commit 03a2235 into kubedl-io:master Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants