added subtract operation #120
Conversation
src/rdd/rdd.rs
Outdated
| @@ -26,28 +26,49 @@ use serde_derive::{Deserialize, Serialize}; | |||
| use serde_traitobject::{Deserialize, Serialize}; | |||
|
|
|||
| mod parallel_collection_rdd; | |||
|
|
|||
| pub use parallel_collection_rdd::*; | |||
There was a problem hiding this comment.
can you remove these extra new lines?
| @@ -838,6 +859,65 @@ pub trait Rdd: RddBase + 'static { | |||
| self.intersection_with_num_partitions(other, self.number_of_splits()) | |||
| } | |||
|
|
|||
| fn subtract<T>(&self, other: Arc<T>) -> SerArc<dyn Rdd<Item = Self::Item>> | |||
There was a problem hiding this comment.
If possible, please add documentation for this method. Similar to the one in Spark will do.
There was a problem hiding this comment.
have added documentation for the function and an example in examples/subtract.rs
| .map_partitions(Box::new(Fn!(|iter: Box< | ||
| dyn Iterator<Item = Option<Self::Item>>, | ||
| >| | ||
| -> Box< |
There was a problem hiding this comment.
I maybe wrong, but it looks like rustfmt is not done. If so, please run cargo fmt.
|
Closes #91 There is an optimized version for this op (could be implemented in the future) in case you want to check it out. It entails quite more changes though, I had somethings close to done (had to debug one error) but haven't been able to finish it up yet due to lack of time. |
|
Is there any reference to the more optimized method you mentioned, I could look into |
tests/test_rdd.rs
Outdated
| let first = sc.parallelize(col1, 4); | ||
| let second = sc.parallelize(col2, 4); | ||
| let ans = first.subtract(Arc::new(second)); | ||
| assert_eq!(ans.collect().unwrap(), vec![19, 12, 10, 1, 0, 2]) |
There was a problem hiding this comment.
This assertion can fail randomly depending on shuffle(as evidenced by CI check fail). You can iterate over one vector and check if an element exists in the other one or convert both of them as hashset and do assert_eq. HashSet is the better way.
|
merging this for now as it is working version. @iduartgomez When you are done, please feel free to modify this operation. |
No description provided.