This challenge is used by Didomi for evaluating candidates for data engineering positions.
This challenge is a chance for engineers at Didomi to see how you code and organize a project to implement a specification.
The expected deliverable is a fully functional project that includes the following:
- Code of the app
- Tests of the app
- Documentation for launching a development environment and running the app
The application should use the following stack:
- Spark
- Python or Scala
Except for these requirements, feel free to use whichever libraries, frameworks or tools you deem necessary.
Your code will be reviewed by multiple engineers at Didomi and will serve as the base for a discussion in interviews.
We want to see how you approach working on a complete project and strongly recommend that you work on this challenge alone. We will particularly focus on your attention to details and expect the code to be professionally structured, commented, documented, and tested.
If anything is unclear, feel free to ask any question that might help you understand the specifications or requirements better.
Your application can be sent to us as a GitHub repository (in which case you are welcome to fork this repository) or as a compressed archive containing all the deliverables.
In some specific cases, companies need to collect consent from consumers before using their data. For instance, app users might need to explicitly consent to share their geolocation before a company can use it for advertising.
As users interact with the Didomi platform, we collect different types of events like:
- "Page view" when a user visits a webpage
- "Consent asked" when a user is asked for consent (ie a consent notice is displayed)
- "Consent given" when a user gives consent (ie has clicked on a Agree or Disagree in the notice)
The goal of this challenge is to build a very simple Spark app that processes events and summarizes various metrics as time-series data.
Download the example data for the challenge.
Events are stored as JSON Lines files with the following format:
{
"timestamp": "2020-01-21T15:19:34Z",
"id": "94cabac0-088c-43d3-976a-88756d21132a",
"type": "pageview",
"domain": "www.website.com",
"user": {
"id": "09fcb803-2779-4096-bcfd-0fb1afef684a",
"country": "US",
"token": "{\"vendors\":{\"enabled\":[\"vendor\"],\"disabled\":[]},\"purposes\":{\"enabled\":[\"analytics\"],\"disabled\":[]}}",
}
}| Property | Values | Description |
|---|---|---|
timestamp |
ISO 8601 date | Date of the event |
id |
UUID | Unique event ID |
type |
pageview, consent.given, consent.asked |
Event type |
domain |
Domain name | Domain where the event was collected |
user.id |
UUID | Unique user ID |
user.token |
JSON-String | Contains status of purposes/vendors |
user.country |
ISO 3166-1 alpha-2 country code | Country of the user |
We consider an event as positive consent when at least one purpose is enabled.
The data is partitioned by date/hour with Hive partition structure.
The Spark job is expected to output the following grouped metrics as a Parquet table:
| Column | Type | Description |
|---|---|---|
datehour |
Dimension | Date and hour (YYYY-MM-DD-HH) dimension |
domain |
Dimension | Domain |
country |
Dimension | User country |
pageviews |
Metric | Number of events of type pageview |
pageviews_with_consent |
Metric | Number of events of type pageview with consent (ie user.consent = true) |
consents_asked |
Metric | Number of events of type consent.asked |
consents_asked_with_consent |
Metric | Number of events of type consent.asked with consent (ie user.consent = true) |
consents_given |
Metric | Number of events of type consent.given |
consents_given_with_consent |
Metric | Number of events of type consent.given with consent (ie user.consent = true) |
avg_pageviews_per_user |
Metric | Average number of events of type pageview per user |
On top of computing the metrics listed above, the following operations must be run:
- Deduplication of events based on event ID