Food waste is a significant issue in the United States, with a substantial portion occurring as a result of grocery spoiling in households. Fridge Buddy aims to combat this problem by empowering users to manage food items efficiently and reduce food waste by transforming traditional refrigerators into smart appliances. Fridge Buddy is a mobile application and an easily mountable touchscreen that tracks item inventory, synchronizes data across multiple devices, notifies users when expiration dates are near, and suggests ways to help users best make use of their grocery items. Users will be able to easily view, add, remove, and update items using a touchscreen interface that easily mounts onto existing refrigerators, supported with an optional mobile application. The data displayed and modified between these two interfaces will be synchronized through a cloud backend, ensuring that user data is always up-to-date. Users will also be provided access to a recipe suggestion feature to help them more optimally use their existing food, prioritizing items that will soon expire. Fridge Buddy empowers users to better manage food at home, reduce waste, and save money on groceries using its low-cost, effective solution.
The Fridge Buddy application comprises of a React Native frontend application for both the touchscreen refrigerator attachment and mobile devices. The application frontend includes four main pages:
- Home Page: The main page for displaying and managing the current user's item inventory. Also used for generating recipes based on selected user items.
- Scanner Page: Camera page for inputting user items via barcode scanning or smartscanning.
- Recipe Page: Page for displaying saved and recently searched recipes, as well as allowing users to search for recipes by name.
- Settings Page: Page for displaying user information, as well as handling user authentication and signing out.
The application's storage and computational backend is powered by Amazon Web Services. The services used in the AWS backend are DynamoDB (AWS’s distributed NoSQL database service, used for storing user, item, and recipe information), Lambda (AWS’s computing platform that allows users to execute specified code as containerized workloads, used for running calls to 3rd party APIs), and Cognito (AWS's identity platform, used for user authentication). The frontend and backend are connected using the AWS Amplify toolkit and React Native library that enable users to connect frontend applications to backend AWS services. Fridge Buddy uses AppSync GraphQL as the middleware by creating a GraphQL schema and resolvers that define how the React Native application can access and call upon the DynamoDB and Lambda backend services.
Figure 1: Fridge Buddy System Block Diagram
Listed below are descriptions of the Fridge Buddy application features, as well as links to documentation PDFs.
- User Manual
- First Prototype Testing Report
- Second Prototype Testing Report
- Final Prototype Testing Report
- Demo Video
The user can view their current item inventory on the Home page of the Fridge Buddy tablet and mobile app. A user’s items will synchronize across devices as long as they are connected to the Internet and logged in with their Fridge Buddy account. Items are stored in the backend using DynamoDB, with CRUD operations being connected to the React Native application using AWS AppSync using a GraphQL API. Items are defined in the backend using the following schema:
- pk: String (User ID, partition key for DynamoDB) (required, not displayed)
- sk: String (Item ID, sort key for DynamoDB) (required, not displayed)
- UPC: String (Universal Product Code used in barcodes) (not required, not displayed)
- name: String (Name of item) (required, displayed)
- category: String (Food category) (optional, displayed)
- calories: String (Number of calories) (optional, displayed)
- exp_date: Int (Expiration Date) (optional, displayed)
- added_date: Int (Date item was added to user) (optional, not displayed)
- quantity: Int (Item quantity) (optional, displayed)
- prod_name: String (General product name used for recipes, generated by app) (required, not displayed)
The following are the possible ways that users can add items. Along with manually adding in item information, the user can also scan items to add to their item inventory using the Scan page using either the Barcode Scan mode or the Smart Scan mode.
At the bottom of the Home page is an 'Add Item' entry, where users can input desired item information (item name (required), category, calorie count, quantity, expiration date), which can then be added to the user's item using the '+' button.
Best method for inputting packaged food items. On either the Fridge Buddy tablet or mobile application, when the user selects the Barcode Scan mode, if a barcode is held up in the camera's view, the app will automatically scan the barcode, extract the UPC, then automatically add the item to the user's inventory including item name, UPC, calorie count, expected expiration date (if found), and food category.
The user can also add an item to their inventory using AI-based food item recognition by going to the Scan page and selecting Smart Scan. This is most useful for produce, prepared foods, and other unpackaged food items. When using the Smart Scanner, the user takes a picture of the desired item(s) to be added using the app camera, then the application will prompt the user to select from five predicted food items that were identified. One or more items can be selected, which will then be sent to Lambda to gather the item information and add all of the selected items to the user inventory.
When items are added using the Barcode scanner or the Smart scanner, the Lambda function handling gathering item information also uses the item's generic product name to predict the item's expiration date. Fridge Buddy does so by using the FoodKeeper database to search for and calculate the expected expiration date of the item from the current date.
The user can search for recipes either by using the Generate Recipes button on the Home page using selected items to generate 5 recipes, or by using the Search bar on the Recipes page.
- Generate Recipes: The user can search for recipes that utilize items in their item inventory by going to the Home page of the Fridge Buddy tablet or mobile app and taking the following steps:
- Press the Select button.
- The user can select the items that they would like to include in the recipe.
- Press the Generate Recipes button to search for recipes based on the user’s selections.
- A list of recipes is displayed on the Recipes page.
- Search Recipe by Name: The user can search for recipes by their name using the Search bar on the Recipes page. They can do so by going to the top of the page, entering the name of their desired recipe, then hitting Enter.
The user can select a recipe to view its full list of recipe names, ingredients, step-by-step instructions, and calorie count. These recipes will disappear if the user logs out or performs a separate recipe search unless saved. To save a recipe, press the Favorite button for the desired recipe (so that the recipe will continue to display even after the next search). Favorited recipes and those from the last recipes search can be viewed on the Recipes page.
Fridge Buddy incorporates AWS Cognito to implement user authentication. The user authentication process is prompted upon the application started with no logged-in user, with user information being displayed on the Settings page. User information stored includes the user email, name, password (protected by Cognito), and username. When a user is created, the user's email will be sent a confirmation email before activating the user account.
To install a development build of Fridge Buddy to an Android device, the following should be taken:
- Clone the Fridge-Buddy/ui repository.
- Set the cloned repository as the current working directory.
- Run
npm ito install all dependencies. - Connect an Android device to your computer, ensuring that USB debugging is enabled.
- USB Debugging can be enabled in the device's developer settings.
- When the device is connected to your computer, you will be prompted to allow USB Debugging. Press "Allow".
- Run
npx expo prebuildto run the prebuild processes. - Once that is done, run
npx expo run:androidto create the development build. This will take some time. - The development build will be installed to the connected Android device.
For getting started, any new team would have to set up their own AWS account. This would require creating an AWS account, setting up a new Amplify project and environment for connecting the React Native application to the backend, and setting up the Cognito, Lambda, DynamoDB, and AppSync services.
- GraphQL: The GraphQL schema used for connecting to AppSync can be found within this repo in the schema.graphql file, then you just have to set up resolvers for all of the mutations and queries defined in the schema.
- Amplify: Getting started with setting up the Amplify CLI and connecting a project to AWS can be found here.
- DynamoDB: DynamoDB itself doesn't require much setup because the schema for the items is not defined in the table itself. Instead, you just have to create a DynamoDB table, as well as setting it up as a data source within the AppSync Schema that will be connected to the resolvers for GraphQL.
- Lambda: The code and packages used for the Lambda function handlers can be found in this repository we created.
- Cognito: We set up Cognito using the Amplify CLI tools for Amplify Auth, which can be found here.
Although we would add on more features if we had more time to work on the project, Fridge Buddy is largely in a completed state. We managed to complete the minimum viable product that we had intended, with reach goals such as creating a custom tablet, local storage of item information using AWS DataStore, voice activation, and implementing receipt scanning ultimately becoming out of scope of the project timeline. Of course, though, there's always room for polishing.
- Setting up AWS: For getting started, any new team would have to set up their own AWS account. This would require creating an AWS account, setting up a new Amplify project and environment for connecting the React Native application to the backend, and setting up the Cognito, Lambda, DynamoDB, and AppSync services.
- GraphQL: The GraphQL schema used for connecting to AppSync can be found within this repo in the schema.graphql file, then you just have to set up resolvers for all of the mutations and queries defined in the schema.
- Amplify: Getting started with setting up the Amplify CLI and connecting a project to AWS can be found here.
- DynamoDB: DynamoDB itself doesn't require much setup because the schema for the items is not defined in the table itself. Instead, you just have to create a DynamoDB table, as well as setting it up as a data source within the AppSync Schema that will be connected to the resolvers for GraphQL.
- Lambda: The code and packages used for the Lambda function handlers can be found in this repository we created.
- Cognito: We set up Cognito using the Amplify CLI tools for Amplify Auth, which can be found here.
- Updating the GraphQL Schema: It can be difficult when updating the GraphQL schema, because the way that it's modeled using the @model directive means that any update to the schema will auto-generate unused fields for the item and user datatypes (which can cause errors if not removed).
- Testing React Native Using Expo Go: Unfortunately, the Amplify Auth library is no longer supported using Expo Go. This means that, although you can test the application as a web application while using Expo Go, it is impossible to create a development build with user authentication implemented at this time.
- Testing Application As a Web Page: The React-native-vision-camera library used for implementing the camera in the barcode and smart scanner is used because the other camera libraries used break when using the barcode scanner. However, this library breaks when trying to use Expo Go for testing the application as a web application. If testing the application using Expo Go as a web application, you will have to disable the camera library.
Fridge Buddy incorporates multiple external APIs and libraries for functionalities such as querying for item information and object recognition. The external APIs used by the application are:
- FoodCentral: United States Department of Agriculture (USDA)’s official API and database for storing food product information such as Universal Product Code (UPC) and nutrition information. This API is called using an AWS Lambda function when users add items using UPC barcodes or object recognition to gather item information (item name, category, calorie count). This API was chosen because of its large size, breadth of information, and its credibility from being created by the USDA.
- Spoonacular: An online API with a large breadth of capabilities including getting a list of recipes using given product names and retrieving recipe nutritional information. Spoonacular was selected because of its generous API free tier for prototyping, as well as its capabilities for gathering recipe information. Although Spoonacular includes barcode lookup functions, its limited database made FoodCentral the better option.
- Clarifai: An online API that uses a pre-trained model to classify food items in an inputted image. This API can recognize multiple food items within a single image and returns a list of all possible food names and their corresponding accuracy percent. Fridge Buddy uses this API with its smart scanner to use object recognition for inputting user items. This API was also selected because its multi-item output allows users to input multiple items with a single image.