storages.md

CSCI640 Inquiry Based Project - Distributed Storage

Goals

This project has the following goals:

• Project sets out to answer a question about distributed storage
• Students find a survey of research work as the starting point of their project
• Students devise a way to test and/or gain understanding about the question they are trying to ask through a hands on implementation of the concept.
  • Should come up with a procedure or method for testing the effectiveness of known solutions or their solutions.
• Students present their findings to the class
• Students write up a document with their findings

Sample Questions

Recommended questions, you are welcome to propose a different one. If you did an inquiry based project in one of my classes you can not answer the same question. Any new question needs approval first.

• What distributed file system works best for large files?
• What distributed file system works best for small files?
• (Harder to test) What is the best distributed file system for limited memory systems?

Distributed Filesystems

You aren't limited to the following; however, you need to choose at least 2 different systems to deploy and compare.

• Hadoop Filesystem (HDFS)
  • For additional references in case the official docs aren't enough:
    • Linode HDFS 3 nodes
  • Recommended to choose as one of them as may benefit to have for the distributed databases project
• Ceph Filesystem
  • For additional references in case the official docs aren't enough:
    • Ceph FS on 3 nodes James Coyle
• GlusterFS
  • For additional references in case the official docs aren't enough:
    • GlusterFS 3 nodes
• MooseFS
  • For additional references in case the official docs aren't enough:
    • Burnham Forensics MooseFS guide
• Lustre Filesystem
  • For additional references in case the official docs aren't enough:
    • Lustre Wiki
    • GitHub Gist - Quick three-node Lustre set-up on CentOS 6

Teams

You can work in a team of two to setup your test environment and help each other troubleshoot technical problems. Each team member will have their own unique question, develop their own testing code/methodology/etc, and submit their own lab report of their results.

Evaluation

Your project will be graded based on the following components.

• Papers - You will need to find a minimum of 2 papers that relate to your topic beyond ones from class or try to solve the question you are wanting to answer. These cannot all come from the same research group/team/author. Also need to be in addition to papers provided in class.
• Presentation - You will need to give a short 5 minute presentation to the class on what you were trying to answer, how you tested, and what findings you discovered
• Paper - You will submit a short paper describing what you were trying to answer, how you tested it, the testing results, and what you learned from these results. This will be in the form of a pdf.
  • What is your question
  • Your experimental setup (machines, etc)
    • Are there any side effects or artifacts related to your experimental setup that might impact your results?
    • Who you worked with, if anyone, to help setup this environment
  • How you ran your tests to answer your question
  • Test results:
    • How accurate are they?
      • Make sure you run enough tests that your results are accurate enough.
    • Do you have variation in your testing?
  • Your conclusion/answer to the question
  • What you learned from answering the question
  • References to your papers and other references
• Code - Any code you used for testing in the process of your inquiry used as part of the evaulation in the submitted paper.

Submission

Submit your pdf to Canvas, and commit any code, extra files, data appendixes, etc to your CSCI640 Repo on branch storage.