Home

Fisheye

Note: this documentation is for an untested version of Fisheye.

Fisheye is FRC team 8230's apriltag-based pose estimation solution. It makes use of both opencv's aruco module, and its solvePNP.

Features

• Apriltag detection & pose estimation
• Works using NetworkTables, making it easy to veiw output using a tool such as AdvantageScope or Glass
• Has dynamic thread allocation, providing more resources to cameras currently viewing apriltags, while keeping other cameras active

Hardware

Before we get to installation, it is important to note that Fisheye has only ever been tested on the Beelink Mini S12 and Beelink SER5, both running Debian. Additionally, it has only been used with Arducam OV2311s and Arducam OV9281s

If you choose to use a different coprocessor, keep in mind that Fisheye is designed to work with more CPU threads than cameras, and without that, dynamic thread allocation will not function. Also keep in mind that you need to at least keep 1 thread free for the main function and the OS (preferably 2)

If you choose to get a different camera, the most important thing is that it is a global shutter camera. This will ensure that there is no motion blur, which makes it impossible to gather data while moving. For a cheaper option than the OV2311s, try the OV9281s. It should work with any camera, so long as it can be given a unique name (all USB arducams should have this functionality, no idea about other stuff), and is UVC compliant

Installation

Dependencies

In order for Fisheye to function, your coprocessor needs the following installed:

• openCV and openCV contrib
  • Ensure that when you build these, it says "YES" next to "v4l/v4l2"
• WPILib's NetworkTables
  • For the sake of time, install only NTCore, set everything else to OFF
• CMake
  • Install using package manager

Setup

To install and compile Fisheye, run the following commands:

git clone [email protected]:koibots8230/Fisheye.git
cd Fisheye
mkdir build
cd build
cmake ../src
make

To run Fisheye, simply run the command ./fisheye. If you are going to use this on a robot, it is necessary to set it up so that this runs on startup.

Configuration

Fisheye does not work out of the box. It must be configured to your specific setup. Configuration is accomplished via the .json files in the config folder

Cameras

This is where you define everything about your cameras. You can put as many cameras in here as you'd like, but they must all actually exist, and follow the format outlined exactly.

You need two main things about your camera: the id, and the calibration data.

Camera Id

This is the unique identifier for your camera. Make sure your camera is given a unique name (if using a USB arducam, here is the tool). Then, simply go to the /dev/v4l/by-id folder, and copy-paste the name found there to the id section like so: "/dev/v4l/by-id/nameYouFound" (Note: if there is two per camera, use the one with 0)

Calibration Data

In order for Fisheye to function, you must calibrate your camera. Calibration is used to remove lens distortion, and if it is missing or inaccurate, Fisheye will provide inaccurate data.

To calibrate, you can use virtually any camera calibration tool. 8230 has a pre-made program to accomplish this, or you can use another option.

Once you have the data, input each number into its spot.

Resolution

The last thing in the cameras.json file is the resolution. You can choose any resolution you'd like, but make sure it is the same resolution that your calibration data was based on.

A higher resolution will mean more accuracy, but will also increase processing time.

Detector

This is where you configure the settings of the apriltag detector. Information on everything except tagSizeMeters can be found here.

tagSizeMeters is simply a measure of how long the side length of your tag is, in meters.

NetworkTables

The only thing here is what your team number is. This is used by NetworkTables to find your roboRIO.

Threading

This is where you configure how Fisheye handles threading.

totalThreads: How many total threads you want to allocate to the cameras. Keep in mind that you should leave at least 1, preferably 2 threads free for main and the operating system.

defaultThreadsPerCamera: How many threads each camera will use if no cameras see an apriltag

minThreadsPerCamera: How many threads a camera will use if a different camera sees an apriltag and it doesn't

Tag Sightings: Explained

Fisheye gives more threads to cameras that actively see apriltags. To do this, it keeps track of if a thread sees a tag. If it does, it adds one to the tag sightings. If a thread does not see a tag, it subtracts one. Tag sightings are individual to each camera.

maxTagSightingsPerCamera: The maximum value that tag sightings can reach. A higher value will mean that cameras take longer to go into and out of priority mode.

minTagSightingsForPriority: The minimum number of tag sightings for a camera to take thread priority. This should be at least one less than your max to account for false negatives.

minThreadOffsetMilliseconds: The minimum amount of time between starting a thread for each camera. This is to ensure that two threads are not calculating pose for the same or very similar images. This should be equal to 1000 divided by your camera's frame rate.

Misc.

If you want a behind-the-scenes look at how Fisheye works, see the Behind The Scenes page.

If you are experiencing an issue with Fisheye, check the Previous Issues page to see if you can find a solution there.