[BUG] Ensure arrow sizing is correct / 64 byte aligned (#2426)

## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- Arrow will overreport the size of a buffer if the underlying buffers
are shared (apache/arrow-rs#5969).
- apache/arrow-rs#5554 Exposes the ability to
enforce alignment at write time. This PR enables this option explicitly
and upgrades arrow to take advantage of it. We don't change from the
default alignment but this is defensive.
- See the comments in get_size() for further understanding of this PR
(https://github.com/chroma-core/chroma/pull/2426/files#diff-03bcd4f01acfa68c46fcc974d3722fa621056b4e0f908d708a2d15028b0e99b1R410)
- Cleans up various error handling and documentation - adding explicit
errors and removing panics as needed
 - New functionality
	 - None

## Test plan
*How are these changes tested?*
Updated all tests in delta.rs to save, load and check the sizes match
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust

## Documentation Changes
None

rust/worker/Cargo.toml

@@ -41,7 +41,7 @@ parking_lot = "0.12.1"
 aws-sdk-s3 = "1.5.0"
 aws-smithy-types = "1.1.0"
 aws-config = { version = "1.1.2", features = ["behavior-version-latest"] }
-arrow = "50.0.0"
+arrow = "52.0.0"
 roaring = "0.10.3"
 tantivy = "0.21.1"
 tracing = "0.1"
@@ -55,6 +55,7 @@ opentelemetry = { version = "0.19.0", default-features = false, features = [
 opentelemetry-otlp = "0.12.0"
 shuttle = "0.7.1"
 regex = "1.10.5"
+flatbuffers = "24.3.25"
 
 [dev-dependencies]
 proptest = "1.4.0"