BETYdb 5.0.2

Update of Sprockets Gem Version

Summary of Changes

Per GitHub recommendation, the version of the sprockets Gem listed in the Gemfile.lock file has been updated from version 3.7.1 to 3.7.2.

Steps Needed for Upgrade

If you upgraded to release BETYdb 5.0 or 5.0.1 after 19:50 UTC on August 9 (2:50 PM CDT on August 9) with a git pull, you probably already got these changes. (Run git log to be sure.)

If you have not yet upgraded to Rails 5, simply follow the instructions for release BETYdb 5.0.

If you upgraded to release BETYdb 5.0 or 5.0.1 before 19:50 UTC on August 9, proceed as follows:

1. Log in to the server.
2. cd to the Rails root directory.
3. Run git pull to get the latest version of the master branch.
4. Run bundle.
5. Restart the Rails server with

   touch tmp/restart.txt
   

BETYdb 5.0.1

Fix for API fuzzy matching

Summary of Changes

Changes to the Rails API introduced in Rails 5.1 caused problems with API version 1 fuzzy matching queries. This fixes that problem.

Steps Needed for Upgrade

If you upgraded to release BETYdb 5.0 after 12:01 AM UTC on August 8 (7:01 PM CDT on August 7) with a git pull, you probably already got this fix. (Run git log to be sure.)

If you have not yet upgraded to Rails 5, simply follow the instructions for release BETYdb 5.0.

If you upgraded to release BETYdb 5.0 before 12:01 AM UTC on August 8, proceed as follows:

1. Log in to the server.
2. cd to the Rails root directory.
3. Run git pull to get the latest version of the master branch.
4. Restart the Rails server with

   touch tmp/restart.txt
   

5. Check that an API query that uses fuzzy matching (e.g. curl "<your BETYdb home page URL>/api/v1/search?treatment=~genotype&key=<your API key>) works.

BETYdb 5.0

Upgrade to Rails 5

Summary of Changes

This release includes a major Rails upgrade—from version 4 to version 5—as well
as a few bug fixes.

Bug fixes

• Add experiments table to data menu in the BETYdb Rails app #488

  Up until now, the only way to visit the experiments table was to type the
  correct URL into the browser address box. This fix adds a link in the Data
  menu.

• Can't create or edit Experiments via the BETYdb Web app #581

  The Rails 4 upgrade introduced a bug preventing users from creating or editing
  experiments via the BETYdb Web app. This problem is fixed by this release.

• Getting ForbiddenAttributesError during search on TERRA-REF instance #582

  Certain searchs via the version 1 API failed with a ForbiddenAttributesError.
  This has been fixed.

Steps Needed for Upgrade

Note: This release is a major upgrade, so a few more steps than usual are
required. There are, however, no database migrations in the release, so users
may upgrade at their leisure and still be able to do data synchronizations.

To begin with, be sure the current directory is the Rails root directory and
that you are logged in as the BETYdb app user (the user that owns the files
under the Rails root directory, and the user that the BETYdb Rails app runs as).
Then do the following:

1. Check that you are currently on the 4.21 release. You can do this by running

   git describe --tags --exact-match
   

   It should return betydb_4.21. It's possible that you are on the correct version but somehow didn't pull
   down the tag name. In that case, you should get

   fatal: no tag exactly matches 'c7f98a8ab58acf0b2f68fde825f60b0dce236ab8'`.
   

   To fix this problem, run git fetch and then re-run git describe --tags --exact-match. If you are not on the correct version, upgrade to BETYdb 4.21
   before doing this upgrade.

2. Run git pull to get the latest version of the master branch.

3. Upgrade Ruby, if needed.

   Rails 5 requires Ruby language version 2.2.2 or newer. If you already have
   Ruby 2.2.2 or later installed, skip to the next step. Otherwise, continue as
   follows:

   If you are using RVM:

   Ensure you are using the right version of Ruby and the right Gem set. Running

   rvm current

   should return ruby-2.3.0@betydb_rails5. If not, make sure you have ruby-2.3.0 installed:

   rvm list

   If it is not listed, run

   rvm install "ruby-2.3.0"

   Now make the Gem set "betydb_rails5":

   rvm gemset create "betydb_rails5"

   Run the following three commands to leave and return to the BETYdb Rails root
   directory and check the current version of Ruby and the the current gemset:

   cd ..
   cd -
   rvm current

   The last command should return "ruby-2.3.0@betydb_rails5", automatically
   creating the required Gem set if it does not exist. Alternatively, you can just
   do

   rvm use "ruby-2.3.0@betydb_rails5" --create

   Note: If you ran the bundler with the --deployment option, Gems will be
   stored under the BETYdb Rails root directory at vendor/bundle. In this
   case, the RVM Gem set does not much matter. You can check if bundle is
   using the --deployment option by running:

   bundle config path

   If you aren't using RVM:

   Make sure you have Ruby version 2.2.2 or later installed, and make sure it is
   the default version used:

   ruby --version

   should return version 2.2.2 or newer.

4. Run the bundler to get all the needed Gems stored in the right place:

   bundle

5. Upgrade Phusion Passenger

   It is strongly recommended you upgrade Phusion Passenger to version 5.3.4. If
   you currently run a version of Passenger downloaded through the bundler, you
   can continue to do so. The bundler will automatically download version 5.3.4
   when you run the bundle command. To build the Apache module, run

   bundle exec passenger-install-apache2-module

   Choose Ruby as the language to build for. After the build is complete, the
   script will print out updated LoadModule, IfModule, PassengerRoot, and
   PassengerDefaultRuby directives for using in your Apache configuration
   files. Edit your configuration files, replacing the existing
   Passenger-related directives with these.

   For more extensive Phusion Passenger installation instructions, consult
   https://www.phusionpassenger.com/library/walkthroughs/deploy/, especially if
   you are not planning to use the copy of Phusion Passenger found in the Gem set
   maintained by the bundler.

6. Whether or not you are using RVM, you may need to update the version of Ruby
   specified in your Apache configuration file. Run

   passenger-config about ruby-command

   to find what value to use with the PassengerRuby directive in the Apache
   configuration file. (If you are using the copy of Passenger downloaded by the
   bundler, you may need to prepend bundle exec to the foregoing command.)

   Edit the configuration file as needed.

   (If your existing configuration files do not override the
   PassengerDefaultRuby directive with a PassengerRuby directive (inside a
   VirtualHost block, say), then you do not need to include this
   PassengerRuby directive. In this case, the path to the Ruby interpreter
   found using the passenger-config command should match the path in the
   PassengerDefaultRuby directive.)

7. If you haven't already done so, set SECRET_KEY_BASE.

   This step should have been done in the upgrade to Rails 4. Here are the
   instructions if it remains to be done:

   Run

   bundle exec rake secret

   to generate a suitable value for this variable. The value must be in the
   environment of the application. One way to do this is to use a directive in
   an Apache configuration file of the form

   SetEnv SECRET_KEY_BASE <secret key>
   

   where <secret key> is the value you generated. This should go in the block
   of the Directory directive corresponding to the path where your BETYdb Rails
   code is located.

8. Compile Rails Assets

   BETYdb now uses the Rails Assets Pipeline. You need to compile these
   assets, and to do so, you need a Javascript runtime. You may already have one
   on your server. If not, you can install one. On CentOS, for example, use
   this command:

   sudo yum install nodejs

   On Ubuntu, replace yum with apt-get.

   To compile the assets, first determine whether you serve your BETYdb site at a
   subURI. For example, if the URL of the BETYdb home page is something like
   https://www.mycollege.edu/bety, then /bety is the subURI and you should
   use the RAILS_RELATIVE_URL_ROOT variable with the compile task like this:

   bundle exec rake assets:precompile RAILS_ENV=production RAILS_RELATIVE_URL_ROOT=/bety

   On the other hand, if you serve BETYdb from the root URL of your site, at
   https://www.mycollege.edu, say, then omit the RAILS_RELATIVE_URL_ROOT
   setting:

   bundle exec rake assets:precompile RAILS_ENV=production

9. Restart the Apache httpd server to enable the updated configuration.

10. Test your site to make sure it is working.

BETYdb 4.21

Add Foreign-Key Constraints and Database Maintenance Scripts

Summary of Changes

This release includes a migration to add all missing foreign-key constraints and changes the update rule of several existing ones to ON UPDATE CASCADE. It includes some Ruby scripts to detect and repair database problems. A few bug fixes are included and new features are added to the Bulk Upload Wizard.

New features

• The Bulk Upload Wizard now supports putting the time of day into the date column and supports a method column (see issue #545).

• Several scripts for diagnosing and fixing problems with BETYdb databases have been added. See the file script/db_maintenance/README for a summary.

Bug fixes

• The migration in this release adds all missing foreign-key constraints.

• The Covariate Show page has been cleaned up (see issue #280).

• The Editing Format page has been fixed so that editing of related variables works properly (see issue #481).

• reCaptcha VI Shutdown #588

  This bug, which prevented new users from signing up, has been fixed.

Steps Needed for Upgrade

1. Log in to the server.
2. cd to the Rails root directory.
3. Check that you are currently on the 4.20 release. You can do this by running

git describe --tags --exact-match

It should return betydb_4.20. It's possible that you are on the correct version but somehow didn't pull
down the tag name. In that case, you should get

fatal: no tag exactly matches 'a763a495a2a9bb0a5ca3192b29d6e4e41832a03c'`.

To fix this problem, run git fetch and then re-run git describe --tags --exact-match.
If you are not on the correct version, upgrade to BETYdb 4.20 before doing this upgrade.

4. Run git pull to get the latest version of the master branch.
5. Get the new recaptcha Gem with

bundle

6. Run the migration in the new release with

bundle exec rake db:migrate RAILS_ENV=production

7. Restart the Rails server with

touch tmp/restart.txt

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (There is one pending test having to do with site-group editing permissions.)

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.20 Upgrade to Rails 4.2

Upgrade to Rails 4.2; several bug fixes.

Summary of Changes

This release upgrades the version of Rails used from 3.2 to 4.2.

New features

• Added Rake tasks for dumping and updating SQL comments using YAML files (issue #542 and pull request #543).

• On the Advanced Search page, traits and yields are now searchable by method name (pull request #506)

• When creating or editing Managements, possible management types now include "thining" and "other" (pull request #546).

Bug fixes

• Issue #533 ("Trait/entity notes don't get populated through Bulk Upload Wizard")

  If a Bulk Upload file has a "notes" column, those notes now show up in the newly-created traits or yields.

• Issue #552 ("Travis fails trying to create postgis extension.")

  Travis now runs RSpec successfully so builds no longer always fail.

Other Changes

• Updated DBMS named in the README file from MySQL to PostgreSQL (long overdue).

• Docker cleanup (pull requests #537 and #550).

Steps Needed for Upgrade

1. Log in to the server.
2. cd to the Rails root directory.
3. Run git pull to get the lastest version of the master branch.
4. Run bundle to upgrade several Gems (including Rails itself) and to add new Gems that are no longer part of Rails proper are that are needed to support legacy Rails behaviors until code is updated to no longer need this legacy support.
5. The file config/secrets.yml expects a variable called SECRET_KEY_BASE to be in the environment. To generate a suitable value for this variable, run rake secret.
6. Ensure this variable is in the environment seen by the HTTP server. If you run your BETYdb app under Apache HTTP server, one way to do this is to put a directive of the form

SetEnv SECRET_KEY_BASE some_secret_key

in the <Directory> section of the configuration file corresponding to the BETYdb instance you have deployed.
7. Restart the HTTP server, e.g. sudo apachectl restart.

[In lieu of steps 6 and 7, you could also edit config/secrets.yml directly, replacing

<%= ENV["SECRET_KEY_BASE"] %>

with the value you generated in step 5; then restart PhusionPassenger by running touch tmp/restart.txt. This is not the preferred way of doing things, however, since Git will then always show this file as modified.]

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (There is one pending test having to do with site-group editing permissions.)

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.19

Publication of the beta API as version 1 (/api/v1); major speed-up of Bulk Upload wizard and several bug fixes.

Summary of Changes

Bug fixes

• Issue #452 ("Unhelpful 'Bulk Upload' error message")

• Issue #523 ("Can't delete variables via the BETYdb Rails app")

• Issue #525 ("Pagination issue in BETY web interface when searching species") When editing a PFT and revising the species list, it is now possible to search for species and page correctly through the resulting list of search results.

• Trying to visit an API endpoint of the form /api/v1/search/19 used to result in a cryptic error. Now it takes one to the /api/docs page (as do other invalid API routes) on which this is no longer listed as a valid route.

Other Changes

• Published beta API as "v1" (version 1). Note that the beta API URLs are still valid.

• Data access level 2 is now called "Internal & Collaborators" instead of "Internal EBI & Collaborators".

• Major speed-up of the validation stage of the Bulk Upload wizard. This now should run up to two-thirds faster.

The remaining changes are probably of most interest to developers:

• Migrated to using YARD (see http://yardoc.org) as the default application documentation generator, replacing RDoc. Running rake doc:app now runs YARD.

• Simplified and better documented the XML schema used to validate XML files used in uploading new traits via the API.

• Better documentation of other code, especially code implementing the Bulk Upload wizard.

• Added extraction of the trait-upload XML schema documentation to the doc:app Rake task.

Steps Needed for Upgrade

Gem Installation

Administrators need to run the bundler to install the YARD gem.

Run bundle install from the Rails root directory.

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (There is one pending test having to do with site-group editing permissions.) The tests can be run using the command

 bundle exec rspec

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.18

Implements API query of sites by geographical coordinates; fixes constraints and sequence numbering of sites_cultivars table

Changes Pertinent to PEcAn Users

Administrators will need to do a database migration.

Summary of Changes

New Features

• Querying sites by geographical coordinates

  It is now possibly to query sites by geographical coordinates. For example, a URL of the form [hostname]/api/beta/sites?containing=40.064,-88.195 will return the site or sites containing the point at latitude 40.064 and longitude -88.195.

Bug fixes

• Issue #521 (Two bugs in sites_cultivars)

  The migration SitesCultivarsUniqueness guarantees that site_id is unique in the sites_cultivars table. This means sites referenced by that table uniquely determine a cultivar.

Steps Needed for Upgrade

Database Changes

Administrators will need to do a database migration.

The database version for this release is 20170712171513.

Gem Installation

No new Gems need be installed.

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (There is one pending test having to do with site-group editing permissions.) The tests can be run using the command

 bundle exec rspec

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.17

Important bug fixes to Bulk Upload Wizard

Summary of Changes

Bug Fixes

• Issue #495 ("Yields with no assigned creator (user_id) are not updatable.")

  Yields that have no assigned creator are now updatable. Moreover, the updater is, in this case, provided the option of being assigned as creator.

• Issue #459 ("Change bulk upload (via wizard or API) to allow for traits not in the trait_covariate_associations table")

  The implementation of this feature was flawed—in fact it would report that uploads were successful when in fact they were not. This has been fixed.

• Issue #520 ("Bulk Upload Wizard shouldn't create unnamed entities if there is only one trait per row of the CSV file")

  It is useless to attach a nameless entity to a single trait. This provides no useful information and only clutters up the database. Therefore, the Bulk Upload Wizard will no longer automatically create such entities.

Steps Needed for Upgrade

Database Changes

There are no new database migrations in this release.

The database version for this release is 20170415183619.

Gem Installation

No new Gems need be installed.

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (Test spec/features/treatment_integration_spec.rb:74 passes intermittently.) There is one pending test having to do with site-group editing permissions.

The tests can be run using the command

 bundle exec rspec

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.16

Adds experiments API endpoint and new table relating sites to cultivars

Changes Pertinent to PEcAn Users

Administrators will need to do a database migration.

Summary of Changes

New Features

• Issue #475 Add sites_cultivars table

  Entries in the sites_cultivars table are used to specify that a given site is used for growing and studying a given cultivar. Trigger functions in the database schema ensure that this relationship is enforced on all traits and yields; that is, if a given row in the traits (or yields) table specifies a given site, and if that site matches an entry in the sites_cultivars table, then the row must also specify the appropriate cultivar as determined by that entry.

  For more details, see https://github.com/PecanProject/bety/files/972071/sites_cultivars.pdf.

  [At some future time this may be changed so that a cultivar may depend on both the site and the date of the trait or yield. This way, a site may be associated with different cultivars and different points in time.]

• New API endpoint for experiments relation

  Information from the experiments table is now available from the API endpoint

  [BETYdb root URL]/api/beta/experiments
  

Other Changes

There are some minor changes to the new variable and edit variable forms, including a newly-available variable type "reflectance index".

Steps Needed for Upgrade

Database Changes

Administrators will need to do a database migration.

The database version for this release is 20170415183619.

Gem Installation

No new Gems need be installed.

Status of RSpec Tests

All non-pending tests pass when run in the default environment. (There is one pending test having to do with site-group editing permissions.) The tests can be run using the command

 bundle exec rspec

Complete details for running the RSpec tests are at https://pecan.gitbooks.io/betydb-documentation/content/automated_tests.html.

BETYdb 4.15.1

Security update. See PecanProject/private#8.