Ra tests

[vaz]

Added functional tests using Ra.

Works with ra on branch 101819816_custom_tests (this PR) - which should be ready to merge and release 1.0 I think.

Changes:

• RAML changes:
  • changed baseUri to /api like in the ini
  • added example body values (used by Ra as default request bodies)
  • defined uriParameters (with examples) for item resources
  • specified expected schema for responses that return an object directly in their response, which is validated by Ra
  • added Location header definition for POST /users to test header validation
• Comments in local.ini.template about making a test.ini with separate db/index
• User schema: allow profile to be null
• Added tests/test_api.py with custom tests and autotests, and tox.ini

[vaz]

The RAML modifications here are fairly minimal.

@postatum 's PR from the original version has quite a bit of detail in the RAML so I'll look into merging in what I can of his as well.

(But, it would be a lot easier and cleaner if some of the issues in ramlfications were resolved first so we could use resourceTypes, traits and root-level declared schemas properly. I've looked at the code and it doesn't seem like there are huge issues to fix...)

[postatum]

Looks good, but I thought we were going to test more than just basic request/response. @jstoiko what do you think?

In local.ini.template note should be updated to also say that auth has to be disabled for tests to work.

PS. I like users example data 😄

[jstoiko]

Nice work @vaz

We should also test custom processors and authentication if that's possible

[jstoiko]

You're right that ResourceTypes would be nice to have as our RAML file is getting bigger.

I met @econchick last week who told me it was almost fixed. I see there's been some progress.

jdiegodcp/ramlfications#23

Any chance we can help?

[vaz]

@postatum so far there are tests for response body matching schema, response header (Location returned by POST), and the routes themselves. I wanted to put back in the automated queryParameter testing (similar to what you had but using example values) though I am having a harder time seeing how to do it in a way that is useful—since they all change behaviour in different ways it's not really possible to tell if a query param is doing its job without a custom test. It's also hard to test RAML's requirement that "all possible query parameters are declared" since we pass field names and other more complex stuff like ES aggregation syntax—the RAML would get incredibly verbose, if it's even possible to cover it all. We could see that it doesn't cause an error to pass params but that doesn't say much—many apps will just ignore extra query params. I think query param testing should have to stay in the realm of custom test. (I need to re-add support for required query params in auto tests, though.)

RAML doesn't require that response codes be declared, only that the stuff you do declare is fulfilled properly, so a "returns a proper response code" test doesn't make sense either if we're following the spec.

It also wants all possible non-standard response headers declared—but we so far aren't really using response headers much or providing an easy ramses-like way to customize them.

I'll add the note about auth.

@jstoiko there's a processor test (POST /users) checking that the lowercase function is called on email.

Also yes, I think it would be beneficial to see if we can lend a hand on some of the more straightforward ramlfications fixes.

I'll put the tasks that were mentioned into Pivotal in a bit.

[econchick]

@jstoiko I can release today (this afternoon SF time - will ping this issue when it's out) - you could help in letting me know if it's the expected behavior when you do try it out. Also note that traits have the same issue as #23 but are not fixed yet.

[econchick]

Hey folks - v0.1.8 of ramlfications has been pushed to PyPI and github. Happy hacking!

[jstoiko]

Awesome! Thank you.

[postatum]

Thanks @econchick !