Document the testing process

[Ekleog]

Hey! In the same idea as #112 ; would it be possible for you to write some documentation for how you test marvin when you develop on it? Creating multiple github accounts / repositories is totally OK, I just have no idea how to run marvin myself :)

[timokau]

Hi! I don't have the time to write proper documentation right now. Here is a draft:

• There are some automated tests that you can run with pytest. There are not enough of them though, and they are limited by the current architecture and GitHub's API.
• I have a test instance of marvin that runs on heroku and responds to PRs on marvins-playground. You can set one up by
  • setting up a playground repository.
  • setting up a marvin instance. If you want to use heroku for this, their free credits should be sufficient. They pause your application when it does not receive any events for a while, which is good for the testing use-case. You could also run it locally of course, but your machine needs to be able to receive webhooks.
  • creating and configuring your own GitHub app here.
    • Permissions: Read access to metadata, Read and write access to issues and pull requests
    • Repository access: Your playground repository
    • Webhook URL: https://<your-heroku-app>.herokuapp.com/webhook
  • configuring heroku to set the WEBHOOK_SECRET, GH_PRIVATE_KEY and GH_APP_ID environment variables.

I usually test the changes in a PR on the playground and also do some sanity-checks to make sure that I didn't break any basic functionality. There is room for improvement before we move on to a less experimental stage (#112). You should be aware that marvin will use the same reviewer list on your playground repository (#93).

I hope that helps. Let me know if you're having trouble with one of the steps. It would be great if you could make some notes to make the contributing experience smoother in the future :)