From 8fab936e97842e99e0c23f09133a6b517f7feb09 Mon Sep 17 00:00:00 2001 From: Gabe Isman Date: Mon, 6 May 2019 11:15:02 -0400 Subject: [PATCH 01/42] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 0276952d..d5e6d156 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,8 @@ Click on the button that says “Manage App”. This takes you behind the scenes ![](docs/scheduler.png) +Unfortunately our email provider Sendgrid requires an additional step to confirm that you are not a spammer. Your new Sendgrid account is now in a "suspended" state, to get it unsuspended you have to contact Sendgrid support. You can do this by clicking the Sendgrid logo on the Resources tab. **You will not be able to log in until you are cleared by Sendgrid.** This usually happens pretty quickly (hours not days). + Now, at the top of the scheduler page, click the link that is the name of your app (“sl-klaxon”). This will take you to your Klaxon’s login screen on the web. ![](docs/login_screen.png) From 91c0852c30b3ad42156e8e840914a158511f670d Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Wed, 18 Dec 2019 19:44:26 +0000 Subject: [PATCH 02/42] [Security] Bump rack from 2.0.6 to 2.0.8 Bumps [rack](https://github.com/rack/rack) from 2.0.6 to 2.0.8. **This update includes a security fix.** - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/master/CHANGELOG.md) - [Commits](https://github.com/rack/rack/compare/2.0.6...2.0.8) Signed-off-by: dependabot-preview[bot] --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index 154083c2..84919713 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -172,7 +172,7 @@ GEM method_source (~> 0.9.0) public_suffix (3.0.3) puma (3.12.1) - rack (2.0.6) + rack (2.0.8) rack-cache (1.9.0) rack (>= 0.4) rack-protection (2.0.5) From 9d9c1233553ba9dc636161042e3ea5fed8295f9d Mon Sep 17 00:00:00 2001 From: Tom Meagher Date: Mon, 23 Dec 2019 14:34:18 -0500 Subject: [PATCH 03/42] add documentation on deleting --- data/help.md | 5 +++++ public/images/delete_button.png | Bin 0 -> 6300 bytes 2 files changed, 5 insertions(+) create mode 100644 public/images/delete_button.png diff --git a/data/help.md b/data/help.md index f58c8c87..0a59f3b9 100644 --- a/data/help.md +++ b/data/help.md @@ -58,6 +58,11 @@ Click on the “Settings” button in the upper right corner of the page and cho Now, choose the channel that you want the Klaxon alerts to go to from the dropdown menu. We’d recommend that you not send them to #General, but maybe create a new channel called #Klaxon. After you create or choose your channel, click the green button that says “Add Incoming Webhooks Integration”. Near the top of the next screen, you should see a red URL next to the label “Webhook URL”. Copy that URL and switch over to your browser window with Klaxon in it. Paste the URL into the box labeled “Webhook URL,” and type the name of the channel you want your Slack alerts to go to into the “Channel” box (this should be the same channel name you used in Slack when you created the integration). Now click the “Create Slack Integration Button”. Now you should be all set. If you want to have the ability to send Klaxon alerts to other channels, for specific reporting teams or for certain projects, you can repeat this process. +## How to remove an alert +If you find that you have set an alert in error, or you are no longer using it and want to clear out space, you can delete any alert. From the "Watching" page, click on the "Edit" button next to the alert you want to remove. Then scroll to the bottom of the Edit page and look for this button: ![Klaxon delete button](/images/delete_button.PNG) + +It should do the trick and will remove the alert and it's stored snapshots from the database. + ## About Klaxon Built and refined in the newsroom of [The Marshall Project](https://www.themarshallproject.org/#.2N8GFLsI0), Klaxon has provided our journalists with many news tips, giving us early warnings and valuable time to pursue stories. The public release of this free and open source software was supported by Knight-Mozilla [OpenNews](https://opennews.org/). With feedback or suggestions, contact us with the form below. To help improve Klaxon for other users, whether you’re a coder or not, read [our guide to how you can contribute to the project.](https://github.com/themarshallproject/klaxon/blob/master/CONTRIBUTING.md) diff --git a/public/images/delete_button.png b/public/images/delete_button.png new file mode 100644 index 0000000000000000000000000000000000000000..c67dacd84e245b22abe6c7be470373dc5d9f2b7b GIT binary patch literal 6300 zcmd6qWmr^QyT^xyfq|h00nuSVL4={ZdkEdLO?WN_as|Zipp>fg8aEO$=73U$L^K$m-QtR!A zAX;!){8V&VT-avgK44Hk(?5`n9ZQOiQw6;CLpd?_W!}01fMj2a0tui?NgMJa0v_T- zL;5@Hx8T7W{f3e5bvM`NL21mQj45~ks&E%E(Ui2v{YT6%YtBO10U8d>mDOxP*Q}#UcPq>>=L1{jvu29JCF8;_Uz@Gbavb zc&emo$-r|+!mgyElDjTTxZQuN&45~d-XHgzC+jZxXwb0N>S9yYm5w#)wu#}^(5|#Q zf`952l!d5Oc4J!h-5_g$ZebpAD#(F$2+a9UcG@2(M6gGb=Y;9;fD`7O`}<`C3Mb$t5R z7gbe9tjreoBgsv_`O_BM@d}O&o@PK5hg5P--(78-;FoBCMlr;lZZB!QiZZwchdmGK zCn4xHg@q3e5lX^8l)|$G7@HA}1XEWLpgM8Rf~COt`CY(HY$Yg&u#>F{;(&er4!ncw zCoNop9VLw;gs&DX;DT=i$msIq0_+AWni0l_>YHJ!hR_m*x`6TWWe$_r3h-#9tdl_B z34NpjH3+FdebRTM35fFQ#vmnNwvf+x&r8TJXt#plQu=w)Ke6~o5@h$@jdB4W_Dodq zd<4<=2via85Jz;mF@M?B;vGS4iP3;&KN2@Yk%s+peoA$MV>$cWv@gAs-iR%>8f-$a!N(*C2M z09|&xaTrJUz0P{Gma4E>@*QTLJ#4@8HwK#MPo%1TVm+@0(w*S%e;A8 zStRgBdz||-))H4nSDn`e_IYuMLLpr*!3Z--H^RPf0~wZ8eDhQ;*;M8&FhjI?ujSgq zXSCH>-Z5vHXK&?Tr6~?dUCeMhrqXzi9FGV-)eE_F*mla|V6`Qq4?)x01^Ga!K=nM0 z+DE9=JIW9%g2GpBuMSfWHRIW=nft*OagN{*V0-G8M<)t7(257Uuyz?S{Z^f;f-}&; z3mM@f4~YKU(LHpG;y|G*TnT2NakRxqpOqZL&MS5DB- z)B%;zr^zqMloy)hzf`s;^emvA7@CNf7$~?bWU(x;WVeJ_>J94*=A_Z)=h@El;d7+v zsftRss9dstVBdyiLtBjNpPm%$)aK;p)EI>tg&4&xFf@%Gdiao?Q=J>6TuXBtF$6Z$|+gZCcDQgu76r}A?uS(UA(=lO^B)URV} zls|pGm*&p4UZ$g~s9Ujh@9R++=O&K_jW1hU{E}?0hPKv`9%Y&ChvN@lO14X)J~-<+ zYX;>@PgRX0re@!dmviTpmTgux>D4G5 z8>2T5-wZTU8M7>sEYdB0^F@l&w^6m>_~KktwHF2^2S(h0aArF{2iv=Jnz;~J$NZvL zN`C#|lw>Fp9yu5JB(g_=k4f_V7X?QJ=Xm}Ei+C^EQ@SADYCa|YQhpyxHmfa*=DFI* z-*2+oX@@^{@6-$TElm1{`LvuH663_-#)UynIZD%S(hWIUI64fT8WMecUEW{LU%vkF zarxdH;v4Q9)tpgv-)!@|QIApyG{C)5 zTQdAY{5qmi6&n?s2DH^Z)nD4Apu;l?XXq;7&hZVEr|>;L5-a?;@xjdf>XWDF$#szU5mh!zb7*j!#2L zu1&++G-5HgWc9u}z52Y`D`T78wC!pO7(re77*#=KI%MR-LrO_Q>&3iF%$xMICnw&f zSHEAqY5{qPtei#di0>TWMujJbSIb59Cil|ZB|tf$ypy%H>?Z6h=o@k_1?EJX*nC7D za1}~e3GnmOJpkoGGN!vy)(ZE1Cv+$Di^&c_XXjUWBrWl6w3OJU(#8){o5X6wRp__{ z17wU>$+}k^26ke154jcRFt&wLm?|{cGd02?2-1(GiBwAwJ*Hi3^#8b zk6oM{FVgb~e0RLF__MCsoYGbMbCIcFwgY_b%3S4pI!f$EYl07xr1$k$DrY+9U`NB( zK>M!(zXf*dZ+>ljJ9WO*>x4Jz*=ivT&I+S-J$@-p-X9LkS$x&_zGJrRY(;HHZ3)%V zlJkN;6_NfuJ$~n8d9gXFz1VCts_pTq2}d1k`Mlx+%YvZJhW>QBsrxPJ+VpeE=RKW9 zy;>bMU8l+%iB1WZqjyJL8*D$yy30nR8l!gg{d60qZD&&N_1{BDT(t`1*PpUv=hAIc zZ2`C2r;oOx=w#@8>B{Np6ImG^IfZ72KUr6_NF=@ME;+sAFl)3ZWB*8u^d)g($CdCd zx`oAG-6OYFc+OvWTjz(})@G<)34gI?K1ZoziFk8BRZsEzmdW;`-<%2^Q@NU=+OB#V z4hO<(O|9-VX!lX|7Y8jSrYlBwHcjTPe?FV)`zGEcex2dZY238qbN;MSW$*kksXof6 zu4Uf!%Z9->&kx(>0qFt7L3LNh9v!+t+n{awUpVzRZ^BdvMi)JreOA*i%ZD3i8!DHH z8hO11uXeBfs!t<#YxX+4lP*?%_3T9PCJIROi^08N2L-=NCnjGn{|>aL{X{1sx+Af0 z%Km%qJ^A$Dhh^TNTb|daIftzts-7PYBIe?F*Wa$=aAvgK4d@HvKN7hm8DoNW@zH!eR_#G|E=r0=ROYBUuhOH zK2`dfigug_`iW%=hz{ETGndN&Kq4V-@?fuHfJQ#~9v1+h>99l@M8mhY-a&b$F1DB5 zF$@g$wE=VsVy{Gbr-mVH8C4bfg;l6FNX*>bHLql|Vm7Rg>IS20di`_W1XPj$Dky_1 zL+9qQaB6By?O$2n2aKh0@AaX>##9_7`9xfuBoL*jDa4%lCvzrYk zx3I7g2Ua%$-~^6!^xfLpC$LK=`%fk(_kXp8X^`{J3MV(5i}T;!7^&EwSrH98Zz~6+oSmbUlRGAd1dp(w z*kAGgW92^?|E7HUAEgk_--Ul$_?J+O^H2SM>*t@F_17$BwjR%Ddv>VBu6e&XniwD*&-J(%BlJ+pbZUyf0l% z9%f|;8f|%*TQ9ZxEjQT&dhMQ~n%oDxS-WMhnWeDPDO}i(cGL6 z=~2EzJQWx7c!`U^Snd7s{6DZV(`y1-n)tUF71>7NB31}wpvQ|%$&y4&rOD(Im>g2c zn39wx3LPn+A<0A<1d zIb4uD-ryBgjt@T2_-y{>c5%3{5t$>Z*!R5!Z{a@m-l<{V^@Yn4l z_6YnTk6du^jb!rEn?Mso?YFAV$y1WcsO9emj;=u$@Lt9sPnb>dp<>Yh6M2WNM?dc8 zgXK2=N!$LH`=MgE{jR$a_G>pgqiecfs7qgS($NajZ}<5Fu|7D%#SR9@zla4QHQrq< z**}+q1ZEGsvdftWtR*=?Y?Pb4G6<+a<|I6Z2P>7Lkq z+{TA#KZPVi!Hm!7nn74nWi#XOEaTT@BgZW5%bvdFLO8{8dqOwsBWm2P5{?UurSw5$ z^NYd{)bxoBP!RNyi7TB=+(=o*6c@_b=q_^bn*>S-B@4DPy8b#q?Yexid_BMTVP>)gSxy9K>tNpxx zNkUJ^?*ygFF{kDwxYO~J$^NcXHEI`V*5=lVRxegGk$ggnh z@y5I>`<09wp8=0?wm-4SH_pCm_A|Hadvfv}-Z)-2iS8fQU$)U;KX|`(<1L$c!-(Cu z6i4M&*cc}ey(#^DOembc^hK@gWk};mbi2qi$QpgMfam#4^q|=@l$h!j^vj_yXKZVJ zcg$3cPg`S{P^!z~6SBGvM=)!-Y0L$qc+>?w!v4cSSjhCig=B=}Fcf0*+gyVN~xdOfynmm<}_Alxy33uH~sBpAMZEge0);TYr?KVl5ww z{sGT_bX_HX;l7>U7!+a~zF8DqG`jM`JUd|6)x#ujdzoet5=m=pBBIR__-M6D)+Vb4z@!hq)*rnWfY-Tx~yz*>VCEBkY zeAnlm8#ly$=6V>V)*ndNV?RFhlX`rbYksSBh65xxZoOuBa5ebgdCH|o&&jpo8qJ`f zY|cq|=~_9XL*X5TX4)cC&5U)*an1cQnNyvtJV~Y*DAjqrR0))cKd`K`KGt9D0RANQ z4olK}0F7}J>-NKnaSqT~pT>ob1|!nWzh7jwV?Qk@h3i#Ys|6pd2n4EN3b0g)2gtK= z*K%E5Gq3A{kX&TiF+HX+tr1hexSuB9TFikF4NnXo)Om4FKjjMfgh#1@?r=ln0I^ZNsQ>&OnMs10Oh@NT{Xd{HJ6Yx_c>`AO+Drj0o}eYx-jUx@63XWd)Yh^@_tq^vYn;V=Ll#N zO;}q)P`Uc7eSU(6bP4AS&q;VXPi>U?bibpoHO#a12SWBet4Ec7s~^X+^wouc0%ubK zK_cf-j{(-7v7Z z(~B!)*PnLl;7~qLHE&J4f$s9CDi<)LLb$%r^+$Wb@;n8=pMk^7B@)pic@-kuW6>+s~$98}#=js#3X|0paOWhW4v?!x0 zGJc?k6DJ_qE^OrzZ^19B>I*ipKiW-eB-Db+1eMWC=rZ8*%+}`FomYBvUM?KXG`3}0 z5md@Fp`K)~c5T>ntmoU`iJuvWa_-O%c<%e{3k0H;SvFV@iiU0zrNii6_N-dj)`gpb zo}abghM(?VVIbq+j+uDIaJm>ZHMA$b1s{X^s8Y7uj!-c|oW(S~K(a{q-DMIKnR#V$ zACl}zAF-?1To88<-Hm*!f?*Q0+pOdD7-E}rhtdY5)%IMD0jYX&;oE!ti`>cbEk_2^6eWB`#)Fb03HljiU+ z1a*KRsJ@!@G)%;Q^TC{k2Rq&mFcEl!8U<{CEMp8pQK`Rk2#R~Og!ji>3i|-}j#6KZ zIS2!Lco-W93AQm5iy|8HlEm8d&|%<^7$arBgWHVZEHDN!sYfIEF)|1k!-`yl9wE30 z=v&MI@D^l`VM}&Q7oW_)KQR<4C5Fi^MnC-rHKj19*##ZL!w{Y~rWtR6kBnFpRhR}o zhf2RA{;LR9tpf+7EQVCEFr Date: Mon, 9 Mar 2020 03:25:14 +0900 Subject: [PATCH 04/42] Use Ruby 2.4.5 on CircleCI --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 79559ff9..46cbccec 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -7,7 +7,7 @@ jobs: # Includes ruby, node, and chrome/phantomjs # TODO: Go back to the non-legacy version of this image, when our test # infrastructure no longer depends on phantomjs - - image: circleci/ruby:2.4 # ...with this image as the primary container; this is where all `steps` will run + - image: circleci/ruby:2.4.5 # ...with this image as the primary container; this is where all `steps` will run environment: # environment variables for primary container BUNDLE_JOBS: 3 BUNDLE_RETRY: 3 From b35010d01cd67429df50a0bb8f08cd947ce28208 Mon Sep 17 00:00:00 2001 From: Tom Meagher Date: Wed, 11 Mar 2020 14:33:02 -0400 Subject: [PATCH 05/42] Update NEWSROOMS.md Adds another newsroom to our list of known users. --- NEWSROOMS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/NEWSROOMS.md b/NEWSROOMS.md index 1248c2a2..b19dadf9 100644 --- a/NEWSROOMS.md +++ b/NEWSROOMS.md @@ -11,6 +11,7 @@ * FiveThirtyEight * Fusion * Gizmodo +* The Globe and Mail * KBIA, in Columbia, Mo. * KPCC in Pasadena, Calif. * The Marshall Project From 0acccd4179cc7b3af1a17657ce7a736b5226ead1 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Mon, 6 Apr 2020 10:11:56 +0000 Subject: [PATCH 06/42] [Security] Bump loofah from 2.2.3 to 2.5.0 Bumps [loofah](https://github.com/flavorjones/loofah) from 2.2.3 to 2.5.0. **This update includes a security fix.** - [Release notes](https://github.com/flavorjones/loofah/releases) - [Changelog](https://github.com/flavorjones/loofah/blob/master/CHANGELOG.md) - [Commits](https://github.com/flavorjones/loofah/compare/v2.2.3...v2.5.0) Signed-off-by: dependabot-preview[bot] --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 84919713..4c631bb8 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -76,7 +76,7 @@ GEM concurrent-ruby (1.1.5) crack (0.4.3) safe_yaml (~> 1.0.0) - crass (1.0.4) + crass (1.0.6) css_parser (1.6.0) addressable database_cleaner (1.7.0) @@ -133,7 +133,7 @@ GEM rb-fsevent (~> 0.9, >= 0.9.4) rb-inotify (~> 0.9, >= 0.9.7) ruby_dep (~> 1.2) - loofah (2.2.3) + loofah (2.5.0) crass (~> 1.0.2) nokogiri (>= 1.5.9) lumberjack (1.0.13) @@ -154,7 +154,7 @@ GEM mustermann (1.0.3) nenv (0.3.0) nio4r (2.3.1) - nokogiri (1.10.1) + nokogiri (1.10.9) mini_portile2 (~> 2.4.0) notiffany (0.1.1) nenv (~> 0.1) From 305cc7d82276a210d17059d642e3cc958f608057 Mon Sep 17 00:00:00 2001 From: Bob Date: Wed, 8 Apr 2020 13:41:48 -0400 Subject: [PATCH 07/42] adds more detailed steps for spinning up dockerized version locally --- docker_example/README.md | 23 +++++++++++++++++++++++ docker_example/env_local.list | 15 +++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 docker_example/README.md create mode 100644 docker_example/env_local.list diff --git a/docker_example/README.md b/docker_example/README.md new file mode 100644 index 00000000..c0fbd402 --- /dev/null +++ b/docker_example/README.md @@ -0,0 +1,23 @@ +## Modify environment variables + +edit env_local.list (found in the docker_example directory) and use your own values for things like database_url, admin_email, etc. + + +## Build the image from The Marshall Project's github repo (using the develop branch) + +`docker build --network host -t klaxon https://github.com/themarshallproject/klaxon.git#develop` + + +## Run it on your local machine as a daemon + +cd into the docker_example directory in this repo and run the following. You only need to be in this directory because we used a relative path to the env_local.list file. + +`docker run -p 8885:3000 --name klaxon_local --env-file ./env_local.list --restart unless-stopped -d klaxon` + + +## View the app + +Go to your browser and hit `127.0.0.1:8885` . + + + diff --git a/docker_example/env_local.list b/docker_example/env_local.list new file mode 100644 index 00000000..a5599188 --- /dev/null +++ b/docker_example/env_local.list @@ -0,0 +1,15 @@ +#We set the postgres url/IP addres here and include username and password in the url +DATABASE_URL=postgres://USERNAME:PASSWORD@192.168.10.40/klaxon +"ADMIN_EMAILS=someuser@companyname.com +RAILS_ENV=production +# We're using the same production secret_key_base value used in non-docker version of the app +SECRET_KEY_BASE=secret_key_value_here +# postfix doesn't need to be installed for this to work. +SMTP_PROVIDER=POSTFIX +# set this to the value of your own smpt server +POSTFIX_ADDRESS=smtp-xxx.companyname.com +POSTFIX_PORT=25 +POSTFIX_DOMAIN=companyname.com +#We have this set to false because we are handling ssl redirect in our nginx vhost. Leaving it true causes "too many redirects" error. If running on your local machine, this should also be set to false because you probably don't have an ssl cert set up for localhost. +KLAXON_FORCE_SSL=false +KLAXON_COMPILE_ASSETS=true From a7471440de15e712b98296105e2cdb264e87dd82 Mon Sep 17 00:00:00 2001 From: Lucia Walinchus <25959269+walinchus@users.noreply.github.com> Date: Fri, 8 May 2020 11:20:54 -0400 Subject: [PATCH 08/42] Update NEWSROOMS.md --- NEWSROOMS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/NEWSROOMS.md b/NEWSROOMS.md index b19dadf9..1b3bc2f4 100644 --- a/NEWSROOMS.md +++ b/NEWSROOMS.md @@ -28,5 +28,6 @@ * The Villages Daily Sun * Washington Post * WRAL, in Raleigh, N.C. +* Eye on Ohio, the Ohio Center for Journalism If you know of others who are finding Klaxon useful, please let us know with an email to klaxon-reports[at]themarshallproject[dot]org From d038657bf3b5cae9b1ddf502565dd5d8410c3a09 Mon Sep 17 00:00:00 2001 From: Tom Meagher Date: Sat, 16 May 2020 17:28:26 -0400 Subject: [PATCH 09/42] adds buttons to view and download captured html for each page snapshot. Addresses #330. --- CONTRIBUTING.md | 6 +-- DEVELOPING.md | 60 +++++++++++++++++++++++++++ app/assets/stylesheets/changes.scss | 7 ++++ app/controllers/changes_controller.rb | 22 ++++++++++ app/views/changes/html.html.erb | 39 +++++++++++++++++ app/views/changes/page.html.erb | 1 + app/views/pages/snapshots.html.erb | 2 +- config/initializers/assets.rb | 1 + config/routes.rb | 2 + 9 files changed, 136 insertions(+), 4 deletions(-) create mode 100644 DEVELOPING.md create mode 100644 app/assets/stylesheets/changes.scss create mode 100644 app/views/changes/html.html.erb diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79ca9ac3..64e54469 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,7 @@ # CONTRIBUTING -Klaxon is open source software built in the newsroom of [The Marshall Project](https://www.themarshallproject.org/), a nonprofit investigative news organization covering the American criminal justice system. It was created by a team of three—Ivar Vong, Andy Rossback and Tom Meagher—and it is subject to the kind of shortcomings any young, small side project might encounter. It may break unexpectedly. It may miss a change in a website, or an email might not fire off correctly. Still, we’ve found it immensely useful in our daily reporting. We want other journalists to benefit from Klaxon and to help us improve it, but keep these caveats in mind and use it at your own risk. +Klaxon is open source software built in the newsroom of [The Marshall Project](https://www.themarshallproject.org/), a nonprofit investigative news organization covering the American criminal justice system. It was created by a team of three—Ivar Vong, Andy Rossback and Tom Meagher—and has been maintained by Gabe Isman in the years since then. It is subject to the kind of shortcomings any young, small side project might encounter. It may break unexpectedly. It may miss a change in a website, or an email might not fire off correctly. Still, we’ve found it immensely useful in our daily reporting. We want other journalists to benefit from Klaxon and to help us improve it, but keep these caveats in mind and use it at your own risk. Our team will keep hacking on Klaxon in spare moments, and we plan to keep it humming for our own use. But we think this project has the potential to help just about any newsroom. For it to succeed and to evolve, it will depend on the contributions from other journalist-developers. We are excited about the prospect of building a community around this project to help maintain it. So when you spot the inevitable bug, please let us know. And if you’d like to help us make this better, or add new functionality to it, we’d love to have your help. @@ -21,10 +21,10 @@ We’re especially excited about other journalist-developers contributing code t If you want to contribute, start by reviewing this advice, inspired by this post on [the Codacy blog](http://blog.codacy.com/2015/12/17/open-source-development-a-few-guidelines/) and [Shauna Gordon-McKeon’s PyCon 2015 talk](https://shaunagm.github.io/personal/pycon2015.html). Before you do anything, read the documentation in our Github repo, particularly this CONTRIBUTING.md file and our [Code of Conduct](CODE_OF_CONDUCT.md). Once you’ve done that, you’re ready to engage with the community, by commenting on issues and participating in the process. -Look for [issues](https://github.com/themarshallproject/klaxon/issues) in Klaxon’s Github repo tagged "help wanted" or “first-timer-friendly”. After you announce you’re working on an issue with a comment, fork the project. Create a new branch for your feature, write a patch and send a pull request to us on Github. +Look for [issues](https://github.com/themarshallproject/klaxon/issues) in Klaxon’s Github repo tagged "help wanted" or “first-timer-friendly”. After you announce you’re working on an issue with a comment, fork the project. If you need help getting Klaxon running on a local development server, [follow these directions](DEVELOPING.md). Create a new branch for your feature, write a patch and send a pull request to us on Github. You can expect that we'll acknowledge your patch and respond with questions or comments, and we’ll expect that you’ll remain engaged with the issue, responding to our questions in a timely manner and iterating on the code until the patch is merged or otherwise closed. ### Discuss -One of the things we’re most excited about as we release Klaxon is seeing how other newsrooms put it to use. We’ll be eager to hear from you about your experiences with the tool. There are several ways we can discuss this together. You can email us directly at [klaxon-reports@themarshallproject.org](mailto:klaxon-reports@themarshallproject.org). We also have a [Google Group email list](https://groups.google.com/forum/#!forum/news-klaxon-users) where Klaxon users and developers working on the project can talk. Users can ask questions of one another, and contributors can discuss changes to the code and adding new functionality. Finally, you can always open or comment on an item in our [Github repo’s issue tracker](https://github.com/themarshallproject/klaxon/issues). +One of the things we’re most excited about as we release Klaxon is seeing how other newsrooms put it to use. We’ll be eager to hear from you about your experiences with the tool. There are several ways we can discuss this together. You can email us directly at [klaxon-reports@themarshallproject.org](mailto:klaxon-reports@themarshallproject.org). We also have a [Google Group email list](https://groups.google.com/forum/#!forum/news-klaxon-users) where Klaxon users and developers working on the project can talk. Users can ask questions of one another, and contributors can discuss changes to the code and adding new functionality. Finally, you can always open or comment on an item in our [Github repo’s issue tracker](https://github.com/themarshallproject/klaxon/issues). \ No newline at end of file diff --git a/DEVELOPING.md b/DEVELOPING.md new file mode 100644 index 00000000..46f673d1 --- /dev/null +++ b/DEVELOPING.md @@ -0,0 +1,60 @@ +# Developing Klaxon Locally # + +If you're interested in [helping improve Klaxon by contributing code](CONTRIBUTING.md), this is a quick guide to how to get it running on your local machine for development. + +First, we'll assume you already have git on your local machine, as well as a standard postgres installation. So go to your terminal, navigate to whatever directory where you keep your projects and clone the Klaxon repo. + +``` +git clone git@github.com:themarshallproject/klaxon.git +``` + +After you've cloned it, `cd` into the Klaxon directory. If you're on a Mac, you'll need to already have [homebrew](https://brew.sh/) and [rbenv](https://github.com/rbenv/rbenv) (a program that manages versions of Ruby) installed. Then you'll want + +These next commands will put the proper (albeit aged) version of Ruby that Klaxon requires on your machine and make it available in this repo's directory. + +``` +brew update && brew upgrade ruby-build +rbenv install +``` + +Next, you'll need to have the proper versions of all of Klaxon's dependency libraries, as well as foreman, Heroku's tool for running a local server for development. Run these commands here. + +``` +gem install bundler +bundle install +gem install foreman +``` + +In order to be allowed to login to Klaxon once it's running on your machine, it'll need to know that your email address is. Create a file named `.env` and paste in these two items: + +``` +ADMIN_EMAILS="my_awesome_email@gmail.com" +HOST='localhost:5000' +``` + +Feel free to substitute in your real email address. + +Now that's set, you'll run a couple of commands for Rails to create Klaxon's database on your machine. + +``` +rake db:create +rake db:migrate +``` + +Now, you should be about ready to get started. This command in the top folder of the Klaxon repo will get your dev server rolling. + +``` +foreman start -f Procfile +``` + +Now, you should be able to go to `localhost:5000` in your web browser and see Klaxon's welcome screen pop up. You'll want to manually add a webpage or two to watch at ```watching/pages/new```. For development purposes, you'll probably want to pick a site that updates pretty regularly. We use `http://www.timeanddate.com/`. + +To get Klaxon to check for updates on the pages you're watching, in another terminal window, run this rake command. + +``` +rake check:all +``` + +Now, when you go to the main Klaxon page, you should start to see changes in the Feed of the latest updates. + +Go forth and add some features, and be sure to send us your [pull requests](/pulls) for features you think other Klaxon users might find handy. diff --git a/app/assets/stylesheets/changes.scss b/app/assets/stylesheets/changes.scss new file mode 100644 index 00000000..91eb391a --- /dev/null +++ b/app/assets/stylesheets/changes.scss @@ -0,0 +1,7 @@ +// Place all the styles related to the Changes controller here. +// They will automatically be included in application.css. +// You can use Sass (SCSS) here: http://sass-lang.com/ + +.ignore-css { + all: initial; +} \ No newline at end of file diff --git a/app/controllers/changes_controller.rb b/app/controllers/changes_controller.rb index 8d980267..38e411ed 100644 --- a/app/controllers/changes_controller.rb +++ b/app/controllers/changes_controller.rb @@ -23,6 +23,27 @@ def resend render json: @change.send_notifications end + def html + @change_id=params[:change_id] + @snapshot = PageSnapshot.find(@change_id) + @page = Page.find(@snapshot.page_id) + @timestamp = @snapshot.created_at + @html = @snapshot.html + end + + def download + @id = params[:change_id] + @change_id = @id + @snapshot = PageSnapshot.find(@id) + @html = @snapshot.html + @parent_page = Page.find(@snapshot.page_id) + @filename = @parent_page.name.gsub(" ","-") + "-" + @snapshot.created_at.to_s.gsub(" ","-") + ".html" + send_data(@html, + :filename => @filename, + :type => 'text/html', + :disposition => 'attachment') + end + private def set_change @@ -32,4 +53,5 @@ def set_change def change_params params.require(:change).permit(:summary) end + end diff --git a/app/views/changes/html.html.erb b/app/views/changes/html.html.erb new file mode 100644 index 00000000..ffead270 --- /dev/null +++ b/app/views/changes/html.html.erb @@ -0,0 +1,39 @@ +<%= stylesheet_link_tag "changes", media: "all" %> + +
+
+
+
+

Latest Captured HTML

+
+
+
+
+

Below is a version of <%= @page&.name || @page&.url %> downloaded on <%= @timestamp.strftime("%A, %B %d, %Y at %H:%M %P") %>.

+

It may take a moment for the captured page to load. The page may look odd because we're not capturing the CSS as well. Sorry.

+
+ +
+ +
+
+ <%= link_to 'Past Snapshots', snapshots_page_path(@page), class: 'btn btn-default btn-lg btn-block' %> + <%= link_to 'Download HTML', download_html_path(@change_id), download:true, class: 'btn btn-default btn-lg btn-block' %> + <%= link_to 'Go to Source', @page.url, class: 'btn btn-default btn-lg btn-block' %> + <%= link_to 'Edit Klaxon', edit_page_path(@page), class: 'btn btn-default btn-lg btn-block' %> + + +
+
+
+
+
+
+ +
<%= raw @html %>
+
+ +
+ + + \ No newline at end of file diff --git a/app/views/changes/page.html.erb b/app/views/changes/page.html.erb index 7eec5389..3a788f5b 100644 --- a/app/views/changes/page.html.erb +++ b/app/views/changes/page.html.erb @@ -14,6 +14,7 @@
<%= link_to 'Past Snapshots', snapshots_page_path(@page), class: 'btn btn-default btn-lg btn-block' %> + <%= link_to 'Snapshot HTML', show_html_path(@after.id), class: 'btn btn-default btn-lg btn-block' %> <%= link_to 'Go to Source', @page.url, class: 'btn btn-default btn-lg btn-block' %> <%= link_to 'Edit', edit_page_path(@page), class: 'btn btn-default btn-lg btn-block' %>
diff --git a/app/views/pages/snapshots.html.erb b/app/views/pages/snapshots.html.erb index 9c8c3017..e1343c37 100644 --- a/app/views/pages/snapshots.html.erb +++ b/app/views/pages/snapshots.html.erb @@ -42,7 +42,7 @@

<%= snapshot.created_at %>

-

+

<%= link_to 'Snapshot HTML', show_html_path(snapshot.id), class: 'btn btn-default btn-sm btn-block' %>

<% end %> diff --git a/config/initializers/assets.rb b/config/initializers/assets.rb index 01ef3e66..a8b6bff4 100644 --- a/config/initializers/assets.rb +++ b/config/initializers/assets.rb @@ -9,3 +9,4 @@ # Precompile additional assets. # application.js, application.css, and all non-JS/CSS in app/assets folder are already added. # Rails.application.config.assets.precompile += %w( search.js ) +Rails.application.config.assets.precompile += %w( changes.css ) \ No newline at end of file diff --git a/config/routes.rb b/config/routes.rb index 9196762b..89f3bdc5 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -28,6 +28,8 @@ end get '/changes/page/:change_id' => 'changes#page', as: :page_change + get '/changes/page/:change_id/html' => 'changes#html', as: :show_html + get '/changes/page/:change_id/html/download' => 'changes#download', as: :download_html post '/changes/resend/:change_id' => 'changes#resend', as: :resend_change_notifications patch '/changes/:change_id' => 'changes#update', as: :change From 0ba6c932aa20a9df31237a3fb04d185ffcc53ff7 Mon Sep 17 00:00:00 2001 From: Tom Meagher Date: Wed, 20 May 2020 18:39:36 -0400 Subject: [PATCH 10/42] better display of html using iframe. Thx @gabeisman for the advice! --- app/assets/stylesheets/changes.scss | 7 ++++--- app/controllers/changes_controller.rb | 7 +++++++ app/views/changes/html.html.erb | 4 ++-- config/routes.rb | 1 + 4 files changed, 14 insertions(+), 5 deletions(-) diff --git a/app/assets/stylesheets/changes.scss b/app/assets/stylesheets/changes.scss index 91eb391a..db623db5 100644 --- a/app/assets/stylesheets/changes.scss +++ b/app/assets/stylesheets/changes.scss @@ -2,6 +2,7 @@ // They will automatically be included in application.css. // You can use Sass (SCSS) here: http://sass-lang.com/ -.ignore-css { - all: initial; -} \ No newline at end of file +iframe { + height: 800px; + width: 100%; +} diff --git a/app/controllers/changes_controller.rb b/app/controllers/changes_controller.rb index 38e411ed..024550b2 100644 --- a/app/controllers/changes_controller.rb +++ b/app/controllers/changes_controller.rb @@ -31,6 +31,13 @@ def html @html = @snapshot.html end + def raw_html + @change_id=params[:change_id] + @snapshot = PageSnapshot.find(@change_id) + return render html: @snapshot.html.html_safe + end + + def download @id = params[:change_id] @change_id = @id diff --git a/app/views/changes/html.html.erb b/app/views/changes/html.html.erb index ffead270..42c8bc45 100644 --- a/app/views/changes/html.html.erb +++ b/app/views/changes/html.html.erb @@ -9,7 +9,7 @@
-

Below is a version of <%= @page&.name || @page&.url %> downloaded on <%= @timestamp.strftime("%A, %B %d, %Y at %H:%M %P") %>.

+

Below is a version of <%= @page&.name || @page&.url %> downloaded on <%= @timestamp.strftime("%A, %B %d, %Y at %l:%M %P %z off UTC") %>.

It may take a moment for the captured page to load. The page may look odd because we're not capturing the CSS as well. Sorry.

@@ -30,7 +30,7 @@
-
<%= raw @html %>
+