Rouge #103
Rouge #103
Conversation
add rouge-l and rouge-w
Codecov Report
@@ Coverage Diff @@
## master #103 +/- ##
==========================================
- Coverage 94.41% 93.63% -0.78%
==========================================
Files 64 65 +1
Lines 1611 1776 +165
==========================================
+ Hits 1521 1663 +142
- Misses 90 113 +23
Continue to review full report at Codecov.
|
| class Ngrams(object): | ||
| """ | ||
| datastructure for n grams. | ||
| if `exclusive`, datastructure is set |
There was a problem hiding this comment.
Can you please formally define Args?
| return ngram_set | ||
|
|
||
|
|
||
| def _frp_rouge_n(eval_count, ref_count, overlapping_count): |
There was a problem hiding this comment.
Should this be factored out as a generic f1 score utility function?
| p_lcs = llcs / n | ||
| if not beta: | ||
| beta = p_lcs / (r_lcs + 1e-12) | ||
| num = (1 + (beta**2)) * r_lcs * p_lcs |
There was a problem hiding this comment.
Do you have a paper or link to learn more about this algorithm?
|
|
||
|
|
||
| def get_rouge_w(evaluated_sentence, reference_sentence, | ||
| f=lambda x: x**2, inv_f=lambda x: math.sqrt(x)): |
There was a problem hiding this comment.
| f=lambda x: x**2, inv_f=lambda x: math.sqrt(x)): | |
| f=lambda x: x**2, inv_f=math.sqrt): |
| Computes ROUGE-W of two sequences, namely evaluated_sentence and reference sentece | ||
| Reference: https://www.aclweb.org/anthology/W04-1013.pdf | ||
| Args: | ||
| evaluated_sentence: a sentence that have been produced by the summarizer |
There was a problem hiding this comment.
Can you make sure all the arguments are defined, including the optional arguments?
| return lcs_seq_wrd(len(x), len(y)) | ||
|
|
||
|
|
||
| def _w_lcs(x, y, func=lambda x: x**2): |
There was a problem hiding this comment.
This algorithm is fairly complex. Can you provide a full test suite to make sure it's correct?
| dictionary. WLCS-based F-measure score, P-score and R-score | ||
| """ | ||
| if strict: | ||
| assert(check_increase(f)) |
There was a problem hiding this comment.
Thanks for adding invariant checks!
| Returns: | ||
| bool, if f(x + y) > f(x) + f(y) or not | ||
| """ | ||
| for i in range(100000): |
There was a problem hiding this comment.
Should this constant (10000) be parameterized?
| return True | ||
|
|
||
|
|
||
| def _frp_rouge_w(wlcs, m, n, f=lambda x: x**2, inv_f=lambda x: math.sqrt(x), |
There was a problem hiding this comment.
Do you need to repeat these lambdas a second time? I'm thinking about the principles of DRY.
| n: number of words in candidate summary | ||
| f: weighting function | ||
| inv_f: inverse function of weighting function | ||
| beta: beta = P_lcs / R_lcs when ∂ F_lcs / ∂ R_lcs = ∂ F_lcs / ∂ P_lcs. |
Sorry for the inconvenience, I should have mentioned that the codes here are mainly for learning the project, and so I didn't consider making it maintainable. I should take the responsibility here. However, I am currently not available to rewrite the codes and do the testing. And I can not promise to do it in the future. I will try to fix the problems when I am available. Rouge. This implementation is more complete than tensorflow one. What is ROUGE and how it works for evaluation of My friend may give more paper references, since ROUGE-L and ROUGE-W are implemented by him (@ST-Saint) |
|
:) No problem! Do you feel like you learned a lot? If you'd like, I can close the PR and you don't have to worry about fixing the implementation. |
Yes, sorry for the inconvenience again. I will open another PR if I rewrite the codes and do the testing. I shouldn't bother you mainly for the learning. I hope I can contribute to the project in the future. Thanks a lot! |
|
I'm happy to help! Thanks for contributing! |
Related to issue #6
implement ROUGE-N, ROUGE-W, ROUGE-L