Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor the Linker, Ranker, Recogniser and Pipeline #282

Open
wants to merge 161 commits into
base: dev
Choose a base branch
from
Open

Refactor the Linker, Ranker, Recogniser and Pipeline #282

wants to merge 161 commits into from

Conversation

thobson88
Copy link
Collaborator

@thobson88 thobson88 commented Dec 15, 2024 •
edited
Loading

Closes #276.

Major refactor. The main changes are:

  • new subclass structure in the Recogniser, Ranker & Linker (and common load and run interface)
  • new dataclasses representing data structures throughout the pipeline: Mention, Sentence, Candidates, Predictions, etc.
  • simplified Pipeline class, with all business logic moved into the Linker

@thobson88
Ignore .DS_Store files
00a3607
@thobson88
Refactor linking methods as Linker subclasses
a3c8c7a 
All tests not marked as 'deezy' are passing
@thobson88
Add RelDisamb Linker subclass & fix tests
a114f2c
@thobson88
Move rel_params to the RelDisambLinker class
769488e
@thobson88
Remove superfluous return value
5422243
@thobson88
Refactor Ranker into subclasses
e8282fb
@thobson88
Modify Ranker class hierarchy; fix unit tests
7a1e7a5
@thobson88
Fix method_name in DeezyMatchRanker
b42f6e5
@thobson88
Fix integration tests
3bf6fa6
@thobson88
Move train method to DeezyMatchRanker
936cc21
@thobson88
Fix docstrings
5219598
@thobson88
Move deezy-specific code to Ranker subclass
c94c166
@thobson88
Update docstrings
c3cf63b
@thobson88
Refactor the Recogniser class
daf97ad
@thobson88
Simplify variable names & rename module to avoid clash
b31d07e
@thobson88
Simplify Pipeline constructor
2e7d53d
@thobson88
Remove superfluous return value
dc97af8
@thobson88
Add Pipeline constructor unit test
ee02a65
@thobson88
Remove inconsistency in treating place_wqid
e13743f
@thobson88
Remove superfluous return value
d28f5ad
@thobson88
Add dataclasses to represent ranking candidates
7c39f6d
@thobson88
Integrate dataclasses into the Ranker
e42192d
@thobson88
Integrate dataclasses into Linker & Pipeline
2088beb
@thobson88
Add method return types
29de527
@thobson88
Move dataclasses to own module
1394718
@thobson88
Add new dataclasses for Ranker & Linker output
6d8fa19
@thobson88
Update dataclasses and integrate into Linker & Pipeline
34260f0
@thobson88
Reorder dataclasses module
004bd53
@thobson88
Rename Ranker cache & load method; Edit docstrings
d54392a
@thobson88
Replace Ranker method_name() with class attribute
ffe8a59
@thobson88
Hide unfinished mkdocs pages
f7094d9
@thobson88
Merge pull request #285 from Living-with-machines/mkdocs
7f94fa7 
Update docs site & migrate to Material for Mkdocs
@thobson88
Merge branch 'mkdocs' into 276-refactor
d7a038e
@thobson88
Fix integration test with microtoponym
ba874ee
@thobson88
Fix Pipeline integration test with microtoponym
a8bd73e
@thobson88
Fix Pipeline integration tests with empty candidates
ce14a48
@thobson88
Move coords field into WikidataLinks dataclass
98e9c25
@thobson88
Fix bugs in Linker
95d8be5
@thobson88
Add static constructor for Recogniser
cdf2719
@thobson88
Fix bug in Ranker
2571ec9
@thobson88
Handle empty candidates in Predictions dataclass
242cf5a
@thobson88
Add BatchJob class, tests & sample files
7bcd279
@thobson88
Add batch-job executable script
021f936
@thobson88
Fix microtoponym handling
5dfc48a
@thobson88
Fix empty candidates handling
0ef17bc
@thobson88
Add batch job API and executable script
6301dc8
@thobson88
Add log level config parameter in batch job
8020a81
@thobson88
Add debug-level logging
f013aea
@thobson88
Avoid non-String input to NER run method
a435949
@thobson88
Avoid non-String input to pipeline run_text_recognition
25a2798
@thobson88
Add BatchJob methods for place of pub info
2f46edc
@thobson88
Fix place of publication handling in BatchJob
81ae5d0
@thobson88
Fix place of publication logging in BatchJob
c480038
@thobson88
Handle case of empty list of WikidataLinks
b0262d6
@thobson88
Fix bug in Candidates dataclass
89ae34b
@thobson88
Fix bugs in REL linking & dataclasses
5c86fbe
@thobson88
Limit to compatible versions of transformers dependency
dbd5a2c
@thobson88
Merge pull request #289 from Living-with-machines/287-transformers
6921f83 
Limit to compatible versions of transformers dependency
@thobson88
Handle empty predictions in BatchJob results
6345f83
@thobson88
Merge pull request #288 from Living-with-machines/286-batch-processing
5bd701e 
Batch processing interface
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rwood-97 rwood-97 rwood-97 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor the Linker, Ranker, Recogniser and Pipeline [PR open]
2 participants
@thobson88 @rwood-97