#3234: Modify create_federal_portfolio script to add requested suborg fields - [AG] #3303

zandercymatics · 2025-01-06T19:45:20Z

Ticket

Resolves #3234

Changes

Modifies the create_federal_portfolio script to add requested suborg fields
Adds to the create_federal_portfolio script to add a post-processing step where for started domain requests under the same portfolio, the federal agency is set to none if the federal agency name and the organization match
Adds string normaliziation to organization_name, requested suborg and suborg city
Adds a "normalize_string" helper function

Context for reviewers

This PR adds some missing data to the create_federal_portfolio script -- we are now adding data to "requested suborganization, suborganization city, and suborganization state_territory" if the domain request is requesting a suborganization name that does not yet currently exist.

Additionally, it does some normalization on org names so we don't get duplicates.

Note: Department of Defense has a lot of good data, which may be helpful when testing this PR.

Setup

This is on getgov-ag so you can view the results there. Specifically, I'd recommend looking at department of defense and its started domains, as well. To test this locally, create a new domain request with the organization_name, state, and state_territory fields filled out. Then add a portfolio to said domain request, and run the script

Code Review Verification Steps

As the original developer, I have

Satisfied acceptance criteria and met development standards

Met the acceptance criteria, or will meet them in a subsequent PR
Created/modified automated tests
Update documentation in READMEs and/or onboarding guide

Ensured code standards are met (Original Developer)

If any updated dependencies on Pipfile, also update dependencies in requirements.txt.
Interactions with external systems are wrapped in try/except
Error handling exists for unusual or missing values

Validated user-facing changes (if applicable)

Tag @dotgov-designers in this PR's Reviewers for design review. If code is not user-facing, delete design reviewer checklist
Verify new pages have been added to .pa11yci file so that they will be tested with our automated accessibility testing
Checked keyboard navigability
Tested general usability, landmarks, page header structure, and links with a screen reader (such as Voiceover or ANDI)

As a code reviewer, I have

Reviewed, tested, and left feedback about the changes

Pulled this branch locally and tested it
Verified code meets all checks above. Address any checks that are not satisfied
Reviewed this code and left comments. Indicate if comments must be addressed before code is merged
Checked that all code is adequately covered by tests
Verify migrations are valid and do not conflict with existing migrations

Validated user-facing changes as a developer

Note: Multiple code reviewers can share the checklists above, a second reviewer should not make a duplicate checklist. All checks should be checked before approving, even those labeled N/A.

New pages have been added to .pa11yci file so that they will be tested with our automated accessibility testing
Checked keyboard navigability
Meets all designs and user flows provided by design/product
Tested general usability, landmarks, page header structure, and links with a screen reader (such as Voiceover or ANDI)
(Rarely needed) Tested as both an analyst and applicant user

As a designer reviewer, I have

Verified that the changes match the design intention

Checked that the design translated visually
Checked behavior. Comment any found issues or broken flows.
Checked different states (empty, one, some, error)
Checked for landmarks, page heading structure, and links

Validated user-facing changes as a designer

Checked keyboard navigability
Tested general usability, landmarks, page header structure, and links with a screen reader (such as Voiceover or ANDI)
Tested with multiple browsers (check off which ones were used)
- Chrome
- Microsoft Edge
- FireFox
- Safari
(Rarely needed) Tested as both an analyst and applicant user

References

Code review best practices

Screenshots

github-actions · 2025-01-06T19:49:44Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T19:50:55Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T19:53:51Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T20:26:24Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T20:47:32Z

🥳 Successfully deployed to developer sandbox ag.

…name

zandercymatics · 2025-01-06T20:55:48Z

src/registrar/management/commands/populate_requested_suborg.py

+        fields_to_update = ["requested_suborganization", "suborganization_city", "suborganization_state_territory"]
+        self.mass_update_records(DomainRequest, filter_conditions, fields_to_update)
+
+    def custom_filter(self, records):


Note: maybe this function should be renamed to something more semantic within the context of its confines.
Basically all this does is after the initial filter, it does an exclude on the existing set of records

github-actions · 2025-01-06T20:56:05Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T21:05:24Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-06T21:15:07Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-08T19:29:48Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-08T22:29:50Z

🥳 Successfully deployed to developer sandbox ag.

src/registrar/management/commands/create_federal_portfolio.py

github-actions · 2025-01-09T15:23:23Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-09T15:24:36Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2025-01-13T15:13:12Z

🥳 Successfully deployed to developer sandbox ag.

dave-kennedy-ecs · 2025-01-13T16:59:45Z

This comment does not apply to your changes, but to the create_federal_portfolio.py file. add_arguments method could use explanation in its method comment for the --branch option.

zandercymatics · 2025-01-13T17:09:14Z

@dave-kennedy-ecs Good find, thanks. I added a better comment to it which you may find helpful, as well

src/registrar/management/commands/create_federal_portfolio.py

dave-kennedy-ecs · 2025-01-13T17:15:22Z

src/registrar/management/commands/create_federal_portfolio.py

@@ -111,6 +170,8 @@ def handle_populate_portfolio(self, federal_agency, parse_domains, parse_request
        if parse_requests or both:
            self.handle_portfolio_requests(portfolio, federal_agency)


Question: why is handle_portfolio_domains only called if created, but handle_portfolio_requests is called regardless of whether it is a creation?

I don't entirely remember, to be honest. This script has "outgrown its shoes" a few times, so honestly probably a leftover

In a follow-on PR I am not checking on "created" for create_suborganizations either, so it probably makes sense to remove that created check, imo

Ok, got it. Thanks for the explanation. I don't think it effects this PR, as it is focused on domain request.

dave-kennedy-ecs · 2025-01-13T17:28:14Z

Review of code is done. Script is easy to read and follow. And appears to meet the ACs. Will now test in AG.

github-actions · 2025-01-13T17:28:42Z

🥳 Successfully deployed to developer sandbox ag.

…age.get.gov into ag/3234-test-existing-script

github-actions · 2025-01-13T17:38:14Z

🥳 Successfully deployed to developer sandbox ag.

dave-kennedy-ecs

Tested successfully locally and on AG sandbox
Approved

github-actions · 2025-01-13T18:07:25Z

🥳 Successfully deployed to developer sandbox ag.

zandercymatics added 3 commits January 6, 2025 11:36

Add suborg data and clean duplicate script

0118e1f

cleanup

d0183d4

Remove unrelated content

6086b95

zandercymatics requested a review from abroddrick as a code owner January 6, 2025 19:45

zandercymatics added 2 commits January 6, 2025 12:46

Update create_federal_portfolio.py

345ca03

Update create_federal_portfolio.py

5e6a798

Use dedicated script

9052d9e

zandercymatics added 2 commits January 6, 2025 13:39

Unit tests and documentation

188c99a

Update test_management_scripts.py

822c8dd

Add rule not to parse cases where the org name matches the portfolio …

6d13f26

…name

zandercymatics changed the title ~~[Draft] #3234: Update create_federal_portfolio to include requested suborganization fields - [AG]~~ [Draft] #3234: Add script to include requested suborganization fields on domain requests - [AG] Jan 6, 2025

zandercymatics requested review from lizpearl, therealslimhsiehdy, Matt-Spence, CocoByte, rachidatecs, dave-kennedy-ecs, erinysong and asaki222 January 6, 2025 20:54

zandercymatics commented Jan 6, 2025

View reviewed changes

Update populate_requested_suborg.py

87b08e3

Exclude approved

8e7ffab

Merge branch 'main' into ag/3234-test-existing-script

1e6a887

zandercymatics commented Jan 9, 2025

View reviewed changes

src/registrar/management/commands/create_federal_portfolio.py Outdated Show resolved Hide resolved

Update src/registrar/management/commands/create_federal_portfolio.py

983fddd

zandercymatics commented Jan 9, 2025

View reviewed changes

src/registrar/management/commands/create_federal_portfolio.py Outdated Show resolved Hide resolved

Update src/registrar/management/commands/create_federal_portfolio.py

26c3e89

zandercymatics commented Jan 9, 2025

View reviewed changes

src/registrar/management/commands/create_federal_portfolio.py Show resolved Hide resolved

Update src/registrar/management/commands/create_federal_portfolio.py

f6b4d17

Merge branch 'main' into ag/3234-test-existing-script

fc236c6

dave-kennedy-ecs self-assigned this Jan 13, 2025

Added better function comment for add_arugments

5308217

zandercymatics commented Jan 13, 2025

View reviewed changes

src/registrar/management/commands/create_federal_portfolio.py Outdated Show resolved Hide resolved

Update src/registrar/management/commands/create_federal_portfolio.py

f6e94da

dave-kennedy-ecs reviewed Jan 13, 2025

View reviewed changes

zandercymatics added 2 commits January 13, 2025 10:30

Lint

0f59a63

Merge branch 'ag/3234-test-existing-script' of github.com:cisagov/man…

2865dcd

…age.get.gov into ag/3234-test-existing-script

dave-kennedy-ecs approved these changes Jan 13, 2025

View reviewed changes

Merge branch 'main' into ag/3234-test-existing-script

6a49519

zandercymatics merged commit 652ebeb into main Jan 13, 2025
10 checks passed

zandercymatics deleted the ag/3234-test-existing-script branch January 13, 2025 18:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3234: Modify create_federal_portfolio script to add requested suborg fields - [AG] #3303

#3234: Modify create_federal_portfolio script to add requested suborg fields - [AG] #3303

zandercymatics commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

zandercymatics Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 13, 2025

dave-kennedy-ecs commented Jan 13, 2025

zandercymatics commented Jan 13, 2025

dave-kennedy-ecs Jan 13, 2025

zandercymatics Jan 13, 2025 •

edited

Loading

dave-kennedy-ecs Jan 13, 2025

dave-kennedy-ecs commented Jan 13, 2025

github-actions bot commented Jan 13, 2025

github-actions bot commented Jan 13, 2025

dave-kennedy-ecs left a comment

github-actions bot commented Jan 13, 2025

		@@ -111,6 +170,8 @@ def handle_populate_portfolio(self, federal_agency, parse_domains, parse_request
		if parse_requests or both:
		self.handle_portfolio_requests(portfolio, federal_agency)

#3234: Modify create_federal_portfolio script to add requested suborg fields - [AG] #3303

#3234: Modify create_federal_portfolio script to add requested suborg fields - [AG] #3303

Conversation

zandercymatics commented Jan 6, 2025 • edited Loading

Ticket

Changes

Context for reviewers

Setup

Code Review Verification Steps

As the original developer, I have

Satisfied acceptance criteria and met development standards

Ensured code standards are met (Original Developer)

Validated user-facing changes (if applicable)

As a code reviewer, I have

Reviewed, tested, and left feedback about the changes

Validated user-facing changes as a developer

As a designer reviewer, I have

Verified that the changes match the design intention

Validated user-facing changes as a designer

References

Screenshots

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

zandercymatics Jan 6, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 13, 2025

dave-kennedy-ecs commented Jan 13, 2025

zandercymatics commented Jan 13, 2025

dave-kennedy-ecs Jan 13, 2025

Choose a reason for hiding this comment

zandercymatics Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

dave-kennedy-ecs Jan 13, 2025

Choose a reason for hiding this comment

dave-kennedy-ecs commented Jan 13, 2025

github-actions bot commented Jan 13, 2025

github-actions bot commented Jan 13, 2025

dave-kennedy-ecs left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 13, 2025

zandercymatics commented Jan 6, 2025 •

edited

Loading

zandercymatics Jan 13, 2025 •

edited

Loading