Argument parsing refactor and urgent mode (#221)

* Coordinate command line on ArgvParser class * Urgent option to focus on avoiding or fixing translations manual builds * Command line option to ignore entity by name Co-authored-by: André L F S Bacci <[email protected]> Co-authored-by: Kamil Tekiela <[email protected]>
php · Feb 14, 2025 · 80d1314 · 80d1314
1 parent c63809f
commit 80d1314
Show file tree

Hide file tree

Showing 7 changed files with 266 additions and 109 deletions.
diff --git a/scripts/translation/README.md b/scripts/translation/README.md
@@ -1,24 +1,69 @@
+# Scripts to check consistency of manual translations
 
-# Useful scripts for maintaining translation consistency of manual
+After a normal `doc-base/configure.php --with-lang=$LANG`, it is possible to
+run the command line tools below to check translated source files
+for inconsistencies. These tools check for structural differences
+that may cause translation build failures or non-validating DocBook XML
+results, and fixing these issues will help avoid build failures.
 
-Some of these scripts only test some file contents or XML structure
-of translated files against their equivalents on `en/` directory.
-Others will try to modify the translations in place, changing the
-translated files. Use with care.
+Some checks are less structural, and as not all translations are identical,
+or use the same conventions, they may not be entirely applicable in all
+languages. Even two translators working on one language may have different
+opinions on how much synchronization is wanted, so not all scripts will be of
+use for all translations.
 
-Not all translations are identical, or use the same conventions.
-Even two translators working on one language may havedifferent
-opinions on how much synchronization is wanted. So not all scripts
-will be of use for all translations.
+Because of the above, it's possible to silence each alert indempendly. These
+scripts will output `--add-ignore` commands that, if executed, will omit the
+specific alerts in future executions.
 
-Because of aboce, it's possible to silence each alert indempendly.
-These scripts will output `--add-ignore` commands that, if executed,
-will omit the specific warming in future executions.
+## First execution
 
-The `lib/` directory contains common code and functionality
-across these scripts.
+The first execution of these scripts may generate an inordinate amount of
+alerts. It's advised to initially run each command separately, and work the
+alerts on a case by case basis. After all interesting cases are fixed,
+it's possible to rerun the command and `grep` the output for `--add-ignore`
+lines, run these commands, and by so, mass ignore the residual alerts.
+
+## qaxml-attributes.php (structural)
+
+`doc-base/scripts/translation/qaxml-attributes.php` checks if all translated
+files have the same tag-attribute-value triplets. Tag's attributes are
+extensively utilized in manual for linking and XIncludes. Translated files
+with missing or mistyped attributes may cause build failures or missing parts.
+
+This script accepts an `--urgent` option, to filter alerts related to `xml:id`
+attributes. This will help translators on languages that are failing to build,
+to focus on mismatches that are probably most related with build fails.
+
+## qaxml-entities.php (structural)
+
+`doc-base/scripts/translation/qaxml-entities.php` checks if all translated
+files contain the same XML Entities References as the original files.
+Unbalanced entities may indicate mistyped or wrongly translated parts. This
+is problematic because some of these entities are "file
+entities", that is, entities that include entire files and even directories,
+so missing or misplaced file entity references almost always cause build
+failures.
+
+This script accepts an `--urgent` option, to filter alerts related to file
+entities. This will help translators on languages that are failing to build,
+to focus on mismatches that are probably most related with build fails.
+
+This script also accepts `-entity` options that will ignore the informed
+entities when generating alerts. This is handy in languages that use some
+"leaf" entities differently than `doc-en`. For example, `doc-de` uses a lot of
+`&zb;` and `&dh;` entities, and could run with `-zb -dh` to avoid generating
+alerts for these entities' differences.
 
-Before using the scripts, it need be configured:
+## Old tools (below)
+
+The tools on `doc-base/scripts/translation/` are slowly being rewritten. While
+this effort is not complete, the previous tools, document below, could be used
+to supply for features yet not completed.
+
+---
+
+Before using the old scripts, they need be configured:
 ```
 php doc-base/scripts/translation/configure.php $LANG_DIR
 ```
@@ -103,12 +148,3 @@ php doc-base/scripts/translation/qaxml.t.php filename
 php doc-base/scripts/translation/qaxml.t.php literal
 php doc-base/scripts/translation/qaxml.t.php varname
 ```
-
-## Initial alerts execution
-
-The first execution of these scripts may generate an inordinate amount of
-alerts. It's advised to initially run each command separately, and work the
-alerts on a case by case basis. After all interesting cases are observed,
-it's possible to rerun the command, and `grep` the output for `--add-ignore`
-lines, and to mass ignore the residual alerts.
-
diff --git a/scripts/translation/libqa/ArgvParser.php b/scripts/translation/libqa/ArgvParser.php
@@ -0,0 +1,79 @@
+<?php /*
++----------------------------------------------------------------------+
+| Copyright (c) 1997-2025 The PHP Group                                |
++----------------------------------------------------------------------+
+| This source file is subject to version 3.01 of the PHP license,      |
+| that is bundled with this package in the file LICENSE, and is        |
+| available through the world-wide-web at the following url:           |
+| https://www.php.net/license/3_01.txt.                                |
+| If you did not receive a copy of the PHP license and are unable to   |
+| obtain it through the world-wide-web, please send a note to          |
+| [email protected], so we can mail you a copy immediately.              |
++----------------------------------------------------------------------+
+| Authors:     André L F S Bacci <ae php.net>                          |
++----------------------------------------------------------------------+
+
+# Description
+
+This class coordinates and centrailzes control for $argv command line
+parameters, used between vairous classes.                             */
+
+class ArgvParser
+{
+    private array $argv;
+    private array $used;
+
+    public function __construct( array $argv )
+    {
+        $this->argv = array_values( array_filter( $argv ) );
+        $this->used = [];
+        $this->used = array_fill( 0 , count( $argv ) , false );
+    }
+
+    public function use( string $arg ) : void
+    {
+        foreach ( $this->argv as $pos => $value )
+            if ( $arg == $value && $this->used[ $pos ] == false )
+            {
+                $this->used[ $pos ] = true;
+                return;
+            }
+        throw new Exception( "Unused '$arg' not found." );
+    }
+
+    public function consume( string $equals = null , string $prefix = null , int $position = -1 ) : string|null
+    {
+        $args = $this->argv;
+        foreach ( $args as $pos => $arg )
+        {
+            if ( $arg == null )
+                continue;
+
+            $foundByEquals = $equals != null && $arg == $equals;
+            $foundByPrefix = $prefix != null && str_starts_with( $arg , $prefix );
+            $foundByPosition = $position == $pos;
+
+            if ( $foundByEquals || $foundByPrefix || $foundByPosition )
+            {
+                $this->argv[ $pos ] = null;
+                $this->used[ $pos ] = true;
+
+                return $arg;
+            }
+        }
+
+        return null;
+    }
+
+    public function complete() : void
+    {
+        foreach ( $this->argv as $pos => $arg )
+            if ( $this->used[ $pos ] == false )
+                fwrite( STDERR , "Unknown argument: {$arg}\n\n" );
+    }
+
+    public function residual() : array
+    {
+        return array_filter( $this->argv );
+    }
+}
diff --git a/scripts/translation/libqa/OutputBuffer.php b/scripts/translation/libqa/OutputBuffer.php
@@ -35,10 +35,7 @@ public function __construct( string $header , string $filename , OutputIgnore $i
         $this->header = $header . ": " . $filename . "\n\n";
         $this->filename = $filename;
         $this->ignore = $ignore;
-
-        $copy = $ignore->residualArgv;
-        array_shift( $copy );
-        $this->options = implode( " " , $copy );
+        $this->options = implode( " " , $ignore->argv->residual() );
     }
 
     public function add( string $text )
@@ -67,7 +64,7 @@ public function addDiff( string $text , int $sourceCount , int $targetCount )
 
     public function addFooter( string $text )
     {
-        // $this->footer[] = $text;
+        $this->footer[] = $text;
     }
 
     public function addLine()
@@ -76,15 +73,24 @@ public function addLine()
             $this->add( "\n" );
     }
 
+    public function contains( string $text ) : bool
+    {
+        foreach( $this->matter as $line )
+            if ( str_contains( $line , $text ) )
+                return true;
+        return false;
+    }
+
     public function print( bool $useAlternatePrinting = false )
     {
         if ( count( $this->matter ) == 0 && count( $this->footer ) == 0 )
             return;
 
+        $hashFile = hash( "crc32b" , $this->filename );
         $hashHead = $this->hash( false );
         $hashFull = $this->hash( true );
 
-        if ( $this->ignore->shouldIgnore( $this , $this->filename , $hashHead , $hashFull ) )
+        if ( $this->ignore->shouldIgnore( $this , $hashFile , $hashHead , $hashFull ) )
             return;
 
         print $this->header;
@@ -103,6 +109,8 @@ public function print( bool $useAlternatePrinting = false )
 
         if ( count( $this->footer ) )
             print "\n";
+
+        print "\n";
     }
 
     private function printMatterAlternate() : void

diff --git a/scripts/translation/libqa/OutputIgnore.php b/scripts/translation/libqa/OutputIgnore.php
@@ -20,57 +20,53 @@
 
 class OutputIgnore
 {
-    public array $residualArgv;
-
     private bool   $appendIgnores = true;
     private bool   $showIgnore = true;
-    private string $filename = ".syncxml.ignores";
+    private string $filename = ".qaxml.ignores";
     private string $argv0 = "";
 
-    public function __construct( array & $argv )
+    public ArgvParser $argv;
+
+    public function __construct( ArgvParser $argv )
     {
-        $this->argv0 = escapeshellarg( $argv[0] );
+        $this->argv = $argv;
+        $this->argv0 = escapeshellarg( $argv->consume( position: 0 ) );
 
-        foreach( $argv as $key => $arg )
-        {
-            if ( str_starts_with( $arg , "--add-ignore=" ) )
-            {
-                $list = $this->loadIgnores();
-                $line = substr( $arg , 13 );
-                if ( ! in_array( $line , $list ) )
-                {
-                    $list[] = $line;
-                    $this->saveIgnores( $list );
-                }
-                exit;
-            }
+        $arg = $argv->consume( prefix: "--add-ignore=" );
 
-            if ( str_starts_with( $arg , "--del-ignore=" ) )
+        if ( $arg != null )
+        {
+            $item = substr( $arg , 13 );
+            $list = $this->loadIgnores();
+            if ( ! in_array( $item , $list ) )
             {
-                $list = $this->loadIgnores();
-                $line = substr( $arg , 13 );
-                $dels = 0;
-                while ( in_array( $line , $list ) )
-                {
-                    $key = array_search( $line , $list );
-                    unset( $list[$key] );
-                    $dels++;
-                }
-                if ( $dels == 0 )
-                    print "Ignore mark not found.\n";
-                else
-                    $this->saveIgnores( $list );
-                exit;
+                $list[] = $item;
+                $this->saveIgnores( $list );
             }
+            exit;
+        }
 
-            if ( $arg == "--disable-ignore" )
+        $arg = $argv->consume( prefix: "--del-ignore=" );
+        if ( $arg != null )
+        {
+            $item = substr( $arg , 13 );
+            $list = $this->loadIgnores();
+            $dels = 0;
+            while ( in_array( $item , $list ) )
             {
-                $this->showIgnore = false;
-                unset( $argv[$key] );
+                $key = array_search( $item , $list );
+                unset( $list[$key] );
+                $dels++;
             }
+            if ( $dels == 0 )
+                print "Ignore mark not found.\n";
+            else
+                $this->saveIgnores( $list );
+            exit;
         }
 
-        $this->residualArgv = $argv;
+        if ( $argv->consume( "--disable-ignore" ) != null )
+            $this->showIgnore = false;
     }
 
     private function loadIgnores()
@@ -87,36 +83,28 @@ public function saveIgnores( $data )
         file_put_contents( $this->filename , $contents );
     }
 
-    public function shouldIgnore( OutputBuffer $output , string $filename , string $hashHeader , string $hashMatter )
+    public function shouldIgnore( OutputBuffer $output , string $hashFile , string $hashHeader , string $hashMatter )
     {
         $ret = false;
 
-        $prefix = "{$filename}:{$hashHeader}:";
-        $ignore = "{$filename}:{$hashHeader}:{$hashMatter}";
+        $prefix = "{$hashFile}-{$hashHeader}-";
+        $active = "{$hashFile}-{$hashHeader}-{$hashMatter}";
         $marks = $this->loadIgnores();
 
         // --add-ignore command
 
-        if ( in_array( $ignore , $marks ) )
-            $ret = true;                        // is already ignored
-        else                                    //
-            if ( $this->showIgnore )            // show add command
-                $output->addFooter( "  php {$this->argv0} --add-ignore=$ignore\n" );
-
-        // Remove valid ignores, leaves outdated ones for listing
-
-        while ( in_array( $ignore , $marks ) )
-        {
-            $key = array_search( $ignore , $marks );
-            unset( $marks[$key] );
-        }
+        if ( in_array( $active , $marks ) )
+            $ret = true;
+        else
+            if ( $this->showIgnore )
+                $output->addFooter( "  php {$this->argv0} --add-ignore=$active\n" );
 
         // --del-ignore command
 
-        if ( $this->showIgnore )                // show del commands (for this file/prefix)
+        if ( $this->showIgnore )
             foreach ( $marks as $mark )
-                if ( $mark != null )
-                    if ( str_starts_with( $mark , $prefix ) )
+                if ( str_starts_with( $mark , $prefix ) )
+                    if ( $mark != $active )
                         $output->addFooter( "  php {$this->argv0} --del-ignore=$mark\n" );
 
         return $ret;

diff --git a/scripts/translation/libqa/all.php b/scripts/translation/libqa/all.php
@@ -18,6 +18,7 @@
 ini_set( 'display_startup_errors' , 1 );
 error_reporting( E_ALL );
 
+require_once __DIR__ . '/ArgvParser.php';
 require_once __DIR__ . '/OutputBuffer.php';
 require_once __DIR__ . '/OutputIgnore.php';
 require_once __DIR__ . '/SyncFileList.php';