Skip to content

INSTANCES: versions, REST hosts, clusters, DataBases

Jump to bottom Edit New page
Stefano Belforte edited this page May 11, 2020 · 12 revisions

Executive Summary

  • CRABClient and TaskWorker will allow to indicate the REST endpoint to use to access CRAB data base as either one of 1. or 2. below:
  1. one of a small set of predefined nicknames :
nickname REST host DB instance
prod cmsweb.cern.ch prod
preprod cmsweb-testbed.cern.ch preprod
k8s cmsweb-k8s-testbed.cern.ch preprod
test cmsweb-test2.cern.ch dev
dev cmsweb-test.cern.ch dev
  1. one pair of strings indicating the REST host fqdn and the DB instance [prod|preprod|dev]
  • QUESTION1: Do we want to support all nicknames test[1-6] as pointing to cmsweb-test[1-6] and dev DB ? In which case, do we still have an use for cmsweb-test and/or cmsweb-dev ?
  • QUESTION2: Is it OK to have this table hardcoded in Client and TaskWorker ? Or do we want to have it in a file on the web, dowloaded when used, and thus able to change at any time ?

CRAB CLIENT

  • NOTE need keep compatibility with current situation where CRABClient users do not need to specify the CRAB service instance, in which case it defaults to "prod"
  • THEREFORE in CRABClient the restHost/dbInstance mode is activate via setting
    • config.General.instance = 'other'
    • in which case the configuration file must contain:
      • config.General.restHost = 'stefanovm.cern.ch # or similar fqdn
      • config.General.dbInstance = one of [prod|preprod|dev]

TASK WORKER

  • Do exactly like for CRABClient, but there is no default instance.
  • Make things more clear by renaming config.TaskWorker.mode to config.TaskWorker.instance
  • THEREFORE in TaskWOrker the restHost/dbInstance mode is activate via setting
    • config.TaskWorker.instance = 'other'
    • in which case the configuration file must contain:
      • config.TaskWorker.restHost = 'stefanovm.cern.ch # or similar fqdn
      • config.TaskWorker.dbInstance = one of [prod|preprod|dev]

Concepts and definitions

  • CRABServer service is a REST interface to CRAB Oracle Data Base
  • A given REST service is usually a DNS alias for a set of actual hosts which implement load balance and HighAvailability. The external users only see the DNS alias which will be called restHost.
  • CRABServer runs inside CMSWEB framework, so it is part of a given CMSWEB cluster
    • Numerous CMSWEB clusters exist
      • cmsweb.cern.ch aka main production one
      • cmsweb-testbed.cern.ch aka testbed
      • cmsweb-k8s-testbed.cern.ch supposedly identical to cmsweb-testbed
      • cmsweb-test.cern.ch K8s developemnt (Valentin's playground)
      • cmsweb-test[1-6].cern.ch test (developers') clusters for application developers
        • cmsweb-test2.cern.ch is reserved for CRAB usage
      • private VM's like stefanovm.cern.ch or stefanovm2
  • Oracle Data Base has several instances, meaning "different data bases"
    • Production on CMS Production Oracle cluster cmsr
    • Preprod on devdb11 username: cmsweb_analysis_preprod
    • Dev on devdb11 username: cmsweb_analysis_dev
    • private like Stefano's or Diego's private DB's on devdb11
  • while Oracle DBA's usually refer to cmsr or devdb11 as instances (again)
  • CRABClient allows to submit to a given 'CRAB instance' which means a given Data Base instance: global (i.e. production) or preprod or dev etc.

History and evolving requirements

  • CRAB was developed at a time when it was easy to get multipl DB instances, but almost inconceivable to have more than two cmsweb clusters (cmsweb.cern.ch and cmsweb-testbed.cern.ch) therefore
    • one CRABServer REST instance is capable to connect to multiple DataBases, i.e. support multiple DB instances
  • So the DataBase instance (prod/preprod/dev) could not be part of the CRABServer Rest configuration, but it was specified as something that the client (clients of the CRBServer REST are CRABClient and CRABTaskWorker) indicates in the URL (API) used. Which is constructed as hostname/crabserver/dbinstance/API
  • in the initial design the view was: the CRABClient (i.e. who submits) is only interested in deciding if to submit to the production or preproduction DataBase (or some private test instance) so the CRABClient configuration file accepts the parameter config.General.instance and "CRAB" would figure out everything
  • in the migration to K8s we have multiple several cmsweb clusters, i.e. multiple REST instances which may e.g. all connect to the same DB instance and want to be able to connect explicitly to one or another such clusters in order to test specific REST instances.

Implementation details and configurations as of April 2020

CRABCache i.e. FileCache

CrabClient and TaskWorker communicate via the Oracle CRAB DataBase (so need a REST hostname and a DB instance name), but the CRABClient also needs to upload a sandbox to be used in job submission. FOr this it uses a dedicated CRABCache service, which also has a REST interface. The URL to be used for the CRABCache file is obtained by querying the CRABServer REST. In other words, each CRABServer REST instance knows which CRABCache service should be used and communicates this to both CRAB Client and CRAB TaskWorker via the query

https://<restHost>/crabserver/<dbInstance>/info?subresource=backendurls

e.g.

https://cmsweb.cern.ch/crabserver/prod/info?subresource=backendurls

CRABServer

import cx_Oracle as DB
import socket
fqdn = socket.getfqdn().lower()
dbconfig = {'preprod': {'.title': 'Pre-production',
                        '.order': 1,
                        '*': {'clientid': 'cmsweb-preprod@%s' %(fqdn),
                              'dsn': 'devdb11',
                              'liveness': 'select sysdate from dual',
                              'password': '*****' ,
                              'schema': 'cmsweb_analysis_preprod',
                              'timeout': 300,
                              'trace': True,
                              'type': DB,
                              'user': 'cmsweb_analysis_preprod'}},
            'prod': {'.title': 'Production',
                     '.order': 0,
                     'GET': {'clientid': 'cmsweb-prod-r@%s' %(fqdn),
                             'dsn': 'cmsr',
                             'liveness': 'select sysdate from dual',
                             'password': '*****',
                             'schema': 'cms_analysis_reqmgr_r',
                             'timeout': 300,
                             'trace': True,
                             'type': DB,
                             'user': 'cms_analysis_reqmgr_r'},
                     '*':  {'clientid': 'cmsweb-prod-w@%s' %(fqdn),
                            'dsn': 'cmsr',
                            'liveness': 'select sysdate from dual',
                            'password': '******',
                            'schema': 'cms_analysis_reqmgr_w',
                            'timeout': 300,
                            'trace': True,
                            'type': DB,
                            'user': 'cms_analysis_reqmgr_w'}}}

since this CRABServerAuth.py file contains passwords, they are not kept in publicly available repositories.

  • the CRABServer REST API machinery detects the Data Base instance from the URL in the HTTP request and selects the appropriate Oracle connection instance.

CRABClient

  • CRABClient configuration file accepts the parameter config.General.instance which can also be passed as an option in the command line, and e.g. crab submit --help lists this option:
 --instance=INSTANCE   Running instance of CRAB service. Valid values are
                        ['test1', 'test3', 'test2', 'prod', 'preprod', 'test',
                        'k8s'].

where it is apparent how in January we added some K8s cluster overloading the parameter "instance" to indicate a particular REST instance instead of the DB instance.

  • this was justified since already config.General.instance was used to indicate a particular REST host in order to support submission to private developer VM's via thins like General.instance = 'stefanovm2.cern.ch'
  • this require that there is always a 1:1 mapping between Data Base instance and REST Host instance, so that the CRAB Client can figure out the two (needed to build the HTTP queries) from a single parameter.
  • the code which maps the General.instance parameter into a REST hostname and a DataBase instance is in https://github.com/dmwm/CRABClient/blob/301de634b1fe16bf11696d975133487cd0094d37/src/python/CRABClient/ClientUtilities.py#L195

CRABTaskWorker

As an user of CRAB DataBase each TaskWorker instance need to identify one REST host to talk to and the DB instance to use.

  • there is a set of pre-defined host/instance pair in the code, each TW instance can pick one of those via the configuration parameter config.TaskWorker.mode in the TaskWorkerConfig.py file. Relevant code is in MasterWorker.py where the value of this configuration parameter is called MODEURL:
MODEURL = {'cmsweb-dev': {'host': 'cmsweb-dev.cern.ch', 'instance':  'dev'},
           'cmsweb-test': {'host': 'cmsweb-test.cern.ch', 'instance': 'preprod'},
           'cmsweb-preprod': {'host': 'cmsweb-testbed.cern.ch', 'instance': 'preprod'},
           'cmsweb-prod': {'host': 'cmsweb.cern.ch', 'instance':  'prod'},
           'test' :{'host': None, 'instance': 'preprod'},
           'private': {'host': None, 'instance':  'dev'},
          }
  • if mode is set to 'test' or 'private', then the host name for the REST needs to be specified in the TaskWorkerConfig.py configuration file via the (badly named) parameter config.TaskWorker.resturl e.g.:
config.TaskWorker.resturl = 'stefanovm.cern.ch'

Changes to better manage current situation

1. CRAB Client

Modify CRAB Client so that the submitter can select REST host and Data Base instance independently

  • be backward compatible with pre-2020 use (it is OK to break compatibility for K8s clusters)
  • introduce instance='other' as a switch to allow specifying restHost and dbInstance
  • get rid of old instance='private' which in the end was a confusing way to allow a indicate restHost while forcing data base instance to dev

2. CRAB Task Worker

Should do like for CrabClient, while taking advantage that here we have freedome with configuration file. keep a smaller set of nicknames (MODEURLs) where both REST host and DB instance are hardcoded. Support also MODERUL='other' which incorporates old test/private in which case both instance and url must be specified :

Add a custom footer
Add a custom sidebar
Clone this wiki locally