Package Ganga :: Package GPIDev :: Package Lib :: Package Job :: Module Job' :: Class Job :: Class _proxyClass

Class _proxyClass

    object --+    
             |    
GPIProxyObject --+
                 |
                Lib.Job.Job'.Job._proxyClass

Job is an interface for submision, killing and querying the jobs :-).

    Basic configuration:

    The    "application"    attribute    defines   what    should    be
    run. Applications  may be generic arbitrary  executable scripts or
    complex, predefined objects.

    The  "backend" attribute  defines  where and  how  to run.  Backend
    object  represents  a resource  or  a  batch  system with  various
    configuration parameters.

    Available applications, backends and other job components may be listed
    using the plugins() function. See help on plugins() function.

    The "status"  attribute represents the  state of Ganga  job object.
    It  is automatically  updated  by the  monitoring  loop. Note  that
    typically at the  backends the jobs have their  own, more detailed
    status.    This    information   is   typically    available   via
    "job.backend.status" attribute.

    Bookkeeping and persistency:
    
    Job objects contain basic book-keeping information: "id", "status"
    and  "name".  Job  objects   are  automatically  saved  in  a  job
    repository which may be a  special directory on a local filesystem
    or a remote database.

    Input/output and file workspace:
    
    There   is  an  input/output   directory  called   file  workspace
    associated with each  job ("inputdir" and "outputdir" properties).
    When a  job is submitted, all  input files are copied  to the file
    workspace  to keep  consistency  of  the input  while  the job  is
    running. Ganga then ships all  files in the input workspace to the
    backend systems in a sandbox.

    The   list  of  input   files  is   defined  by   the  application
    (implicitly).  Additional files may be explicitly specified in the
    "inputsandbox" attribute.

    Job splitting:

    The "splitter" attributes defines how a large job may be divided into
    smaller subjobs.  The subjobs are automatically created when the main
    (master) job is submitted. The "subjobs" attribute gives access to
    individual subjobs. The "master" attribute of a subjob points back to the
    master job.

    Merging:

    The "merger" attribute defines how the output of the subjobs may be merged.
    Merging is not perfromed automatically and it is triggered by the merge() method.
        
    Datasets: PENDING
    Datasets are highly application and virtual organisation specific.
    

Properties:

     info             JobInfo . ('jobinfos' object, default=None,comparable)

     status           current state of the job, one of "new", "submitted",
                      "running", "completed", "killed", "unknown", "incomplete".
                      (simple property, default='new',protected,comparable)

     inputdata        dataset definition (typically this is specific either to
                      an application, a site or the virtual organization.
                      ('datasets' object, default=None,comparable,optional)

     name             optional label which may be any combination of ASCII
                      characters. (simple property, default='',comparable)

     inputdir         location of input directory (file workspace). (simple
                      property, default=None,transient,protected,optional)

     merger           optional output merger. ('mergers' object,
                      default=None,comparable,optional)

     do_auto_resubmit Automatically resubmit failed subjobs. (simple property,
                      default=False,comparable)

     inputsandbox     list of File objects shipped to the worker node . ('files'
                      object, list, default=[],comparable)

     outputdir        location of output directory (file workspace). (simple
                      property, default=None,transient,protected,optional)

     fqid             fully qualified job identifier. (simple property,
                      default=None,transient,protected,optional)

     application      specification of the application to be executed.
                      ('applications' object, default=None,comparable)

     master           master job. ('jobs' object,
                      default=None,transient,protected,optional)

     outputdata       dataset definition (typically this is specific either to
                      an application, a site or the virtual organization.
                      ('datasets' object, default=None,comparable,optional)

     time             provides timestamps for status transitions. ('jobtime'
                      object, default=None,protected)

     splitter         optional splitter. ('splitters' object,
                      default=None,comparable,optional)

     outputsandbox    list of filenames or patterns shipped from the worker
                      node. (simple property, list, default=[],comparable)

     subjobs          list of subjobs (if splitting). ('jobs' object, list,
                      default=[],protected,comparable,optional)

     id               unique Ganga job identifier generated automatically.
                      (simple property, default='',protected)

     backend          specification of the resources to be used (e.g. batch
                      system). ('backends' object, default=None,comparable)

Nested Classes

[hide private]

_impl
This is a Ganga.GPI.Job implementation class.

Instance Methods

[hide private]

__eq__(self, x)
Equality operator (==), compare the Job properties which are declared as [comparable].

source code

__init__(self, *args, **kwds)
GPI Job object constructor: Job() : create job with default settings; Job(j) : make a copy of j; Job(j,x=a,...): make a copy of j and set property 'x' to a, etc..

source code

__ne__(self, x)
Non-equality operator (!=).

source code

__repr__(self)
Return an short representation of Job object.

source code

__setattr__(self, x, v)
Set a property of Job with consistency and safety checks.

source code

__str__(self)
Return a printable string representing Job object as a tree of properties.

source code

copy(self)
Make an identical copy of self.

source code

fail(...)
Deprecated.

source code

force_status(...)
Force job to enter the "failed" or "completed" state.

source code

kill(...)
Kill the job.

source code

merge(...)
Merge the output of subjobs.

source code

peek(...)
Allow viewing of job output (and input) files

source code

remove(...)
Remove the job.

source code

resubmit(...)
Resubmit a failed or completed job.

source code

submit(...)
Submits a job.

source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __sizeof__, __subclasshook__

Class Variables

[hide private]

application = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor obj...

backend = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object ...

do_auto_resubmit = <Ganga.GPIDev.Base.Proxy.ProxyDataDescripto...

fqid = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at ...

id = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x...

info = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at ...

inputdata = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor objec...

inputdir = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object...

inputsandbox = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor ob...

master = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object a...

merger = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object a...

name = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at ...

outputdata = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor obje...

outputdir = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor objec...

outputsandbox = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor o...

splitter = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object...

status = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object a...

subjobs = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object ...

time = <Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at ...

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

init(self, *args, **kwds)
(Constructor)

source code

GPI Job object constructor: Job() : create job with default settings; Job(j) : make a copy of j; Job(j,x=a,...): make a copy of j and set property 'x' to a, etc..

Overrides: object.__init__

repr(self)
(Representation operator)

source code

Return an short representation of Job object.

Overrides: object.__repr__

setattr(self, x, v)

source code

Set a property of Job with consistency and safety checks. Setting a [protected] or a unexisting property raises AttributeError.

Overrides: object.__setattr__

str(self)
(Informal representation operator)

source code

Return a printable string representing Job object as a tree of properties.

Overrides: object.__str__

fail(...)

source code

Deprecated. Use force_status('failed') instead.

force_status(...)

source code

Force job to enter the "failed" or "completed" state. This may be used for marking jobs "bad" jobs or jobs which are stuck in one of the internal ganga states (e.g. completing).

To see the list of allowed states do: job.force_status(None)

kill(...)

source code

Kill the job. Raise JobError exception on error.

merge(...)

source code

Merge the output of subjobs.

By default merge all subjobs into the master outputdir. The output location and the list of subjobs may be overriden. The options (keyword arguments) are passed onto the specific merger implementation. Refer to the specific merger documentation for more information about available options.

peek(...)

source code


Allow viewing of job output (and input) files

Arguments other than self:
filename : name of file to be viewed
           => For backends where this is enabled, the filename
              for a job in the "running" state is relative to
              the job's work directory unless the filename begins
              with "../".  In all other cases, the filename is
              relative to the job's output directory
command  : command to be used for viewing the file
           => If no command is given, then the command defined in
              the [File_Associations] section of the Ganga
              configuration file(s) is used

Example usage:

   # examine contents of output/work directory
   peek()

   # examine contents of output directory,
   # even in case of job in "running" state
   peek( "../output" )

   # examine contents of input directory
   peek( "../input" )

   # View ROOT histograms, running root.exe in a new terminal window
   peek( "histograms.root", "root.exe &&" )

   # View contents of file in output/work directory, using
   # command defined in configuration file
   peek( "output.txt" )

   # View ROOT histograms in ouput/work directory,
   # running root.exe in a new terminal window
   peek( "histograms.root", "root.exe &&" )
              
Return value: None

remove(...)

source code

Remove the job.

If job has been submitted try to kill it first. Then remove the file workspace associated with the job.

If force=True then remove job without killing it.

resubmit(...)

source code

Resubmit a failed or completed job.  A backend object may
be specified to change some submission parameters (which
parameters may be effectively changed depends on a
particular backend implementation).

Example:
 b = j.backend.copy()
 b.CE = 'some CE'
 j.resubmit(backend=b)

Note: it is not possible to change the type of the backend in this way.

submit(...)

source code

Submits a job. Return true on success.

First the application is configured which may generate additional input files, preprocess executable scripts etc. Then backend handler is used to submit the configured job. The job is automatically checkpointed to persistent storage.

The default values of keep_going and keep_on_fail are controlled by [GPI_Semantics] configuration options.

When the submission fails the job status is automatically reverted to new and all files in the input directory are deleted (keep_on_fail=False is the default behaviour unless modified in configuration). If keep_on_fail=True then the job status is moved to the failed status and input directory is left intact. This is helpful for debugging anf implements the request #43143.

For split jobs: consult https://twiki.cern.ch/twiki/bin/view/ArdaGrid/GangaSplitters#Subjob_submission

Class Variable Details

[hide private]

application

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9410>

backend

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9810>

do_auto_resubmit

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9210>

fqid

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9390>

id

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9790>

info

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x189ded0>

inputdata

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x189dfd0>

inputdir

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9110>

inputsandbox

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9290>

master

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9490>

merger

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9190>

name

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9090>

outputdata

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9510>

outputdir

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9310>

outputsandbox

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9690>

splitter

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9610>

status

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x189df50>

subjobs

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9710>

time

Value:

<Ganga.GPIDev.Base.Proxy.ProxyDataDescriptor object at 0x18a9590>

Class _proxyClass

__init__(self, *args, **kwds) (Constructor)

__repr__(self) (Representation operator)

__setattr__(self, x, v)

__str__(self) (Informal representation operator)

fail(...)

force_status(...)

kill(...)

merge(...)

peek(...)

remove(...)

resubmit(...)

submit(...)

application

backend

do_auto_resubmit

fqid

id

info

inputdata

inputdir

inputsandbox

master

merger

name

outputdata

outputdir

outputsandbox

splitter

status

subjobs

time

init(self, *args, **kwds)
(Constructor)

repr(self)
(Representation operator)

setattr(self, x, v)

str(self)
(Informal representation operator)