Problem
SAGE addresses problems associated with complex
processes involving large amounts of data, many files, and many
tools in IC hardware development. Often there are dozens of steps,
executed over many iterations, on a large collection of evolving
data files. Even in the hands of a seasoned expert, these task
flows are error prone.
Add to that the real-life dynamics of flows in which
unexpected problems occur and workarounds involving new tools
or scripts are inserted into the allegedly "golden"
flow that worked last time, assuming anyone really knows what
that was!
SAGE conceptually categorizes these types of problems
into five areas:
· Data
Integrity
· Flow
Standardization
· Flow
Integrity
· Flow
Management
· Project
Tracking
First, there is the integrity of the data. Was the correct version
of each file used in the process, or did some out-of-date or stray
creep in to the process? What records were kept to prove
this?
Next, how do you capture the expert knowledge that
went into the development of the flow in an understandable format?
Not everyone is an expert, and maybe he left last week. We call
this problem flow standardization. This includes the details about
invoking tools. What are the specific commands and command-line
arguments? What environment variables are necessary for a tool?
This is known as tool encapsulation.
Given that the correct data are used, were all the
steps performed correctly so far? Were the right tools used at
the right time? And what step is next? We call this flow integrity.
Once there is a golden flow that everyone understands,
what happens when it tarnishes? Some bug in some tool crops up
that requires a complicated workaround. Or perhaps a new tool
needs to be added that replaces or supplements the old flow. Often
an expert (you hope) gets in and tackles this, weaving a bunch
of scripts together. Now that that's done, how do you capture
it, maintain it, and deploy it elsewhere? This is the problem
of flow management.
Finally, given a flow and all the tools and files
involved in your complex project, how can you tell where you're
right now? What jobs ran? Which ones failed or passed? Where are
the reports associated with that step in the flow? This all comes
under the heading of project tracking.
Features
SAGE tackles these problems using a
rich set of features and standards. For rapid startup deployment,
SAGE has standard directory structures for tools, libraries, and
projects. These can also be customized to accommodate existing
infrastructures.
For data integrity, SAGE manages the location and
status of the tools, files, and scripts in the flow. SAGE understands
their logical dependencies, knowing whether they're up-to-date
not. In this sense, SAGE acts like UNIX make.
Keeping track of all the pieces of a flow is a difficult
task. Usually, this is done on paper, after the fact, and with
varying amounts of accuracy. SAGE provides a living, working vision
of the flow, down to the lowest level of detail. The SAGE model
of a flow is a working standard that can be understood and re-used.
For flow integrity, SAGE maintains a database identifying
the exact tools used, when they were used, and what the results
were. At any given time, SAGE tracks what tasks are ready
to perform next.
As a flow evolves new steps are added, some are
deleted, and file and tool locations may change. SAGE includes
a flow editor, providing the means to adapt flows, capture them,
and implement them.
SAGE controls the operation of the flow by launching
jobs, sequentially or in parallel, waiting for previous steps
to complete or for the required prerequisites to be met. SAGE
can run jobs locally, or distribute them using any job queuing
system.
Given this detailed knowledge about the flow, SAGE
is in a position to provide up-to-the-minute job status for each
file tracked by SAGE. The metrics and reports collected at then
end of each job reflecting the success or failure of the job as
well as the tool-specific results are available for display. SAGE
keeps track of the tool log files and provides an environment
for summarizing them at any time.
SAGE comprises these major elements:
