Migrating a large, multi-team, simulation environment.
Storage Technology Corporation
We had a large legacy simulation environment comprised of thousand's of source files, representing many different boards and ASICs. Many different designers, and development teams, used this environment for creating new designs, as well as supporting old designs. Data management was rudimentary which led to many time consuming problems during regression testing and/or re-creating a previous simulation environment. This paper describes the migration of this environment to a more stable environment and the use of ClioSoft's SOS tool in the data/configuration management process.
We have a very large verilog simulation environment The source files in this environment represent all the boards and ASICs that have been developed over the last 9 years. Some of this legacy code is still in use today, while other code serves as the basis for development of newer versions of boards and ASICs. In addition new code is being added that represents completely new boards and ASICs. This environment is constantly changing, with code being modified and/or created by multiple design teams, normally each at a different stage of development, consisting of 6-12 designers with varying levels of experience.
Some of the reasons that convinced us that we needed to migrate to a better design data management system were:
· A complex simulation environment. With frequent sharing of design data between, and within, design teams it was difficult to keep a “stable” environment for very long. This was not going to work well when migrating to a compiled simulator, such as VCS™, Model Tech™, or NC-Verilog™, where all the environment changes would cause unnecessary recompiles.
· Our ASIC vendor was no longer going to be supporting our development flow. We previously used Mentor Graphics Quicksim™ for ASIC signoff. But with 0.2u, and newer ASICs, verilog was going to have to be our signoff simulator.
· The re-creation of two previously simulated designs was a problem. The first was a processor and memory upgrade, with the goal of quick verification. The design team had difficulty in re-creating the verification environment and an estimated 2-3 week project took 10 weeks. The second entailed the validation of different versions of the same board. Multiple versions of the processor board were in use in the engineering lab. Analysis of various failures, on the different versions, was difficult because the design team could not recreate the multiple board versions for simulation.
This paper describes the steps we took to migrate our simulation environment to a more board centric environment, that is, organizing our design data around a particular board type. By doing this and improving our management of our design data we could improve our design process and enable better team collaboration.
2.0 Past Environment
There was an ad-hoc approach to data management. RCS was generally limited to verilog source files. Support files which configured the simulation environment, and drove the testbenches, sometimes had no revision control at all. In addition, for those files that had revision control, there was no tracking of what revisions went together to represent a particular working configuration. This made it nearly impossible to later determine the exact simulation environment that was used before sending a board and/or ASIC to layout? Also, because of this some designs were only simulated with “compatible” hardware, that is older boards that would never be present in a real sub-system. Conversely, the same design would not be simulated with the boards that would be in a real sub-system.
The different types of files were checked into a flat file space with prefixes providing the only clue of ownership. Over 1800 source files, and 5000 support files, representing approximately 18 different boards and ASICs. With the one directory approach, some designers only checked-in known good working code at the completion of their design. This meant that all the files were Revision 1.1. and the detailed development history resided with the original designer. When that designer went to another project , or left the company, that design history was lost..
There were many data dependancies that were caused by every board and ASIC referencing global header files. These header files contained verilog “`defines”, that could change almost daily as other design teams added or changed their definitions. Additionally, there are design support files, containing test specific `defines, which configure the simulation environment for that test i.e. which boards to use, how many processors to use, etc.. These test specific files were read after the global files causing many warning messages about "Text Macro Redefinition”, making it very difficult to analyze log files and find configuration problems.
Each design team, and even each designer, was free to handle their design workspace however they wanted. Thus it was difficult for designers to support each other since there was usually a learning curve involved with learning how another design team, or designer, configured their environment and ran their simulations.
There was no managed control of when the central area, the flat file space containing the files for the other boards/ASICs a designer was using in their simulation, was updated. It was easy to make a mistake that could impact ALL design teams since everyone used some portion of the central area. Because anyone could change this central area at any time, and a design’s regression run could take 4-5 days, it was possible for the simulation environment to “break” in the middle of a teams regression run. It could be very time consuming to figure what had changed and resolve the problem.
The verilog library search path was the same for all boards and ASICs since it was determined by the verilog command line options –y and –v. The long search paths, coupled with the large number of files in some of the directories, slowed down compilation.
New Designers (and even experienced ones) had difficulty in this environment.
It was clear that our problem had two parts, a data usage component and a data management component. We wanted to create a more stable environment for each design team as well as for each designer. This would help to avoid, or minimize, situations where one designers/teams changes could impact another designers/teams development. To achieve this we determined that Data and Configuration Management was essential and came up with the following goals.
· Let each board and/or ASIC have their own file space. The goal being that each board/ASIC should stand and compile on its own.
· Minimize data inter-dependencies. Header files are localized to each board, or ASIC, with a “board.vh” or “ASIC.vh” file. The “.vh” file contains definitions that are specific to that design. Global only definitions can be found in the “motherboard.vh” file.
· Compartmentalize library searching for each board and/or ASIC by using the verilog “`uselib” directive. This is more efficient because you can optimize the library search path and search only those libraries necessary for this design, speeding up compilation. This also eliminates the duplicate module name problem.
· Obvious configurable parameters within the board moved up to the board interface level. These parameters would then be passed down from the motherboard to the board at simulation run-time.
· Log files shall be warning free, no text macro redefinitions are allowed; Test specific `define’s are read first to see what hardware is needed. This is preferable with a compiled simulator to minimize recompilation and will aid in the debugging.
· Only Production hardware configurations to be simulated, verified, and then stored as a working configuration for future retrieval.
· Be able to re-create any stage in the design, from RTL to gates, in any configuration.
4.0 The Board Centric Data Management Environment
In our simulation environment we do sub-system level simulations using multiple types of boards and ASICs. For this reason it made sense to organize our design data around a board type. Each board type has an identical file structure to hold different pieces of the design data. Some examples of the directories are:
· Bin – This holds tools/scripts specific to the board.
· Syn – This holds the synthesis scripts for the ASICs on the board. Most often broken into specific ASIC subdirectories.
· BLM – This holds the behaviourable and synthesizeable RTL code for the board and ASICs. Sometimes broken into ASIC specific subdirectories.
· SSP – This System Simulation Plan directory holds the verilog command files for running simulations. It contains many subdirectories for such things as test specific definitions, “object” files necessary to support the tests, verilog logfiles, etc..
In addition, a design team typically consists of a project lead, several design engineers creating and modifying the RTL code, and a verification engineer who develops the testbenches. Each of these users requires a “workspace” where they can do their development before making their work available to other team members. Each of these workspaces needed to be isolated, so that designers would feel free to check in/out of a repository as often as necessary without fear of breaking the simulation environment. At the same time, each designer needs a set of “stable” data that they can reference for the other boards and ASICs they need to support their tests. By referencing this stable area they can minimize the amount of data in their workspace. By doing these two things, creating isolated workspaces and a stable reference area, we can improve design team and designer collaboration. Figure 1 shows the concept and structures we use to accomplish this.
Figure 1. Data hierarchy and usage model.
· Repository – This holds the history of all our design data, boards and ASICs. It contains the revision history of all source files, scripts, test definitions, etc.. It is the source from which all “workspaces” are created (working, regression, stable) and provides each designer with visibility, and access, into all the project data. It is a secure area owned by a special “administrator” account so that designers cannot accidentally delete the repository.
· Stable – This workspace is created from files in the Repository and holds the RTL code for all known good boards/ASICs. By referencing the files in stable, designers are able to find other working boards/ASICs for use in their own simulation tests. This means that the designer only needs local copies of the files for the board/ASIC they are working with. In order to make the simulation environment more difficult to break there are predefined checkpoints that a designs RTL must go through in order to be placed in stable.. All RTL code that is a candidate for stable is first verified in the “stable regression” workspace. The tests run in stable regression are selected by each design team, from their boards regression tests, to check the basic functionality of their board and its interfaces. After these tests have passed the files are “tagged”, a “snapshot” is taken of the stable area to save its configuration, and then the stable workspace is updated with this new RTL code. This workspace has revision control.
· Regression – This workspace is also created from files in the Repository and is the workspace where a board/ASIC is verified to be correct. This workspace is generally used by the verification engineer to run the designs regression tests. The design must simulate clean here before moving into stable, one of the checkpoint requirements. After the design has passed the files are “tagged” and submitted as a stable candidate. This workspace has revision control.
· Working – This workspace is also created from files in the Repository and is the workspace in which the designer works on changes for the design, or creates a new design. The local development directories are identical in structure to the Regression, Stable, and other team members workspaces, except that it only contains design data for the design of interest. When a designer decides his portion of the design is ready for regression, he will “tag” his files and submit them to his teams regression workspace.This workspace has revision control.
The following figure, Figure 2, show the typical flow of RTL code from a designers workspace, through his teams regression workspace, and into stable.
Figure 2. Typical flow of RTL code to “stable”.
5.0 Design Data Management – The Choices and Decision
We had several choices of tools for our data management needs. Freely available tools, such as RCS and CVS, as well as third-party tools. Because there were no resources allocated for tool development, onging tool administration, and user training, these had to be kept to a minimum. The tool had to be simple, intuitive, and easy to use, so that a designer would not have any reason not to use it . Tools that were manually intensive, and therefore harder to use, were avoided.
RCS provided only basic capabilities and is not very well suited to managing a large project with many teams, designers, files and directories. It is manually intensive with only a command line interface. It was felt that extensive script development, and user training in how to use them, would have been necessary to handle our environment. The scripts and user training also presented an ongoing administrative need. Because it would be commnad line driven, it was felt that it would not be particularly user friendly. Additionally, it is unsupported. For these reasons RCS was not chosen for our data management needs.
CVS provided all the capabilities we needed and is well suited to managing a large project. By using tkCVS we would also gain a gui. It was felt that some script development would be necessary to manage things such as “sticky tags”, and setting up “watch lists” for visibility into other designers/teams development. User training would be minimal. As with RCS, the scripts and user training presented an ongoing, though minimal, administrative need. Additionally, it is also unsupported. For these reasons CVS/tkCVS were not chosen for our data management needs.
We chose ClioSoft and their SOS tool for our data management needs because it provided all the capabilites we needed right out-of-the-box and is supported. SOS provides revision control (with concurrent checkout), branch and merge management, tagging, and snapshots (a permanent tag) via an easy to use gui interface (a commandline interface is also provided). The gui makes it very easy for designers to learn and use the tool. The icons (“flags”), that propogate up through the gui provide visual information such as what files are not at the lateset revision, or what files are unmerged. There are also status fields that can provide information such as who has a file checked out, change summary for a file, etc. All of these provide greater visibilty into what other designers are doing and improve team collaboration. The tool is very easy to configure and customize to extend functionality. It supports collections of files and directories. This is important to us so that we can see what directories and files have been renamed, added, or deleted, with the deleted directories and files being recoverable from the repository. We can also recreate a previous directory structure if needed. Figure 3 shows the SOS gui.
Figure 3. The SOS Graphical User Interface
By compartmentalizing our design data and placing more of it under revision control we are more easily able to reproduce any working configurations. By giving each designer an isolated workspace and having “checkpoint” requirements before data can be placed in the “stable” area, the designers have a better environment in which to do their development. They are free to make changes without fear of breaking another designers/teams simulation environment. Because our logfiles are intended to be warning free, designers are less likely to miss potential problems arising from a warning. And finally, though CVS/tkCVS would have met our needs we felt that trading tool development, administration, and user training costs, for a supported third-party tool was a good trade-off. By using SOS we have made it much easier to view and manage our design data, which has improved team collaboration.