More Than Just a Number

Discover your age in years months days hours and fascinating milestones you've reached

AUXTR - Auxiliary Trace

Enhanced Definition

AUXTR, or Auxiliary Trace, is a comprehensive system-wide tracing facility within z/OS that records significant system events, including program execution flow, SVC calls, I/O operations, and dispatcher activity. It is primarily used by system programmers and support personnel for in-depth problem diagnosis and performance analysis of the operating system and its components.

Key Characteristics

- System-Wide Scope: Unlike component-specific traces, AUXTR captures events across the entire z/OS system, providing a holistic view of system activity.
- Event Recording: It records various system events such as SVC (Supervisor Call) entries and exits, Program Interrupts, I/O operations, Dispatching events, and Branching activity.
- Output Destination: Trace records are typically written to the Generalized Trace Facility (GTF) data sets, which are then processed offline. It can also be directed to internal buffers.
- Control via TRACE CT: AUXTR is controlled using the z/OS TRACE CT operator command, allowing dynamic activation, deactivation, and modification of trace options.
- Performance Impact: Due to its extensive data collection, enabling AUXTR can introduce significant CPU overhead and I/O activity, potentially impacting system performance.
- Authorization Required: Activating or modifying AUXTR requires appropriate operator console authority or RACF authorization.

Use Cases

- Diagnosing System Hangs or Loops: Analyzing the trace data can reveal which programs or system services are stuck in a loop or causing a system-wide hang.
- Troubleshooting Performance Degradation: Identifying bottlenecks by examining I/O activity, SVC calls, and dispatcher wait times during periods of poor performance.
- Debugging Complex Inter-Component Issues: Tracing the interactions between different z/OS components or subsystems to pinpoint the source of an elusive problem.
- Analyzing Specific System Events: Capturing detailed information about specific types of SVCs, Program Interrupts, or I/O operations to understand their frequency and context.
- Post-Mortem Analysis: Providing critical diagnostic data for dumps or system failures when other diagnostic tools are insufficient.

Related Concepts

AUXTR is a core part of the z/OS System Trace facility, which also includes Branch Trace, Instruction Trace, SVC Trace, and I/O Trace. It often writes its output to GTF (Generalized Trace Facility) data sets, which act as a central repository for various system and component traces. The collected GTF data is then typically analyzed using IPCS (Interactive Problem Control System) to format and interpret the trace records. While Component Trace (CTRACE) focuses on specific software components, AUXTR provides a broader, system-level perspective.

Best Practices:

Enable Judiciously: Only activate AUXTR when necessary for problem diagnosis, as its performance impact can be substantial.
Limit Scope: If possible, use TRACE CT options to limit the types of events traced (e.g., specific ASIDs, SVC numbers) to reduce overhead and data volume.
Monitor Performance: Continuously monitor system performance metrics (CPU, I/O) while AUXTR is active to ensure it doesn't exacerbate existing problems.
Ensure GTF Capacity: Configure GTF with sufficient buffer space and appropriate data set sizes to prevent trace data loss or GTF performance issues.
Disable Promptly: Deactivate AUXTR as soon as the necessary diagnostic data has been collected to minimize its impact on the production environment.
Use IPCS for Analysis: Leverage IPCS commands and GTF trace formatting options to effectively analyze the voluminous trace data and extract meaningful insights.