--- ---

Specs & Tests

Specifying and testing DocOps Lab projects

Most DocOps Lab projects include a specs/ directory in the base. This path is for all “definitional” code and content, including:

  • YAML- or JSON-formatted schema or definition documents

  • “natural language” requirement/specification documents

  • test scripts in RSpec or other formats

  • test data files

  • test configurations

The typical structure of this path is:

specs/
  docs/             # natural language PRDs
    *.adoc
  data/             # defs, schemas, test data
    *.yml,*.yaml
  tests/
    *.rb, *.sh      # test scripts
    rspec/          # RSpec test files
      spec_helper.rb
      *_spec.rb
    results/        # Test output logs

Specifications, Requirements, and Definitions

DocOps Lab project development is “docs-driven”, meaning we write up our code requirements and specifications in natural language documents before we start coding. These files tend to take the following form:

README.adoc

Especially during early development, fairly detailed product documentation is stored here.

specs/docs/*.adoc

Natural language product requirement documents (PRD) specifying dependencies, features, behaviors, and inter-component contracts and interfaces.

specs/data/*.yml

YAML-formatted definition files that designate actual attributes and content of the app, typically including config-def.yml for configuration properties defaults and docs.

specs/data/*-schema.yaml

SGYML-formatted schema files that define the structure and constraints user-editable YAML files, such as configuration files.

Testing Infrastructure Standards

DocOps Lab projects follow consistent testing patterns to ensure reliability and maintainability across the ecosystem. These standards cover test organization, configuration files, and data management used by testing frameworks.

RSpec Configuration

All Ruby-based projects should include:

.rspec

Configuration file specifying RSpec options and test file patterns. Standard configuration:

--format documentation
--color
--pattern 'specs/tests/rspec/**/*_spec.rb'
specs/tests/rspec/spec_helper.rb

Shared configuration, helper methods, and sample data for tests. Should include:

  • Bundler setup and project requires

  • RSpec configuration (monkey patching disabled, expect syntax)

  • Helper methods for temporary file/directory creation

  • Sample data generators for testing

  • Cleanup procedures for test artifacts

Standard Rake Tasks

All Ruby gem projects with tests should implement these standard Rake tasks in their Rakefile:

bundle exec rake rspec

Run RSpec test suite using the standard pattern matcher.

bundle exec rake cli_test

Validate command-line interface functionality. May test basic CLI loading, help output, version information.

bundle exec rake yaml_test

Validate YAML configuration files and data structures. Should test all project YAML files for syntax correctness.

bundle exec rake pr_test

Comprehensive test suite for pre-commit and pull request validation. Typically includes: RSpec tests, CLI tests, YAML validation.

bundle exec rake install_local

Build and install the project locally for testing.

Note that non-gem projects may have some or all of these tasks, as applicable.

Test Categories

Tests should be organized into these categories:

Unit Tests

  • Module loading and initialization

  • Class structure validation

  • Basic functionality verification

  • Individual method testing

Integration Tests

  • Data processing workflows

  • Template rendering operations

  • Configuration loading scenarios

  • API client functionality (where applicable)

Validation Tests

  • File format compliance (YAML, JSON)

  • Configuration schema validation

  • Template syntax verification

  • Command-line option parsing

Test Data Management

Projects should utilize:

Demo/Sample Data

Rich sample data in dedicated demo directories (e.g., ../projectname-demo/). Used for integration testing and examples.

Generated Test Data

Programmatically generated test data using helper methods. Ensures consistent, controlled test scenarios.

Temporary Files

Automatic creation and cleanup of temporary files/directories. Prevents test pollution and ensures isolated test environments.

Test Documentation

Each project’s specs/tests/README.adoc should:

Reference shared standards

Link to https://docopslab.org/docs/testing.

Document project-specific patterns

Cover unique testing approaches.

Provide quick start instructions

Enable new contributors to run tests immediately.

Explain integration with demo data

Show how sample data is used.

List available test commands

Document all relevant Rake tasks and their purposes.

Testing the Documentation

Not only do tests need to be documented, but documentation needs to be systematically tested.

These tests are mainly performed via the docopslab-dev tool.

DocOps Lab performs markup-syntax linting, prose linting, HTML linting, HTML link testing, and code-block testing.

Syntax linting

Evaluates AsciiDoc and RDoc source syntax for proper style and formatting. We use Vale for prose linting and custom RuboCop for RDoc comment.

HTML linting and link testing

Validates generated HTML documentation for proper structure, formatting, and link integrity. See Docs Link Testing with HTMLProofer.

Code-block testing

Extracts and executes code blocks from documentation to ensure accuracy and functionality. This is performed using the Sourcerer library, which is presently part of the ReleaseHx gem but will soon be released as a standalone gem.

Docs Link Testing with HTMLProofer

HTML validation for Jekyll sites and documentation builds.

Base config

.config/.vendor/docopslab/htmlproofer.yml

Project config

.config/htmlproofer.yml

Sync command

bundle exec rake labdev:sync:configs

Enable in manifest

Add htmlproofer tool with enabled: true

HTMLProofer validates links, images, and HTML structure in built sites. Only enabled for projects that generate HTML output (Jekyll sites, etc.).

Base HTMLProofer Configuration
---
# DocOps Lab HTML-Proofer Base Configuration
# This provides sensible defaults for DocOps Lab projects

# URL checking options
check_external_hash: false    # Skip checking external URL fragments (can be slow)
check_img_http: false         # Allow HTTP images for now
enforce_https: false          # Don't enforce HTTPS for all links

# URLs to ignore (patterns and strings)
ignore_urls:
  - /localhost/               # Skip local development URLs
  - /127\.0\.0\.1/           # Skip local IPs  
  - /example\.com/           # Skip example URLs
  - /foo\.bar/               # Skip placeholder URLs
  - /fonts.googleapis.com/   # Skip Google Fonts
  - /fonts.gstatic.com/      # Skip Google Fonts static

# Files to ignore (patterns)
ignore_files:
  - slides/                  # Skip slides directories

# Other options
check_favicon: false         # Don't require favicon
check_html: true            # Check HTML structure
check_opengraph: false      # Skip OpenGraph validation
disable_external: false     # Check external links (can be disabled for speed)
For full HTMLProofer configuration options, see the official docs.

Continuous Integration Standards

DocOps Lab projects follow consistent CI/CD patterns. These broadly include pre-commit/pre-push testing, GitHub Actions integration, and release testing.

Pre-push Testing

bundle exec rake pr_test

This test runs the complete local suite, including RSpec tests as well as any established CLI testing.

GitHub Actions Integration

Projects should implement GitHub Actions workflows that:

  • Run on pull requests

  • Execute the complete pr_test suite

  • Test across multiple Ruby versions (where applicable)

  • Validate documentation quality

  • Report test coverage

Release Testing

Before any merge to main and pre-release.

bundle exec rake install_local
bundle exec rake pr_test

Release testing also involves examining artifacts (packaged gems, Docker image, documentation) before it is published. See below for basics and DocOps Lab Release Process (General) for details.

Test Results and Artifacts

Test execution should generate:

specs/tests/results/

Directory for test output, logs, and reports. Automatically created during test runs.

.rspec_status

RSpec status persistence file for --only-failures and --next-failure flags.

Test results should include:

  • Pass/fail status for all test categories

  • Performance metrics (where applicable)

  • Coverage reports

  • Artifact validation logs