Skip to content

Latest commit

 

History

History
397 lines (263 loc) · 8.24 KB

utilities.md

File metadata and controls

397 lines (263 loc) · 8.24 KB

Concourse Utilities Reference

This document describes the development tools and scripts available in the utils/ directory.

Table of Contents

Overview

The utils/ directory contains scripts and tools that support Concourse development:

utils/
├── compile-thrift-*.sh      # Thrift compilation scripts
├── generate-*-docs.sh       # Documentation generators
├── install-git-hooks.sh     # Git hooks installer
├── install-snapshot.sh      # Snapshot installer
├── build-ete-test-plugin.sh # Test plugin builder
├── run-bash-tests.sh        # Bash test runner
├── codegen/                 # Code generation scripts
├── bats/                    # Bash testing framework
├── ronn/                    # Manual page generator
├── jq/                      # JSON processor
├── fpm/                     # Package builder
└── apidoc/                  # API documentation config

Thrift Compilation Scripts

These scripts compile the Thrift IDL files in interface/ to generate client/server code for various languages.

compile-thrift-all.sh

Compiles Thrift interfaces for all supported languages.

./utils/compile-thrift-all.sh

When to use: After modifying any .thrift file in interface/.

compile-thrift-java.sh

Compiles Thrift interfaces for Java only.

./utils/compile-thrift-java.sh

Output: Generated Java files in concourse-driver-java/src/main/java/com/cinchapi/concourse/thrift/

When to use: When only Java bindings need to be regenerated.

compile-thrift-python.sh

Compiles Thrift interfaces for Python.

./utils/compile-thrift-python.sh

Output: Generated Python files in concourse-driver-python/concourse/thriftapi/

compile-thrift-php.sh

Compiles Thrift interfaces for PHP.

./utils/compile-thrift-php.sh

Output: Generated PHP files in concourse-driver-php/src/

compile-thrift-ruby.sh

Compiles Thrift interfaces for Ruby.

./utils/compile-thrift-ruby.sh

Output: Generated Ruby files in concourse-driver-ruby/lib/

Prerequisites

These scripts require the Apache Thrift compiler:

# macOS
brew install thrift

# Ubuntu/Debian
apt-get install thrift-compiler

# Verify installation
thrift --version

Documentation Generation

generate-cash-docs.sh

Generates documentation for the CaSH (Concourse Shell) commands.

./utils/generate-cash-docs.sh

Output: Shell command documentation in docs/shell/

How it works:

  1. Extracts method signatures from the Concourse API
  2. Uses ronn to generate man pages and HTML documentation

generate-driver-docs.sh

Generates API documentation for the Java driver.

./utils/generate-driver-docs.sh

Output: JavaDoc in concourse-driver-java/build/docs/javadoc/

generate-rest-api-docs.sh

Generates REST API documentation.

./utils/generate-rest-api-docs.sh

Output: API documentation based on the configuration in utils/apidoc/apidoc.json

Development Tools

install-git-hooks.sh

Installs Git hooks for the repository.

./utils/install-git-hooks.sh

Installed hooks:

  • pre-push: Runs code formatting checks before pushing

When to use: Once, after cloning the repository.

install-snapshot.sh

Installs a snapshot build of Concourse to your local system.

./utils/install-snapshot.sh

What it does:

  1. Builds the installer from source
  2. Installs to a local directory
  3. Useful for testing local changes

build-ete-test-plugin.sh

Builds test plugins for end-to-end testing of the plugin framework.

./utils/build-ete-test-plugin.sh [java_version]

Arguments:

  • java_version (optional): Target Java version (8, 11, 17, 21)

Output: Plugin ZIP files in concourse-server/src/test/resources/plugins/

When to use: When testing plugin framework changes across Java versions.

Code Generation

codegen/StatefulConcourseServiceGenerator.groovy

A Groovy script that generates stateful service wrappers.

groovy utils/codegen/StatefulConcourseServiceGenerator.groovy

Purpose: Generates wrapper classes that maintain state across Thrift service calls.

When to use: After modifying the Thrift service interface when stateful tracking is needed.

Testing Tools

run-bash-tests.sh

Runs bash-based tests using the BATS framework.

./utils/run-bash-tests.sh [test_file]

Arguments:

  • test_file (optional): Specific test file to run

Example:

# Run all bash tests
./utils/run-bash-tests.sh

# Run specific test file
./utils/run-bash-tests.sh concourse-bash-tests/post_install/install.bats

bats/

The Bash Automated Testing System (BATS) framework for testing shell scripts.

Location: utils/bats/

Usage: Tests are defined in .bats files:

#!/usr/bin/env bats

@test "concourse starts successfully" {
    run concourse start
    [ "$status" -eq 0 ]
}

@test "concourse shell connects" {
    run concourse shell --password admin -e "ping()"
    [ "$status" -eq 0 ]
    [[ "$output" == *"true"* ]]
}

Running BATS tests:

./utils/bats/bin/bats path/to/test.bats

Bundled Tools

ronn/

A Ruby-based tool for generating manual pages from Markdown.

Location: utils/ronn/

Usage:

./utils/ronn/bin/ronn --roff --html docs/shell/add.ronn

Input: .ronn files (extended Markdown) Output: Man pages (.1, .7) and HTML files

Example .ronn file:

add(1) -- Add a value to a key in a record
=============================================

## SYNOPSIS

`add` <key>, <value>, <record>

## DESCRIPTION

Adds the specified **value** to the **key** in **record** if
the mapping does not already exist.

## RETURN VALUE

Returns `true` if the value is added, `false` otherwise.

jq/

Pre-compiled jq binaries for JSON processing.

Location: utils/jq/

Available binaries:

  • jq-osx-amd64 - macOS (Intel)
  • jq-linux64 - Linux (64-bit)

Helper script: utils/jq/jq.sh - Selects the appropriate binary

Usage:

# Parse JSON output
echo '{"name":"test"}' | ./utils/jq/jq.sh '.name'

fpm/

Effing Package Management - tool for creating system packages.

Location: utils/fpm/

Purpose: Used in the build process to create distribution packages (.deb, .rpm, etc.)

Usage (typically automated in Gradle):

./utils/fpm/bin/fpm -s dir -t deb -n concourse ...

apidoc/

Configuration for API documentation generation.

Location: utils/apidoc/

Files:

  • apidoc.json - Configuration for the apidoc tool

Configuration example:

{
  "name": "Concourse REST API",
  "version": "0.12.0",
  "description": "RESTful API for Concourse",
  "title": "Concourse API Documentation"
}

Quick Reference

Common Workflows

After Modifying Thrift Files

# Regenerate all language bindings
./utils/compile-thrift-all.sh

# Or just Java
./utils/compile-thrift-java.sh

Setting Up Development Environment

# Install git hooks
./utils/install-git-hooks.sh

# Build and install locally
./utils/install-snapshot.sh

Generating Documentation

# Shell docs
./utils/generate-cash-docs.sh

# API docs
./utils/generate-driver-docs.sh
./utils/generate-rest-api-docs.sh

Running Bash Tests

# All tests
./utils/run-bash-tests.sh

# Specific file
./utils/run-bash-tests.sh concourse-bash-tests/post_install/install.bats

Script Dependencies

Script Requires
compile-thrift-*.sh Apache Thrift compiler
generate-cash-docs.sh Ruby, ronn
generate-driver-docs.sh Java, Gradle
run-bash-tests.sh Bash, bats
build-ete-test-plugin.sh Java, Gradle, Docker (optional)

Related Documentation