Fork me on GitHub

Apache Hivemall Release Guide

This document describes the release process of Apache Hivemall particularly for Release Managers.

For general information of the Apache Incubator release process, please refer Incubator Release Management and ASF Release Poloicy page.

Prerequisites

If it is your first time doing an Apache release, then there is some initial setup involved. Follow this guide for the initial setup.

  1. Notifying the community the overall plan for the release
  2. Ensure JIRA Issues are appropriately tagged for the Release
    • Check this link for CLOSED/RESOLVED issues that does not have FixVersion.
    • Then, ensure that all JIRA issues that are addressed in this release are marked with the release version in the FixVersion field of the issue. Here is an example for v0.5.0 release.

UPDATE CHANGELOG

  1. Create a release note in JIRA following this instructions.
  2. Download a release note as ChangeLog.html using JIRA.

TEST YOUR SETTINGS

Try installing locally artifacts with activation apache-release profile. The following command will build artifacts, sources and sign.

Note: Apache Hivemall keeps compatibility to Java 7.

# JDK 7 is required for packaging
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`

# Java 8 is required for building Spark 2.2 module
export JAVA8_HOME=`/usr/libexec/java_home -v 1.8`

# Try to create artifacts
export MAVEN_OPTS=-XX:MaxPermSize=256m

# (Optional) Workaround for SSL error `Received fatal alert: protocol_version`
export MAVEN_OPTS="$MAVEN_OPTS -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2"

mvn -Papache-release clean install

Ensure that all unit tests passes. License check by Apache RAT (mvn apache-rat:check) will be ran by the above commands.

SNAPSHOT RELEASE

PREPARE YOUR POMS FOR RELEASE

1) Switch to master syncing to ASF master

git checkout master
git fetch
git pull # or, git reset --hard asf/master

2) Set version string for a snapshot

version=X.Y.Z
# echo "${version}-incubating-SNAPSHOT"

./bin/set_version.sh --pom --version "${version}-incubating-SNAPSHOT"

Note: --pom option SHOULD only used for SNAPSHOT release.

git commit -a -m "Prepare for the next Snapshot release of X.Y.Z"
git push asf master

3) Perform a dryRun

version=X.Y.Z
# RC should start at 1 and increment if early release candidates fail to release
rc=1
next_version=X.(Y+1).Z

# Confirm that version and rc is defined.
echo "Release version: ${version}-incubating"
echo "Development version: ${next_version}-incubating-SNAPSHOT"

mvn -Papache-release release:prepare \
  -DautoVersionSubmodules=true -DdryRun=true \
  -Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
  -Dtag=v${version}-rc${rc} -DreleaseVersion=${version}-incubating -DdevelopmentVersion=${next_version}-incubating-SNAPSHOT

Note: -rcX is only used for git tag. Release candidates do NOT have -rcX for artifact versions in pom. Release candidates are just for voting.

PUBLISH A SNAPSHOT

mvn deploy \
  -Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true

Note: You SHOULD verify the deployment under the Maven Snapshot repository at Apache.

PREPARE THE RELEASE

BRANCH THE RELEASE

1) Create a branch

git checkout -b vX.Y.Z

Note: Branch name starts with v and does not include -rcX.

2) Send e-mail announcing the release branch

To: dev@hivemall.incubator.apache.org
CC: private@hivemall.incubator.apache.org
Subject: New release branch X.Y.Z

Hello Hivemall developers and friends,

We now have a release branch for X.Y.Z release. Trunk has been bumped to X.Y.Z-SNAPSHOT.

  https://github.com/apache/incubator-hivemall/tree/vX.Y.Z

I'll be going over the JIRAs to move every non-blocker from this release to the next release. Release voting will be posted soon.

PREPARE A RELEASE

1) Set a release version

version=X.Y.Z
rc=1
next_version=X.(Y+1).Z

# Confirm that version and rc is defined.
echo "${version}-incubating (Release Candidates ${rc})"
# 0.5.0-incubating (Release Candidates 3)

2) Update version strings in source codes.

./bin/set_version.sh --version "${version}-incubating"

# Check list of files to be committed.
git status
git commit -a -m "Bumped version string to ${version}-incubating"

3) Prepare sets the version numbers in POM and creates a tag.

mvn -Papache-release release:clean release:prepare \
  -DautoVersionSubmodules=true -DdryRun=false \
  -Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
  -Dtag=v${version}-rc${rc} -DreleaseVersion=${version}-incubating -DdevelopmentVersion=${next_version}-incubating-SNAPSHOT

4) Update version strings for the development iteration

./bin/set_version.sh --version "${next_version}-incubating-SNAPSHOT"

# Check list of files to be committed.
git stauts
git commit --amend -a

5) Push release branch and tag to remote ASF repository

# Push the release branch
git push asf v${version}

# Push the release tag
git push asf v${version}-rc${rc}

STAGE THE RELEASE FOR A VOTE

The release will automatically be inserted into a temporary staging repository for you.

mvn -Papache-release release:perform \
  -Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
  -Dgoals=deploy -DlocalRepoDirectory=. -DlocalCheckout=true

Verify nexus release artifacts

  1. Verify the staged artifacts in the nexus repo
  • Go to https://repository.apache.org/ and login
  • Under Build Promotion, click Staging Repositories
  • In the Staging Repositories tab there should be a line with profile org.apache.hivemall
  • Navigate through the artifact tree and make sure that all javadoc, sources, tests, and jars have .asc (GPG signature). Refer the ASF page for artifacts verification.
  1. Close the nexus staging repo

Attach signatures for shaded jars

Shaded jars does not have signatures. So, attach signatures to them as follows:

cd target/

# Sign to the artifacts created by maven-shade-plugin
for f in `ls hivemall-*-with-dependencies*.jar`; do
  gpg --armor --output ${f}.asc --detach-sig ${f}
  shasum -a 512 ${f} | cut -f1 -d' '  > ${f}.sha512
done

# Verify GPG sign
for file in `find . -type f -iname 'hivemall-*-with-dependencies*.jar.asc'`; do
  echo ${file}
  gpg --verify ${file}
  echo
done

Upload the artifacts via subversion to a staging area

  • Prepare release artifacts in SVN repository
# Checkout release SVN repository
mkdir -p dist/dev/incubator
cd dist/dev/incubator
svn co https://dist.apache.org/repos/dist/dev/incubator/hivemall/
cd hivemall

# Download release artifacts
wget -e robots=off --no-check-certificate \
 -r -np --reject=html,txt,tmp -nH --cut-dirs=7 \
 https://repository.apache.org/content/repositories/orgapachehivemall-1001/org/apache/hivemall/hivemall/${version}-incubating/

# Rename directory for RC
mv ${version}-incubating ${version}-incubating-rc${rc}

# Put ChangeLog
cd ${version}-incubating-rc${rc}
# Put ChangeLog generated by JIRA
cp ~/Downloads/ChangeLog.html .

# Put Shaded jars
cp ~/incubator-hivemall/target/hivemall-*-with-dependencies*.jar* .

# Remove deprecated md5 and sha1
# See https://www.apache.org/dev/release-distribution.html#sigs-and-sums

ls *.md5 *.sha1
rm *.md5 *.sha1

# Add SHA-512 checksum

for f in hivemall-0.5.2-incubating-site.xml hivemall-0.5.2-incubating-source-release.zip hivemall-0.5.2-incubating.pom; do
  shasum -a 512 ${f} | cut -f1 -d' '  > ${f}.sha512
  ls ${f}.sha512
done
  • Push release arfifacts to ASF svn repository
cd dist/dev/incubator/hivemall
# ls ${version}-incubating-rc${rc}

svn add ${version}-incubating-rc${rc}/
svn commit -m "Put hivemall version ${version}-incubating-rc${rc} artifacts"

Verify Release Artifacts

PPMC voting

Create a VOTE email thread on dev@hivemall.incubator.apache.org to record votes as replies.

To: dev@hivemall.incubator.apache.org
CC: private@hivemall.incubator.apache.org
Subject: [VOTE] Release Apache Hivemall (Incubating) v0.5.0 Release Candidate #3

Hi all,

Apache Hivmeall 0.5.0 release candidate #1 is now available for a vote within dev community.

Links to various release artifacts are given below. Please review and cast your vote.

    - The source tarball, including signatures, digests, ChangeLog, etc.:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/
	- Sources for the release:
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip.asc (PGP Signature)
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip.sha512 (SHA512 Hash)
    - Git tag for the release:
      https://git-wip-us.apache.org/repos/asf?p=incubator-hivemall.git;a=shortlog;h=refs/tags/v0.5.0-rc1
    - The Nexus Staging URL:
      https://repository.apache.org/content/repositories/orgapachehivemall-1001/
    - KEYS file for verification:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/KEYS
    - For information about the contents of this release, see:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/ChangeLog.html

Find Podling releases policies in
    https://incubator.apache.org/policy/incubation.html#releases
    http://www.apache.org/legal/release-policy.html

Artifacts verification how-to can be found in
    http://hivemall.incubator.apache.org/verify_artifacts.html

The vote will be open for at least 72 hours and until necessary number of votes are reached.

At lease three +1 from PPMC members are required and we welcome your vote!

[ ] +1  approve (Release this package as Apache Hivemall 0.5.0-incubating-rc1)
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Here is my +1 (binding).

Regards,
Makoto

Once 72 hours has passed (which is generally preferred) and/or at least three +1 (binding) votes have been cast with no -1 (binding) votes, send an email closing the vote and pronouncing the release candidate a success.

To: dev@hivemall.incubator.apache.org
Subject: [RESULT][VOTE]: Release Apache Hivemall (Incubating) <release version>

Hi all,

The Apache Hivemall <release version> vote is now closed and has passed as follows:

From PPMC members:
+1 myui (Makoto)
+1 maropu (Takeshi)
+1 lewuathe (Kai)
+1 takuti (Takuya)
+1 ozawa (Tsuyoshi)

From Mentors:
+1 daniel (IPMC)

Other non-binding votes:
+1 Kento (committer)

I will send a vote mail to incubator-general mail list to collect three IPMC +1's votes.
https://incubator.apache.org/policy/incubation.html#releases.

If voting succeeds, v0.5.0 will be released based on v0.5.0-rc2 with removing rc2.

Thanks all of the voters.

Thanks,
Makoto
on behalf of Apache Hivemall PPMC

IPMC Voting

What if vote succeed, then vote in general@incubator.apache.org. Three +1 Incubator PMC votes are required as described in this page.

To: general@incubator.apache.org
Subject: [VOTE] Release Apache Hivemall (Incubating) v0.5.0-RC2

Hi all,

The Apache Hivemall community has approved a proposal to release Apache Hivemall v0.5.0 by v0.5.0-RC2.

We now kindly request that the Incubator PMC members review and vote on this incubator release candidate.

The PPMC vote thread is located here:
    https://www.mail-archive.com/dev@hivemall.incubator.apache.org/msg00450.html (vote)
	https://www.mail-archive.com/dev@hivemall.incubator.apache.org/msg00460.html (vote result)

Links to various release artifacts are given below.

    - The source tarball, including signatures, digests, ChangeLog, etc.:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc2/
    - Sources for the release:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc2/hivemall-0.5.0-incubating-rc2-source-release.zip
    - Git tag for the release:
      https://git-wip-us.apache.org/repos/asf?p=incubator-hivemall.git;a=shortlog;h=refs/tags/v0.5.0-rc2
    - The Nexus Staging URL:
      https://repository.apache.org/content/repositories/orgapachehivemall-1002/
    - KEYS file for verification:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/KEYS
    - For information about the contents of this release, see:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc2/ChangeLog.html

Artifacts verification how-to can be found in
    http://hivemall.incubator.apache.org/verify_artifacts.html

Please vote accordingly:

[ ] +1  approve (Release this package as Apache Hivemall 0.5.0-incubating)
[ ] -1  disapprove (and reason why)

The vote will be open for at least 72 hours.

Regards,
Makoto
on behalf of Apache Hivemall PPMC

Once 72 hours has passed (which is generally preferred) and/or at least three +1 (binding) votes have been cast with no -1 (binding) votes, send an email closing the vote and pronouncing the release candidate a success.

To: general@incubator.apache.org
CC: dev@hivemall.incubator.apache.org
Subject: [RESULT][VOTE]: Release Apache Hivemall (Incubating) <release version>

Hi all,

The vote to release Apache Hivemall (incubating) version v0.5.0
Release Candidate 3 has passed with 3 +1 binding votes and no other votes.

+1 binding:
 ..
 ..
 ..

Vote thread:
 https://lists.apache.org/thread.html/86a0d8df057c6c7b0675f8bbdb211fe6602545c68de2bf9b7b8204c1@%3Cgeneral.incubator.apache.org%3E

I'd like to thank all voters who helped verifying the release.

The Apache Hivemall community will proceed to publish the 0.5.0-incubating release and send announcements.

Best,
Makoto
on behalf of Apache Hivemall PPMC

Finalize release

Once the Incubator general vote has passed, we can publish the source release and artifacts.

Publish to Maven Central in Nexus

  • Go to https://repository.apache.org/ and login
  • Under Build Promotion, click Staging Repositories
  • In the Staging Repositories tab there should be a line with profile org.apache.hivemall
  • click Release.
    • You shall now see your artifacts in the release repository created for you. In some hours, it should also appear in Maven Central.

Update JIRA

Update the JIRA versions page to close all issues, mark the version as released, and set the date to the date that the release was approved. You may also need to make a new release entry for the next release.

Source Release

The Hivemall source release are checked into SVN under: https://dist.apache.org/repos/dist/release/incubator/hivemall

# cd ~/work/dist
# mkdir -p release/incubator
# cd release/incubator
# svn co https://dist.apache.org/repos/dist/release/incubator/hivemall/
# cd hivemall

# Put release artifacts to release directory
mkdir 0.5.0-incubating
cp ../../../dev/incubator/hivemall/0.5.0-incubating-rc3/ChangeLog.html 0.5.0-incubating/
cp ../../../dev/incubator/hivemall/0.5.0-incubating-rc3/hivemall-0.5.0-incubating*.zip* 0.5.0-incubating/
cp ../../../dev/incubator/hivemall/0.5.0-incubating-rc3/hivemall*.jar* 0.5.0-incubating/
ls 0.5.0-incubating/

# Commit to SVN
svn add 0.5.0-incubating/
svn commit -m "Put hivemall version 0.5.0-incubating release artifacts"

Check release artifacts in the SVN repository: https://dist.apache.org/repos/dist/release/incubator/hivemall/0.5.0-incubating/ (Mirror)

Merge release branch for the next development iteration

If IPMC vote succeed, then merge the release branch into the master branch.

Publish the websit

Update download page etc.

Announcing the release

Make an announcement about the release on the user@hivemall.incubator.apache.org, dev@hivemall.incubator.apache.org, general@incubator.apache.org, and announce@apache.org list as per the Apache Announcement Mailing Lists page.

[ANNOUNCE] Apache Hivemall 0.5.0-incubating released

The Apache Hivemall (incubating) project team is proud to announce
Apache Hivemall 0.5.2-incubating has been released.
This is the 2nd Apache release as an Apache Incubator project.

Apache Hivemall is a scalable machine learning library implemented as
Hive UDFs/UDAFs/UDTFs. Hivemall runs on Hadoop-based data processing
frameworks, specifically on Apache Hive, Apache Spark, and Apache Pig.

The release artifacts and ChangeLog can be found at:

 http://hivemall.incubator.apache.org/download.html

Release artifacts in Maven Central:

 https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.hivemall%22%20AND%20v%3A%220.5.2-incubating%22

Find more about our project at:
 - Project Site:  http://hivemall.incubator.apache.org/
 - Github mirror: https://github.com/apache/incubator-hivemall
 - Mailing list(s): dev@hivemall.incubator.apache.org
                    user@hivemall.incubator.apache.org

Thanks,
Makoto
on behalf of Apache Hivemall PPMC

=====
*Disclaimer*
Apache Hivemall (incubating) is an effort undergoing incubation at
The Apache Software Foundation (ASF), sponsored by the name of Apache
Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure, communications,
and decision making process have stabilized in a manner consistent
with other successful ASF projects. While incubation status is not
necessarily a reflection of the completeness or stability of the code,
it does indicate that the project has yet to be fully endorsed by the ASF.